Endoscopic stereo matching data transfer methods, devices, equipment and media

Abstract

This invention provides a data migration method, apparatus, device, and medium for endoscopic stereo matching, relating to the field of data processing technology. The method includes: generating a data transfer method, apparatus, device, and medium for endoscopic stereo matching, comprising pairs of simulated endoscopic images and corresponding geometric ground truth data, wherein the geometric ground truth data includes at least an absolute depth map and a visual control map for geometric constraints; acquiring a lightweight style model, using the simulated endoscopic images as a visual base map, and using the absolute depth map and the visual control map as constraints; inputting the visual base map and constraints into a generation pipeline composed of the lightweight style model and the geometric control network; performing texture style transfer on the visual base map through the lightweight style model; and applying geometric constraints to the generation process according to the constraints through the geometric control network; and outputting a domain-transferred image with realistic texture style and maintaining the geometric structure of the simulated endoscopic images.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data migration method for endoscopic stereo matching, a data migration device for endoscopic stereo matching, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In deep learning-based 3D measurement technologies, especially in complex scenarios such as industrial endoscopes, simulation data generation or style transfer methods are often used to obtain the image-depth ground truth pairs needed for training. While this approach is simple to operate and can partially alleviate the problem of scarce real data, it has a series of problems in practical applications. For example, simulation data differs significantly from the real environment in key photometric characteristics such as material reflection, complex noise, and lens distortion, leading to a sharp decline in the performance of the trained model in real-world scenarios. At the same time, related style transfer techniques often destroy the geometric structure of the image, causing edge shifting or shape distortion, resulting in inaccurate depth ground truth values ​​that cannot meet the stringent accuracy requirements of 3D measurement for geometric position. Summary of the Invention

[0003] The present invention provides a data migration method, apparatus, electronic device, and computer-readable storage medium for endoscopic stereo matching, in order to solve or partially solve the problem of performance degradation of models in real industrial scenarios caused by reliance on simulation data training.

[0004] This invention discloses a data migration method for endoscopic stereo matching, comprising: Generate a set of simulated endoscope images containing pairs of simulated endoscope images and corresponding geometric truth data, wherein the geometric truth data includes at least an absolute depth map and a visual control map for geometric constraints. A lightweight style model is obtained, and the simulated endoscope image is used as the visual base map, while the absolute depth map and the visual control map are used as constraints. The visual base map and the constraints are input into the generation pipeline consisting of the lightweight style model and the geometric control network. The lightweight style model performs texture style transfer on the visual base map, and the geometric control network applies geometric constraints to the generation process according to the constraints, outputting a domain-transferred image with a realistic texture style and maintaining the geometric structure of the simulated endoscope image.

[0005] In some feasible implementations, the generation of pairs of simulated endoscopic images and corresponding geometric truth data of the simulated endoscopic images includes: Construct a virtual industrial endoscope inspection scenario; A corresponding binocular camera system is configured in the industrial endoscope inspection scenario, and the binocular camera system is driven to move in the industrial endoscope inspection scenario; The binocular camera system synchronously outputs simulated RGB images of the left and right views for each rendered frame as the simulated endoscope image, as well as the absolute depth map, normalized depth map, normal vector map, and edge contour map of the simulated endoscope image. The normalized depth map, the normal vector map, and the edge contour map constitute the visual control map.

[0006] Among some feasible implementation methods are: The absolute depth map is stored in 32-bit floating-point format; The absolute depth map is normalized by mapping the physical distance values ​​to a preset range and converting them into an 8-bit grayscale image to generate the normalized depth map.

[0007] In some feasible implementations, the construction of a virtual industrial endoscope inspection scenario includes: Create virtual scenes in a 3D rendering engine and select cavity models and industrial parts from the model library; The cavity model is rendered in the virtual scene, and the industrial parts are randomly scattered in the cavity model. At least one of rotation, scaling and translation is applied to the industrial parts to simulate the geometric differences of different engine models or different detection positions. Obtain the base material of the industrial part and assign the base material to the industrial part to obtain a virtual industrial endoscope inspection scene.

[0008] In some feasible implementations, driving the binocular camera system to move in the industrial endoscopy inspection scenario includes: Obtain camera parameters for the binocular camera system, the camera parameters including optical parameters and path planning information; Configure the optical parameters for the binocular camera system; According to the path planning information, the binocular camera system is controlled to advance along the centerline inside the industrial endoscope inspection scene, and random shaking and rotation are added during the advancement.

[0009] Among some feasible implementation methods are: Acquire real industrial endoscope images and generate a text description with specific trigger words for each of the industrial endoscope images; Obtain a pre-trained generative base model, wherein the generative base model includes at least a variational autoencoder, a text encoder, and a UNet network; Keep the pre-trained weights in the base model unchanged, and insert a trainable low-rank adaptation matrix in parallel through the weight matrix of the UNet network. The variational autoencoder is used to map the image captured by the industrial endoscope into a first latent vector, and the text encoder is used to encode the text description into a text feature vector. Noise is added to the first latent vector to obtain a noisy second latent vector. The second hidden vector, the text feature vector, and the time step information corresponding to the added noise are input into the UNet network after inserting the low-rank adaptation matrix to perform noise prediction and obtain the predicted noise. Calculate the error between the predicted noise and the added noise, and based on this error, update only the parameters of the low-rank adaptation matrix using the backpropagation algorithm, while freezing the original weights of the UNet network until training converges. Save the parameters of the trained low-rank adaptation matrix as the lightweight style model.

[0010] In some feasible implementations, the geometric control network includes at least one of a depth control unit, a normal control unit, and an edge control unit. The process of performing texture style transfer on the visual base map using the lightweight style model, and applying geometric constraints to the generation process according to the constraints through the geometric control network, to output a domain-transferred image with a realistic texture style and preserving the geometric structure of the simulated endoscope image, includes: Get text prompts with specific trigger words; The visual base image is encoded into the latent space to obtain the first target latent vector; Obtain the redraw magnitude for the first target latent vector, add Gaussian noise to the first target latent vector according to the redraw magnitude, and obtain the second target latent vector after adding noise; The normalized depth map is input into the depth control unit for feature extraction to obtain depth features; The normal vector map is input into the normal control unit for feature extraction to obtain normal features; The edge contour map is input into the edge control unit for feature extraction to obtain edge features; The depth feature, the normal feature, and the edge feature are weighted and superimposed to obtain the geometric constraint feature; During the iterative denoising process, the second target latent vector, the geometric constraint features, and the text prompt words are input into a generative network that integrates the lightweight style model to predict noise and update the latent vector. After the denoising iteration is completed, the final latent vector is decoded into a pixel image to obtain the domain-transferred image.

[0011] This invention also discloses a data migration device for endoscopic stereo matching, comprising: The simulation module is used to generate a set of simulated endoscope images containing pairs of simulated endoscope images and geometric truth data corresponding to the simulated endoscope images. The geometric truth data includes at least an absolute depth map and a visual control map for geometric constraints. The model acquisition module is used to acquire a lightweight style model, and uses the simulated endoscope image as the visual base map, and the absolute depth map and the visual control map as constraints. The migration module is used to input the visual base map and the constraints into the generation pipeline composed of the lightweight style model and the geometric control network, perform texture style migration on the visual base map through the lightweight style model, and apply geometric constraints to the generation process according to the constraints through the geometric control network, and output a domain-transferred image with a realistic texture style and maintaining the geometric structure of the simulated endoscope image.

[0012] In some feasible implementations, the simulation module is specifically used for: Construct a virtual industrial endoscope inspection scenario; A corresponding binocular camera system is configured in the industrial endoscope inspection scenario, and the binocular camera system is driven to move in the industrial endoscope inspection scenario; The binocular camera system synchronously outputs simulated RGB images of the left and right views for each rendered frame as the simulated endoscope image, as well as the absolute depth map, normalized depth map, normal vector map, and edge contour map of the simulated endoscope image. The normalized depth map, the normal vector map, and the edge contour map constitute the visual control map.

[0013] Among some feasible implementation methods are: The storage module is used to store the absolute depth map in 32-bit floating-point format; The normalization module is used to normalize the absolute depth map, map the physical distance values ​​to a preset range, and convert them into an 8-bit grayscale image to generate the normalized depth map.

[0014] In some feasible implementations, the simulation module is specifically used for: Create virtual scenes in a 3D rendering engine and select cavity models and industrial parts from the model library; The cavity model is rendered in the virtual scene, and the industrial parts are randomly scattered in the cavity model. At least one of rotation, scaling and translation is applied to the industrial parts to simulate the geometric differences of different engine models or different detection positions. Obtain the base material of the industrial part and assign the base material to the industrial part to obtain a virtual industrial endoscope inspection scene.

[0015] In some feasible implementations, the simulation module is specifically used for: Obtain camera parameters for the binocular camera system, the camera parameters including optical parameters and path planning information; Configure the optical parameters for the binocular camera system; According to the path planning information, the binocular camera system is controlled to advance along the centerline inside the industrial endoscope inspection scene, and random shaking and rotation are added during the advancement.

[0016] Among some feasible implementation methods are: The image acquisition module is used to acquire real industrial endoscope images and generate a text description with specific trigger words for each of the industrial endoscope images; The model acquisition module is used to acquire a pre-trained generative base model, which includes at least a variational autoencoder, a text encoder, and a UNet network. A matrix insertion module is used to keep the pre-trained weights in the base model unchanged and insert trainable low-rank adaptation matrices in parallel in the bypass of the weight matrix of the UNet network. The vector mapping module is used to map the image captured by the industrial endoscope into a first latent vector using the variational autoencoder, and to encode the text description into a text feature vector using the text encoder, and to add noise to the first latent vector to obtain a noisy second latent vector. The prediction module is used to input the second hidden vector, the text feature vector, and the time step information corresponding to the added noise into the UNet network after inserting the low-rank adaptation matrix to perform noise prediction and obtain the predicted noise. The training module is used to calculate the error between the predicted noise and the added noise, and based on the error, update only the parameters of the low-rank adaptation matrix through the backpropagation algorithm, while freezing the original weights of the UNet network until the training converges. The parameters of the low-rank adaptation matrix obtained by training are then saved as the lightweight style model.

[0017] In some feasible implementations, the geometric control network includes at least one of a depth control unit, a normal control unit, and an edge control unit, and the migration module is specifically used for: Get text prompts with specific trigger words; The visual base image is encoded into the latent space to obtain the first target latent vector; Obtain the redraw magnitude for the first target latent vector, add Gaussian noise to the first target latent vector according to the redraw magnitude, and obtain the second target latent vector after adding noise; The normalized depth map is input into the depth control unit for feature extraction to obtain depth features; The normal vector map is input into the normal control unit for feature extraction to obtain normal features; The edge contour map is input into the edge control unit for feature extraction to obtain edge features; The depth feature, the normal feature, and the edge feature are weighted and superimposed to obtain the geometric constraint feature; During the iterative denoising process, the second target latent vector, the geometric constraint features, and the text prompt words are input into a generative network that integrates the lightweight style model to predict noise and update the latent vector. After the denoising iteration is completed, the final latent vector is decoded into a pixel image to obtain the domain-transferred image.

[0018] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0019] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0020] The embodiments of the present invention have the following advantages: In this embodiment of the invention, by generating paired simulated endoscope images and their corresponding geometric ground truth data, a data source with accurate geometric benchmarks is constructed for the stereo matching training task. This solves the problem of difficulty in obtaining depth ground truth in real industrial scenarios. Furthermore, by acquiring a lightweight style model and combining it with a geometric control network to form a generation pipeline, this generation pipeline gains the ability to learn and transfer complex texture styles from real images. During the generation process, the simulated image is used as a visual base map, and its corresponding visual control map is input as a constraint condition. The tasks of "texture style transfer" and "geometric structure preservation" are processed collaboratively, ensuring that the original geometric structure of the simulated image, which strictly corresponds to the absolute depth map, is completely preserved during the transfer of texture to realism. This results in the output domain-transferred image possessing both the appearance features of real industrial scenarios and the accurate geometric structure of the simulated data source, effectively reducing the domain difference between simulation and real scenarios. This provides a reliable data foundation for training a stereo matching model with high accuracy and strong robustness in real complex environments, thereby improving the model's performance in real industrial scenarios. Attached Figure Description

[0021] Figure 1 This is a flowchart of the steps of a data migration method for endoscopic stereo matching provided in an embodiment of the present invention; Figure 2 This is a structural block diagram of a data migration device for endoscopic stereo matching provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] As an example, in deep learning-based 3D measurement technology, especially in complex scenarios such as industrial endoscopes (e.g., the interior of aircraft engines), simulation data generation or style transfer methods are often used to obtain the image-depth ground truth pairs needed for training. However, while this approach is simple to operate and can partially alleviate the problem of scarce real data, it faces a series of problems in practical applications. For instance, simulation data differs significantly from the real environment in key photometric characteristics such as material reflection, complex noise, and lens distortion, leading to a sharp decline in the performance of the trained model in real-world scenarios. Simultaneously, related style transfer techniques often disrupt the geometric structure of images, causing edge shifts or shape distortions, resulting in inaccurate generated depth ground truth pairs that fail to meet the stringent precision requirements of 3D measurement for geometric position.

[0024] To address this issue, this invention generates a data source with precise geometric references for stereo matching training tasks by generating paired simulated endoscopic images and their corresponding geometric ground truth data. This solves the problem of obtaining depth ground truth in real industrial scenarios. Furthermore, by acquiring a lightweight style model and integrating it with a geometric control network to form a generation pipeline, this pipeline gains the ability to learn and transfer complex texture styles from real images. During the generation process, the simulated image is used as a visual base map, and its corresponding visual control map is input as a constraint condition. This collaboratively processes the tasks of "texture style transfer" and "geometric structure preservation," ensuring that the original geometric structure of the simulated image, which strictly corresponds to the absolute depth map, is completely preserved during the transfer of texture to realism. This results in the output domain-transferred image possessing both the appearance features of real industrial scenarios and the precise geometric structure of the simulated data source, effectively reducing the domain difference between simulation and real scenarios. This provides a reliable data foundation for training a stereo matching model with high accuracy and strong robustness in real complex environments, thereby improving the model's performance in real industrial scenarios.

[0025] Reference Figure 1 The diagram illustrates a flowchart of a data migration method for endoscopic stereo matching provided in an embodiment of the present invention, which may specifically include the following steps: Step 101: Generate a set of simulated endoscope images containing pairs of simulated endoscope images and geometric truth data corresponding to the simulated endoscope images. The geometric truth data includes at least an absolute depth map and a visual control map for geometric constraints. In this embodiment of the invention, a source domain dataset capable of providing accurate geometric information can first be constructed. Since it is nearly impossible to obtain images with millimeter-level depth ground truth in real industrial endoscopic scenarios (such as the interior of an aircraft engine), a 3D rendering engine can be used to construct a virtual endoscopic inspection environment, and "image-ground truth" data pairs can be generated in batches through automated scripts. These data can then serve as the "geometric skeleton" for subsequent domain migration, ensuring that the 3D structure of the object remains unchanged regardless of texture changes.

[0026] The simulated endoscope image refers to an RGB color image generated by a 3D rendering engine, simulating an industrial endoscope captured in a virtual scene. It may contain similar perspectives, lighting, and object layouts to a real endoscope, but its texture is synthesized by the rendering engine and lacks the complex textures, noise, and dirt found in a real scene. Optionally, the simulated endoscope image is typically stored in common image formats (such as PNG, JPEG, etc.), and this invention does not impose any limitations on this.

[0027] Geometric ground truth data can be a dataset that strictly corresponds to simulated endoscopic images and describes the three-dimensional geometric structure of a scene. This data is difficult to obtain in the real world but can be accurately extracted in virtual rendering. Geometric ground truth data includes at least absolute depth maps and visual control maps. An absolute depth map is a two-dimensional image recording the Euclidean distance from the camera's optical center to the surface of every visible object in the scene. Its pixel values ​​represent the actual physical distance (usually in millimeters or meters). Absolute depth maps can be stored in a high-precision format (e.g., 32-bit floating-point Open EXR format) to preserve the original physical scale. This is used to accurately calculate the disparity ground truth based on the stereo vision principle D=(f×B) / Z (where D is disparity, f is focal length, B is baseline, and Z is depth), and is a key label for supervising stereo matching network training. Correspondingly, a visual control map is a specially processed geometric information map expressed as a visual image. It is used as a constraint in the subsequent generation stage to guide the generative model, ensuring that the generation process follows the original geometric structure. Furthermore, the visual control map can be an 8-bit grayscale or RGB image, including but not limited to normalized depth maps, normal vector maps, edge contour maps, etc. The visual control map is not used directly for measurement, but serves as a visual representation of geometric conditions for geometric control networks (such as Control Net) to identify and impose constraints.

[0028] In some feasible implementations, the process of acquiring simulated endoscopic images and geometric ground truth data can be achieved by first constructing a virtual industrial endoscopic inspection scene, then configuring a corresponding binocular camera system within this scene, and driving the binocular camera system to move within the scene. For each rendered frame, the binocular camera system synchronously outputs simulated RGB images of the left and right views as simulated endoscopic images, along with absolute depth maps, normalized depth maps, normal vector maps, and edge contour maps for the simulated endoscopic images. This constructs a large-scale simulated dataset with accurate geometric benchmarks, solving the problem of obtaining true depth values ​​in real industrial scenarios. Simultaneously, the synchronous rendering by the binocular camera system ensures that the left and right views and their corresponding absolute depth maps and visual control maps are strictly aligned at the pixel level, providing reliable geometric supervision signals for the subsequent training of the stereo matching network. Furthermore, the multiple geometric information outputs of the normalized depth map, normal vector map, and edge contour map lay the data foundation for multi-dimensional geometric constraints in the generation stage. The normalized depth map, normal vector map, and edge contour map constitute the visual control map.

[0029] In this embodiment of the invention, the process of constructing an industrial endoscope inspection scene can first create a virtual scene in a 3D rendering engine, and select a cavity model and industrial parts from the model library. Then, the cavity model is rendered in the virtual scene, and industrial parts are randomly scattered in the cavity model. At least one of rotation, scaling and translation changes is applied to the industrial parts to simulate the geometric differences of different engine models or different inspection positions. Then, the basic material of the industrial parts is obtained and assigned to the industrial parts to obtain a virtual industrial endoscope inspection scene.

[0030] To simulate the complex internal environment of industrial endoscopes (such as borescopes for aero-engines), a single static model cannot be used alone; a diverse set of parametric scenes must be constructed. In practice, corresponding virtual scenes can be created in a 3D rendering engine (such as Blender), and cavity models and various industrial parts can be selected from a pre-built 3D asset library. The cavity model can be a 3D model simulating the central control structure of the object being inspected (such as the annular casing of an aero-engine, the inner wall of the combustion chamber, bends or tube bundles in a piping system, etc.). Industrial parts can be specific components or features installed inside the cavity or constituting the cavity structure that need to be observed and inspected, such as aero-engine blades (rotor blades, stator blades), rivets, bolts, welds, pipe joints, etc. Alternatively, to train defect detection algorithms, 3D defect models such as pits, cracks, scratches, and corrosion points can be added to the surface or edges of the parts.

[0031] The asset library can include aircraft engine blades (rotors / stator), combustion chamber walls, rivets, bolts, welds, and typical defect models (such as dents and cracks). Using Blender's Geometry Nodes or Python scripts (such as the Blender Python API), ring-shaped or tubular cavity structures are randomly generated in virtual space. Then, blades, screws, and other parts are randomly scattered on the cavity walls, and random rotation, scaling, and translation transformations are applied to these parts to simulate the geometric differences between different engine models or different inspection locations. Subsequent textures are mainly synthesized from generative large models. At this stage, basic PBR (Physically Based Rendering) materials (such as Base Color and Roughness) can be assigned to the objects so that the rendering engine can correctly calculate the lighting and reflection logic, ensuring the lighting rationality of the initial rendered image. Through this process, a virtual but geometrically rich and varied industrial endoscope inspection scene is finally obtained.

[0032] After constructing a virtual industrial endoscope scene, a corresponding binocular camera system can be configured within the scene and driven to move within it. When driving the binocular camera system, camera parameters can be configured, including optical parameters and path planning information. Optical parameters can be the field of view and depth of field. The field of view can preferably be set to a wide range of 80° to 100° to simulate the wide-angle imaging characteristics of a real industrial endoscope, ensuring the camera can capture a sufficiently wide internal field of view to cover key inspection areas such as blades and pits. The depth of field can be set to focus on typical inspection surfaces (such as blade surfaces), creating physically correct defocus blur in background areas far from the focal plane. This natural depth-of-field effect helps subsequent deep learning models understand the hierarchical relationships within the scene, thereby improving the perception of 3D spatial structures. Path planning information includes parameters such as the camera module's trajectory, step size, speed, and random perturbation amplitude within the cavity, used to define the camera system's motion behavior in the virtual scene.

[0033] Based on camera parameters, on the one hand, optical parameters can be configured for the binocular camera system to ensure that the generated simulated images are highly consistent with real industrial endoscopes in terms of optical imaging characteristics (such as field of view, depth of field blur, etc.), providing a realistic optical prior for subsequent domain transfer. On the other hand, according to path planning information, the binocular camera system can be controlled to advance along the centerline inside the industrial endoscope inspection scene, and random shaking and rotation can be added during the advancement process. This achieves a realistic simulation of the unavoidable slight shaking, changes in forward speed, and angle deflection during manual handheld endoscope operation. As a result, the generated image dataset covers a variety of shooting conditions, such as frontal view, side view, tilt, and even extreme close-to-the-surface, greatly enriching the diversity of the data. The stereo matching model trained based on this data has stronger robustness and generalization ability when facing the ever-changing shooting postures in real inspection.

[0034] In some examples, two virtual cameras (Left_Cam and Right_Cam) are created in the industrial endoscope inspection scene and bound to a parent empty object (Rig) with a fixed baseline distance, such as 2mm to 5mm, to simulate the specifications of a real industrial endoscope. Simultaneously, the camera's optical parameters need to be set, including the field of view (FOV) (e.g., 80°–100° wide-angle) and depth of field (DOF), to ensure physically correct blurring of the distant background when focusing on the blade surface. This helps the subsequent model understand spatial hierarchy. Then, according to preset path planning information, the binocular camera system can be controlled to advance along the centerline within the industrial endoscope inspection scene, with random jitter and rotation added during the advance to simulate the unstable operation of a person manually holding an endoscope, ensuring that the shooting angle covers various working conditions such as frontal view, side view, and extreme close-to-the-surface views.

[0035] Furthermore, after configuring the binocular camera system, to achieve simultaneous output of multiple data streams during a single rendering, a multi-data-stream output pipeline can be configured using the compositor node of the rendering engine. This pipeline can simultaneously output simulated endoscopic images, absolute depth maps, and visual control maps. The simulated endoscopic images are the conventionally rendered color images, which can be used as the visual basemap for subsequent style transfer. The absolute depth map can be directly extracted from the Z-buffer of the rendering layer, representing the Euclidean distance (in meters or millimeters) from the camera's optical center to the object's surface. Optionally, to preserve physical scale, the absolute depth map can be stored in a 32-bit floating-point format (such as OpenEXR or TIFF). This depth map will be used for supervised training of the subsequent stereo matching network, ensuring that accurate parallax ground truth values ​​can be calculated according to the formula.

[0036] Furthermore, the visual control map can include at least a normalized depth map, a normal vector map, and an edge contour map. The absolute depth map can be normalized by mapping physical distance values ​​to a preset range and converting it to an 8-bit grayscale image to generate the normalized depth map. This process preserves relative depth relationships, making it suitable as a visual control input. For example, adding a MapRange node to the compositor maps the distance from the nearest to the farthest clipping face to the [0, 1] range, then converts it to a grayscale image of 0-255 (e.g., near white, far black or near black, far white), thus obtaining the corresponding normalized depth map. The normalized depth map can be used to guide the generative model to recognize the near-far occlusion relationship of objects without involving physical measurements. The normal vector map can output surface normal information in camera space, recording the orientation of the object's surface, and strongly constraining the generative model to correctly render the object's convex and three-dimensional appearance. Edge contour maps can be generated by using Blender's Freestyle rendering layer or Sobel operator post-processing nodes to extract the geometric contours of objects in the scene, producing a line drawing with black background and white lines. This clearly defines the boundaries of objects (such as leaves, screws, etc.) and prevents the generated model from blurring or distorting the edges of objects during style transfer.

[0037] Optionally, for the above process, corresponding scripts can be written to automate batch generation. Specifically, unattended data generation can be achieved by writing Python scripts. The scripts can loop through a set number (e.g., 10,000 pairs) of generation tasks: randomly resetting scene parameters each frame (e.g., fine-tuning leaf positions, changing light angles, etc.), moving the camera to the next path point, executing rendering commands, and simultaneously saving multi-channel data files. The final generated dataset can include: simulated RGB images of the left / right views (e.g., Left_RGB.png, Right_RGB.png), physical depth ground truth values ​​of the left / right views (e.g., Left_Depth_Truth.exr, Right_Depth_Truth.exr, 32-bit floating-point format), and corresponding visual control maps (e.g., Left_Depth_Norm.png, Right_Depth_Norm.png; Left_Normal.png, Right_Normal.png; Left_Edge.png, Right_Edge.png), etc. This data forms the basis for subsequent steps.

[0038] Through the above steps, this invention can obtain a large number of simulation data pairs with absolutely accurate geometric structures and various geometric constraints. The absolute depth maps in these data pairs can be directly used for training the stereo matching network, while the visual control maps serve as a "geometric lock" in the generation stage, ensuring that subsequent style transfer does not change the pixel-level geometric positions.

[0039] Step 102: Obtain the lightweight style model, and use the simulated endoscope image as the visual base map, and the absolute depth map and the visual control map as constraints. After obtaining the source domain data with accurate geometry, the texture style of the target domain can be introduced, such as real industrial endoscope images. Since real image data is limited and does not require paired data, in this embodiment of the invention, a low-rank adaptive fine-tuning technique can be used to learn style features from a small number of real images and embed the learned style into a generative large model to obtain a corresponding lightweight style model. Then, the simulated endoscope image can be used as the visual base map, and the absolute depth map and visual control map can be used as constraints to achieve the corresponding domain transfer.

[0040] In some feasible implementations, the lightweight style model construction process can begin by acquiring real industrial endoscope images and generating text descriptions with specific trigger words for each image. Next, a pre-trained generative base model is obtained, which includes at least a variational autoencoder, a text encoder, and a UNet network. The pre-trained weights in the base model are kept unchanged, and a trainable low-rank adaptation matrix is ​​inserted in parallel through the sidepath of the UNet network's weight matrix. Then, the variational autoencoder maps the industrial endoscope images to a first latent vector, and... The text encoder encodes the text description into a text feature vector and adds noise to the first latent vector to obtain a noisy second latent vector. The second latent vector, the text feature vector, and the time step information corresponding to the added noise are input into the UNet network after inserting the low-rank adaptation matrix to perform noise prediction and obtain the predicted noise. Finally, the error between the predicted noise and the added noise is calculated, and based on this error, the parameters of the low-rank adaptation matrix are updated only through the backpropagation algorithm, while the original weights of the UNet network are frozen until the training converges. The parameters of the low-rank adaptation matrix obtained by training are saved as a lightweight style model.

[0041] In practical implementation, a small number (e.g., 50-100) of real industrial endoscope images can be collected first. These images do not need to contain corresponding depth ground truth or camera parameters; only RGB images are required. Simultaneously, the dataset covers different exposure conditions, varying degrees of wear (carbon buildup, scratches), and typical imaging artifacts (such as lens smudges and noise). Then, the images are preprocessed, uniformly adjusted to a resolution suitable for large model training (e.g., 1024×1024 pixels), and normalized. Next, a corresponding text description is generated for each image. This can be done through manual annotation or automated algorithms (such as BLIP, WD14 models), and a specific trigger word is uniformly embedded in all text descriptions. For example, a custom character "endo-style" could be used, with example label formats such as "endo-style, close-up, metalsurface, oilstains, highisonoise, darkbackground...". This establishes a strong association between the trigger word and the specific style of the endoscope, allowing the model to subsequently invoke specific style features through this trigger word.

[0042] Next, a pre-trained large-scale text-to-image model, such as SDXL or FLUX, can be selected. Optionally, taking SDXL as an example, its core components may include a variational autoencoder (VAE), text encoders, and a UNet network. When training LoRA, all pre-trained weights in the base model are kept unchanged to preserve its ability to understand the general world. In the UNet weight matrix W∈R... d×k Bypass, parallel insertion of two low-rank matrices A∈R r×k and B∈R d×r , where the rank r If min(d, k) (e.g., r = 64 or r = 128), the updated weights can be expressed as W = W0 + BA. Compared to full fine-tuning, LoRA only requires training a very small number of parameters (matrices A and B) to efficiently capture texture styles in specific domains and is less prone to overfitting.

[0043] Furthermore, after inserting a trainable low-rank adaptation matrix, model training can be performed. The model training process is essentially a denoising learning process: First, the input real endoscopic RGB image x is compressed and mapped to a low-dimensional latent space using the VAE's encoder, obtaining the latent vector z0. Then, random noise following a Gaussian distribution is superimposed on the latent vector z0. After t-step diffusion, the noisy latent vector z is obtained. t Simultaneously, a text encoder is used to encode the corresponding text description (including trigger words) into a text feature vector. (The last part, "z," appears to be a typo and can be left as is.)t The current time step t and the text feature vector are fed into a UNet network with a low-rank adaptation matrix inserted for noise prediction, resulting in predicted noise. θ Next, the predicted noise and the actual noise are calculated. The differences between the models (such as mean squared error, MSE, and loss) are calculated using the backpropagation algorithm, but only the parameters of the inserted low-rank matrix are updated; the weights of the base model remain frozen. This process is repeated until the loss function converges. The trained low-rank adaptation matrix parameters are then saved as a LoRA weight file, which is the lightweight model. Optionally, this model file can be a very small file (e.g., tens of MB) for easy deployment.

[0044] Through the above process, the constructed lightweight style model can extract texture style from a small number of real images, and can be flexibly invoked through trigger words, so that the lightweight style model can be responsible for giving the simulated image realistic texture, noise, gloss and other appearance characteristics in subsequent generation steps.

[0045] Step 103: Input the visual base map and the constraints into the generation pipeline composed of the lightweight style model and the geometric control network. Perform texture style transfer on the visual base map through the lightweight style model, and apply geometric constraints to the generation process according to the constraints through the geometric control network, and output a domain-transferred image with realistic texture style and maintaining the geometric structure of the simulated endoscope image.

[0046] After generating the corresponding simulation data and training the corresponding lightweight style model through the aforementioned process, a geometric control network can be combined to perform graph-to-graph diffusion generation under strict geometric constraints. By inputting the visual base map and constraints into the generation pipeline composed of the lightweight style model and the geometric control network, the lightweight style model performs texture style transfer on the visual base map, and the geometric control network applies geometric constraints to the generation process according to the constraints. The output is a domain-transferred image with realistic texture style and maintaining the geometric structure of the simulated endoscope image. This makes the output domain-transferred image have both the appearance features of the real industrial scene and the accurate geometric structure of the simulation data source, effectively reducing the domain difference between the simulation and the real scene. This provides a reliable data foundation for training a stereo matching model with high accuracy and strong robustness in real complex environments, thereby improving the performance of the model in real industrial scenes.

[0047] In some feasible implementations, the geometric control network includes at least one of a depth control unit, a normal control unit, and an edge control unit. First, text prompts with specific trigger words are acquired. Then, the visual base map is encoded into the latent space to obtain a first target latent vector. Next, the redraw magnitude of the first target latent vector is obtained, and Gaussian noise is added to the first target latent vector according to the redraw magnitude to obtain a second target latent vector with added noise. The normalized depth map is input into the depth control unit for feature extraction to obtain depth features. The normal vector map is input into the normal control unit for feature extraction to obtain normal features. The edge contour map is input into the edge control unit for feature extraction to obtain edge features. Then, the depth features, normal features, and edge features are weighted and superimposed to obtain geometric constraint features. During iterative denoising, the second target latent vector, geometric constraint features, and text prompts are input together into a generative network that incorporates a lightweight style model to predict noise and update the latent vectors. Finally, after the denoising iteration is completed, the final latent vector is decoded into a pixel image to obtain the domain-transferred image.

[0048] In the specific implementation, the SDXL base model (including VAE, text encoder, UNet, etc.) is first loaded. Then, the LoRA weight file (i.e., the lightweight style model) trained in the previous process is merged into the UNet weights of the base model in a weighted summation manner, and the hybrid weight (LoRAScale) is assumed to be set to 0.6~1.0. This value determines the degree to which the generated image retains the original SDXL capability and reflects the realistic style of LoRA endoscopy. Next, text prompts containing specific trigger words are constructed, such as: "endo-style, close-up, metalsurface, oilstains, highisonoise, specularreflection, darkbackground". These prompts can be converted into feature vectors by the text encoder and used to guide the texture features generated in the image during the denoising process. Finally, the pre-trained Control Net model weights (adapted to the SDXL architecture) are loaded. According to the conditional graph type output in the previous embodiment, one or more independent Control Net model units are selected to form parallel control. Optional: Unit A (Depth Control): Inputs a normalized depth map and uses the Control Net-Depth model to constrain the 3D spatial layout (Z-axis information) of the scene at a macro level.

[0049] Unit B (Normal Control): Input normal vector map, using Control Net-Normal model, is responsible for constraining the curvature orientation of the object's surface to ensure that the metallic specular highlight is consistent with the geometry.

[0050] Unit C (Edge Control): Input edge contour map, using Control Net-Canny model, responsible for locking the high-frequency pixel boundaries of components to prevent texture overflow.

[0051] Furthermore, the Control Net can essentially be a trainable copy with the exact same structure as the main UNet encoder, connected to the main network via zero-convolutional layers. During generation, this copy extracts the spatial structure information from the input geometric condition map, converts it into feature residuals, and injects them additively into the decoder layer of the main network. This forces the spatial structure of the image to remain consistent with the input conditions without compromising the texture generation capabilities of the base model. Correspondingly, the residual features F output by the multiple Control Nets... depth F normal F edge A linear weighted summation will be performed: F total_residual =ω d F depth +ω n F normal +ω e F edge Optionally, in this embodiment, weights can be preferably configured. , , The final synthesized F total_residual It is injected into the main network, forming a comprehensive geometric constraint.

[0052] Taking the left view as an example, the Left_RGB.png output in the aforementioned embodiment is read as the visual base map. It is mapped to a latent space vector z0 using a VAE encoder. Gaussian noise is added to z0 according to the set denoising strength (δ) (preferably within the range of 0.5 to 0.9) to obtain the initial noise vector z. T The redraw magnitude determines the degree to which the original image texture is destroyed; the larger the δ value, the more reliant the AI ​​is on regenerating the texture. Simultaneously, the three geometric feature maps output from the aforementioned embodiments—Left_Depth_Norm.png, Left_Normal.png, and Left_Edge.png—are read in parallel. The resolution of these three images is uniformly adjusted (e.g., to 1024×1024) to match the size of the visual base image, and then converted into the tensor format (Batch, Channel, Height, Width) required by the network. Then, the processed tensors can be used as guiding inputs c. depth c normal cedge The data is then sent to the corresponding Control Net control unit to prepare for feature extraction, thereby obtaining the corresponding depth features, normal vector features, geometric features, etc.

[0053] Furthermore, based on the extracted multipath geometric features, the system can simultaneously process the input tensor c. depth c normal c edge Multi-scale feature extraction is performed to capture geometric information from macroscopic layout to microscopic edges, and then the data is processed according to a preset weight ω. d ω n ω e The feature maps output by the three sources are linearly weighted and superimposed to synthesize the total geometric residual feature F. total_residual Among them, depth features ensure the accuracy of scene context, normal features ensure the correctness of object surface undulations, and edge features ensure pixel-level locking of component contours. Next, the system can perform denoising based on the LoRA latent space. Specifically, the main UNet network (with LoRA weights fused) combines the feature vectors of the text prompt words and the current noisy image z... t and the geometric features F injected by Control Net total_residual Predicting noise θ To ensure the generated images can serve as reliable training data for stereo matching, the proportion of Control Net guided steps can be set to 1.0 (i.e., 100%). This means that geometric constraints remain enabled from the first to the last step of denoising, effectively preventing subtle geometric drift in the later stages of denoising. Based on the predicted noise, the latent vector z is processed using a preset sampling algorithm (such as Eulera or DPM++). t The process involves making corrections to gradually recover clear features from the initial random noise. At this stage, LoRA is responsible for determining color and texture (such as metallic sheen, oil stains, and noise), while Control Net ensures the accuracy of geometric features. After the iterations are complete, the VAE decoder can be used to decode the final latent vector z'0 into a pixel-level image, thus obtaining the final realistic endoscopic image.

[0054] Repeat the above process to independently redraw the left and right views output in the previous embodiments, resulting in a pair of images with realistic appearance and correct stereo matching relationship. Since the generation process of the left and right views is based on the same set of geometric control maps (but each corresponds to its own view), and the geometric constraints ensure that the pixel-level position remains unchanged, the generated left and right images still maintain accurate epipolar geometric relationships, and their corresponding absolute depth maps are still valid and can be directly used for supervised training of the stereo matching network.

[0055] It should be noted that the embodiments of the present invention include, but are not limited to, the examples described above. It is understood that those skilled in the art can make further settings according to actual needs under the guidance of the ideas in the embodiments of the present invention, and the present invention does not limit such settings.

[0056] In this embodiment of the invention, by generating paired simulated endoscope images and their corresponding geometric ground truth data, a data source with accurate geometric benchmarks is constructed for the stereo matching training task. This solves the problem of difficulty in obtaining depth ground truth in real industrial scenarios. Furthermore, by acquiring a lightweight style model and combining it with a geometric control network to form a generation pipeline, this generation pipeline gains the ability to learn and transfer complex texture styles from real images. During the generation process, the simulated image is used as a visual base map, and its corresponding visual control map is input as a constraint condition. The tasks of "texture style transfer" and "geometric structure preservation" are processed collaboratively, ensuring that the original geometric structure of the simulated image, which strictly corresponds to the absolute depth map, is completely preserved during the transfer of texture to realism. This results in the output domain-transferred image possessing both the appearance features of real industrial scenarios and the accurate geometric structure of the simulated data source, effectively reducing the domain difference between simulation and real scenarios. This provides a reliable data foundation for training a stereo matching model with high accuracy and strong robustness in real complex environments, thereby improving the model's performance in real industrial scenarios.

[0057] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are provided for illustrative purposes: As examples, suppose we need to train a stereo matching network for detecting defects in aero-engine blades. First, following step 101 in this embodiment, a virtual engine cavity scene containing various blades, dents, and cracks can be constructed in Blender. Blades are randomly placed using a script, the binocular camera baseline is set to 3mm, the field of view is 90°, and the camera is slowly moved along a preset path with slight random jitter. For each frame rendered, left RGB, right RGB, a 32-bit floating-point depth map (.exr), a normalized depth map (grayscale.png), a normal vector map (RGB.png), and an edge contour map (grayscale.png) are output simultaneously. A total of 10,000 sets of data are generated.

[0058] Next, following step 102 in this embodiment, 80 real endoscopic photographs of the engine interior were collected, including images with different lighting conditions and varying levels of oil contamination. Each image was labeled with a text description, and the trigger word "endo-style" was uniformly added. The SDXL base model was loaded, and a LoRA matrix with a rank of 64 was inserted into its UNet. The model was trained using the aforementioned images and text, updating only the LoRA parameters to obtain a lightweight style model file (approximately 60MB).

[0059] Then, following the steps in step 103 of this embodiment, a generation pipeline is constructed. The SDXL model is loaded, and LoRA weights are merged (scale=0.8). Three Control Net units are loaded, corresponding to depth, normal, and edge respectively. A set of left-view data is selected from the data generated in step 101: left RGB as the visual base map, and left normalized depth map, left normal vector map, and left edge contour map as geometric conditions. The redraw magnitude is set to 0.75, with the prompts "endo-style, close-up, turbine blade, metal surface, oilstains, highisonoise". Iterative denoising begins. In each step, the three Control Nets extract geometric features, weighted and fused, and then injected into the main UNet, working in conjunction with the LoRA style guidance. After 50 steps, a new left-view image is obtained, which looks like a real endoscope image, with a metallic sheen, oil stains, and noise on the surface, but the edge and depth relationships of the blades are completely consistent with the original simulation image. The right-view image is processed similarly to obtain a pair of images. The image pair can be combined with the corresponding absolute depth map from step 101 to train the stereo matching network.

[0060] The above methods enable the large-scale generation of training data with realistic appearance and geometric accuracy, effectively solving the problems of scarce real data and unrealistic simulation data.

[0061] Furthermore, in some alternative embodiments and variations, those skilled in the art can adjust and optimize each step according to specific needs.

[0062] For example, in step 101 of this embodiment, in addition to Blender, other engines that support Python scripts and physically based rendering, such as Unreal Engine or Unity, can also be used. Parameters such as camera baseline and field of view can be adjusted according to the specific endoscope model. The visual control graph can also include semantic segmentation graphs, albedo graphs, etc., to provide more dimensional geometric constraints.

[0063] In step 102 of this embodiment, in addition to SDXL, the base model can also be a newer model such as StableDiffusion3 or FLUX. The rank of LoRA can be adjusted according to the amount of training data and style complexity; for example, the rank can be appropriately increased for scenes with large style differences. Regularization terms can also be added during training to prevent overfitting.

[0064] In step 103 of this embodiment, the redrawing magnitude δ can be adjusted according to the style intensity of the generated image: a smaller δ (e.g., 0.5) will make the generated image closer to the original simulation image with less style change; a larger δ (e.g., 0.9) will make the style of the generated image closer to the style learned by LoRA, but may slightly reduce geometric fidelity, in which case the weights of the Control Net need to be increased. The weights of each channel of the Control Net can also be dynamically adjusted according to the actual effect. For example, if the edge contour is very important, ω can be appropriately increased. e Furthermore, the generation process can employ different samplers (such as DDIM, PLMS) to balance speed and quality.

[0065] In addition, the present invention can be extended to other stereo vision tasks that require domain transfer, such as autonomous driving and robot grasping, simply by adjusting the virtual scene and the real domain image accordingly.

[0066] In some examples: Example 1: The randomization process in parametric modeling In Blender, the position of the blades is defined using a Python script within a random range: x-axis offset ±10mm, y-axis offset ±5mm, z-axis rotation 0–360°, and scaling 0.8–1.2. Light intensity is randomly set between 500 and 1500 lumens, and the light angle varies randomly. Before each render, the script re-executes these randomization operations to ensure that each generated frame has a different geometric layout and lighting conditions, thus increasing data diversity.

[0067] Example 2: Compositor node settings for multi-channel rendering Enable Z-channels, normal channels, and UV channels in the BlenderCycles rendering layer. In the compositor editor, add a "MapRange" node to map the nearest distance (e.g., 5mm) and farthest distance (e.g., 100mm) of the Z-channel to 0-1, then multiply by 255 using the "Math" node to output a grayscale image. Simultaneously, use the "Freestyle" rendering layer and the "Line Art" modifier to extract edges, outputting a black and white line drawing. Save all outputs in different file formats using the "File Output" node.

[0068] Example 3: Details of text annotation during LoRA training Suppose a real image shows a blade with carbon deposits, surrounded by oil stains, and captured in a dark environment. The description generated using the BLIP model might be "a close-up of a metal surface with dark stains." After manual correction and the addition of trigger words, it becomes "endo-style, turbine blade with carbon deposits, oilstains, dark background, high iso noise." All image descriptions include "endo-style," allowing the model to learn to associate this term with the overall style of the endoscopic image.

[0069] Example 4: Feature weighted fusion during the generation process Suppose that in a certain generation, the depth map shows clear distinction between the front and back layers of the blade, but the edge contours are complex. To improve the edge sharpness, ω is set... d =0.9, ω n =0.8, ω e =1.2, which makes the contribution of edge features greater, ensuring sharp leaf contours in the generated image. The fused feature F total_residual Inject UNet at each time step to guide the generation process.

[0070] Example 5: Independent generation and consistency verification of left and right views After generating the left view, apply the exact same Control Net and LoRA settings to the right view, but input the corresponding geometric conditions for the right view. Since the geometric condition maps of the left and right views are strictly corresponding (obtained through stereo rendering), the generated left and right images naturally maintain epipolar constraints. To verify this, the disparity maps of the generated left and right images can be calculated and compared with the true disparity values ​​derived from the original absolute depth map; the error should be within an acceptable range.

[0071] Example 6: Comparison of effects of different redraw amplitudes Two sets of experiments were set up: one with δ=0.5 and the other with δ=0.9. The generated results showed that the image with δ=0.5 retained more texture details of the original simulation image, but style transfer was not significant; the image with δ=0.9 was closer to a real endoscopic photograph, with significant metallic sheen and noise, but occasional minor edge blurring. The two can be balanced by adjusting the Control Net weights, for example, by adjusting ω at δ=0.9. e Increasing it to 1.5 can suppress edge blurring.

[0072] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0073] Reference Figure 2 The diagram illustrates a structural block diagram of a data migration device for endoscopic stereo matching provided in an embodiment of the present invention, which may specifically include the following modules: The simulation module 201 is used to generate a set of simulated endoscope images containing pairs of simulated endoscope images and geometric truth data corresponding to the simulated endoscope images. The geometric truth data includes at least an absolute depth map and a visual control map for geometric constraints. The model acquisition module 202 is used to acquire a lightweight style model, and use the simulated endoscope image as a visual base map, and the absolute depth map and the visual control map as constraints. The migration module 203 is used to input the visual base map and the constraints into the generation pipeline composed of the lightweight style model and the geometric control network, perform texture style transfer on the visual base map through the lightweight style model, and apply geometric constraints to the generation process according to the constraints through the geometric control network, and output a domain-transferred image with a realistic texture style and maintaining the geometric structure of the simulated endoscope image.

[0074] In some feasible implementations, the simulation module 201 is specifically used for: Construct a virtual industrial endoscope inspection scenario; A corresponding binocular camera system is configured in the industrial endoscope inspection scenario, and the binocular camera system is driven to move in the industrial endoscope inspection scenario; The binocular camera system synchronously outputs simulated RGB images of the left and right views for each rendered frame as the simulated endoscope image, as well as the absolute depth map, normalized depth map, normal vector map, and edge contour map of the simulated endoscope image. The normalized depth map, the normal vector map, and the edge contour map constitute the visual control map.

[0075] Among some feasible implementation methods are: The storage module is used to store the absolute depth map in 32-bit floating-point format; The normalization module is used to normalize the absolute depth map, map the physical distance values ​​to a preset range, and convert them into an 8-bit grayscale image to generate the normalized depth map.

[0076] In some feasible implementations, the simulation module 201 is specifically used for: Create virtual scenes in a 3D rendering engine and select cavity models and industrial parts from the model library; The cavity model is rendered in the virtual scene, and the industrial parts are randomly scattered in the cavity model. At least one of rotation, scaling and translation is applied to the industrial parts to simulate the geometric differences of different engine models or different detection positions. Obtain the base material of the industrial part and assign the base material to the industrial part to obtain a virtual industrial endoscope inspection scene.

[0077] In some feasible implementations, the simulation module 201 is specifically used for: Obtain camera parameters for the binocular camera system, the camera parameters including optical parameters and path planning information; Configure the optical parameters for the binocular camera system; According to the path planning information, the binocular camera system is controlled to advance along the centerline inside the industrial endoscope inspection scene, and random shaking and rotation are added during the advancement.

[0078] Among some feasible implementation methods are: The image acquisition module is used to acquire real industrial endoscope images and generate a text description with specific trigger words for each of the industrial endoscope images; The model acquisition module is used to acquire a pre-trained generative base model, which includes at least a variational autoencoder, a text encoder, and a UNet network. A matrix insertion module is used to keep the pre-trained weights in the base model unchanged and insert trainable low-rank adaptation matrices in parallel in the bypass of the weight matrix of the UNet network. The vector mapping module is used to map the image captured by the industrial endoscope into a first latent vector using the variational autoencoder, and to encode the text description into a text feature vector using the text encoder, and to add noise to the first latent vector to obtain a noisy second latent vector. The prediction module is used to input the second hidden vector, the text feature vector, and the time step information corresponding to the added noise into the UNet network after inserting the low-rank adaptation matrix to perform noise prediction and obtain the predicted noise. The training module is used to calculate the error between the predicted noise and the added noise, and based on the error, update only the parameters of the low-rank adaptation matrix through the backpropagation algorithm, while freezing the original weights of the UNet network until the training converges. The parameters of the low-rank adaptation matrix obtained by training are then saved as the lightweight style model.

[0079] In some feasible implementations, the geometric control network includes at least one of a depth control unit, a normal control unit, and an edge control unit, and the migration module 203 is specifically used for: Get text prompts with specific trigger words; The visual base image is encoded into the latent space to obtain the first target latent vector; Obtain the redraw magnitude for the first target latent vector, add Gaussian noise to the first target latent vector according to the redraw magnitude, and obtain the second target latent vector after adding noise; The normalized depth map is input into the depth control unit for feature extraction to obtain depth features; The normal vector map is input into the normal control unit for feature extraction to obtain normal features; The edge contour map is input into the edge control unit for feature extraction to obtain edge features; The depth feature, the normal feature, and the edge feature are weighted and superimposed to obtain the geometric constraint feature; During the iterative denoising process, the second target latent vector, the geometric constraint features, and the text prompt words are input into a generative network that integrates the lightweight style model to predict noise and update the latent vector. After the denoising iteration is completed, the final latent vector is decoded into a pixel image to obtain the domain-transferred image.

[0080] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0081] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described data migration method embodiment for endoscopic stereo matching and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0082] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the data migration method embodiment for stereoscopic matching of endoscopes described above, and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0083] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program code.

[0085] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0089] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0090] The above provides a detailed description of the data migration method and device for endoscopic stereo matching provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data transfer method for endoscopic stereo matching, characterized in that, include: Generate a set of simulated endoscope images containing pairs of simulated endoscope images and corresponding geometric truth data, wherein the geometric truth data includes at least an absolute depth map and a visual control map for geometric constraints. A lightweight style model is obtained, and the simulated endoscope image is used as the visual base map, while the absolute depth map and the visual control map are used as constraints. The visual base map and the constraints are input into the generation pipeline consisting of the lightweight style model and the geometric control network. The lightweight style model performs texture style transfer on the visual base map, and the geometric control network applies geometric constraints to the generation process according to the constraints, outputting a domain-transferred image with a realistic texture style and maintaining the geometric structure of the simulated endoscope image.

2. The method according to claim 1, characterized in that, The generation of the simulated endoscope images, comprising pairs of simulated endoscope images and corresponding geometric ground truth data, includes: Construct a virtual industrial endoscope inspection scenario; A corresponding binocular camera system is configured in the industrial endoscope inspection scenario, and the binocular camera system is driven to move in the industrial endoscope inspection scenario; The binocular camera system synchronously outputs simulated RGB images of the left and right views for each rendered frame as the simulated endoscope image, as well as the absolute depth map, normalized depth map, normal vector map, and edge contour map of the simulated endoscope image. The normalized depth map, the normal vector map, and the edge contour map constitute the visual control map.

3. The method according to claim 2, characterized in that, Also includes: The absolute depth map is stored in 32-bit floating-point format; The absolute depth map is normalized by mapping the physical distance values ​​to a preset range and converting them into an 8-bit grayscale image to generate the normalized depth map.

4. The method according to claim 2, characterized in that, The construction of the virtual industrial endoscope inspection scenario includes: Create virtual scenes in a 3D rendering engine and select cavity models and industrial parts from the model library; The cavity model is rendered in the virtual scene, and the industrial parts are randomly scattered in the cavity model. At least one of rotation, scaling and translation is applied to the industrial parts to simulate the geometric differences of different engine models or different detection positions. Obtain the base material of the industrial part and assign the base material to the industrial part to obtain a virtual industrial endoscope inspection scene.

5. The method according to claim 2 or 4, characterized in that, The method of driving the binocular camera system to move in the industrial endoscopy inspection scenario includes: Obtain camera parameters for the binocular camera system, the camera parameters including optical parameters and path planning information; Configure the optical parameters for the binocular camera system; According to the path planning information, the binocular camera system is controlled to advance along the centerline inside the industrial endoscope inspection scene, and random shaking and rotation are added during the advancement.

6. The method according to claim 1, characterized in that, Also includes: Acquire real industrial endoscope images and generate a text description with specific trigger words for each of the industrial endoscope images; Obtain a pre-trained generative base model, wherein the generative base model includes at least a variational autoencoder, a text encoder, and a UNet network; Keep the pre-trained weights in the base model unchanged, and insert a trainable low-rank adaptation matrix in parallel through the weight matrix of the UNet network. The variational autoencoder is used to map the image captured by the industrial endoscope into a first latent vector, and the text encoder is used to encode the text description into a text feature vector. Noise is added to the first latent vector to obtain a noisy second latent vector. The second hidden vector, the text feature vector, and the time step information corresponding to the added noise are input into the UNet network after inserting the low-rank adaptation matrix to perform noise prediction and obtain the predicted noise. Calculate the error between the predicted noise and the added noise, and based on this error, update only the parameters of the low-rank adaptation matrix using the backpropagation algorithm, while freezing the original weights of the UNet network until training converges. Save the parameters of the trained low-rank adaptation matrix as the lightweight style model.

7. The method according to claim 1 or 6, characterized in that, The geometric control network includes at least one of a depth control unit, a normal control unit, and an edge control unit. The process of performing texture style transfer on the visual base map using the lightweight style model, and applying geometric constraints to the generation process according to the constraints using the geometric control network, to output a domain-transferred image with a realistic texture style and maintaining the geometric structure of the simulated endoscope image, includes: Get text prompts with specific trigger words; The visual base image is encoded into the latent space to obtain the first target latent vector; Obtain the redraw magnitude for the first target latent vector, add Gaussian noise to the first target latent vector according to the redraw magnitude, and obtain the second target latent vector after adding noise; The normalized depth map is input into the depth control unit for feature extraction to obtain depth features; The normal vector map is input into the normal control unit for feature extraction to obtain normal features; The edge contour map is input into the edge control unit for feature extraction to obtain edge features; The depth feature, the normal feature, and the edge feature are weighted and superimposed to obtain the geometric constraint feature; During the iterative denoising process, the second target latent vector, the geometric constraint features, and the text prompt words are input into a generative network that integrates the lightweight style model to predict noise and update the latent vector. After the denoising iteration is completed, the final latent vector is decoded into a pixel image to obtain the domain-transferred image.

8. A data migration device for endoscopic stereo matching, characterized in that, include: The simulation module is used to generate a set of simulated endoscope images containing pairs of simulated endoscope images and geometric truth data corresponding to the simulated endoscope images. The geometric truth data includes at least an absolute depth map and a visual control map for geometric constraints. The model acquisition module is used to acquire a lightweight style model, and uses the simulated endoscope image as the visual base map, and the absolute depth map and the visual control map as constraints. The migration module is used to input the visual base map and the constraints into the generation pipeline composed of the lightweight style model and the geometric control network, perform texture style migration on the visual base map through the lightweight style model, and apply geometric constraints to the generation process according to the constraints through the geometric control network, and output a domain-transferred image with a realistic texture style and maintaining the geometric structure of the simulated endoscope image.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-7.

Images