Degraded image target object region ground truth labeling data generation method and system

Abstract

The present application relates to the technical fields of computer vision, image annotation, three-dimensional reconstruction, image restoration, digital twin and dataset annotation construction, and particularly relates to a method and system for generating ground truth annotation data of a target object region in a degraded image. The method comprises: obtaining a clear image target ground truth three-dimensional representation data; obtaining a real two-dimensional degraded image of the target collected under harsh imaging conditions; estimating or registering a spatial pose of the target relative to an image collection device according to an appearance, contour, key point, semantic region or artificial interaction information of the target in the real two-dimensional degraded image; rendering, projecting or perspective transforming the ground truth three-dimensional representation based on the spatial pose to generate target ground truth data corresponding to a perspective of the target in the real two-dimensional degraded image; and generating annotation data corresponding to the real two-dimensional degraded image according to the target ground truth data. The present application provides training and evaluation data support for image restoration, target detection and other visual model tasks.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image annotation, 3D reconstruction, image restoration, digital twins, and dataset annotation construction, and particularly to a method and system for generating ground truth annotation data for target object regions in degraded images. Background Technology

[0002] In real-world vision applications, images often contain complex backgrounds, varying lighting conditions, and various interfering factors. However, many vision tasks are truly focused on the target itself within the image. For a target, its clear appearance, true contours, visible regions, semantic regions, spatial pose, and geometric structure are crucial ground truth data for model training and performance evaluation. Compared to acquiring ground truth data for the entire image or scene, generating accurate ground truth and labeled data around the target is more beneficial for supporting downstream vision tasks such as image restoration, target recognition, segmentation, pose estimation, and 3D perception.

[0003] For example, in image restoration tasks, a clear appearance of the target in the degraded image is needed as a basis for supervision or evaluation of the restoration results; in target detection and segmentation tasks, the target's bounding box, pixel-level mask, semantic region, or instance annotation are required; in target pose estimation and 3D vision tasks, the target's spatial pose, depth, surface normals, or 3D geometric information relative to the image acquisition device is needed. The quality of the above-mentioned target-level ground truth and annotation data directly affects the training effect and evaluation reliability of the visual model.

[0004] However, under adverse imaging conditions such as fog, rain, snow, underwater conditions, smoke, dust, low light, strong scattering media, turbid media, strong reflection, motion blur, and sensor noise, it is difficult to obtain true-value data of the target in a real image. Adverse imaging conditions cause images to be affected by scattering, absorption, blurring, occlusion, color shift, reduced contrast, and noise, degrading the target's appearance, contours, texture, and boundary information. Since adverse imaging conditions and clear imaging conditions are usually difficult to coexist in the same time, space, and viewpoint, it is difficult to directly obtain true-value data of the target that strictly corresponds to the actual degraded image.

[0005] In existing technologies, one type of method uses manual annotation to label targets in degraded images. However, when target boundaries are blurred, visibility is low, texture is missing, or color shifts are severe, the accuracy, consistency, and repeatability of manual annotation are difficult to guarantee. Another type of method uses simulation to synthesize degraded images from clear images or clear scenes. However, the synthesized degraded models often fail to accurately simulate scattering, occlusion, illumination changes, and sensor noise in real complex environments, resulting in significant domain differences between the synthesized data and the real degraded images. Some methods acquire paired data through multi-sensor systems or controlled acquisition environments, but these methods have high acquisition costs, complex system setups, and limited applicability, making it difficult to scale to large-scale, harsh real-world imaging conditions.

[0006] Therefore, a new method is needed to generate clear ground truth data and labeled data corresponding to the viewpoint of targets in real degraded images using ground truth 3D representations without simultaneously obtaining degraded and clear images. This would provide reliable data support for image restoration, image enhancement, object detection, semantic segmentation, instance segmentation, target pose estimation, and the training and evaluation of 3D vision models. Summary of the Invention

[0007] The technical problem this invention aims to solve is to provide a method and system for generating ground truth data and annotations for target object regions in degraded images. This enables the generation of clear ground truth data and annotations corresponding to the target's viewpoint, spatial pose, and visible area in degraded images acquired under harsh imaging conditions. This invention is applicable to degraded images acquired under adverse imaging conditions such as fog, rain, snow, underwater, smoke, dust, low light, strong scattering media, turbid media, strong reflection, motion blur, and sensor noise. It can be used to generate clear ground truth data and corresponding annotations for targets in degraded images, and provides data support for image restoration, image enhancement, target detection, instance segmentation, semantic segmentation, target pose estimation, and visual model training and evaluation.

[0008] The technical solution of the present invention is as follows: A method for generating ground truth annotation data for a target object region in a degraded image includes: first, acquiring a clear image target ground truth three-dimensional representation data; second, acquiring a real two-dimensional degraded image of the target acquired under adverse imaging conditions; then, estimating or registering the spatial pose of the target relative to the image acquisition device based on the target's appearance, contour, key points, semantic regions, or human interaction information in the real two-dimensional degraded image; further, rendering, projecting, or transforming the ground truth three-dimensional representation based on the spatial pose to generate target ground truth data corresponding to the target's viewpoint in the real two-dimensional degraded image; and finally, generating annotation data corresponding to the real two-dimensional degraded image based on the target ground truth data.

[0009] The three-dimensional representation can be any three-dimensional representation that can describe the three-dimensional shape, appearance, texture, material, color, transparency, spatial structure, or renderable attributes of the target, including but not limited to three-dimensional Gaussian representation, neural radiation field representation, three-dimensional mesh model, CAD model, point cloud model, voxel model, symbolic distance field implicit surface model, digital twin model, three-dimensional assets in game engine, or other renderable three-dimensional models.

[0010] The adverse imaging conditions are at least one of the following: foggy weather, rainy weather, snowy weather, underwater, smoke and dust, sand and dust, low light, strong scattering medium, turbid medium, strong reflection, motion blur, or sensor noise.

[0011] The spatial pose can be the camera pose, the target pose, the relative pose between the target and the image acquisition device, the target rotation parameters, the target translation parameters, the target scale parameters, the camera intrinsic parameters, the camera extrinsic parameters, or a combination thereof.

[0012] The method for determining the spatial pose is at least one of the following: manual interactive registration, semi-automatic registration, automatic registration, key point matching, contour matching, edge matching, feature matching, PnP solving, differentiable rendering optimization, neural network pose estimation, template matching, or multimodal alignment.

[0013] The rendering, projection, or viewpoint transformation of the three-dimensional representation of the true value includes: generating target true value data under the corresponding viewpoint in a three-dimensional rendering engine, neural rendering model, graphics rendering pipeline, differentiable renderer, or three-dimensional reprojection module according to the spatial pose.

[0014] The target ground truth data is at least one of the following: a clear target image, a target foreground image, a target region mask, a target boundary contour, a depth map, a surface normal map, a semantic label map, an instance label map, a transparency map, a visibility map, a pose label, or a reference image used for image restoration.

[0015] The process of generating annotation data corresponding to the real two-dimensional degraded image based on the target ground truth data includes: mapping, projecting, registering, or fusing the target ground truth data into the image coordinate system of the real two-dimensional degraded image to obtain annotation results that are consistent with or correspond to the size and coordinates of the real two-dimensional degraded image.

[0016] The labeled data is at least one of the following: target region mask, target boundary contour, pixel-level semantic category, instance annotation, transparency map, visibility map, depth annotation, normal annotation, or pose annotation.

[0017] The labeled data is used to train or evaluate image dehazing, image deraining, underwater image enhancement, low-light enhancement, image denoising, image super-resolution, object detection, semantic segmentation, instance segmentation, pose estimation, or 3D visual understanding models.

[0018] The beneficial effects of this invention are: (1) The present invention can solve the problem of difficulty in synchronously acquiring clear true data under real adverse imaging conditions. It can generate corresponding clear true data and annotation data for targets in real degraded images without simultaneously acquiring images with completely consistent time, space and viewpoint under adverse imaging conditions and clear imaging conditions.

[0019] (2) The present invention expands the scope of application of the target true value data construction method. It is not limited to a single three-dimensional representation. It can be applied to three-dimensional Gaussian representation, neural radiation field representation, three-dimensional mesh model, CAD model, point cloud model, voxel model, digital twin model, graphics engine three-dimensional assets and other renderable three-dimensional models.

[0020] (3) The present invention can reduce the cost and uncertainty of manual annotation. By estimating or registering the target spatial pose and performing rendering, projection or viewpoint transformation based on the target 3D representation, more stable target boundaries, target regions and supervision data can be obtained, thereby improving the consistency and reliability of annotation data.

[0021] (4) The present invention can generate multiple forms of target ground truth data and labeled data simultaneously, including one or more of clear target images, target region masks, depth maps, surface normal maps, target boundary contours, visibility maps, semantic labels, instance labels and pose information, which can meet the data requirements of image restoration, image enhancement, target detection, semantic segmentation, instance segmentation, pose estimation and three-dimensional vision tasks.

[0022] (5) The present invention uses two-dimensional degraded images acquired under real harsh imaging conditions as input, and uses the three-dimensional representation of the true values ​​to generate true data and labeled data corresponding to its viewpoint and spatial pose. Compared with the data construction method that relies entirely on synthetic degraded images, it can reduce the domain difference between pure synthetic data and real degraded images to a certain extent and improve the applicability of data to real application scenarios. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the overall process of generating ground truth annotation data for a target object region in a degraded image, as provided in an embodiment of the present invention.

[0024] Figure 2 This is a structural module diagram of a system for generating ground truth annotation data for target object regions in degraded images, provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the generation of ground truth and labeled data for spatial pose estimation and target 3D representation rendering, as provided in an embodiment of the present invention. Detailed Implementation

[0026] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0027] Example 1: Overall Method Flow See Figure 1 This embodiment provides a method for generating ground truth annotation data for target object regions in degraded images. The method includes steps S101 to S111.

[0028] S101, Obtain a true 3D representation. This 3D representation describes the target's 3D shape, appearance, texture, material, color, transparency, spatial structure, and / or renderable attributes. The 3D representation can be obtained through 3D scanning, multi-view reconstruction, CAD design, neural rendering training, graphics engine modeling, or digital twin construction.

[0029] S102, acquire a realistic two-dimensional degraded image of the target under adverse imaging conditions. The realistic two-dimensional degraded image can be acquired by an RGB camera, industrial camera, underwater camera, vehicle-mounted camera, drone camera, surveillance camera, or mobile terminal camera. The adverse imaging conditions include at least one of the following: fog, rain, snow, underwater, smoke, dust, low light, strong scattering medium, turbid medium, strong reflection, motion blur, and sensor noise.

[0030] S103, determine the target region from the real two-dimensional degraded image. The target region can be obtained through manual specification, object detection models, semantic segmentation models, instance segmentation models, interactive segmentation models, edge detection, or contour extraction.

[0031] S104, determine the spatial pose of the target relative to the image acquisition device. The spatial pose can be determined based on the target's contour, key points, edges, texture, semantic region, instance region, bounding box, or human interaction information. The spatial pose may include camera pose, target pose, relative pose between the target and the image acquisition device, target rotation parameters, target translation parameters, target scale parameters, camera intrinsic parameters, camera extrinsic parameters, or combinations thereof.

[0032] S105, set virtual camera parameters or rendering parameters based on spatial pose. The virtual camera parameters or rendering parameters may include virtual camera intrinsic parameters, virtual camera extrinsic parameters, target pose, target scale, field of view, image resolution, lighting parameters, or background parameters.

[0033] S106, Based on the virtual camera parameters or rendering parameters, render, project, or transform the 3D representation of the true value. This step can be performed in a 3D rendering engine, neural rendering model, graphics rendering pipeline, differentiable renderer, or 3D reprojection module.

[0034] S107, Generate target ground truth data corresponding to the target viewpoint in the real two-dimensional degraded image. The corresponding viewpoint refers to a viewpoint that is the same as or substantially consistent with the observation viewpoint of the target in the real two-dimensional degraded image. The target ground truth data may include one or more of the following: a clear target image, a target foreground image, a reference image for image restoration, a depth map, a surface normal map, or target pose information.

[0035] S108, Generate labeled data based on the target ground truth data. The labeled data may include one or more of the following: target region mask, target boundary contour, pixel-level semantic category, instance annotation, transparency map, visibility map, depth annotation, normal annotation, or pose annotation. The target region mask may be generated by 3D geometric projection, transparency channel, depth threshold, segmentation model, or interactive segmentation method.

[0036] S109, map the target ground truth data and annotation data to the coordinate system of the real two-dimensional degraded image. The mapping method may include projection, registration, homography transformation, dense optical flow, local deformation correction, or differentiable rendering alignment, so that the target ground truth data and annotation data correspond to the target position in the real two-dimensional degraded image.

[0037] S110, generate a target ground truth and annotation dataset corresponding to the real 2D degraded image. The dataset includes the real 2D degraded image, and one or more of the following: a sharp target image, a target region mask, a depth map, a surface normal map, a target boundary contour, a semantic label, an instance label, a visibility label, or a pose label corresponding to its coordinates.

[0038] S111, the target ground truth and labeled dataset are used for visual model training or evaluation. The visual model training or evaluation may include one or more of the following: image restoration, image enhancement, object detection, semantic segmentation, instance segmentation, target pose estimation, 3D visual perception, or model performance evaluation.

[0039] Example 2: System Structure See Figure 2 This embodiment provides a system for generating ground truth annotation data for target object regions in degraded images, including: a target ground truth 3D representation acquisition module, a degraded image acquisition module, a target region determination module, a spatial pose determination module, a ground truth data generation module, an annotation data generation module, a coordinate mapping module, and a dataset construction module.

[0040] The target truth 3D representation acquisition module is used to acquire the target truth 3D representation. The 3D representation is used to describe the target's 3D shape, appearance, texture, material, color, transparency, spatial structure, and / or renderable attributes.

[0041] The degraded image acquisition module is used to acquire real two-dimensional degraded images of the target under adverse imaging conditions. Adverse imaging conditions include at least one of the following: fog, rain, snow, underwater, smoke, dust, low light, strong scattering medium, turbid medium, strong reflection, motion blur, and sensor noise.

[0042] The target region determination module is used to determine the target region in the real two-dimensional degraded image. The target region can be obtained through manual specification, object detection models, semantic segmentation models, instance segmentation models, interactive segmentation models, edge detection, or contour extraction.

[0043] The spatial pose determination module is used to estimate or register the spatial pose of the target relative to the image acquisition device based on the target region, target contour, key points, edges, texture, semantic region, instance region, bounding box, or human interaction information.

[0044] The truth data generation module is used to render, project, or transform the three-dimensional representation of the true value based on the spatial pose, and generate target truth data corresponding to the target viewpoint in the real two-dimensional degraded image.

[0045] The annotation data generation module is used to generate annotation data based on the target ground truth data. The annotation data includes one or more of the following: target region mask, target boundary contour, depth map, surface normal map, pixel-level semantic category, instance annotation, transparency map, visibility map, or pose information.

[0046] The coordinate mapping module is used to map the target ground truth data and annotation data to the coordinate system of the real two-dimensional degraded image, so that the target ground truth data and annotation data correspond to the target position in the real two-dimensional degraded image.

[0047] The dataset construction module is used to generate a target ground truth and labeled dataset that can be used for model training or evaluation based on the real two-dimensional degraded image, the target ground truth data, and the labeled data.

[0048] Example 3: Spatial Pose Estimation and Target 3D Representation Re-rendering Based on 3D-2D Matching See Figure 3In one specific implementation example, the system first extracts the target region, target contour, or target key points from a real two-dimensional degraded image acquired under adverse imaging conditions, using non-adverse conditions. Then, it constructs a true three-dimensional representation of the target (e.g., a sea cucumber) using multi-view geometry or neural network methods. Next, the system performs three-dimensional-to-two-dimensional registration with the target observation information in the two-dimensional degraded image to obtain the spatial pose of the target relative to the image acquisition device.

[0049] After obtaining the spatial pose, the system renders, projects, and transforms the 3D representation of the ground truth based on the spatial pose to obtain clear ground truth data corresponding to the target viewpoint in the real 2D degraded image. Further, the system can generate annotation data such as target region masks, depth maps, surface normal maps, target boundary contours, transparency maps, visibility maps, or pose information based on the clear ground truth data, and map the annotation data back to the coordinate system of the real 2D degraded image.

[0050] In addition, it should be noted that Figure 3 The 3D Gaussian representation (3DGS), Neural Radiation Field (NeRF), 3D Mesh model, CAD model, and graphics engine 3D assets mentioned are merely examples of the 3D representations and do not constitute a limitation on the form of 3D representation. Any 3D representation that can generate corresponding viewpoint target ground truth data or labeled data based on spatial pose can be used as the 3D representation in this invention.

[0051] Example 4: Implementation methods of different 3D representations In different embodiments of the present invention, the three-dimensional representation of the target truth value can adopt different data forms or model forms. As long as the three-dimensional representation can generate target truth data and annotation data from the corresponding viewpoint based on the spatial pose between the target and the image acquisition device, it can be applied to the present invention.

[0052] Implementation based on 3D Gaussian representation: In one implementation, the true 3D representation is a 3D Gaussian representation. The system sets virtual camera parameters, target pose parameters, and / or rendering parameters in the 3D Gaussian representation based on the target spatial pose estimated or registered in the real 2D degraded image, and renders one or more of the following from the corresponding viewpoint: a clear target image, a target region mask, a depth map, a transparency map, and a visibility map.

[0053] Implementation based on a 3D mesh model or CAD model: In one implementation, the ground truth 3D representation is a 3D mesh model or CAD model. The system utilizes the geometric structure, texture information, material information, and color information in the 3D mesh model or CAD model to generate target ground truth data and annotation data from the corresponding viewpoint based on the spatial pose through the graphics rendering pipeline. The annotation data may include one or more of the following: target region mask, target boundary contour, depth map, surface normal map, semantic labels, instance labels, or pose labels.

[0054] Implementation of 3D assets based on graphics engines: In one implementation, a true 3D representation is imported into Unity, Unreal, or other graphics engines as a renderable 3D asset. Based on the target spatial pose estimated or registered from a real 2D degraded image, the system sets virtual camera parameters, target pose parameters, target scale parameters, lighting parameters, material parameters, and image resolution parameters in the graphics engine to generate one or more of the following from the corresponding viewpoint: a clear target image, a target region mask, a depth map, a surface normal map, a target boundary contour, a semantic label map, an instance label map, or pose information.

[0055] Implementation based on a neural rendering model: In one implementation, the ground truth 3D representation is a neural radiation field, a symbolic distance field implicit surface model, a neural texture field, or other neural rendering models. Based on the estimated or registered camera pose, target pose, or the relative pose between the target and the image acquisition device, the system queries or renders the neural rendering model to generate a clear target image, target foreground image, depth map, transparency map, target region mask, or other target ground truth data and annotation data from the corresponding viewpoint.

Claims

1. A method for generating ground truth labeling data of a degraded image target object region, characterized in that, The method includes: S101, Obtain the three-dimensional representation of the target's true value; S102, Acquire a real two-dimensional degraded image of the target under adverse imaging conditions; S103, Determine the target region from the real two-dimensional degraded image; S104, determine the spatial pose of the target relative to the image acquisition device; the spatial pose is based on the target's contour, key points, and edges. Determined by edge, texture, semantic region, instance region, bounding box, or human interaction information; S105, sets virtual camera parameters or rendering parameters based on spatial pose; S106, Based on the virtual camera parameters or rendering parameters, render, project, or transform the three-dimensional representation of the true value; S107, Generate target ground value data corresponding to the target viewpoint in the real two-dimensional degraded image; S108, Generate labeled data based on the target ground truth data; S109, map the target ground truth data and labeled data to the coordinate system of the real two-dimensional degraded image; S110, Generate the target ground truth and labeled dataset corresponding to the real two-dimensional degraded image.

2. The method according to claim 1, wherein The three-dimensional representation can be a three-dimensional Gaussian representation, a neural radiation field representation, a three-dimensional mesh model, a CAD model, a point cloud model, a voxel model, a symbolic distance field implicit surface model, a digital twin model, or a three-dimensional asset in a game engine.

3. The method according to claim 1, wherein The adverse imaging conditions are at least one of the following: foggy weather, rainy weather, snowy weather, underwater, smoke and dust, sand and dust, low light, strong scattering medium, turbid medium, strong reflection, motion blur, or sensor noise.

4. The method according to claim 1, wherein The spatial pose is the camera pose, the target pose, the relative pose between the target and the image acquisition device, the target rotation parameters, the target translation parameters, the target scale parameters, the camera intrinsic parameters, the camera extrinsic parameters, or a combination thereof; The method for determining the spatial pose is at least one of the following: manual interactive registration, semi-automatic registration, automatic registration, key point matching, contour matching, edge matching, feature matching, PnP solving, differentiable rendering optimization, neural network pose estimation, template matching, or multimodal alignment.

5. The method according to claim 1, wherein Rendering, projecting, or transforming the three-dimensional representation of the true value includes: generating target true value data under the corresponding viewpoint in a three-dimensional rendering engine, neural rendering model, graphics rendering pipeline, differentiable renderer, or three-dimensional reprojection module according to the spatial pose.

6. The method according to claim 1, wherein The target ground truth data is at least one of the following: a clear target image, a target foreground image, a target region mask, a target boundary contour, a depth map, a surface normal map, a semantic label map, an instance label map, a transparency map, a visibility map, a pose label, or a reference image used for image restoration.

7. The method according to claim 1, wherein Generating annotation data corresponding to the real two-dimensional degraded image based on the target ground truth data includes: mapping, projecting, registering, or fusing the target ground truth data into the image coordinate system of the real two-dimensional degraded image to obtain annotation results that are consistent with or correspond to the size and coordinates of the real two-dimensional degraded image.

8. The method according to claim 1, wherein The labeled data is at least one of the following: target region mask, target boundary contour, pixel-level semantic category, instance annotation, transparency map, visibility map, depth annotation, normal annotation, or pose annotation.

9. A system for generating ground truth annotation data for target object regions in degraded images, used to implement the method described in any one of claims 1-8, characterized in that, The system includes: a target ground truth 3D representation acquisition module, a degraded image acquisition module, a target region determination module, a spatial pose determination module, a ground truth data generation module, a labeled data generation module, a coordinate mapping module, and a dataset construction module.

10. The degraded image target object region truth labeling data generation system according to claim 9, characterized in that, The target truth three-dimensional representation acquisition module is used to acquire the target truth three-dimensional representation; The degraded image acquisition module is used to acquire a real two-dimensional degraded image of the target under adverse imaging conditions; The target region determination module is used to determine the target region in the real two-dimensional degraded image; The spatial pose determination module is used to estimate or register the spatial pose of the target relative to the image acquisition device based on the target region, target contour, key points, edges, texture, semantic region, instance region, bounding box, or human interaction information. The truth data generation module is used to render, project, or transform the three-dimensional representation of the true value based on the spatial pose, and generate target truth data corresponding to the target viewpoint in the real two-dimensional degraded image. The annotation data generation module is used to generate annotation data based on the target truth data; The coordinate mapping module is used to map the target ground truth data and annotation data to the coordinate system of the real two-dimensional degraded image, so that the target ground truth data and annotation data correspond to the target position in the real two-dimensional degraded image; The dataset construction module is used to generate a target ground truth and labeled dataset that can be used for model training or evaluation based on the real two-dimensional degraded image, the target ground truth data, and the labeled data.

Images