An implicit view reconstruction method and system for data augmentation

Abstract

The application provides an implicit view reconstruction method and system for data enhancement, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a multi-view image sequence of a target scene and corresponding camera pose parameters; performing view space distribution analysis based on the camera pose parameters, dividing the view space into multiple regions, and selecting images from each region to form a sparse input view set; mapping the sparse input view set and the corresponding pose parameters to a three-dimensional implicit feature space to obtain a scene feature representation, and rendering a new view image from the scene feature representation based on a target new view pose parameter; constructing a view parameter space with the camera pose parameters as observation view samples, smoothing and expanding the original distribution of the view parameter space to generate expanded view parameters, and rendering an expanded view image from the scene feature representation based on the expanded view parameters; and merging the new view image, the expanded view image and the multi-view image sequence to obtain an enhanced image data set, thereby realizing view expansion and optimization of the multi-view image sequence.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an implicit viewpoint reconstruction method and system for data augmentation. Background Technology

[0002] In the field of robot vision, the training effect of deep learning models is highly dependent on the scale and quality of the training data. Currently, robot data acquisition is mostly done manually, which is significantly affected by subjective factors such as operator habits, movement trajectories, and work rhythm. The acquired image sequences generally suffer from problems such as viewpoint jitter, pose deviation, and inconsistent lighting, directly affecting the training effect and generalization ability of subsequent models. To alleviate the lack of data and improve data quality, existing technologies typically employ two types of image transformation methods for data augmentation. The first type is traditional image transformation methods, including geometric transformations such as flipping, rotating, cropping, and scaling; color transformations such as brightness adjustment, color jittering, and various noise injections. The second type is generative augmentation methods based on deep learning, represented by generative adversarial networks and diffusion models, which synthesize new image samples by learning the distribution characteristics of the training data.

[0003] However, both types of image transformation methods have their inherent limitations. Regarding the first type of traditional image transformation method, its transformation method is fixed and operates only in two-dimensional pixel space, failing to generate entirely new perspectives or scene states not present in the original data. It also lacks the ability to simulate perspective changes caused by deviations in the robot's motion trajectory during acquisition. As for the second type of generative enhancement method, most existing solutions are designed independently for monocular images, neglecting the epipolar geometric constraints between multi-view images. If each perspective image is generated independently, the position and appearance of the same target will be inconsistent across different perspectives, disrupting the original multi-view geometric consistency. This makes the enhanced data unsuitable for downstream tasks requiring multi-view input, such as stereo vision or 3D reconstruction. More critically, during multi-view acquisition by the robot, the extrinsic pose parameters of each camera include rotation matrices and translation vectors, containing rich spatial geometric information. However, neither of the aforementioned methods effectively utilizes this information to constrain and guide the generation process during enhancement, resulting in an inability to precisely control the perspective and content of the generated image based on the spatial pose of the acquisition device.

[0004] Furthermore, the spatial distribution of manually collected data perspectives is often uneven. Due to the disordered nature of the operator's movement trajectory, some perspective areas have dense samples while others have sparse or even missing samples. This distribution bias has not been effectively corrected in existing enhancement methods. Simultaneously, target objects in the acquired images often partially lie at the image boundaries. Existing methods cannot adequately fill in such boundary areas, further limiting the effective utilization and quality consistency of the dataset. Summary of the Invention

[0005] In view of this, the present invention provides an implicit viewpoint reconstruction method and system for data augmentation, aiming to solve the technical problem of insufficient dataset usability caused by insufficient viewpoint coverage in image data. The technical solution is as follows: Firstly, an implicit view reconstruction method for data augmentation is provided, including: Acquire multi-view image sequences of the target scene and the camera pose parameters corresponding to each frame; Based on the camera pose parameters of the multi-view image sequence, the view space distribution is analyzed, the view space is divided into multiple regions, and images with satisfactory image quality are selected from each region to form a sparse input view set. The sparse input view set and the corresponding camera pose parameters are mapped to a three-dimensional implicit feature space to obtain the scene feature representation of the target scene. Based on the specified target new view pose parameters, the corresponding view features are locked from the scene feature representation, and the new view image corresponding to the target new view is rendered. A viewpoint parameter space is constructed using the camera pose parameters of the multi-view image sequence as observation viewpoint samples. The original distribution of the viewpoint parameter space is smoothly amplified to generate multiple amplified viewpoint parameters with better coverage uniformity than the original distribution. Based on the amplified viewpoint parameters, a corresponding amplified viewpoint image is generated from the scene feature representation. The new perspective image, the augmented perspective image, and the multi-view image sequence are merged to obtain the enhanced image dataset.

[0006] Secondly, an implicit view reconstruction system for data augmentation is provided, comprising: The data acquisition module is used to acquire multi-view image sequences of the target scene and the camera pose parameters corresponding to each frame of the image; The viewpoint filtering module is used to perform viewpoint spatial distribution analysis based on the camera pose parameters of the multi-viewpoint image sequence, divide the viewpoint space into multiple regions, and select images with satisfactory image quality from each region to form a sparse input view set. The implicit view reconstruction module is used to map the sparse input view set and the corresponding camera pose parameters to a three-dimensional implicit feature space to obtain the scene feature representation of the target scene, lock the corresponding view features from the scene feature representation based on the specified target new view pose parameters, and render the new view image corresponding to the target new view. The viewpoint smoothing and augmentation module is used to construct a viewpoint parameter space using the camera pose parameters of the multi-view image sequence as observation viewpoint samples, smoothly augment the original distribution of the viewpoint parameter space, generate multiple augmented viewpoint parameters with better coverage uniformity than the original distribution, and generate corresponding augmented viewpoint images from the scene feature representation based on the augmented viewpoint parameters. The dataset construction module is used to merge the new perspective image, the augmented perspective image, and the multi-view image sequence to obtain an enhanced image dataset.

[0007] By means of the above technical solution, the present invention provides an implicit view reconstruction method and system for data augmentation, which maps a sparse input view set and the corresponding camera pose parameters to a three-dimensional implicit feature space, and generates a new view image based on the target new view pose parameters; at the same time, the original distribution of the view parameter space constructed by the camera pose parameters is smoothly expanded, thereby realizing the effective expansion and distribution optimization of the multi-view image sequence in the view dimension.

[0008] Based on the above technical principles, the embodiments of the present invention can achieve the following technical effects: (1) The sparse input view set and the corresponding camera pose parameters are mapped to the three-dimensional implicit feature space to obtain the scene feature representation of the target scene. Based on the new view pose parameters of the target, the new view image is obtained by rendering from the scene feature representation. This technique realizes the generation of any continuous new view, which fundamentally breaks through the limitation of traditional data augmentation methods that only transform in two-dimensional pixel space. Moreover, no explicit three-dimensional reconstruction is required. Only a small number of sparse input views are needed to generate new view images covering the continuous view space, which reduces the manpower and time costs of data collection.

[0009] (2) The camera pose parameters of the multi-view image sequence are used as observation view samples to construct the view parameter space. The original distribution of the view parameter space is smoothly amplified to generate amplified view parameters with better coverage uniformity than the original distribution. Based on the amplified view parameters, the amplified view image is generated. This technique effectively eliminates the view distribution deviation and disturbance introduced by disordered movement due to manual operation. The enhanced image dataset has more uniform coverage and more continuous distribution characteristics in the view dimension, which improves the overall quality of the enhanced image dataset in the view space.

[0010] (3) Based on the annotation information of the target object in the pre-acquired multi-view image sequence, and based on the geometric transformation relationship between the camera pose parameters corresponding to each image and the camera pose parameters corresponding to the new view image or the augmented view image, the annotation information is projected and mapped to the corresponding new view image or the augmented view image. This achieves automatic migration of the annotation information without the need for manual re-annotation, which greatly reduces the data preparation cost. At the same time, this invention does not depend on a specific sensor model or scene type. It is applicable to any data acquisition system with RGB image acquisition and pose recording capabilities. It has good versatility and deployment efficiency in various application scenarios such as industrial grasping, indoor perception of service robots, and road data enhancement for autonomous driving. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. In the drawings: Figure 1 An application scenario diagram of an implicit view reconstruction method for data augmentation provided in an embodiment of the present invention; Figure 2 A flowchart of an implicit view reconstruction method for data augmentation provided in an embodiment of the present invention; Figure 3 A block diagram of an implicit view reconstruction system for data augmentation provided in an embodiment of the present invention; Figure 4 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0013] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0014] Figure 1 This diagram illustrates an application scenario of an implicit view reconstruction method for data augmentation, provided by an embodiment of the present invention. For example... Figure 1 As shown, this application scenario includes a data acquisition device and a target scene. The data acquisition device is mounted on the robot platform and can be a monocular RGB camera, a binocular RGB camera, or a multi-view RGB camera, used to acquire images of the target scene. The target scene is the environment in which the robot needs to perceive and interact, such as a material bin and the various metal parts placed inside in an industrial grasping scenario, household items in an indoor perception scenario for a service robot, and roads and surrounding vehicles and pedestrians in an autonomous driving scenario.

[0015] During the data acquisition process, the operator is not required to follow a strict movement path, allowing a certain degree of free movement and perspective changes. This simulates the disordered movement caused by manual operation in a real data acquisition scenario. The data acquisition device can move around the target scene along any trajectory, capturing images from multiple perspectives at different positions and orientations. Each frame corresponds to a set of camera pose parameters, which characterize the orientation and position of the data acquisition device in space when acquiring that frame. These parameters mainly include the camera's rotation matrix and translation vector in the world coordinate system. ,in, Let i be the camera pose parameters corresponding to the i-th frame image. Let the camera rotation matrix of the data acquisition device be the matrix used to acquire the i-th frame of the image. Let be the camera translation vector of the data acquisition device when acquiring the i-th frame of the image.

[0016] Camera pose parameters can be acquired in several ways. The first method involves fusing the outputs of the inertial measurement unit (IMU) and visual odometry (VAD) built into the data acquisition device. The IMU measures the acceleration and angular velocity of the acquisition device during its motion, while the VAD estimates the relative motion of the data acquisition device by tracking and matching visual features in consecutive frames of images. Fusing the outputs of both yields high-frequency, high-precision camera pose parameter estimation results. The second method involves acquiring these parameters through an external motion capture system. External motion capture systems, such as optical or inertial motion capture systems, install markers or sensors on the data acquisition device, allowing the external device to track the device's position and attitude in space in real time. This method yields high-precision and stable camera pose parameters, suitable for acquisition in laboratory environments. The third method involves acquiring these parameters through a structured light or lidar-based positioning system. Structured light positioning systems estimate the pose of the data acquisition device by projecting coded light spots and analyzing their deformation on the target surface. LiDAR positioning systems locate the device's position and orientation by scanning the environment to obtain a 3D point cloud and matching it with a pre-set map. The above three acquisition methods can be selected based on the accuracy requirements and equipment conditions of the actual application scenario.

[0017] The data acquisition device stores the acquired images in chronological order of acquisition time, forming a multi-view image sequence, and each frame in the multi-view image sequence corresponds to a camera pose parameter.

[0018] Figure 2 This is a flowchart illustrating an implicit view reconstruction method for data augmentation, provided as an embodiment of the present invention. Figure 2 As shown, the implicit view reconstruction method for data augmentation mainly includes steps S101 to S105.

[0019] Step S101: Obtain the multi-view image sequence of the target scene and the camera pose parameters corresponding to each frame.

[0020] The multi-view image sequence of the target scene and the camera pose parameters corresponding to each frame of the image are obtained from the data acquisition device by real-time acquisition or batch reading. The multi-view image sequence is composed of multiple frames of images arranged in chronological order of acquisition time. Each frame of the image records the visual appearance information of the target scene at a certain acquisition time. The camera pose parameters correspond one-to-one with the image frames.

[0021] Step S102: Based on the camera pose parameters of the multi-view image sequence, perform view space distribution analysis, divide the view space into multiple regions, and select images that meet the image quality conditions from each region to form a sparse input view set.

[0022] View space refers to an abstract space with the center of the target scene as the origin and the spatial orientation and distance of the camera relative to the origin as coordinates. The camera pose parameters of each frame can be mapped to a point in this view space. The position of this point reflects the observation direction and observation distance of the camera relative to the target scene when the frame is captured.

[0023] After obtaining the multi-view image sequence and the camera pose parameters corresponding to each frame, the distribution of the camera pose parameters in the view space of each frame is analyzed: First, the rotation matrix and translation vector in the camera pose parameters of each frame in the multi-view image sequence are converted into a spherical coordinate system. The three dimensions of the spherical coordinate system are azimuth, elevation, and radial distance. The azimuth represents the camera's rotation angle relative to the center of the target scene in the horizontal plane, with a value ranging from 0 to 360 degrees; the elevation represents the camera's pitch angle relative to the horizontal plane, with a value ranging from 0 to 180 degrees; and the radial distance represents the straight-line distance from the camera's optical center to the center of the target scene. Thus, the camera pose parameters of each frame are mapped to a coordinate point in the spherical coordinate system.

[0024] Then, cluster analysis is performed on the coordinate points mapped to the spherical coordinate system to divide the view space into multiple regions. Cluster analysis is implemented using clustering algorithms, such as K-means clustering, density-based clustering, or hierarchical clustering. The clustering algorithm takes the azimuth and elevation coordinates of each coordinate point in the spherical coordinate system as input, groups the coordinate points according to their spatial distance, and assigns coordinate points with similar spatial distances to the same category. After clustering, each category corresponds to a region in the view space, each region contains at least one coordinate point, different regions correspond to different view directions, and all regions collectively cover the entire view space.

[0025] After dividing the view space into regions, for each region, images meeting certain quality criteria are selected from all images falling within that region. Meeting these criteria means that at least one of the following: image sharpness, exposure, and target object visibility reaches a corresponding threshold. Sharpness evaluates the sharpness of edges and textures in an image and can be measured by calculating the spatial gradient or high-frequency components of the image. Exposure evaluates whether the overall brightness of the image is within a suitable range—neither too bright nor too dark—and can be measured by calculating the average brightness and brightness histogram distribution. Target object visibility evaluates whether key targets in the scene are clearly identifiable and unobstructed in the image, and can be measured by calculating the contrast of the target object's region or the target detection confidence. For each region, the frame with the highest sharpness, most reasonable exposure, and highest target object visibility is selected as the optimal image for that region, and this frame is used as an element in the sparse input view set.

[0026] After traversing all regions of the view space, the best images selected from each region are aggregated to form a sparse input view set. This sparse input view set contains a small number of representative images from different view directions, which can cover the main distribution range of the entire view space with minimal data volume.

[0027] Step S103: Map the sparse input view set and the corresponding camera pose parameters to a three-dimensional implicit feature space to obtain the scene feature representation of the target scene. Based on the specified target new view pose parameters, lock the corresponding view features from the scene feature representation and render the new view image corresponding to the target new view.

[0028] After obtaining the sparse input view set, each frame of the sparse input view set and its corresponding camera pose parameters are input into the basic pose encoder for processing. The basic pose encoder is a transformer-based encoding network used to map the images in the sparse input view set and their corresponding camera pose parameters into latent features for 3D perception. For each frame of the sparse input view set, the basic pose encoder first extracts the local visual features of the image through convolutional layers, including low-level visual information such as edge features, texture features, color features, and shape features. Subsequently, the basic pose encoder fuses the extracted local visual features with the rotation matrix and translation vector in the corresponding camera pose parameters. That is, the camera pose parameters are encoded into pose feature vectors of the same dimension as the image features through a multilayer perceptron, and the image features and pose feature vectors are concatenated or added in the feature dimension to obtain image features fused with camera pose parameters. On this basis, the basic pose encoder enhances the image features of each frame fused with camera pose parameters through a self-attention mechanism, so that each spatial location in the image features of each frame can pay attention to the information of other spatial locations in the image, thereby extracting the global long-range dependencies of the image. Subsequently, the basic pose encoder interacts with the image features of different input images through a cross-attention mechanism, establishing feature correspondences between different input viewpoints, so that the feature representations of the same spatial point in the same scene can be correlated and aligned in different viewpoint images.

[0029] After the above processing, the basic pose encoder outputs a unified scene feature representation in a 3D implicit feature space. The 3D implicit feature space is an abstract high-dimensional feature space where each feature point encodes the geometric position and appearance information of the corresponding 3D point in the target scene. Furthermore, feature representations from different viewpoints establish geometric correspondences within this space through a cross-attention mechanism. The scene feature representation is a compact representation of the target scene in the 3D implicit feature space, encompassing the entire scene's geometric structure, appearance texture, and geometric correspondences between different viewpoints.

[0030] After obtaining the scene feature representation, a new target viewpoint pose parameter is specified according to the actual needs of the downstream task. This new target viewpoint pose parameter is the desired orientation and desired position of the new target viewpoint camera, also represented in the form of a rotation matrix and a translation vector. Downstream tasks refer to visual perception tasks performed based on the augmented image dataset, such as object detection, image segmentation, depth estimation, or 3D reconstruction.

[0031] Pose matching is performed between the target's new viewpoint pose parameters and the scene feature representation. Specifically, the viewpoint features in the scene feature representation that are closest to the target's new viewpoint pose parameters are searched. Viewpoint features are a subset of features in the scene feature representation corresponding to a specific viewpoint direction, and this subset of features encodes the image content that should be presented when observing the target scene from that specific viewpoint. The specific pose matching process is as follows: First, the similarity between the target new viewpoint pose parameters and the pose parameters corresponding to each viewpoint feature in the scene feature representation is calculated. The viewpoint feature with the highest similarity is selected as the viewpoint feature to match the target new viewpoint pose parameters. After obtaining the viewpoint feature, pixel-by-pixel rendering is performed based on this viewpoint feature along the gaze direction determined by the target new viewpoint pose parameters to obtain the new viewpoint image corresponding to the target new viewpoint. Pixel-by-pixel rendering means that for each pixel position in the output image, the gaze direction corresponding to the pixel position is calculated based on the coordinates of the pixel position in the image plane and the camera intrinsic and extrinsic parameters determined by the target new viewpoint pose parameters. The corresponding color and density values ​​are sampled along the gaze direction in the viewpoint features, and the sampling results are filled into the output image pixel by pixel to finally form a complete new viewpoint image. The above pixel-by-pixel rendering process is executed by the viewpoint image decoder, which is a decoding network based on a lightweight transformer structure. It receives the scene feature representation output by the basic pose encoder and the target new viewpoint pose parameters as input, and directly renders and outputs the two-dimensional image corresponding to the target new viewpoint pose parameters.

[0032] In practical applications, explicit 3D reconstruction can be used to replace the above-mentioned new perspective generation method based on 3D implicit feature space. Specifically, based on traditional multi-view stereo vision technology or neural radiation field technology, explicit 3D reconstruction is performed on sparse input view sets, and then new perspective images are obtained through rendering.

[0033] In a specific example, when the specified target new viewpoint pose parameters exceed the viewpoint range covered by the sparse input view set—that is, when the target new viewpoint pose parameters exceed the viewpoint space region formed by the camera pose parameters corresponding to each image in the sparse input view set, including at least one of the azimuth, elevation, or radial distance in the target new viewpoint pose parameters exceeding the aforementioned range—there are no viewpoint features in the scene feature representation that are sufficiently close to the target new viewpoint pose parameters, making it impossible to obtain a new viewpoint image through pose matching and pixel-by-pixel rendering. In this case, using the scene feature representation as a condition, a generative extrapolation of the target new viewpoint exceeding the range is performed through a progressively denoised diffusion generation process to obtain a new viewpoint image. This generative extrapolation process is executed by a diffusion decoder, which is a decoding network based on a conditional diffusion model. It uses the scene feature representation output by the basic pose encoder as a condition and achieves generative extrapolation through probability sampling of the latent feature space.

[0034] The aforementioned progressive denoising diffusion generation process refers to starting with a randomly sampled Gaussian noise image and gradually transforming it into a target image with reasonable structure and texture through multiple iterative denoising operations. The entire denoising process is constrained by the scene feature representation, ensuring that the generated image maintains consistency with the geometric structure and appearance of the target scene. This diffusion generation process is defined by the following formula: , in, For the generated new perspective image, For diffusion decoder, The scene feature representation output by the basic pose encoder, For the target's new perspective pose parameters, Random noise sampled from a standard normal distribution. In each denoising step, the diffusion decoder takes the noise image and scene feature representation at the current iteration time as input, outputs the predicted noise component, subtracts the predicted noise component from the noise image at the current iteration time to obtain the noise image at the next iteration time, and after a preset number of iterations, the final output image is the new viewpoint image corresponding to the new viewpoint pose parameters of the target.

[0035] Through the above-described progressive denoising diffusion generation process, reasonable inferences and completions can be made for uncollected viewpoint regions, further expanding the viewpoint coverage boundary of multi-view image sequences.

[0036] Step S104: Using the camera pose parameters of the multi-view image sequence as observation view samples to construct a view parameter space, smoothly amplify the original distribution of the view parameter space to generate multiple amplified view parameters with better coverage uniformity than the original distribution, and generate corresponding amplified view images from the scene feature representation based on the amplified view parameters.

[0037] Using all camera pose parameters corresponding to a multi-view image sequence as observation viewpoint samples, the original distribution in the viewpoint parameter space is constructed. The viewpoint parameter space is a high-dimensional space with each dimension of the camera pose parameters as coordinate axes, and its dimensions include the three Euler angle components of the rotation matrix and the three position components of the translation vector. Each observation viewpoint sample corresponds to a data point in the viewpoint parameter space, and the aggregation of all data points in the viewpoint parameter space constitutes the original distribution.

[0038] Because the operator's movement trajectory is random and disordered during manual data collection, the original distribution often exhibits unevenness, with dense data points in certain viewpoints and sparse or even missing data points in other viewpoints. This uneven distribution can cause downstream visual models to overfit to dense viewpoints and underfit to sparse viewpoints during training, thus affecting the model's generalization ability. Downstream visual models refer to deep learning models used to perform downstream tasks, such as object detection models based on convolutional neural networks, segmentation models based on transformers, or depth estimation models based on stereo matching. In this embodiment, the generated new viewpoint images and augmented viewpoint images are used to expand the training dataset of the downstream visual models to improve their generalization ability and robustness.

[0039] To address the issue of uneven original distribution, a viewpoint diffusion model is employed to smoothly augment the original distribution in the viewpoint parameter space. The viewpoint diffusion model is a generative network based on a conditional diffusion model. Its training process uses observed viewpoint samples as conditions to learn the distribution patterns of camera pose parameters in the viewpoint parameter space. Specifically, firstly, all camera pose parameters corresponding to a multi-view image sequence are used as observed viewpoint samples to extract the statistical features of the original distribution, including the mean, variance, covariance relationships between dimensions, and the boundary range of the original distribution. Based on this, the viewpoint diffusion model gradually adds noise to the viewpoint parameters through a forward diffusion stage, and then learns to recover the viewpoint parameters from the noise through a backward diffusion stage, thereby mastering the overall understanding of the original distribution and the mapping relationship from the noise distribution to the true distribution. In this embodiment, camera pose parameters and viewpoint parameters refer to the same physical quantity. The difference between the two is that camera pose parameters focus on describing the actual position and orientation of the camera in physical space when the image is acquired, while viewpoint parameters focus on the coordinates of this physical quantity after mapping it to the viewpoint parameter space.

[0040] After the viewpoint diffusion model is trained, it is used to smoothly expand the original distribution of the viewpoint parameter space. This involves iterative sampling within the neighborhood of the original distribution to generate candidate viewpoint parameters with a higher density than the original distribution. Then, a coverage uniformity evaluation is performed on the candidate viewpoint parameters, selecting those that result in a more consistent coverage density across the viewpoint parameter space as the final expanded viewpoint parameters. The coverage uniformity evaluation involves dividing the viewpoint parameter space into multiple equally sized grid regions, counting the number of candidate viewpoint parameters in each grid region, and calculating the variance of the number of candidate viewpoint parameters between different grid regions. If this variance is less than a preset threshold, the coverage uniformity of the current candidate viewpoint parameters is considered to meet the requirements.

[0041] The expanded view parameters generated using the view diffusion model can be expressed as follows: , in, For the newly generated amplified viewpoint parameters, For the perspective diffusion model, These are contextual view samples sampled from camera pose parameters of multi-view image sequences. It is random noise.

[0042] After the above smoothing amplification, multiple amplified viewpoint parameters are obtained. The distribution density of these amplified viewpoint parameters in the viewpoint parameter space is more uniform than that of the original distribution, which can effectively eliminate the aggregation and sparsity phenomena present in the original distribution.

[0043] In practical applications, traditional interpolation methods can be used to replace the smooth amplification method based on the viewpoint diffusion model. Specifically, linear interpolation or spline interpolation of the viewpoint parameters in the viewpoint parameter space can be performed to generate amplified viewpoint parameters.

[0044] After obtaining the augmented viewpoint parameters, each augmented viewpoint parameter is used as the target viewpoint. Viewpoint features corresponding to each augmented viewpoint parameter are extracted from the scene feature representation, and these extracted viewpoint features are rendered to generate an augmented viewpoint image. This process is similar in principle to generating a new viewpoint image within the coverage area in step S13; that is, for each augmented viewpoint parameter, the closest viewpoint feature is found in the scene feature representation, and then pixel-by-pixel rendering is performed along the line of sight to obtain the augmented viewpoint image corresponding to that parameter. This rendering process is also executed by the viewpoint image decoder.

[0045] Step S105: The new perspective image, the augmented perspective image, and the multi-view image sequence are merged to obtain the enhanced image dataset.

[0046] First, standardized cropping operations are performed on the new and expanded viewpoint images. Specifically, one of the following cropping methods is used: center cropping, random cropping, and adaptive cropping based on the target object's position. Center cropping involves cropping a fixed-size sub-image from the central region of the image, centered on the image's center point, according to a preset target size. Random cropping involves randomly selecting a starting position within the image range and cropping a sub-image from that position according to a preset target size. Random cropping increases data diversity and improves the generalization ability of downstream models. Adaptive cropping based on the target object's position involves first detecting the location of the target object in the image, and then cropping a sub-image containing the target object, centered on the target object's center, according to a preset target size. Adaptive cropping ensures that the target object is in a prominent position in the cropped image. Regardless of the cropping method used, all cropped new and expanded viewpoint images have the same target resolution, such as 224×224 pixels, 256×256 pixels, or 512×512 pixels.

[0047] Next, quality assessment is performed on the new and augmented viewpoint images. Quality assessment involves calculating a sharpness score and a structural integrity score for each image. The sharpness score evaluates the edge sharpness and texture clarity of the image, while the structural integrity score evaluates whether the shape of the target object in the image is complete and whether there is distortion or missing parts. Only images whose sharpness and structural integrity scores both meet preset thresholds are retained for merging with the multi-viewpoint image sequence; images that fail the quality assessment are discarded to ensure the quality of the augmented dataset.

[0048] Finally, the new perspective images, augmented perspective images, and multi-view image sequences are merged to form the enhanced image dataset. The merging method involves storing all images in the same dataset while maintaining the correlation between each frame and its corresponding camera pose parameters. For new perspective images, the corresponding camera pose parameters are the specified target new perspective pose parameters; for augmented perspective images, the corresponding camera pose parameters are the augmented perspective parameters. These pose parameters are stored along with the images in the enhanced image dataset. In other words, the enhanced image dataset simultaneously contains the original captured real images and the virtual perspective images generated through implicit perspective reconstruction, significantly expanding the viewpoint coverage and improving the uniformity of viewpoint distribution compared to the original real images.

[0049] Furthermore, when the multi-view image sequence contains annotation information of target objects, such as target detection boxes or segmentation masks, this annotation information can be automatically transferred to the generated new and augmented view images through geometric transformation relationships, eliminating the need for manual re-annotation. Specifically, the annotation information of target objects in the multi-view image sequence is obtained. This annotation information is either a target detection box or a segmentation mask, where the target detection box is in the form of a rectangle, used to identify the position range of the target object in the image; the segmentation mask is in the form of pixel-level annotations, used to identify the precise pixel area occupied by the target object in the image. Based on the geometric transformation relationship between the camera pose parameters corresponding to each image in the multi-view image sequence and the camera pose parameters corresponding to the new or augmented view image, the annotation information is projected and mapped to the corresponding new or augmented view image. The geometric transformation relationship refers to the mapping relationship from two-dimensional image coordinates under one view in the multi-view image sequence to two-dimensional image coordinates under another view. This mapping relationship is calculated using the camera pose parameters and camera intrinsic parameters of each of the two views. Through this geometric transformation relationship, the coordinates of each key point or contour point in the annotation information of the original real image can be transformed to the corresponding coordinate positions in the target image, thereby realizing the automatic migration of annotation information and enabling the generated new perspective image and augmented perspective image to carry annotation information consistent with the original real image.

[0050] To illustrate the effectiveness of the implicit view reconstruction method for data augmentation provided in this embodiment of the invention, the following detailed explanation uses robot vision data augmentation in an industrial grasping scenario as an example.

[0051] In industrial grasping scenarios, robots need to grasp various metal parts from bins, and the vision system, as the robot's perception core, directly affects the grasping success rate due to its detection accuracy. However, due to the randomness and disorder of the operator's movement trajectory during manual acquisition, the acquired image sequences often suffer from uneven viewpoint distribution, i.e., samples are dense in some viewpoint directions while sparse in others, leading to a decrease in the detection accuracy of downstream vision models at specific viewpoints. The purpose of this invention is to expand the viewpoint coverage of the dataset and optimize the viewpoint distribution through implicit viewpoint reconstruction technology, providing a higher quality dataset for downstream vision model training.

[0052] In this application example, an Intel RealSense D435 RGB-D camera was used as the data acquisition device. This camera has both color image acquisition and depth information sensing capabilities, with a resolution of 1920×1080 and a frame rate of 30fps. Pose recording was achieved through the fusion of the camera's built-in inertial measurement unit and visual odometry, outputting a six-DOF camera pose for each frame. This six-DOF camera pose includes three degrees of freedom of position information and three degrees of freedom of attitude information. The acquisition method involved manually holding the camera and moving it around a target object, such as various metal parts in a bin, acquiring image sequences from different angles and distances. Free movement was allowed during the acquisition process to simulate the disordered movement caused by manual operation in real data acquisition scenarios. A total of 500 frames of images and their corresponding camera pose parameters were acquired.

[0053] In the data augmentation stage, the camera pose parameters of 500 frames of images are first mapped to a spherical coordinate system. The K-means clustering algorithm is used to divide the view space into 8 regions. The frame with the highest sharpness in each region is selected as the sparse input view set, resulting in 8 sparse input images. Then, the 8 sparse input images and their corresponding camera pose parameters are input into the basic pose encoder to obtain the scene feature representation of the target scene. Next, 200 new target view pose parameters are uniformly sampled within the neighborhood of the original view distribution. Each new target view pose parameter is sequentially input into the view image decoder to generate 200 new view images. For new target view pose parameters that exceed the view range covered by the sparse input view set, a diffusion decoder is used for generative extrapolation to smooth edge noise and resolve image blurring. All generated new view images are center-cropped, and the output resolution is uniformly set to 512×512 to ensure that the new view images have the same size specifications as the original images. Finally, the original 500 frames of images are merged with the 200 generated new view images to form the augmented image dataset.

[0054] The enhanced image dataset was input into the downstream visual model for training. The effectiveness of the implicit viewpoint reconstruction method for data augmentation provided in this invention was verified by comparing it with a baseline model trained using only the original 500 frames of images. Experimental results show that the original 500 frames of images exhibit significant clustering in the viewpoint parameter space, mainly concentrated in the azimuth range of 60 to 120 degrees. After smoothing augmentation using the viewpoint diffusion model, the newly generated 200 viewpoints achieved uniform coverage within the range of 0 to 180 degrees, significantly improving the uniformity of data coverage in the viewpoint dimension. Correspondingly, after training on the enhanced image dataset, the downstream visual model showed significant improvement in target detection accuracy under varying viewpoint conditions, verifying the effectiveness of the method in improving dataset viewpoint coverage and model generalization ability.

[0055] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of the present invention, which will not be elaborated here.

[0056] Figure 3 This is a block diagram of an implicit view reconstruction system for data augmentation, provided as an embodiment of the present invention. Figure 3 As shown, the implicit view reconstruction system for data augmentation includes a data acquisition module 201, a view selection module 202, an implicit view reconstruction module 203, a view smoothing augmentation module 204, and a dataset construction module 205.

[0057] The data acquisition module 201 is used to acquire multi-view image sequences of the target scene and the camera pose parameters corresponding to each frame of the image.

[0058] The viewpoint filtering module 202 is used to perform viewpoint spatial distribution analysis based on the camera pose parameters of the multi-viewpoint image sequence, divide the viewpoint space into multiple regions, and select images with image quality that meet the conditions from each region to form a sparse input view set.

[0059] The implicit view reconstruction module 203 is used to map the sparse input view set and the corresponding camera pose parameters to a three-dimensional implicit feature space to obtain the scene feature representation of the target scene, lock the corresponding view features from the scene feature representation based on the specified target new view pose parameters, and render the new view image corresponding to the target new view.

[0060] The viewpoint smoothing and augmentation module 204 is used to construct a viewpoint parameter space using the camera pose parameters of the multi-view image sequence as observation viewpoint samples, smooth and augment the original distribution of the viewpoint parameter space to generate multiple augmented viewpoint parameters with better coverage uniformity than the original distribution, and generate corresponding augmented viewpoint images from the scene feature representation based on the augmented viewpoint parameters.

[0061] The dataset construction module 205 is used to merge the new perspective image, the augmented perspective image, and the multi-view image sequence to obtain an enhanced image dataset.

[0062] In a specific example, the implicit view reconstruction module 203 described above is specifically used to lock the corresponding view features from the scene feature representation based on the specified target new view pose parameters and render the new view image corresponding to the target new view when the target new view pose parameters do not exceed the view range covered by the sparse input view set, including: The target new viewpoint pose parameters are matched with the scene feature representation, and the viewpoint features corresponding to the target new viewpoint pose parameters are extracted from the scene feature representation. Based on the aforementioned viewpoint features, pixel-by-pixel rendering is performed along the line of sight determined by the new viewpoint pose parameters of the target to obtain the new viewpoint image.

[0063] In a specific example, the implicit view reconstruction module 203 described above is further used to obtain the new view image by using the scene feature representation as a condition and performing generative extrapolation on the new view that exceeds the range through a progressive denoising diffusion generation process when the target new view pose parameters exceed the view range covered by the sparse input view set.

[0064] In a specific example, the aforementioned viewpoint smoothing amplification module 204 is specifically used to smooth and amplify the original distribution of the viewpoint parameter space, generating multiple amplified viewpoint parameters with better coverage uniformity than the original distribution, including: The distribution pattern of camera pose parameters in the viewpoint parameter space is learned using the observed viewpoint samples; Based on the distribution pattern, iterative sampling is performed in the neighborhood of the original distribution to generate candidate view parameters with a density higher than that of the original distribution; The candidate viewing angle parameters are evaluated for coverage uniformity, and the augmented viewing angle parameters that make the coverage density of each region in the viewing angle parameter space tend to be consistent are selected.

[0065] In a specific example, the dataset construction module 205 described above is specifically used to obtain the annotation information of the target object in the multi-view image sequence, wherein the annotation information is one of the target detection box or the segmentation mask; Based on the geometric transformation relationship between the camera pose parameters corresponding to each image in the multi-view image sequence and the camera pose parameters corresponding to the new view image or the augmented view image, the annotation information is projected and mapped onto the corresponding new view image or the augmented view image.

[0066] This embodiment provides an implicit view reconstruction system for data augmentation, which is used to execute the implicit view reconstruction method for data augmentation provided in the above embodiment. Its implementation method and principle are the same. For details of the implementation method of each module, please refer to the relevant description of the above method embodiment, which will not be repeated here.

[0067] This invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the implicit view reconstruction method for data augmentation of any of the above embodiments.

[0068] In an exemplary embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The illustrated electronic device includes a processor and a memory. The processor and memory are connected, for example, via a bus. Optionally, the electronic device may also include a transceiver, which integrates software programs or hardware such as applications, application programming interfaces (APIs), middleware, and kernels. It should be noted that in practical applications, the transceiver is not limited to one unit, and the structure of this electronic device does not constitute a limitation on the embodiments of the present invention.

[0069] The processor can be a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0070] A bus can include a pathway for transmitting information between the aforementioned components or connecting these devices to hardware such as input / output interfaces, display devices, and communication interfaces. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0071] The memory can be ROM (Read Only Memory), which is another type of static storage device that stores static information and instructions; RAM (Random Access Memory), which is another type of dynamic storage device that stores information and instructions; it can also be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0072] The memory stores computer program code that executes the present invention, and its execution is controlled by a processor. The processor executes the computer program code stored in the memory to implement the content shown in the foregoing method embodiments.

[0073] Among them, electronic devices include, but are not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0074] This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the implicit view reconstruction method for data augmentation of any of the above embodiments at runtime.

[0075] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0076] Those skilled in the art will understand that the technical solution of the present invention, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (such as a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of the present invention when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0077] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device performs all or part of the steps of the methods described in the various embodiments of the present invention.

[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to depart from the protection scope of the present invention.

Claims

1. An implicit view reconstruction method for data augmentation, characterized in that, include: Acquire multi-view image sequences of the target scene and the camera pose parameters corresponding to each frame; Based on the camera pose parameters of the multi-view image sequence, the view space distribution is analyzed, the view space is divided into multiple regions, and images with satisfactory image quality are selected from each region to form a sparse input view set. The sparse input view set and the corresponding camera pose parameters are mapped to a three-dimensional implicit feature space to obtain the scene feature representation of the target scene. Based on the specified target new view pose parameters, the corresponding view features are locked from the scene feature representation, and the new view image corresponding to the target new view is rendered. A viewpoint parameter space is constructed using the camera pose parameters of the multi-view image sequence as observation viewpoint samples. The original distribution of the viewpoint parameter space is smoothly amplified to generate multiple amplified viewpoint parameters with better coverage uniformity than the original distribution. Based on the amplified viewpoint parameters, a corresponding amplified viewpoint image is generated from the scene feature representation. The new perspective image, the augmented perspective image, and the multi-view image sequence are merged to obtain the enhanced image dataset.

2. The implicit view reconstruction method for data augmentation according to claim 1, characterized in that, Mapping the sparse input view set and its corresponding camera pose parameters to a 3D implicit feature space to obtain a scene feature representation of the target scene includes: The images in the sparse input view set and the camera pose parameters corresponding to each image are jointly mapped to obtain the three-dimensional perception information of each image in the three-dimensional implicit feature space. Based on the cross-view geometric correspondence between the three-dimensional perception information of each image, establish the association constraints between different viewpoints; Based on the aforementioned association constraints, the 3D perception information of each image is aggregated to output a scene feature representation of the target scene.

3. The implicit view reconstruction method for data augmentation according to claim 2, characterized in that, When the target new view pose parameters do not exceed the view range covered by the sparse input view set, the step of locking the corresponding view features from the scene feature representation based on the specified target new view pose parameters and rendering the new view image corresponding to the target new view includes: The target new viewpoint pose parameters are matched with the scene feature representation, and the viewpoint features corresponding to the target new viewpoint pose parameters are extracted from the scene feature representation. Based on the aforementioned viewpoint features, pixel-by-pixel rendering is performed along the line of sight determined by the new viewpoint pose parameters of the target to obtain the new viewpoint image.

4. The implicit view reconstruction method for data augmentation according to claim 1, characterized in that, When the target new perspective pose parameters exceed the perspective range covered by the sparse input view set, the target new perspective image is obtained by generative extrapolation of the out-of-range target new perspective through a progressive denoising diffusion generation process, using the scene feature representation as a condition.

5. The implicit view reconstruction method for data augmentation according to claim 1, characterized in that, The original distribution of the viewpoint parameter space is smoothly amplified to generate multiple amplified viewpoint parameters with better coverage uniformity than the original distribution, including: The distribution pattern of camera pose parameters in the viewpoint parameter space is learned using the observed viewpoint samples; Based on the distribution pattern, iterative sampling is performed in the neighborhood of the original distribution to generate candidate view parameters with a density higher than that of the original distribution; The candidate viewing angle parameters are evaluated for coverage uniformity, and the augmented viewing angle parameters that make the coverage density of each region in the viewing angle parameter space tend to be consistent are selected.

6. The implicit view reconstruction method for data augmentation according to claim 1, characterized in that, The image quality conditions include: at least one of the image sharpness, exposure, and visibility of the target object reaching the corresponding threshold.

7. The implicit view reconstruction method for data augmentation according to claim 1, characterized in that, Based on the camera pose parameters of the multi-view image sequence, a viewpoint spatial distribution analysis is performed, dividing the viewpoint space into multiple regions, including: The camera pose parameters of the multi-view image sequence are mapped to a spherical coordinate system, which takes the center of the target scene as the origin and uses azimuth and elevation angles to represent the viewpoint direction of each frame image. Clustering algorithms are used to cluster the pose parameters of each camera after mapping to the spherical coordinate system. Based on the clustering results, the view space is divided into a preset number of regions, each region corresponding to a different view direction.

8. The implicit view reconstruction method for data augmentation according to claim 1, characterized in that, Before merging the new perspective image, the augmented perspective image, and the multi-view image sequence, the method further includes: The new viewpoint image and the augmented viewpoint image are cropped using one of the following methods: center cropping, random cropping, and adaptive cropping based on the target object position, so that the cropped new viewpoint image and the augmented viewpoint image have the same resolution.

9. The implicit view reconstruction method for data augmentation according to claim 1 or 8, characterized in that, The method further includes: Obtain the annotation information of the target object in the multi-view image sequence, wherein the annotation information is either a target detection box or a segmentation mask; Based on the geometric transformation relationship between the camera pose parameters corresponding to each image in the multi-view image sequence and the camera pose parameters corresponding to the new view image or the augmented view image, the annotation information is projected and mapped onto the corresponding new view image or the augmented view image.

10. An implicit view reconstruction system for data augmentation, characterized in that, include: The data acquisition module is used to acquire multi-view image sequences of the target scene and the camera pose parameters corresponding to each frame of the image; The viewpoint filtering module is used to perform viewpoint spatial distribution analysis based on the camera pose parameters of the multi-viewpoint image sequence, divide the viewpoint space into multiple regions, and select images with satisfactory image quality from each region to form a sparse input view set. The implicit view reconstruction module is used to map the sparse input view set and the corresponding camera pose parameters to a three-dimensional implicit feature space to obtain the scene feature representation of the target scene, lock the corresponding view features from the scene feature representation based on the specified target new view pose parameters, and render the new view image corresponding to the target new view. The viewpoint smoothing and augmentation module is used to construct a viewpoint parameter space using the camera pose parameters of the multi-view image sequence as observation viewpoint samples, smoothly augment the original distribution of the viewpoint parameter space, generate multiple augmented viewpoint parameters with better coverage uniformity than the original distribution, and generate corresponding augmented viewpoint images from the scene feature representation based on the augmented viewpoint parameters. The dataset construction module is used to merge the new perspective image, the augmented perspective image, and the multi-view image sequence to obtain an enhanced image dataset.

Images