Transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement

Abstract

The invention discloses a transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement, and the method comprises the steps: carrying out the modeling of collected multi-channel multi-frame video data into a fourth-order tensor, carrying out the normalization processing of the fourth-order tensor, decomposing the fourth-order tensor into a third-order depth tensor and a fourth-order RGB tensor, and maintaining the image space, time and channel information. For different tensors, a transform domain high-order tensor denoising model is constructed, and mixed noise is removed by using a tube rank and constraint conditions, including suppression of low rank and sparse noise. An optimization problem is converted through an augmented Lagrange expression, and alternating optimization is carried out by adopting an alternating direction multiplier method frame until convergence is carried out to obtain denoised tensor data. And finally, splicing the de-noised depth and RGB information on a channel dimension through tensor inverse remodeling, and recovering a complete RGBD video data structure. According to the method, mixed noise is effectively removed, the data definition and reducibility are improved, high-quality input data are provided for a 3D machine vision system, and the detection and measurement precision and robustness are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a video denoising method, and in particular to a transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement, which is suitable for data processing and denoising applications of 3D machine vision systems and belongs to the technical field of industrial vision and image processing. Background Art

[0002] In the fields of modern industrial automation, precision manufacturing, and intelligent detection, 3D machine vision systems have been widely used in various detection and measurement scenarios. These systems rely on RGBD (red, green, blue, depth) video data to obtain the three-dimensional shape, position, and texture information of the target object, and support the analysis of the appearance, structure, and precise size of the object. This type of 3D vision technology has become one of the key means to improve the efficiency of automated production lines and product quality control. However, in the process of collecting and transmitting RGBD video data, due to the performance limitations of the sensor, the complexity of the environment, and the limitations of data processing, various noises are often generated, resulting in a decrease in image quality, which affects the detection and measurement accuracy of the 3D machine vision system. Effective noise removal has become an important research direction in 3D machine vision systems.

[0003] Currently, the types of noise that affect the quality of RGBD video data mainly include photosensitive noise, compression noise, quantization noise, measurement noise, and spatial alignment noise. The manifestations and causes of these noises in practical applications are different:

[0004] 1. Photosensitive noise: Photosensitive noise comes from the physical limitations of RGB sensors. Especially in low light conditions, the sensor is easily affected by factors such as electronic thermal noise, resulting in random fluctuations in the pixel values ​​of the collected image. This type of noise will significantly reduce the signal-to-noise ratio of the image and affect the clarity of image details.

[0005] 2. Compression noise: In order to reduce data transmission pressure and storage space, RGBD video data is usually processed by lossy compression. Quantization and encoding during the compression process will introduce compression noise, resulting in loss of image details and degradation of image quality. This noise is particularly significant in areas with more high-frequency details, manifesting as block distortion or blurring.

[0006] 3. Quantization noise: In the process of digitizing RGBD data, the bit depth of the sensor limits the sampling accuracy and generates quantization noise. Especially in depth image data, quantization noise causes discontinuity of depth information in the image, which appears as a step-like error, affecting 3D reconstruction and accurate measurement.

[0007] 4. Measurement noise: Measurement noise mainly occurs in depth data acquisition, especially on the surface of objects with strong light, reflection or transparent materials, it is difficult for the depth sensor to obtain the true depth value. This noise usually manifests as jitter, distortion or loss of depth data, which has a negative impact on the accurate measurement and reconstruction of the 3D vision system.

[0008] 5. Spatial alignment noise: Since RGB and depth images are usually acquired by different sensors, there is a certain physical distance or angle difference between the two. In order to realize the combined application of RGBD data, RGB images and depth images usually need to be spatially aligned. However, errors in the alignment process will cause the RGB image and depth image to shift or misalign, thereby introducing spatial alignment noise in 3D reconstruction.

[0009] In view of the noise from these different sources, various denoising methods have been proposed in the prior art. These methods can be roughly divided into three categories: denoising methods based on filtering, denoising methods based on deep learning, and denoising methods based on sparse representation and dictionary learning.

[0010] Filter-based denoising methods are a type of traditional denoising techniques, including mean filtering, Gaussian filtering, median filtering, and bilateral filtering. These methods reduce noise by smoothing the image, especially for photosensitive noise and quantization noise. Mean filtering suppresses random noise by taking the average value of pixels in a local area, but it also causes blurring of image details. Gaussian filtering uses weighted averaging to reduce noise while retaining some edge information, but the effect is still limited when processing complex images. Bilateral filtering can better retain edge details while removing noise, but the amount of calculation is large and there is an efficiency bottleneck on high-dimensional data. In general, filter-based denoising methods are easy to implement, but they are prone to losing image detail information while removing noise, especially in the processing of RGBD videos.

[0011] Denoising methods based on deep learning have made great progress in the field of image processing in recent years. Deep learning denoising methods usually rely on a large amount of annotated data, and automatically learn and remove noise features by training convolutional neural networks (CNNs), generative adversarial networks (GANs) or other models. Deep learning denoising methods can handle different types of noise and have good detail retention capabilities. However, deep learning methods also face some limitations. First, deep learning models require large-scale annotated data for training, the data acquisition cost is high, and the training process is complex and time-consuming. Second, deep learning models often have "black box" characteristics, and the denoising mechanism inside the model is difficult to explain. Especially in RGBD video data, the types and distributions of noise are complex and diverse, and the denoising effect of deep learning models on unseen noise may be significantly reduced. Therefore, when deep learning denoising methods are applied to 3D machine vision systems, there are certain problems of insufficient scalability and interpretability.

[0012] Denoising methods based on sparse representation and dictionary learning are a new type of method that has gradually developed in recent years. Sparse representation methods assume that images are sparse in a certain transform domain. Based on this assumption, sparse dictionaries are constructed to represent image signals as a combination of sparse coefficients. By sparsely processing images in the transform domain, sparse representation methods can effectively remove Gaussian noise and compression noise. However, sparse representation methods face some bottlenecks when processing RGBD video data. First, the assumption of sparse representation does not work well under structural noise and non-Gaussian noise, which limits its application effect in complex noise backgrounds. Second, the computational complexity of sparse representation and dictionary learning methods is high, especially in video data processing, which requires a long computing time. In addition, sparse representation methods usually rely on fixed transform domains and sparse assumptions, while actual RGBD video data contains multiple types of noise and rich structural information, making it difficult to remove all noise through a single sparse representation.

[0013] In summary, existing denoising techniques still face the following challenges in processing RGBD video data for 3D machine vision systems:

[0014] 1. Balance between denoising and detail preservation: Preserving image details while achieving effective denoising is an indispensable requirement in RGBD video data processing, but existing methods often find it difficult to achieve a good balance between denoising and detail preservation.

[0015] 2. Versatility and interpretability of the method: 3D machine vision systems have a wide range of application scenarios and a variety of noise types. Existing denoising methods have poor versatility and adaptability, especially those based on deep learning, whose "black box" characteristics limit their application in diverse scenarios.

[0016] 3. Computational efficiency and feasibility: RGBD video data is high-dimensional data, and computational efficiency is crucial to practical applications. However, the existing denoising methods generally have low computational efficiency when processing video data, and it is difficult to meet real-time processing requirements.

[0017] Therefore, in the practical application of 3D machine vision systems, there is an urgent need for a denoising technology that can effectively denoise and retain detail information and has high computational efficiency and versatility. Summary of the invention

[0018] In order to solve the above problems in the prior art, the present invention proposes a transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement, comprising the following steps:

[0019] The collected multi-channel multi-frame video data is represented as a fourth-order tensor, wherein the dimensions of the fourth-order tensor correspond to the length, width, number of channels and time dimension of the image respectively;

[0020] Normalize the fourth-order tensor and adjust the data to a uniform scale range;

[0021] The normalized fourth-order tensor is decomposed into a third-order depth tensor and a fourth-order RGB tensor, where the third-order depth tensor retains the spatial and temporal information of the image, and the fourth-order RGB tensor contains RGB channels and time dimensions.

[0022] The third-order depth tensor and the fourth-order RGB tensor are denoised respectively, and a transform domain high-order tensor denoising model is constructed. The mixed noise in the data is removed by using the pipe rank and constraint conditions, including the suppression of low-rank and sparse noise.

[0023] The optimization problem is transformed into an augmented Lagrangian expression, and the variables are alternately optimized using the alternating direction multiplier method framework until the convergence condition is reached to obtain the denoised tensor data.

[0024] The denoised third-order depth tensor is expanded to a fourth-order tensor and concatenated with the denoised fourth-order RGB tensor in the channel dimension to obtain the complete denoised fourth-order tensor and restore the structure of the RGBD video data.

[0025] The decomposed third-order depth tensor and fourth-order RGB tensor are denoised respectively, and a transform domain high-order tensor denoising model is constructed to remove mixed noise using the following optimization problem:

[0026]

[0027] in, Represents a high-order tensor The rank function of and denote Gaussian noise and sparse noise respectively, and α1 and α2 are regularization parameters.

[0028] In the optimization solution process, the convex substitution function of the tube rank is used to rewrite the optimization problem as follows:

[0029]

[0030] in, It represents the pipe rank of a high-order tensor in the transform domain, which can be the discrete Fourier domain or the discrete cosine domain.

[0031] The optimization problem is converted into an augmented Lagrangian expression:

[0032]

[0033] The alternating direction method of multipliers (ADMM) framework is used to and Perform alternating optimization until convergence conditions are reached;

[0034] in:

[0035] represents the real tensor to be solved;

[0036] Represents the Gaussian noise component in the tensor;

[0037] Represents the sparse noise component in the tensor;

[0038] Represents the observed fourth-order tensor data;

[0039] Represents a high-order tensor The transform domain pipe rank of

[0040] Used to suppress Gaussian noise, is the Frobenius norm of Gaussian noise, α1 is the regularization parameter;

[0041] Used to suppress sparse noise, For sparse noise norm, α2 is the regularization parameter;

[0042] represents the Lagrange multiplier term, where is the Lagrange multiplier, <·,·> is the inner product operation;

[0043] represents the penalty term of the constraint, where β is the penalty parameter.

[0044] The denoised third-order depth tensor It is expanded to a fourth-order tensor by the following formula:

[0045]

[0046] And in the channel dimension with the denoised fourth-order RGB tensor Splice to get the complete fourth-order tensor after denoising The splicing formula is as follows:

[0047]

[0048] in:

[0049] Represents the denoised third-order depth tensor with dimension [L, W, T], where:

[0050] L: The length of the image, which indicates the number of pixels in the horizontal direction.

[0051] W: The width of the image, which indicates the number of pixels in the vertical direction.

[0052] L: Time dimension, indicating the number of video frames.

[0053] Represents an expanded fourth-order depth tensor with dimensions [L, W, 1, T]; the newly added "1" is the depth channel dimension;

[0054] reshape(·): Array reshaping operation, used to expand a third-order deep tensor to a fourth-order tensor;

[0055] Represents the denoised fourth-order RGB tensor with dimension [L, W, 3, T];

[0056] 3: represents the dimensions of red, green and blue of the RGB three channels;

[0057] Represents the expanded fourth-order deep tensor with dimension [L,W,1,T];

[0058] Represents the final denoised fourth-order tensor with dimensions [L, W, 4, T], where "4" represents the combination of the RGB three channels plus the depth channel;

[0059] cat(3,·): indicates concatenation operation on the third dimension (channel dimension).

[0060] Beneficial effects:

[0061] The present invention effectively removes low-rank and sparse noise in the data by modeling multi-channel and multi-frame video data as a fourth-order tensor and using a transform domain high-order tensor denoising model, thereby achieving accurate suppression of mixed noise. The augmented Lagrangian expression and the alternating direction multiplier method are used to improve the efficiency of the optimization process and the convergence of the algorithm. Through tensor inverse reshaping, the denoised depth and RGB information are integrated into a unified structure to restore high-quality RGBD video data, providing clear and accurate input data for 3D machine vision inspection and measurement, thereby improving the detection accuracy and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:

[0063] Figure 1 : A schematic diagram showing the RGBD video data processing flow. DETAILED DESCRIPTION

[0064] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0065] This embodiment provides a transform domain high-order tensor video denoising method for a 3D machine vision system. The method effectively removes mixed noise in the data by modeling the collected RGBD video data as a fourth-order tensor and combining the pipe rank denoising strategy of the transform domain high-order tensor, thereby significantly improving the clarity and restoration quality of the video data. Figure 1 The specific implementation process of this method is described in detail.

[0066] First, this method represents the RGBD video data collected in the 3D machine vision system as a fourth-order tensor Where L represents the length of the image, W represents the width of the image, C represents the number of channels (usually including three color channels, RGB, and one depth channel), and T represents the time dimension. This tensor structure can effectively integrate RGB and depth information to form a complete data model to process color and depth information under a unified data framework.

[0067] In practical applications, due to the instability of electronic components and the interference of the external environment, different types of noise are inevitably mixed into the collected RGBD video data. This method is mainly aimed at Gaussian noise and sparse noise Perform denoising. Assume that the real noise-free RGBD video data is The noise-degraded video data collected It can be expressed as:

[0068]

[0069] in, represents the real noise-free video data that we want to recover from Y. and They represent Gaussian noise and sparse noise components respectively. Gaussian noise usually appears as a weak disturbance with random distribution, while sparse noise is mostly sudden noise, such as salt and pepper noise. These noises will cause distortion of video data and affect the accuracy of visual detection and analysis.

[0070] In order to transform this denoising problem into a solvable optimization problem, this method uses the low-rank prior characteristics of high-order tensors to construct a complete multi-noise removal process. Specifically, this method mainly includes the following steps: tensor normalization, tensor reshaping, and transform domain high-order tensor denoising algorithm.

[0071] Step 1: Tensor Normalization

[0072] For the collected fourth-order tensor data First, normalize it. The purpose of normalization is to adjust the data to a uniform scale range to improve the stability and consistency of subsequent data processing and reduce processing errors that may be caused by differences in data ranges. The specific normalization formula is as follows:

[0073]

[0074] Parameter Description:

[0075] Represents the original collected fourth-order tensor data, with the dimension [L×W×C×T], where L is the length of the image, W is the width of the image, C is the number of channels (usually including three RGB color channels and one depth channel), and T is the time dimension.

[0076] Representing a tensor The minimum value in . It is obtained by counting the values ​​of the entire tensor range and is used to determine the lower limit of the data. During the normalization process, For the original tensor Remove the offset from the , so that the data range is adjusted starting from the lowest value.

[0077] Representing a tensor The maximum value in . It is obtained by counting the values ​​of the entire tensor range and is used to determine the upper limit of the data. In normalization, As the denominator, it represents the width of the value interval of the original data.

[0078] Represents a normalized tensor with the same dimensions as is consistent, that is, [L×W×C×T]. After normalization, The value range is compressed to the interval [0,1] to facilitate subsequent processing.

[0079] After normalization, we get a new tensor The data range is limited to [0,1]. This normalization operation not only improves the consistency of the data, but also effectively reduces the calculation errors caused by extreme values, laying a reliable data foundation for the subsequent tensor reshaping and denoising. This step ensures that each channel and time frame data is processed at a unified scale, making the subsequent denoising algorithm more stable and effective when processing multi-channel video data.

[0080] Step 2: Reshape the tensor

[0081] After normalization, the fourth-order tensor Contains RGB and depth data. In order to further improve the efficiency and accuracy of denoising, Perform a reshape operation to decompose it into two independent tensors: a third-order depth tensor and a fourth-order RGB tensor, so as to optimize the data characteristics of different channels. The specific decomposition steps are as follows:

[0082] First, from Extract the data of the depth channel and generate a third-order depth tensor This tensor retains the spatial and temporal variations of the depth information, which helps in the subsequent depth data denoising process.

[0083] Secondly, from Extract the RGB channel data to form a fourth-order RGB tensor This tensor contains the color information of the RGB three channels and the frame sequence of the time dimension, which is used to describe the color details and time-varying characteristics of each frame image in the video.

[0084] Through this tensor reshaping operation, RGB and depth data can be modeled separately, so that more targeted processing can be performed according to the characteristics of different types of data during the denoising process. This structural decomposition provides the basis for the precise execution of the subsequent denoising algorithm, allowing this method to more efficiently remove mixed noise in video data.

[0085] Step 3: Transform domain high-order tensor denoising algorithm

[0086] In this embodiment, the third-order depth tensor and a fourth-order RGB tensor The core of this denoising algorithm is to effectively remove Gaussian noise and sparse noise in the data by constructing a high-order tensor denoising model in the transform domain and using the pipe rank and constraint conditions. The specific optimization problem is stated as follows:

[0087]

[0088] This formulation describes a multi-noise removal optimization problem for high-dimensional tensor data, where the goal is to minimize the tube rank function And two regularization terms to achieve low-rank representation of tensors and suppress the interference of Gaussian noise and sparse noise, where:

[0089] Represents the observed original fourth-order tensor data, including the target tensor Gaussian noise and sparse noise Mixed results.

[0090] Represents the real tensor data to be solved, and its low-rank structure reflects the effective information in the tensor.

[0091] Represents the Gaussian noise component in the tensor, which obeys the normal distribution and is random.

[0092] Represents sparse noise components in a tensor, such as salt and pepper noise or bursty outliers, with sparse distribution characteristics.

[0093] Represents a high-order tensor The tube rank function is used to measure the low-rank characteristics of the tensor. The introduction of the tube rank function helps capture the low-dimensional structural information of the tensor, thereby retaining the main features of the data.

[0094] represents the Gaussian noise suppression term, where is the Frobenius norm of Gaussian noise, which is used to measure the quadratic error of noise. α1 is the regularization parameter, which is used to adjust the weight of Gaussian noise suppression.

[0095] represents the sparse noise suppression term, where For sparse noise The norm is used to measure the sparsity of noise, and α2 is the regularization parameter used to adjust the weight of sparse noise suppression.

[0096] In order to reduce the complexity of the solution, a convex substitution function of the tube rank is introduced and the optimization problem is rewritten as:

[0097]

[0098] Represents a high-order tensor The convex substitute function of the pipe rank in the transform domain is used to approximate the original pipe rank function in a more solvable form. By decomposing the tensor into frequency domain data through the transform domain method, the low rank characteristics of the tensor can be more efficiently represented.

[0099] α1 and α2: The two regularization parameters control the strength of suppressing Gaussian noise and sparse noise respectively. A larger α1 will strengthen the constraints on random noise, while a larger α2 will suppress sparse noise more effectively.

[0100] This optimization problem achieves multi-noise removal of high-dimensional tensor data by minimizing the tube rank and noise terms.

[0101] Through this method, the convex substitution function of the transform domain tube rank reduces the computational complexity of the optimization process while ensuring the denoising effect, making the processing of high-dimensional tensors more efficient, thereby improving the accuracy and adaptability of the 3D machine vision system in denoising processing.

[0102] Step 4: Augmented Lagrangian function and ADMM framework solution

[0103] In order to solve the above optimization problem and realize multi-noise removal, the augmented Lagrangian function is further introduced to decompose the problem. Convert the optimization problem into an augmented Lagrangian expression. Augmented Lagrangian function The definition is as follows:

[0104]

[0105] in, is the Lagrange multiplier, β is the penalty parameter, which is used to adjust the constraint term so that Better approximation to the original data This ensures the stability and convergence of the solution process.

[0106] Formula parameter explanation:

[0107] Represents the real tensor to be solved, which contains the main structural information of the original data. One of the optimization goals is to minimize its transform domain rank.

[0108] Represents the Gaussian noise component in the tensor, whose random distribution characteristics are constrained by the quadratic norm.

[0109] Represents the sparse noise component in the tensor, which usually appears as anomalies or burst noise. Its sparse characteristics are Norm constraints.

[0110] Represents the observed fourth-order tensor data, which is and combination.

[0111] Represents a high-order tensor The transformed domain pipe rank is minimized by a convex surrogate function of the pipe rank, preserving the low rank characteristics of the tensor.

[0112] Used to suppress Gaussian noise, is the Frobenius norm of Gaussian noise, which measures the square error of Gaussian noise. α1 is the regularization parameter, which controls the weight of Gaussian noise suppression.

[0113] Used to suppress sparse noise, For sparse noise norm, which measures the sparsity of sparse noise, and α2 is the regularization parameter, which controls the weight of sparse noise suppression.

[0114] represents the Lagrange multiplier term, where is the Lagrange multiplier, and <·,·> is the inner product operation. This term is used to ensure the constraint condition The optimization process is gradual.

[0115] represents the penalty term of the constraint, where β is the penalty parameter, which is used to control the impact of the constraint term on the optimization objective. This term ensures that Gradually approaching the original data

[0116] In the solution process, the alternating direction method of multipliers (ADMM) framework is used to alternately optimize and solve each variable. Specifically, the ADMM framework updates the variables in sequence. and Gradually approach the optimal solution of the optimization problem. The specific solution steps are as follows:

[0117] Sub-problem 1: First update Variables, minimize the objective function:

[0118]

[0119] The closed-form solution to this subproblem is:

[0120]

[0121] Among them, Proj 1 / β Express The variables are projected to ensure that the rank constraint is satisfied.

[0122] The variable updated in the current iteration step represents the actual tensor data after optimization.

[0123] Represents a high-order tensor The transform domain rank is used to preserve the low rank characteristics of the data.

[0124] The penalty term is adjusted by β so that Gradually approaching the observed data

[0125] Project 1 / β (·): indicates the value of a variable The projection operation is used to ensure Satisfy the pipe rank constraint, where is the adjustment parameter of the projection.

[0126] Sub-question 2: Next update variables, by minimizing the following objective function:

[0127]

[0128] The closed-form solution to this problem is:

[0129]

[0130] With this update, The change gradually approaches the optimal noise suppression effect.

[0131] The updated Gaussian noise variable of the current iteration.

[0132] The Frobenius norm of Gaussian noise is used to quantify the square error of Gaussian noise. α1 is the regularization parameter that controls the intensity of Gaussian noise suppression.

[0133] <·,·>: represents the inner product operation, which is used to introduce the Lagrange multiplier term so that The constraints are gradually satisfied.

[0134] The penalty term is used to reduce the degree of constraint violation, and β is the penalty parameter.

[0135] The closed-form solution to this subproblem is:

[0136]

[0137] molecular Represents the noise estimate under the current constraints.

[0138] Denominator 2α1+β: The weight used to balance the regularization constraint and penalty term of Gaussian noise.

[0139] Sub-question 3: Update Variables, minimize the objective function:

[0140]

[0141] The closed-form solution to this problem is:

[0142]

[0143] Among them, soft(·) represents the soft threshold operation, which is used to Suppress the sudden noise so that it can be effectively removed.

[0144] The sparse noise variable after the current iteration step update.

[0145] Sparse noise The norm is used to quantify the sparsity of sparse noise, and α2 is the regularization parameter that controls the strength of sparse noise suppression.

[0146] The closed-form solution to this subproblem is:

[0147]

[0148] Represents the sparse noise estimate under the current constraints.

[0149] Update auxiliary variables: Finally update the Lagrange multiplier variable:

[0150]

[0151] This step is gradually adjusted value, and It tends to converge and ensures that the final constraints are met.

[0152] The Lagrange multiplier for the current iteration, used to adjust the constraints.

[0153] Indicates the degree of default under the current constraints.

[0154] β: Penalty parameter, used to adjust the weight of the constraint.

[0155] This step is continuously updated The constraints are gradually satisfied and finally converged.

[0156] Through the ADMM framework, alternate optimization and variable:

[0157] Subproblem 1: Optimizing the real tensor with a rank constraint

[0158] Sub-problem 2: Suppressing Gaussian noise via the Frobenius norm

[0159] Sub-question 3: By Norm to remove sparse noise

[0160] Lagrange multiplier update: ensure that constraints are satisfied step by step.

[0161] Finally, the ADMM framework completes multi-noise removal and restores high-quality RGBD video data tensors by setting convergence conditions.

[0162] By alternating the above steps, the ADMM framework can continuously optimize the solutions of each variable and gradually approach the global optimal solution. The algorithm determines the termination of iteration by setting the convergence condition, and finally obtains the RGBD video data tensor after denoising, realizing the effective removal of mixed noise in the video data.

[0163] Step 5: Inverse reshape of the tensor

[0164] After the denoising process is completed, the denoised RGB tensor and denoised depth tensor are recorded as and In order to restore the complete RGBD video data structure, the third-order depth tensor needs to be expanded to a fourth-order tensor so that it can be spliced ​​with the fourth-order RGB tensor in the channel dimension. The specific expansion operation is as follows:

[0165]

[0166] Represents the denoised third-order depth tensor with dimension [L, W, T], where:

[0167] Represents an expanded fourth-order depth tensor with dimensions [L,W,1,T]. The newly added "1" is the depth channel dimension, which is used to keep consistent with the channel structure of the RGB tensor.

[0168] reshape(·): Array reshaping operation, used to expand the third-order depth tensor to a fourth-order tensor to meet the dimensional requirements of concatenation with the RGB tensor.

[0169] This operation converts the third-order depth tensor It is expanded to a fourth-order tensor of shape [L, W, 1, T], where L and W represent the length and width of the image, 1 is the newly added depth channel dimension, and T represents the time dimension. Through this expansion operation, the dimension of the depth tensor is consistent with the RGB tensor, so that smooth splicing can be achieved in the channel dimension.

[0170] Next, the expanded depth tensor And the denoised RGB tensor Splicing is performed on the channel dimension (the third dimension) to obtain the final denoised fourth-order tensor The splicing formula is as follows:

[0171]

[0172] Represents the denoised fourth-order RGB tensor with dimensions [L, W, 3, T], where:

[0173] 3: Represents the dimension of the RGB three channels (red, green, and blue).

[0174] Represents an expanded fourth-order depth tensor with dimension [L,W,1,T]. This tensor contains the spatial and temporal distribution of depth information.

[0175] Represents the final denoised fourth-order tensor with dimensions [L, W, 4, T], where "4" represents the combination of the RGB three channels plus the depth channel. This tensor integrates the denoised RGB and depth information and is the complete RGBD video data.

[0176] cat(3,·): represents a concatenation operation on the third dimension (channel dimension). The three channels of RGB data and the single channel of depth data are combined into a complete four-channel tensor.

[0177] Since the RGB tensor is a fourth-order tensor and the depth tensor is a third-order tensor, in order to achieve data integration, the depth tensor needs to be expanded to a fourth-order tensor through a reshape operation so that a single channel is added in the channel dimension to keep it consistent with the structure of the RGB tensor.

[0178] The denoised RGB tensor and the expanded depth tensor are concatenated along the channel dimension using the cat(3,·) operation to generate a denoised fourth-order tensor containing complete RGBD information.

[0179] Complete Data Recovery:

[0180] Through the above expansion and splicing operations, the denoised RGB and depth information are reintegrated into a unified structure, providing clear and accurate denoised video data for subsequent 3D machine vision applications.

[0181] This tensor inverse reshaping step ensures that the denoised RGB and depth information can be reintegrated into a unified structure, providing clear and accurate denoised video data for subsequent 3D machine vision applications.

[0182] The above description is only a preferred embodiment of the present invention, so all equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims

1. A transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement, characterized by: The following steps are involved: The collected multi-channel multi-frame video data is represented as a fourth-order tensor, wherein the dimensions of the fourth-order tensor correspond to the length, width, number of channels and time dimension of the image respectively; Normalize the fourth-order tensor and adjust the data to a uniform scale range; The normalized fourth-order tensor is decomposed into a third-order depth tensor and a fourth-order RGB tensor, where the third-order depth tensor retains the spatial and temporal information of the image, and the fourth-order RGB tensor contains RGB channels and time dimensions; The third-order depth tensor and the fourth-order RGB tensor are denoised respectively, and a transform domain high-order tensor denoising model is constructed. The mixed noise in the data is removed by using the pipe rank and constraint conditions, including the suppression of low-rank and sparse noise. The optimization problem is transformed into an augmented Lagrangian expression, and the variables are alternately optimized using the alternating direction multiplier method framework until the convergence condition is reached to obtain the denoised tensor data. The denoised third-order depth tensor is expanded to a fourth-order tensor and concatenated with the denoised fourth-order RGB tensor in the channel dimension to obtain the complete denoised fourth-order tensor and restore the structure of the RGBD video data.

2. The transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement according to claim 1, characterized in that: The decomposed third-order depth tensor and fourth-order RGB tensor are denoised respectively, and a transform domain high-order tensor denoising model is constructed to remove mixed noise using the following optimization problem: in, Represents a high-order tensor The rank function of and denote Gaussian noise and sparse noise respectively, and α1 and α2 are regularization parameters.

3. The transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement according to claim 2, characterized in that: In the optimization solution process, the convex substitution function of the tube rank is used to rewrite the optimization problem as follows: in, It represents the pipe rank of a high-order tensor in the transform domain, which can be the discrete Fourier domain or the discrete cosine domain.

4. The transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement according to claim 1, characterized in that: Convert the optimization problem to augmented Lagrangian The expression is: The alternating direction method of multipliers (ADMM) framework is used to and Perform alternating optimization until convergence conditions are reached; in: represents the real tensor to be solved; Represents the Gaussian noise component in the tensor; Represents the sparse noise component in the tensor; Represents the observed fourth-order tensor data; Represents a high-order tensor The transform domain pipe rank of Used to suppress Gaussian noise, is the Frobenius norm of Gaussian noise, α1 is the regularization parameter; Used to suppress sparse noise, is the l1 norm of sparse noise, α2 is the regularization parameter; represents the Lagrange multiplier term, where is the Lagrange multiplier, <·,·> is the inner product operation; represents the penalty term of the constraint, where β is the penalty parameter.

5. The transform domain high-order tensor video denoising method for 3D machine vision inspection and measurement according to claim 1, characterized in that: The denoised third-order depth tensor It is expanded to a fourth-order tensor by the following formula: And in the channel dimension with the denoised fourth-order RGB tensor Splice to get the complete fourth-order tensor after denoising The splicing formula is as follows: in: Represents the denoised third-order depth tensor with dimension [L, W, T], where: L: The length of the image, which indicates the number of pixels in the horizontal direction; W: The width of the image, which indicates the number of pixels in the vertical direction; T: time dimension, indicating the number of video frames; Represents an expanded fourth-order depth tensor with dimensions [L,W,1,T]; the newly added "1" is the depth channel dimension; reshape(·): Array reshaping operation, used to expand a third-order deep tensor to a fourth-order tensor; Represents the denoised fourth-order RGB tensor with dimension [L, W, 3, T]; 3: represents the dimensions of red, green and blue of the RGB three channels; Represents the expanded fourth-order deep tensor with dimension [L, W, 1, T]; Represents the final denoised fourth-order tensor with dimensions [L,W,4,T], where "4" represents the combination of RGB three channels plus the depth channel; cat(3,·): indicates concatenation operation on the third dimension (channel dimension).

Images