Intelligent sorting device reconstruction method based on diffusion model

Abstract

The present invention relates to an intelligent sorting device reconstruction method based on a diffusion model. The technical features thereof lie in: inputting into a latent diffusion model a reference image from an object observation viewpoint, target viewpoint images, a viewpoint difference value and text features; adding noise to a plurality of target viewpoint images, and then performing transformation to obtain a joint feature; performing local interpolation to obtain features at different levels from a specific camera viewpoint; fusing the features into a latent diffusion model network by means of a mip-attention module, using the latent diffusion model network to perform denoising, and performing reconstruction by means of a neural implicit surface; and adding a normal-line function to a signed distance function, and optimizing the neural implicit surface by adding high-frequency information. In the present invention, spatial feature information is extracted and constructed, a diffusion process is guided, and a surface is then continuously optimized, so that a more refined surface is reconstructed, thereby realizing a three-dimensional reconstruction function for high-quality industrial devices; and the present invention has the characteristics of high efficiency, fast speed, high convenience, etc.

Description

Intelligent sorting equipment reconstruction method based on diffusion model TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional reconstruction of industrial equipment, and in particular to an intelligent sorting equipment reconstruction method based on a diffusion model. BACKGROUND

[0002] Three-dimensional reconstruction of industrial equipment is an important research direction in the field of computer graphics and computer vision, mainly applied to restoring the three-dimensional structure and shape of industrial equipment from a single two-dimensional image. Three-dimensional content is an important topic in the field of computer graphics and computer vision. In the past few decades, manual creation has dominated in real scenes, which requires tedious professional modeling. Now creators rely on special software tools (such as Blender, Maya3D, 3DS Max, etc.) and time-consuming manual adjustments to realize imagination and convert it into virtual objects. In recent years, the industry is developing towards intelligence, and digital twin technology is an important means to realize intelligent manufacturing, and three-dimensional reconstruction has broad prospects and applications.

[0003] Three-dimensional reconstruction technology is mainly divided into three categories: multi-view reconstruction, intra-domain single-view reconstruction, and zero-sample single-view reconstruction. Multi-view reconstruction requires different camera positions and poses to take multiple images of the same object, and the information between multiple views is associated for reconstruction. Intra-domain single-view reconstruction requires prior information of a single image and the geometry of the object to infer the three-dimensional structure of the object. Zero-sample single-view reconstruction requires the fusion of a single image and a multi-modal network for inference, thereby achieving the goal of zero-sample three-dimensional reconstruction. Among them, the representative is the application of contrastive text-to-image pre-training model (Contrastive Language-Image Pre-training, CLIP for short), which uses CLIP to guide three-dimensional generation and operation according to text prompts. Reconstructing a three-dimensional object from a single perspective image requires strong prior knowledge to infer information at different camera positions and poses, and it is challenging to infer information about different directions of the object from a single image. Neural rendering is usually used to reconstruct the implicit representation of the object. General neural surface reconstruction methods are difficult to learn high-frequency geometric details of industrial equipment, so the reconstructed shape is often too smooth. Position encoding and other methods are usually used to learn high-frequency information from the original input, but such methods require additional three-dimensional supervision, increasing data costs and limiting generalization performance. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art, provide a diffusion model-based intelligent sorting equipment reconstruction method, which can learn features and infer information in two-dimensional data using deep learning methods, establish an industrial equipment three-dimensional model, and realize the three-dimensional reconstruction function of the industrial equipment structure, solving the problems of insufficient prior information and insufficient reconstruction of surface details in the existing reconstruction of industrial equipment.

[0005] The present application solves the existing technical problems by adopting the following technical solutions:

[0006] A diffusion model-based intelligent sorting equipment reconstruction method, comprising the following steps:

[0007] Step 1, input the reference image of the object observation view, the target view image, the view angle difference and the text feature into the latent diffusion model;

[0008] Step 2, combine the reference image and the target view image input in step 1, and use a hybrid adjustment mechanism to combine the difference between the observation view and the target view with the CLIP feature through the text attention layer of the diffusion model, and input into the latent diffusion model network;

[0009] Step 3, add noise to the multiple target view images and transform to obtain the joint feature f;

[0010] Step 4, transform a three-dimensional convolutional neural network into a spatial feature F, and perform local interpolation on the spatial feature F to obtain features at different levels under a specific camera view;

[0011] Step 5, fuse the features obtained in step 4 into the latent diffusion model network through the mip-attention module, thereby fusing features at different levels, using the latent diffusion model network for denoising to obtain the image after removing noise, and finally the output of the diffusion model network is the target view image after denoising;

[0012] Step 6, reconstruct the different view images output by steps 1 to 5 through the method of neural implicit surface, embed the signed distance function into the equation of volume rendering, and model the relationship between distance and transparency;

[0013] Step 7, add a normal function to the signed distance function to optimize the neural implicit surface by adding high-frequency information.

[0014] Further, the implementation method of step 1 is: first, input the reference image of the object observation view As input, where H represents the height of the input image, W represents the width of the input image, and the channel number of the input image is 3; then input the target view image X={x1,x2...x nThe input after adding noise, the size of the target view image is the same as the reference image of the object observation view, and the camera view difference between the observation direction and the target direction is D; the text feature is extracted by inputting the reference image of the object observation view into the CLIP pre-training model.

[0015] Further, the step 2 is implemented by a camera transformation, which is represented as: Wherein And are the view-dependent camera rotation and translation, respectively, and the camera view difference between the observation direction and the target direction is D; in the implementation process, first, the CLIP text feature is embedded into the input image with spliced view information, and the text attention is used to encode the view difference and the input image text feature; then, the input image is connected with the image being denoised in the channel, so that the model retains the details of the object being synthesized.

[0016] Further, the implementation method of the step 3 is: constructing a joint feature for the input noise image, the input noise image includes B, N, 4, H, W, wherein B refers to batch size, N refers to image number, 4 refers to channel number, H refers to image height, and W refers to image width; extract two-dimensional picture feature f a , and embed time feature t and view feature v to obtain joint feature f b =c(f a ,t,v), convolve and up-sample the joint feature, combine the feature maps of each level with the up-sampled feature maps to obtain joint feature f, and the final scale of the joint feature f includes B, C, V 3 , wherein B refers to batch size, C refers to channel number, and V 3 refers to spatial vertex.

[0017] Further, the implementation method of the step 4 is: the joint feature f obtained in the step 3 is sequentially subjected to a three-dimensional convolution layer, a normalization layer, a SiLU activation function, and a three-dimensional transposed convolution layer to obtain features with different widths, and through feature fusion operation, spatial features F with different depths and scales are obtained.

[0018] Further, the implementation method of the step 5 is: the mip-attention module uses discrete wavelet transform to decompose the feature, concatenates the extracted high-frequency component h and low-frequency component l in the channel, and obtains the feature fused at different levels through the normalization layer and the ReLu activation function.

[0019] Further, the step 5 uses a noise predictor for denoising, and the noise predictor is represented as: ε θ (x t ;D,t,z t ), wherein xt denotes the current time target view image, D denotes the camera view difference value of the observation direction and the target direction, t denotes the current time, z t denotes the current time target noise image.

[0020] Further, the implementation method of the step 6 is: taking a group of target object images I = {I1, I2...I N} as input, obtaining the final reconstructed object V, using a volume rendering method to calculate the final color of each pixel by using the color c and the volume density σ of the sampling points along the ray r, encoding the opacity as a signed distance field, and making the three-dimensional surface S as a hidden function representation; the volume rendering method adopts the following formula: Wherein, P denotes the number of sampling points along the ray, T denotes the opacity, and d denotes the ray direction.

[0021] Further, the implementation method of the step 7 is: adding a normal function to the signed distance function to increase high-frequency information, and the signed distance function is represented as: f(p) = f(p + λ·f n ′(p)n), wherein f n ′ denotes the normal function, and the normal function f n ′ represents that the point p is mapped to the point p' along the surface normal n, p denotes a point on the surface, and n denotes the normal of the surface at the point; the distance between the two points is represented as a dynamic parameter λ, and the dynamic parameter is represented as: λ = |d p -d p ′|.

[0022] The advantages and positive effects of the present application are:

[0023] 1. The present application extracts and constructs spatial feature information, guides the diffusion process, and then continuously optimizes the surface, thereby reconstructing a more detailed surface, realizing the three-dimensional reconstruction function of high-quality industrial equipment, and having the characteristics of high efficiency, quickness and convenience.

[0024] 2. The present application uses two-dimensional images of industrial equipment as input, without other sensors or equipment, and the implementation method is more convenient, and compared with the traditional method of scanning objects, the present application is more efficient in obtaining two-dimensional image data and generation mode.

[0025] 3. The present application fully utilizes spatial features, and a three-dimensional model of industrial equipment can be reconstructed by using single image information, effectively overcoming the problems of too smooth surface and insufficient high-frequency geometric detail information in the reconstruction process of industrial equipment.

[0026] 4. The present application fully utilizes three-dimensional spatial features to guide the diffusion process under the condition of insufficient prior knowledge of industrial equipment, thereby improving the generation quality.

[0027] 5、The application uses a normal function to optimize the surface in the industrial equipment reconstruction process, and increases the geometric details of the object surface reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0028] Fig. 1 is a flowchart of the intelligent sorting equipment reconstruction method of the application;

[0029] Fig. 2 is a network structure diagram of the intelligent sorting equipment reconstruction method of the application;

[0030] Fig. 3 is a mip-attention structure diagram of the intelligent sorting equipment reconstruction method of the application;

[0031] Fig. 4 is a normal function optimization structure diagram of the intelligent sorting equipment reconstruction method of the application. DETAILED DESCRIPTION

[0032] The embodiments of the application are further described in detail below with reference to the accompanying drawings.

[0033] The application proposes an intelligent sorting equipment reconstruction method based on a diffusion model, as shown in Fig. 1, including the following steps:

[0034] Step 1, input the reference image of the observation view, the target view image, the view angle difference and the text feature into the latent diffusion model, and input the information according to step 2.

[0035] In this step, an RGB image of an object observation view is input as follows: wherein H refers to the height of the input image, W refers to the width of the input image, and the channel number of the input image is 3. The target view image X={x1, x2...x n} is input after adding noise, and the size is the same as that of the observation view image. The camera view angle difference between the observation direction and the target direction is D. The text feature is obtained by inputting the reference image of the object observation view into the CLIP pre-training model.

[0036] Step 2, combine the reference image and the target view image, and use a hybrid adjustment mechanism to combine the difference between the observation view and the target view and the CLIP feature through the text attention of the diffusion model, and input the whole into the latent diffusion model network.

[0037] In this step, the camera transformation can be represented as: wherein and Respective camera rotation and translation of the view angle. The camera view angle difference between the observation direction and the target direction is D. First, the CLIP text feature is embedded into the input image spliced with the view angle information, and the text attention is used to encode the view angle difference and the input image text feature. Then, the input image is channel-connected with the image being denoised, so that the model retains the details of the object being synthesized.

[0038] Step 3, transform the multiple target view angle images after adding noise to obtain the joint feature f.

[0039] In this step, the joint feature is constructed for the input noise image. The input feature image is (B, N, 4, H, W), where B refers to the batch size, N refers to the number of images, 4 refers to the number of channels, H refers to the height of the image, and W refers to the width of the image.

[0040] Extract the two-dimensional picture feature f a At the same time, the time feature t and the view angle feature v are embedded to obtain the joint feature f b = c(f a , t, v), the joint feature is convolved and up-sampled, and the feature maps of each level are combined with the up-sampled feature maps to obtain the joint feature f with a final scale of (B, C, V 3 ), where B refers to the batch size, C refers to the number of channels, and V 3 refers to the spatial vertex.

[0041] Step 4, transform a three-dimensional convolutional neural network into a spatial feature F, and perform local interpolation on the spatial feature F to obtain features at different levels under a specific camera view angle.

[0042] In this step, the joint feature f obtained in step 3 is sequentially subjected to a three-dimensional convolutional layer, a normalization layer, a SiLU activation function, and a three-dimensional transposed convolutional layer to obtain features with different widths. Through feature fusion operations, spatial features F with different depths and scales are obtained.

[0043] Step 5, fuse the features obtained in step 4 into the latent diffusion model network through the mip-attention module, thereby fusing features at different levels. The latent diffusion model network is used for denoising to obtain an image after removing noise. The final network output is the denoised target view angle image.

[0044] In this step, the spatial features F obtained in step 4 are extracted through the mip-attention module and fused into the diffusion model network. Among them, the mip-attention module uses discrete wavelet transform to decompose the features, and the high-frequency components h and low-frequency components l extracted are channel spliced, and then normalized layer and ReLu activation function are used to obtain discrete frequency features. The mip-attention module adopts multi-head attention, calculates the size of the output dimension of each linear transformation layer according to the dimension size of each attention head, and then performs linear transformation on the feature dimension and the number of attention heads to obtain the matrix q, and performs linear transformation on the depth information, feature dimension and number of attention heads to obtain the matrix k, v. After multiplying q and k, the matrix is multiplied and scaled and normalized, and the result is multiplied by v to obtain the intermediate feature, that is, Attn(q, k, v) = softmax(MatMul(q T k))v. The intermediate features f = {f0, f1…f n} obtained at each different depth and dimension are respectively passed down to the lower layer after being spliced according to the different spatial feature F dimensions and channel numbers to obtain the final result, which is input into the latent diffusion model for guidance. The noise predictor can be represented as: ε θ (x t ;D,t,z t ). Wherein, x t refers to the current time target view image, D refers to the camera view angle difference value of the observation direction and the target direction, t refers to the current time, and z t refers to the current time target noise image.

[0045] Step 6, the different view image set output by steps 1 to 5 is reconstructed by the method of neural implicit surface, the signed distance function is embedded into the equation of volume rendering, and the relationship between distance and transparency is modeled.

[0046] In this step, a set of target object images I = {I1, I2…I N} are input to obtain the final reconstructed object V, wherein the volume rendering method is used to calculate the final color of each pixel according to the color c and volume density σ of the sampling points along the ray r, and the volume rendering formula is: Where P refers to the number of sampling points along the ray, T refers to the opacity, and d refers to the ray direction. The opacity is encoded as a signed distance field, so that the three-dimensional surface S is represented as an implicit function.

[0047] Step 7, add the normal function to the signed distance function to optimize the neural implicit surface by adding high-frequency information.

[0048] In this step, high frequency information is added by adding a normal function to the signed distance function, which is represented as: f(p) = f(p + λ·f n ′(p)n), where f n ′ refers to the normal function, the normal function f n ′ represents mapping the point p along the surface normal n to the point p', p refers to a point on the surface, n refers to the normal of the surface at the point, and f(p) = 0 represents that the point is on the surface. Using f n , it represents mapping the point p' along the reverse direction of the surface normal to the point p, then f n ′(p) = f n (p′). The distance between the two points is taken as a dynamic parameter to adjust the surface in a finer granularity, so that the surface detail information is more sufficient, and the dynamic parameter can be represented as λ = |d p -d p ′|, from the above formula, it can be seen that the dynamic parameter is adjusted by the distance between the point p and the point p'.

[0049] The network is trained using the color loss and the signed distance function, the difference between the rendered color and the real color of the object is minimized through the color loss function, and the color loss is represented as: Where C i is the pixel color, is the generated object color, and R is the L1 loss. The signed distance function is regularized by the potential term, and the potential loss is represented as: The weights are added to the two loss functions respectively, α is the inverse of the number of training data, β is the inverse of the product of the sampling point size and the number of training data, and the specific loss is as follows: L = αl color +βl eikonal .

[0050] Through the above steps, the three-dimensional reconstruction function of the industrial equipment structure based on the diffusion model is realized.

[0051] The unmentioned part of the present application is applicable to the prior art.

[0052] It should be emphasized that the embodiments described in the present application are illustrative rather than limiting, and therefore the present application includes but is not limited to the embodiments described in the specific embodiments, and any other embodiments derived by those skilled in the art according to the technical solutions of the present application also belong to the scope of protection of the present application.

Claims

1. A method for reconstruction of an intelligent sorting device based on a diffusion model, characterized in that: The method comprises the following steps: Step 1, input the reference image of the object observation view angle, the target view angle image, the view angle difference and the text feature into the latent diffusion model; Step 2, combine the reference image and the target view angle image input in step 1, and use a hybrid adjustment mechanism to combine the difference between the observation view angle and the target view angle with the CLIP feature through the text attention layer of the diffusion model, and input into the latent diffusion model network; Step 3, add noise to the multiple target view angle images and transform to obtain the joint feature f; Step 4, transform a three-dimensional convolutional neural network into a spatial feature F, and perform local interpolation on the spatial feature F to obtain features of different levels under a specific camera view angle; Step 5, fuse the features obtained in step 4 into the latent diffusion model network through the mip-attention module, thereby fusing features of different levels, using the latent diffusion model network for denoising to obtain the image after removing the noise, and finally the output of the diffusion model network is the target view angle image after denoising; Step 6, reconstruct the different view angle image set output by steps 1 to 5 through the neural implicit surface method, embed the signed distance function into the equation of volume rendering, and model the relationship between distance and transparency; Step 7, add the normal function to the signed distance function to optimize the neural implicit surface by adding high-frequency information.

2. The method of claim 1, wherein: The implementation method of the step 1 is: first, input the reference image of the object observation view angle as input, wherein H represents the height of the input image, W represents the width of the input image, and the channel number of the input image is 3; then input the target view angle image X = {x1, x2...x n added with noise, the size of the target view angle image is the same as that of the reference image of the object observation view angle, the camera view angle difference between the observation direction and the target direction is D; and the text feature is extracted by inputting the reference image of the object observation view angle into a CLIP pre-training model.

3. The method of claim 1, wherein: The step 2 is implemented by a camera transformation, which is represented as: wherein and are the camera rotation and translation associated with the view angle, and the camera view angle difference between the observation direction and the target direction is D; in the implementation process, first, the CLIP text feature is embedded into the input image spliced with the view angle information, and the text attention is used to encode the view angle difference and the input image text feature; Then, the input image is connected with the image being denoised in the channel to make the model retain the details of the object being synthesized.

4. The method of claim 1, wherein: The implementation method of step 3 is: joint feature construction is performed on the input noise image, and the input noise image includes B, N, 4, H and W, wherein B represents batch size, N represents image quantity, 4 represents channel number, H represents image height, and W represents image width; two-dimensional picture features f a At the same time, the time feature t and the view feature v are embedded to obtain the joint feature f b a c(f a ,t,v), convolution and up-sampling are performed on the joint feature, the feature maps of various levels are combined with the up-sampled feature maps to obtain the joint feature f, and the final scale of the joint feature f includes B, C and V 3 , wherein B represents batch size, C represents channel number, and V 3 represents a spatial vertex.

5. The method of claim 1, wherein: The implementation method of step 4 is that the joint feature f obtained in step 3 is sequentially subjected to a three-dimensional convolution layer, a normalization layer, a SiLU activation function, and a three-dimensional transpose convolution layer to obtain features of different widths, and through feature fusion operation, spatial features F of different depths and scales are obtained.

6. The method of claim 1, wherein: The implementation method of step 5 is that the mip-attention module uses discrete wavelet transform to decompose the features, channels are spliced for the extracted high-frequency component h and low-frequency component l, and a normalization layer and a ReLu activation function are used to obtain features fused at different levels.

7. The method of claim 1, wherein: The step 5 uses a noise predictor for denoising, which is expressed as: ε θ (x t ;D,t,z t ), where x t denotes the current time target view image, D denotes the camera view difference between the observation direction and the target direction, t denotes the current time, and z t denotes the current time target noise image.

8. The method of claim 1, wherein: The implementation method of the step 6 is: taking a group of target object images I = {I1, I2, …, I N} as input, obtaining a final reconstructed object V, using a volume rendering method to calculate the final color of each pixel by using the color c and the volume density σ of the sampling points along the ray r, encoding the opacity as a signed distance field, and making the three-dimensional surface S as a hidden function representation; the volume rendering method adopts the following formula: Wherein, P refers to the number of sampling points along the ray, T refers to the opacity, and d refers to the ray direction.

9. The method of claim 8, wherein: The implementation method of the step 7 is: adding a normal function to a signed distance function to increase high-frequency information, the signed distance function is expressed as: f(p) = f(p + λ·f n ′(p)n), wherein f n ′ indicates the normal function, the normal function f n ′ represents that the point p is mapped to the point p' along the surface normal n, p indicates a point on the surface, and n indicates the normal of the surface at the point; the distance between two points is represented by a dynamic parameter λ, which is expressed as: λ = |d p -d p′ |

Images