Method and system for reconstructing and monomerizing three-dimensional target

Abstract

The invention relates to a method and a system for reconstructing and monomerizing a three-dimensional target. The method comprises the following steps of: shooting an image and a video by using an unmanned aerial vehicle; using a coomap tool to extract pose information and shooting information; performing first segmentation by using a SegFormer model to obtain an original target image and a corresponding original mask; processing the original target image to generate a target image; performing image similarity measurement by using a twin neural network to obtain a matching similarity and a pose overlapping degree; carrying out weighted calculation, screening out target images belonging to the same type, and carrying out second segmentation to obtain a three-dimensional target image and a corresponding target mask; rendering by using three-dimensional Gaussian splash to obtain a rendering result, and initializing a label; differentiated matching is carried out on the rendering result and the corresponding target mask, specifically, a label distribution optimization problem is constructed and solved, and an optimal label distribution scheme is obtained; and performing reconstruction and monomerization based on the optimal label distribution scheme.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional modeling technology, and mainly to a method and system for reconstructing and singulating three-dimensional objects. Background Art

[0002] Understanding and interacting with three-dimensional environments poses a formidable and crucial technical challenge within the field of computer vision. A core goal of research in this field focuses on accurately perceiving and segmenting three-dimensional structures. As we continue to explore more complex 3D scene representations, achieving this goal becomes increasingly challenging, with the task significantly increasing in complexity as we strive to maintain accuracy while minimizing time.

[0003] While existing 3D object segmentation methods have demonstrated impressive performance, they suffer from high algorithmic complexity and relatively long processing times. Among them, ISRF (Iterative Self-Refinement for Fine-Grained 3D Object Segmentation) uses iterative self-refinement to gradually optimize the 3D object segmentation results to achieve higher segmentation accuracy. However, this iterative process requires multiple calculations and updates, increasing the algorithm's complexity and processing time. The computational burden of ISRF is particularly heavy when processing large-scale 3D data, making real-time performance a major challenge. SA3D (Sparse Anisotropic 3D Convolutional Neural Networks) utilizes sparse, asymmetric 3D convolutional neural networks to process 3D data. This approach offers advantages in capturing the spatial features of 3D data. However, due to the high computational complexity of 3D convolutional neural networks, especially when the network structure is complex, the processing time of the SA3D algorithm increases significantly, making it unsuitable for applications requiring fast response.

[0004] For example, the Chinese invention patent with the publication number "CN116681839A" discloses a "Real Scene 3D Object Reconstruction and Singulation Method Based on Improved NeRF", which specifically discloses "S10, obtain multiple 2D images in the scene; S20, use the SAM full segmentation model to hierarchically segment the 2D image; S30, input the segmented object into the twin neural network to complete the object image pairing; S40, perform image optimization based on the image feature similarity and posture overlap relationship; S50, use the NeRF neural network as the basic model, construct a real scene 3D object reconstruction model Sen-NeRF based on improved NeRF, obtain the segmentation map of the target object at different angles as a label, and combine the basic information of the original image , rendering is performed through the real-scene 3D target reconstruction model Sen-NeRF; S60, intercepting the mesh model scene target to complete the singulation of the real-scene 3D target reconstruction model Sen-NeRF", but this method uses the SAM full segmentation model for hierarchical segmentation, which may have limitations in the segmentation effect of some complex scenes or specific targets and cannot accurately segment; in addition, during the singulation process, this method intercepts the mesh model scene target to complete the singulation, and cannot fully consider the complex relationship and spatial position between the targets, resulting in limited accuracy and effect of the singulation; at the same time, this method uses multiple models for combination, which will increase the complexity of the entire solution, and has high requirements on computing resources and technical capabilities, and is difficult to implement and debug. Summary of the Invention

[0005] In order to solve the above-mentioned problems existing in the prior art, the present application provides a method and system for reconstructing and singulating a three-dimensional object.

[0006] The technical solution of this application is as follows:

[0007] In one aspect, the present invention provides a method for reconstructing and singulating a three-dimensional object, the method comprising:

[0008] Using a drone to shoot images and videos of a preset scene, wherein the FFmpeg tool is used to extract frames from the video to obtain video images; using the colmap tool to extract posture information and shooting information of the video images, and obtaining an image set based on the shot images and video images;

[0009] Performing a first segmentation on the image set using the SegFormer model to obtain an original target image and a corresponding original mask; processing the original target image to generate a preset number of target images, where each target image contains only one candidate 3D target;

[0010] Using a twin neural network 2-channels network, the target image is measured for image similarity to obtain matching similarity and pose overlap; performing a weighted calculation on the matching similarity and pose overlap to obtain a final similarity score; comparing the final similarity score with a preset similarity threshold to filter out target images of the same type; performing a second segmentation on the target image to obtain a three-dimensional target image and a corresponding target mask;

[0011] Rendering a 3D target image using 3D Gaussian splatting yields a rendering result, which is then initialized with labels. Differential matching is performed between the rendering result and the corresponding target mask, specifically constructing a label assignment optimization problem, including an objective function that minimizes differential matching and a constraint that the labels must be within a preset range. Solving the label assignment optimization problem yields an optimal label assignment solution.

[0012] The three-dimensional object in the three-dimensional object image is reconstructed and singulated based on the optimal label assignment scheme.

[0013] Preferably, the colmap tool is used to extract the pose information and shooting information of the video image, and the pose information is expressed as (x k ,y k ,z k ,α k ,γ k ), where x k Indicates the X-axis coordinate of the k-th video image, y k Indicates the Y-axis coordinate of the k-th video image, z k Indicates the Z-axis coordinate of the kth video image, α k represents the angle between the center point of the kth video image and the X axis, γ k represents the angle between the center point of the k-th video image and the Z axis; k represents the index value of the k-th video image;

[0014] The shooting information includes the shooting position and shooting angle of the video image.

[0015] Preferably, the SegFormer model is used to perform the first segmentation of the image set, specifically:

[0016] The Transformer encoder is used to extract the multi-scale features of the image, which can be expressed as follows:

[0017]

[0018] Where, represents the output features of the i-th image set of the l-th layer Transformer decoding layer; l represents the index value of the l-th layer Transformer decoding layer; L represents the total number of Transformer decoding layers; Transformer represents the Transformer decoding layer; i represents the index value of the i-th image set;

[0019] The output features are fused based on the lightweight multi-layer perceptron decoder, which can be expressed as follows:

[0020]

[0021] Where, represents the fusion feature map of the i-th image set; MLP represents the multi-layer perceptron decoder; Upsample represents the upsampling function; Indicates channel splicing;

[0022] The target image and the corresponding original mask are obtained, which can be expressed as follows:

[0023]

[0024] Where M j (a, b) represents the pixel value of the original mask of the j-th target image in the a-th row and b-th column, where the pixel value of 1 represents the target area and the pixel value of 0 represents the background area; M j represents the original mask of the j-th target image; a represents the row index of the a-th row; b represents the column index of the b-th column; A represents the number of rows of pixel values; B represents the number of columns of pixel values; σ represents the activation function; represents the pixel value of the fusion feature map of the i-th image set in the a-th row and b-th column; j represents the index value of the j-th target image; ξ represents the preset segmentation threshold;

[0025] Processing the original target image, specifically removing and filling the background, to generate a preset number of target images, where each target image contains only one candidate three-dimensional target;

[0026] The background removal is expressed as follows:

[0027] E j (a,b)=Y j (a,b)⊙M j (a,b);

[0028] Where, E j represents the pixel value of the jth target image in row a and column b after background removal, where E j (a,b)=1 represents the target area, E j (a, b) = 0 represents the background area; Yj (a,b) represents the pixel value of the jth target image at row a and column b; ⊙ represents element-by-element multiplication;

[0029] The background filling specifically retains the pixel values of the target area and fills the pixel values of the background area, which can be expressed as follows:

[0030] O j (a,b)=E j (a,b)⊙M j (a,b)+U⊙M j (a,b);

[0031] Where, O j (a, b) represents the pixel value of the jth target image in the ath row and the bth column after background filling; U represents the background filling constant.

[0032] Preferably, the image similarity measurement of the target image is performed using a twin neural network 2-channels network, specifically:

[0033] The image similarity measurement specifically comprises the following steps: the twin neural network 2-channels network receives an input target image; selects a preset number of target images whose current shooting information is within a preset distance range for matching; each matching step specifically comprises extracting a pair of candidate target images from the selected target image, and calculating the similarity and overlap between the candidate target images;

[0034] The extraction process is repeated until all the selected preset number of target images are similarly matched to obtain the matching similarity and pose overlap;

[0035] The matching similarity is expressed as follows:

[0036]

[0037] Where Loss represents the loss function of the twin neural network 2-channels network; ω represents the preset network weight; λ represents the preset weight decay rate; max represents the maximum value function; y q represents the matching label of the qth pair of candidate target images; o q represents the matching similarity of the qth pair of candidate target images; q represents the index value of the qth pair of candidate target images; Q represents the number of matching pairs of extracted candidate target images; n represents the index value of the nth candidate target image; m represents the index value of the mth candidate target image;

[0038] The posture overlap is expressed as follows:

[0039]

[0040] Where, Pos q represents the degree of pose overlap; exp represents the exponential function; α n represents the angle between the nth candidate target image and the X axis; α m represents the angle between the mth candidate target image and the X axis; γ n represents the angle between the mth candidate target image and the Y axis; γ m Represents the angle between the mth candidate target image and the Y axis.

[0041] Preferably, the matching similarity and the pose overlap are weighted to obtain a final similarity score, which can be expressed as:

[0042]

[0043] Where S q represents the final similarity score of the qth pair of candidate target images; Indicates the preset weighted weight;

[0044] Compare the final similarity score with the preset similarity threshold and filter out target images of the same type. q >S thr , it means that the two-dimensional target in the nth candidate target image and the mth candidate target image is the same target, that is, y q =1; otherwise, it means that the two-dimensional target in the nth candidate target image and the mth candidate target image are not the same target, that is, y q =-1, where S thr Indicates the preset similarity threshold;

[0045] The target image is segmented for the second time to obtain a three-dimensional target image and a corresponding target mask.

[0046] Preferably, the three-dimensional target image is rendered using three-dimensional Gaussian splashing to obtain a rendering result, and label initialization is performed, specifically as follows:

[0047] Initialize the three-dimensional Gaussian points as G g ={x g ,y g ,z g ,c g ,μ g ,η g}, where G g represents the g-th three-dimensional Gaussian point, g represents the index value of the g-th three-dimensional Gaussian point, x g Indicates the X-axis coordinate of the g-th three-dimensional Gaussian point, y g Indicates the Y-axis coordinate of the g-th three-dimensional Gaussian point, z grepresents the Z-axis coordinate of the g-th three-dimensional Gaussian point, c g Indicates the color of the g-th three-dimensional Gaussian point, μ g Indicates the rendering degree of the g-th three-dimensional Gaussian point, η g represents the transmittance of the g-th three-dimensional Gaussian point;

[0048] Projecting the three-dimensional Gaussian points into two-dimensional space can be expressed as:

[0049]

[0050] Where a g Represents the row index of the pixel value after projection of the g-th three-dimensional Gaussian point; b g Represents the column index of the pixel value after projection of the g-th three-dimensional Gaussian point; K represents the external parameter matrix of the preset drone shooting angle; R represents the internal parameter matrix of the preset drone shooting angle;

[0051] Render the three-dimensional target image and obtain the rendering result, which is expressed as follows:

[0052]

[0053] X={X1,...,X z ,...,X Z};

[0054] Where, X z represents the rendering result of the z-th 3D target image; X z (a g ,b g ) represents the rendering result of the zth three-dimensional target image in the a g Row b g The pixel value of the column; X represents the rendering result; X Z represents the rendering result of the Z-th 3D target image; z represents the index value of the z-th 3D target image; Z represents the number of 3D target images;

[0055] And perform label initialization specifically to initialize the labels of the rendering results of all three-dimensional target images as H z =0, where H z Represents the original label of the z-th 3D target image, where H z =0 indicates that the candidate 3D target in the rendering result of the z-th 3D target image does not participate in the reconstruction and singulation.

[0056] Preferably, the rendering result and the corresponding target mask are differentially matched, specifically:

[0057] The label assignment optimization problem is constructed, including the objective function with the goal of minimizing differential matching, which is expressed as follows:

[0058]

[0059] In the formula, I() represents the indicator function, where I(,0) represents the background area and I(,1) represents the target area;

[0060] The constraint condition that the label is within the preset range is expressed as follows:

[0061]

[0062] Where GS represents the number of three-dimensional Gaussian points; st represents the constraint condition;

[0063] To solve the label allocation optimization problem, the iterative solution process is as follows: Then H z =0; if Then H z =1, where H z = 1 means that the candidate 3D target in the rendering result of the zth 3D target image participates in the reconstruction and monomerization; until the maximum number of iterations is reached or the label assignment optimization problem converges, the iteration is stopped and the optimal label assignment solution {H1,...,H z ,....,H Z}.

[0064] On the other hand, the present invention also provides a three-dimensional object reconstruction and singulation system, the system comprising an image acquisition module, an image processing module, an image screening module, a label assignment module, and a singulation module, wherein:

[0065] The image acquisition module is used to use a drone to shoot images and videos of a preset scene, wherein the FFmpeg tool is used to extract frames from the video to obtain video images; the colmap tool is used to extract the pose information and shooting information of the video image, and an image set is obtained based on the shot image and the video image; and the image set is transmitted to the target screening module;

[0066] The image processing module is used to perform a first segmentation on the image set using the SegFormer model to obtain an original target image and a corresponding original mask; the original target image is processed to generate a preset number of target images, where each target image contains only one candidate three-dimensional target;

[0067] The image screening module is used to measure the image similarity of the target image using a twin neural network 2-channels network to obtain matching similarity and pose overlap; perform weighted calculation on the matching similarity and pose overlap to obtain a final similarity score; compare the final similarity score with a preset similarity threshold to screen out target images of the same type; perform a second segmentation on the target image to obtain a three-dimensional target image and a corresponding target mask;

[0068] The label assignment module is configured to render a three-dimensional target image using three-dimensional Gaussian splatting to obtain a rendering result and initialize labels; perform differential matching between the rendering result and the corresponding target mask, specifically constructing a label assignment optimization problem, including an objective function with the goal of minimizing differential matching and a constraint that the labels must be within a preset range; and solve the label assignment optimization problem to obtain an optimal label assignment solution.

[0069] The singulation module reconstructs and singulates the three-dimensional object in the three-dimensional object image based on the optimal label assignment scheme.

[0070] On the other hand, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, it implements a three-dimensional target reconstruction and singulation method as described in any embodiment of the present invention.

[0071] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a three-dimensional object reconstruction and singulation method as described in any embodiment of the present invention.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] 1) The present invention provides a method and system for reconstructing and singulating three-dimensional objects. The colmap tool is used to extract pose and shooting information from video images. The SegFormer model is used to perform the first segmentation of the image set. This covers the extraction of key information from both pose and image segmentation, improving the comprehensiveness of image preprocessing.

[0074] 2) The present invention provides a method and system for reconstructing and singulating three-dimensional objects. This system uses a twin neural network (2-channels network) to measure the similarity of target images and performs a weighted calculation to derive a final similarity score. This score is then compared with a preset similarity threshold to filter images. This improves the accuracy of image screening and enables more precise selection of target images of the same type.

[0075] 3) The present invention provides a method and system for reconstructing and singulating three-dimensional targets, which uses three-dimensional Gaussian splattering to render three-dimensional target images and constructs an integer programming problem for label assignment, thereby enhancing the scientific nature of the singulation process and enabling it to more reasonably and accurately complete the three-dimensional target singulation work in complex and changing scenarios, thereby improving the quality and reliability of the singulation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0078] The present invention provides the following technical solution: a method and system for reconstructing and singulating a three-dimensional object.

[0079] Example 1

[0080] See Figure 1 This embodiment provides a method for reconstructing and singulating a three-dimensional object, and the specific steps include:

[0081] S1. Using a drone to shoot images and videos of a preset scene, wherein FFmpeg tool is used to extract frames from the video to obtain video images;

[0082] In this embodiment, the drone shooting method is to shoot in a circular direction vertically from the ground to the sky;

[0083] S2, using colmap tool to extract the posture information and shooting information of the video image; the posture information is expressed as (x k ,y k ,z k ,α k ,γ k ), where x k Indicates the X-axis coordinate of the k-th video image, y k Indicates the Y-axis coordinate of the k-th video image, z k Indicates the Z-axis coordinate of the kth video image, α k represents the angle between the center point of the kth video image and the X axis, γ k represents the angle between the center point of the k-th video image and the Z axis; k represents the index value of the k-th video image;

[0084] The shooting information includes the shooting position and shooting angle of the video image;

[0085] S3, obtaining an image set based on the photographed images and video images;

[0086] S4. Use the SegFormer model to perform the first segmentation of the image set and use the Transformer encoder to extract the multi-scale features of the image, which can be expressed as:

[0087]

[0088] Where, represents the output features of the i-th image set of the l-th layer Transformer decoding layer; l represents the index value of the l-th layer Transformer decoding layer; L represents the total number of Transformer decoding layers; Transformer represents the Transformer decoding layer; i represents the index value of the i-th image set;

[0089] The output features are fused based on the lightweight multi-layer perceptron decoder, which can be expressed as follows:

[0090]

[0091] Where, represents the fusion feature map of the i-th image set; MLP represents the multi-layer perceptron decoder; Upsample represents the upsampling function; Indicates channel splicing;

[0092] The original target image and the corresponding original mask are obtained, which can be expressed as follows:

[0093]

[0094] Where M j (a, b) represents the pixel value of the original mask of the j-th target image in the a-th row and b-th column, where the pixel value of 1 represents the target area and the pixel value of 0 represents the background area; M j represents the original mask of the j-th target image; a represents the row index of the a-th row; b represents the column index of the b-th column; A represents the number of rows of pixel values; B represents the number of columns of pixel values; σ represents the activation function; represents the pixel value of the fusion feature map of the i-th image set in the a-th row and b-th column; j represents the index value of the j-th target image; ξ represents the preset segmentation threshold;

[0095] S5, processing the original target image, specifically removing and filling the background, to generate a preset number of target images, wherein each target image contains only one candidate 3D target;

[0096] The background removal is expressed as follows:

[0097] E j (a,b)=Y j (a,b)⊙M j (a,b);

[0098] Where, E j represents the pixel value of the jth target image in row a and column b after background removal, where E j (a,b)=1 represents the target area, E j (a, b) = 0 represents the background area; Y j (a,b) represents the pixel value of the jth target image at row a and column b; ⊙ represents element-by-element multiplication;

[0099] The background filling specifically retains the pixel values of the target area and fills the pixel values of the background area, which can be expressed as follows:

[0100] O j (a,b)=E j (a,b)⊙M j (a,b)+U⊙M j (a,b);

[0101] Where, O j (a, b) represents the pixel value of the jth target image in row a and column b after background filling; U represents the background filling constant;

[0102] S6. Using a twin neural network 2-channels network to measure the image similarity of the target image, and obtain matching similarity and pose overlap;

[0103] The image similarity measurement specifically comprises the following steps: the twin neural network 2-channels network receives an input target image; selects a preset number of target images whose current shooting information is within a preset distance range for matching; each matching step specifically comprises extracting a pair of candidate target images from the selected target image, and calculating the similarity and overlap between the candidate target images;

[0104] The extraction process is repeated until all the selected preset number of target images are similarly matched to obtain the matching similarity and pose overlap;

[0105] The matching similarity is expressed as follows:

[0106]

[0107] Where Loss represents the loss function of the twin neural network 2-channels network; ω represents the preset network weight; λ represents the preset weight decay rate; max represents the maximum value function; y q represents the matching label of the qth pair of candidate target images; o q represents the matching similarity of the qth pair of candidate target images; q represents the index value of the qth pair of candidate target images; Q represents the number of matching pairs of extracted candidate target images; n represents the index value of the nth candidate target image; m represents the index value of the mth candidate target image;

[0108] The posture overlap is expressed as follows:

[0109]

[0110] Where, Pos q represents the degree of pose overlap; exp represents the exponential function; α n represents the angle between the nth candidate target image and the X axis; α m represents the angle between the mth candidate target image and the X axis; γ n represents the angle between the mth candidate target image and the Y axis; γ m represents the angle between the mth candidate target image and the Y axis;

[0111] S7. Perform weighted calculation on the matching similarity and pose overlap to obtain the final similarity score, which is expressed as:

[0112]

[0113] Where S q represents the final similarity score of the qth pair of candidate target images; Indicates the preset weighted weight;

[0114] Compare the final similarity score with the preset similarity threshold and filter out target images of the same type. q >S thr , it means that the two-dimensional target in the nth candidate target image and the mth candidate target image is the same target, that is, y q =1; otherwise, it means that the two-dimensional target in the nth candidate target image and the mth candidate target image are not the same target, that is, y q =-1, where S thr Indicates the preset similarity threshold;

[0115] S8, performing a second segmentation on the target image to obtain a three-dimensional target image and a corresponding target mask;

[0116] S9, rendering the three-dimensional target image using three-dimensional Gaussian splashing to obtain a rendering result, and performing label initialization;

[0117] Initialize the three-dimensional Gaussian points as G g ={x g ,y g ,z g ,c g ,μ g ,η g}, where G g represents the g-th three-dimensional Gaussian point, g represents the index value of the g-th three-dimensional Gaussian point, x g Indicates the X-axis coordinate of the g-th three-dimensional Gaussian point, y g Indicates the Y-axis coordinate of the g-th three-dimensional Gaussian point, z g represents the Z-axis coordinate of the g-th three-dimensional Gaussian point, c g Indicates the color of the g-th three-dimensional Gaussian point, μ g Indicates the rendering degree of the g-th three-dimensional Gaussian point, η g represents the transmittance of the g-th three-dimensional Gaussian point;

[0118] Projecting the three-dimensional Gaussian points into two-dimensional space can be expressed as:

[0119]

[0120] Where a g Represents the row index of the pixel value after projection of the g-th three-dimensional Gaussian point; b g Represents the column index of the pixel value after projection of the g-th three-dimensional Gaussian point; K represents the external parameter matrix of the preset drone shooting angle; R represents the internal parameter matrix of the preset drone shooting angle;

[0121] Render the three-dimensional target image and obtain the rendering result, which is expressed as follows:

[0122]

[0123] X={X1,...,X z ,...,X Z};

[0124] Where, X z represents the rendering result of the z-th 3D target image; X z (a g ,b g ) represents the rendering result of the zth three-dimensional target image in the a g Row b g The pixel value of the column; X represents the rendering result; X Zrepresents the rendering result of the Z-th 3D target image; z represents the index value of the z-th 3D target image; Z represents the number of 3D target images;

[0125] And perform label initialization specifically to initialize the labels of the rendering results of all three-dimensional target images as H z =0, where H z Represents the original label of the z-th 3D target image, where H z = 0 means that the candidate 3D target in the rendering result of the z-th 3D target image does not participate in the reconstruction and singulation;

[0126] S10, performing differential matching on the rendering result and the corresponding target mask, specifically constructing a label assignment optimization problem, including an objective function with the goal of minimizing differential matching, which is expressed as follows:

[0127]

[0128] In the formula, I() represents the indicator function, where I(,0) represents the background area and I(,1) represents the target area;

[0129] The constraint condition that the label is within the preset range is expressed as follows:

[0130]

[0131] Where GS represents the number of three-dimensional Gaussian points; st represents the constraint condition;

[0132] S11, solve the label allocation optimization problem, the iterative solution process is specifically as follows: Then H z =0; if Then H z =1, where H z = 1 means that the candidate 3D target in the rendering result of the zth 3D target image participates in the reconstruction and monomerization; until the maximum number of iterations is reached or the label assignment optimization problem converges, the iteration is stopped and the optimal label assignment solution {H1,...,H z ,....,H Z};

[0133] S12: reconstructing and singulating the three-dimensional object in the three-dimensional object image based on the optimal label assignment scheme.

[0134] Example 2

[0135] This embodiment provides a three-dimensional object reconstruction and singulation system, which includes an image acquisition module, an image processing module, an image screening module, a label assignment module, and a singulation module, wherein:

[0136] The image acquisition module is used to use a drone to shoot images and videos of a preset scene, wherein the FFmpeg tool is used to extract frames from the video to obtain video images; the colmap tool is used to extract the pose information and shooting information of the video image, and an image set is obtained based on the shot image and the video image; and the image set is transmitted to the target screening module;

[0137] The image processing module is used to perform a first segmentation on the image set using the SegFormer model to obtain an original target image and a corresponding original mask; the original target image is processed to generate a preset number of target images, where each target image contains only one candidate three-dimensional target;

[0138] The image screening module is used to measure the image similarity of the target image using a twin neural network 2-channels network to obtain matching similarity and pose overlap; perform weighted calculation on the matching similarity and pose overlap to obtain a final similarity score; compare the final similarity score with a preset similarity threshold to screen out target images of the same type; perform a second segmentation on the target image to obtain a three-dimensional target image and a corresponding target mask;

[0139] The label assignment module is configured to render a three-dimensional target image using three-dimensional Gaussian splatting to obtain a rendering result and initialize labels; perform differential matching between the rendering result and the corresponding target mask, specifically constructing a label assignment optimization problem, including an objective function with the goal of minimizing differential matching and a constraint that the labels must be within a preset range; and solve the label assignment optimization problem to obtain an optimal label assignment solution.

[0140] The singulation module reconstructs and singulates the three-dimensional object in the three-dimensional object image based on the optimal label assignment scheme.

[0141] Example 3

[0142] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a three-dimensional object reconstruction and singulation method as described in any embodiment of the present invention is implemented.

[0143] Example 4

[0144] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for reconstructing and singulating a three-dimensional object as described in any embodiment of the present invention is implemented.

[0145] It is worth noting that the system, electronic device and computer-readable storage medium described in the present invention are all based on the same principles as the method described in Example 1, and will not be repeated here.

[0146] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for reconstructing and singulating a three-dimensional object, characterized in that: The method comprises: Using a drone to shoot images and videos of a preset scene, wherein the FFmpeg tool is used to extract frames from the video to obtain video images; using the colmap tool to extract posture information and shooting information of the video images, and obtaining an image set based on the shot images and video images; Performing a first segmentation on the image set using the SegFormer model to obtain an original target image and a corresponding original mask; processing the original target image to generate a preset number of target images, where each target image contains only one candidate 3D target; Using a twin neural network 2-channels network, the target image is measured for image similarity to obtain matching similarity and pose overlap; performing a weighted calculation on the matching similarity and pose overlap to obtain a final similarity score; comparing the final similarity score with a preset similarity threshold to filter out target images of the same type; performing a second segmentation on the target image to obtain a three-dimensional target image and a corresponding target mask; Rendering a 3D target image using 3D Gaussian splatting yields a rendering result, which is then initialized with labels. Differential matching is performed between the rendering result and the corresponding target mask, specifically constructing a label assignment optimization problem, including an objective function that minimizes differential matching and a constraint that the labels must be within a preset range. Solving the label assignment optimization problem yields an optimal label assignment solution. The three-dimensional object in the three-dimensional object image is reconstructed and singulated based on the optimal label assignment scheme.

2. A method for reconstructing and singulating a three-dimensional object according to claim 1, characterized in that: The colmap tool is used to extract the pose information and shooting information of the video image. The pose information is expressed as (x k ,y k ,z k ,α k ,γ k ), where x k Indicates the X-axis coordinate of the k-th video image, y k Indicates the Y-axis coordinate of the k-th video image, z k Indicates the Z-axis coordinate of the kth video image, α k represents the angle between the center point of the kth video image and the X axis, γ k represents the angle between the center point of the k-th video image and the Z axis; k represents the index value of the k-th video image; The shooting information includes the shooting position and shooting angle of the video image.

3. The method for reconstructing and singulating a three-dimensional object according to claim 1, wherein: The SegFormer model is used to perform the first segmentation of the image set, specifically: The Transformer encoder is used to extract the multi-scale features of the image, which can be expressed as follows: Where, represents the output features of the i-th image set of the l-th layer Transformer decoding layer; l represents the index value of the l-th layer Transformer decoding layer; L represents the total number of Transformer decoding layers; Transformer represents the Transformer decoding layer; i represents the index value of the i-th image set; The output features are fused based on the lightweight multi-layer perceptron decoder, which can be expressed as follows: Where, represents the fusion feature map of the i-th image set; MLP represents the multi-layer perceptron decoder; Upsample represents the upsampling function; Indicates channel splicing; The target image and the corresponding original mask are obtained, which can be expressed as follows: Where M j (a, b) represents the pixel value of the original mask of the j-th target image in the a-th row and b-th column, where the pixel value of 1 represents the target area and the pixel value of 0 represents the background area; M j represents the original mask of the j-th target image; a represents the row index of the a-th row; b represents the column index of the b-th column; A represents the number of rows of pixel values; B represents the number of columns of pixel values; σ represents the activation function; represents the pixel value of the fusion feature map of the i-th image set in the a-th row and b-th column; j represents the index value of the j-th target image; ξ represents the preset segmentation threshold; Processing the original target image, specifically removing and filling the background, to generate a preset number of target images, where each target image contains only one candidate three-dimensional target; The background removal is expressed as follows: E j (a,b)=Y j (a,b)⊙M j (a,b); Where, E j represents the pixel value of the jth target image in row a and column b after background removal, where E j (a,b)=1 represents the target area, E j (a, b) = 0 represents the background area; Y j (a,b) represents the pixel value of the jth target image at row a and column b; ⊙ represents element-by-element multiplication; The background filling specifically retains the pixel values of the target area and fills the pixel values of the background area, which can be expressed as follows: O j (a,b)=E j (a,b)⊙M j (a,b)+U⊙M j (a,b); Where, O j (a, b) represents the pixel value of the jth target image in the ath row and the bth column after background filling; U represents the background filling constant.

4. A three-dimensional object reconstruction and singulation method according to claim 3, characterized in that: The image similarity measurement of the target image is performed using a twin neural network 2-channels network, specifically: The image similarity measurement specifically comprises the following steps: the twin neural network 2-channels network receives an input target image; selects a preset number of target images whose current shooting information is within a preset distance range for matching; each matching step specifically comprises extracting a pair of candidate target images from the selected target image, and calculating the similarity and overlap between the candidate target images; The extraction process is repeated until all the selected preset number of target images are similarly matched to obtain the matching similarity and pose overlap; The matching similarity is expressed as follows: In the formula, Loss represents the loss function of the twin neural network 2-channels network; ω represents the preset network weight; λ represents the preset weight decay rate; max represents the maximum value function; y q represents the matching label of the qth pair of candidate target images; o q represents the matching similarity of the qth pair of candidate target images; q represents the index value of the qth pair of candidate target images; Q represents the number of matching pairs of extracted candidate target images; n represents the index value of the nth candidate target image; m represents the index value of the mth candidate target image; The posture overlap is expressed as follows: Where, Pos q represents the degree of pose overlap; exp represents the exponential function; α n represents the angle between the nth candidate target image and the X axis; α m represents the angle between the mth candidate target image and the X axis; γ n represents the angle between the mth candidate target image and the Y axis; γ m Represents the angle between the mth candidate target image and the Y axis.

5. A three-dimensional object reconstruction and singulation method according to claim 4, characterized in that: The matching similarity and pose overlap are weighted to obtain the final similarity score, which is expressed as: Where S q represents the final similarity score of the qth pair of candidate target images; Indicates the preset weighted weight; Compare the final similarity score with the preset similarity threshold and filter out target images of the same type. q >S thr , it means that the two-dimensional target in the nth candidate target image and the mth candidate target image is the same target, that is, y q =1; Otherwise, it means that the two-dimensional target in the nth candidate target image and the mth candidate target image are not the same target, that is, y q =-1, where S thr Indicates the preset similarity threshold; The target image is segmented for the second time to obtain a three-dimensional target image and a corresponding target mask.

6. A three-dimensional object reconstruction and singulation method according to claim 5, characterized in that: Use 3D Gaussian splashing to render the 3D target image, obtain the rendering result, and initialize the label, specifically: Initialize the three-dimensional Gaussian points represented by G g ={x g ,y g ,z g ,c g ,μ g ,η g }, where G g represents the g-th three-dimensional Gaussian point, g represents the index value of the g-th three-dimensional Gaussian point, x g Indicates the X-axis coordinate of the g-th three-dimensional Gaussian point, y g Indicates the Y-axis coordinate of the g-th three-dimensional Gaussian point, z g represents the Z-axis coordinate of the g-th three-dimensional Gaussian point, c g Indicates the color of the g-th three-dimensional Gaussian point, μ g Indicates the rendering degree of the g-th three-dimensional Gaussian point, η g represents the transmittance of the g-th three-dimensional Gaussian point; Projecting the three-dimensional Gaussian points into two-dimensional space can be expressed as: Where a g Represents the row index of the pixel value after projection of the g-th three-dimensional Gaussian point; b g Represents the column index of the pixel value after projection of the g-th three-dimensional Gaussian point; K represents the external parameter matrix of the preset drone shooting angle; R represents the internal parameter matrix of the preset drone shooting angle; Render the three-dimensional target image and obtain the rendering result, which is expressed as follows: X={X1,...,X z ,...,X Z }; Where, X z represents the rendering result of the z-th 3D target image; X z (a g ,b g ) represents the rendering result of the zth three-dimensional target image in the a g Row b g The pixel value of the column; X represents the rendering result; X Z represents the rendering result of the Z-th 3D target image; z represents the index value of the z-th 3D target image; Z represents the number of 3D target images; And perform label initialization specifically to initialize the labels of the rendering results of all three-dimensional target images as H z =0, where H z Represents the original label of the z-th 3D target image, where H z =0 indicates that the candidate 3D target in the rendering result of the z-th 3D target image does not participate in the reconstruction and singulation.

7. A three-dimensional object reconstruction and singulation method according to claim 6, characterized in that: Differentiate the rendering result and the corresponding target mask, specifically: The label assignment optimization problem is constructed, including the objective function with the goal of minimizing differential matching, which is expressed as follows: In the formula, I() represents the indicator function, where I(,0) represents the background area and I(,1) represents the target area; The constraint condition that the label is within the preset range is expressed as follows: Where GS represents the number of three-dimensional Gaussian points; st represents the constraint condition; To solve the label allocation optimization problem, the iterative solution process is as follows: Then H z =0; if Then H z =1, where H z = 1 means that the candidate 3D target in the rendering result of the zth 3D target image participates in the reconstruction and singulation; until the maximum number of iterations is reached or the label assignment optimization problem converges, the iteration is stopped and the optimal label assignment solution {H1,...,H z ,....,H Z }.

8. A three-dimensional object reconstruction and singulation system, characterized in that: The system includes an image acquisition module, an image processing module, an image screening module, a label assignment module and a singulation module, wherein: The image acquisition module is used to use a drone to shoot images and videos of a preset scene, wherein the FFmpeg tool is used to extract frames from the video to obtain video images; the colmap tool is used to extract the pose information and shooting information of the video images, and an image set is obtained based on the shot images and video images; the image set is transmitted to the image processing module; The image processing module is used to perform a first segmentation on the image set using the SegFormer model to obtain an original target image and a corresponding original mask; the original target image is processed to generate a preset number of target images, where each target image contains only one candidate three-dimensional target; The image screening module is used to measure the image similarity of the target image using a twin neural network 2-channels network to obtain matching similarity and pose overlap; perform weighted calculation on the matching similarity and pose overlap to obtain a final similarity score; compare the final similarity score with a preset similarity threshold to screen out target images of the same type; perform a second segmentation on the target image to obtain a three-dimensional target image and a corresponding target mask; The label assignment module is configured to render a three-dimensional target image using three-dimensional Gaussian splatting to obtain a rendering result and initialize labels; perform differential matching between the rendering result and the corresponding target mask, specifically constructing a label assignment optimization problem, including an objective function with the goal of minimizing differential matching and a constraint that the labels must be within a preset range; and solve the label assignment optimization problem to obtain an optimal label assignment solution. The singulation module reconstructs and singulates the three-dimensional object in the three-dimensional object image based on the optimal label assignment scheme.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for reconstructing and singulating a three-dimensional object as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for reconstructing and singulating a three-dimensional object as claimed in any one of claims 1 to 7 is implemented.

Images