3D reconstruction method of target components in flexible manufacturing system based on multimodal large model

Abstract

The present invention discloses a method for 3D reconstruction of a target component of a flexible manufacturing system based on a multimodal large model, which specifically comprises: arranging a plurality of cameras with different shooting angles in a target component area of ​​the flexible manufacturing system, wherein the cameras shoot 2D pictures of the target component at different viewing angles at the same time, preprocessing the 2D pictures to generate a target image; using a multimodal target detection GLIP model to locate the target image with the assistance of a prompt word to obtain a target area; inputting the target area into a semantic segmentation model SAM2 for segmentation to obtain a segmentation result; inputting the segmentation result into a 3D Gaussian splash module to generate 3D point cloud information; and using a pixel-level reconstruction algorithm to convert the 3D point cloud information into a grid model to obtain a 3D reconstruction result of the target component.

Description

Technical Field

[0001] The present invention relates to the field of flexible manufacturing, and in particular, mainly to a three-dimensional reconstruction method of a target component of a flexible manufacturing system based on a multi-modal large model. Background Art

[0002] With the advancement of science and technology, flexible manufacturing systems, as automated manufacturing systems that adapt to multi-variety and small-batch production, are usually composed of machining centers, automated logistics systems, and computer control systems. By introducing intelligent sensors and advanced control algorithms, production process parameters can be monitored in real time and adjusted dynamically, which improves the flexibility and adaptability of production. In the context of flexible manufacturing and intelligent manufacturing, the three-dimensional reconstruction of the manufactured target components is of great significance. At present, accurate three-dimensional reconstruction can provide designers with more intuitive and accurate product models in the digital design link, improve design efficiency and quality; provide engineers with accurate model information in simulation analysis, improve accuracy and reliability; and provide an accurate model basis for automated control and optimization in the manufacturing process. At the same time, in order to achieve three-dimensional reconstruction, it is necessary to accurately find the target components from the captured pictures, which involves target detection and semantic segmentation tasks. The use of multimodal large models for this task has become a research hotspot in the field of artificial intelligence images, but it still needs to be further optimized to improve performance. In addition, although Gaussian splashing technology has made breakthroughs in the field of three-dimensional reconstruction, there is still room for improvement in its application in flexible manufacturing systems.

[0003] Traditional target detection and semantic segmentation technologies only target targets of fixed categories. Once the target category changes, data collection, labeling, and model training and tuning need to be re-performed. This solution cannot adapt to the frequent changes of target components in flexible manufacturing scenarios.

[0004] Traditional 3D modeling takes a lot of time. The emergence of neural network radiation field provides a new solution for 3D reconstruction, but the reconstruction time is still very long. Summary of the invention

[0005] Based on the above problems in the prior art, according to one aspect of the present invention, a method for 3D reconstruction of target components of a flexible manufacturing system based on a multi-modal large model is proposed, comprising:

[0006] S1. Using a flexible manufacturing system, multiple cameras with different shooting angles are arranged in a target component area of ​​a vehicle. The cameras take 2D images of target components at different viewing angles at the same time. The target components include a vehicle body, a vehicle window, a vehicle frame, and a vehicle tire. The 2D images are preprocessed to generate a target image.

[0007] S2. Using a multimodal target detection GLIP model to locate the target image with the aid of a prompt word to obtain a target area. The model training process of the multimodal target detection GLIP model specifically includes:

[0008] Extracting a feature map from the target image using a visual encoder, and extracting a feature vector from the prompt word using a text encoder;

[0009] Extracting target regions from the feature map, each target region corresponds to a target frame, performing feature pooling on the target region to obtain target region features;

[0010] The target region feature is fused with the feature vector through a cross attention mechanism, and the correlation between the target region feature and the feature vector is calculated, and at the same time, the object in the target region feature is analyzed for correlation with the object in the feature vector;

[0011] Optimizing the multimodal object detection GLIP model using object detection loss and text-image matching loss;

[0012] S3, inputting the target area into the semantic segmentation model SAM2 for segmentation, and obtaining a segmentation result;

[0013] S4, inputting the segmentation result into a three-dimensional Gaussian splash module to generate 3D point cloud information;

[0014] S5. Convert the 3D point cloud information into a mesh model using a pixel-level reconstruction algorithm to obtain a three-dimensional reconstruction result of the target component, and obtain a manufacturing model of the target vehicle based on the three-dimensional reconstruction result of the target component.

[0015] Furthermore, the specific formula of the cross attention mechanism is as follows:

[0016]

[0017] Among them, Q represents the query matrix, specifically the target area features, K represents the key matrix, specifically the feature vector extracted from the prompt word, which is used to calculate the similarity with the query matrix, V represents the value matrix, specifically the feature vector extracted from the prompt word, which is used to generate the final attention output, D represents the dimension of the feature vector, and Softmax represents the activation function.

[0018] The specific formula of the target detection loss is as follows:

[0019]

[0020] in, represents the true category label of the i-th target area, represents the predicted category label, represents the true bounding box of the i-th target region, represents the predicted bounding box, represents the cross entropy loss, represents the regression loss, N represents the total number of labels, represents the target detection loss function.

[0021] GLIP is the target detection version of the CLIP algorithm open sourced by OpenAI. Traditional target detection models locate and classify the detected targets, but these categories are limited to labeled data in the training data. They do not have the ability to detect unlabeled categories. The GLIP algorithm uses a cross-attention mechanism to fuse image features and text features to find the correlation between image elements and text elements. During the training process, the region of interest in the image is extracted for target box comparison, and the objects in the region are analyzed for correlation with various objects in the text, and the images and texts labeled in the training set are paired. Since the text feature extraction network has natural language understanding capabilities, when text information not included in the training set is input during the test phase, the GLIP network can still compare the correlation between the object elements and image elements in the current text and give reasonable classification correspondence and target positioning.

[0022] Furthermore, the step S3 also includes: the semantic segmentation model SAM2 first extracts image features from the target image, and then performs segmentation and result prediction based on the input target area;

[0023] Among them, the semantic segmentation model SAM2 also introduces a memory module, which uses a cross-attention mechanism to fuse the image features, prediction results and prompt words of the historical frames, and provides them to the current frame as a reference.

[0024] With the target detection area given by GLIP, this area can be further used as the prompt information of SAM2 to segment the target area. SAM2 first extracts image features from the original image, and then predicts the segmentation result based on the position of the target detection box. This process is consistent with SAM1. In addition, the most prominent feature of SAM2 is the addition of a memory module, which uses a cross-attention mechanism to fuse the image features, prediction results, and prompt information of historical frames and provides them to the current frame as a reference, thereby achieving association and consistency between multiple frames. SAM2's image encoding, prompt information encoding, mask decoding, and memory encoding and decoding modules have all been pre-trained on large data sets, and have strong zero-sample generalization capabilities, which are very suitable for application scenarios with uncertain targets in this solution.

[0025] Furthermore, the step S4 specifically includes:

[0026] Use SFM reconstruction technology to restore the scene of the input segmentation result and obtain point cloud data;

[0027] The three-dimensional Gaussian splash module uses Gaussian ellipsoid superposition technology to fit the opacity and color of the point cloud data, and at the same time, performs perspective projection and rasterization on the overlapping Gaussian ellipsoids to obtain a projection result image;

[0028] Using the projection result image and the real image corresponding to the projection viewing angle to perform MSE loss optimization, and obtain a loss optimization value;

[0029] The loss optimization value is used to perform gradient descent optimization on the parameters of the Gaussian ellipsoid, thereby changing the position and shape of the ellipsoid.

[0030] The specific formula for MSE loss optimization is as follows:

[0031]

[0032] Where H represents the height of the target image, W represents the width of the target image, C(u,v) represents the color value of the pixel (u,v) in the projection result image, and I(u,v) represents the color value of the pixel (u,v) in the target image.

[0033] Three-dimensional Gaussian sputtering (3DGS) is a new perspective synthesis and three-dimensional model reconstruction method that has become popular after neural network radiance field (NeRF). First, SfM is used to recover the structural information of the scene or object from the image sequence, that is, the point cloud data here. 3DGS uses Gaussian ellipsoids to fit the opacity and color of the approximate point cloud data (corresponding to the MLP expression of NeRF), and then perspective projection and rasterization are performed on these overlapping Gaussian ellipsoids to obtain the projection result image. The MSE calculation is performed using the projection result and the real image corresponding to the projection perspective to obtain the loss function. The loss value is used to perform gradient descent optimization on the parameters of the Gaussian ellipsoid to change the position and shape of the ellipsoid. Compared with NeRF, 3DGS is easier to perform parallel calculations because the calculations of each Gaussian ellipsoid in its rendering process are relatively independent, and multiple ellipsoids can be processed at the same time, making full use of the parallel computing capabilities of modern CPU and other hardware to further improve the rendering speed. In the volume rendering process of NeRF, there is a certain dependency between the sampling points, and the efficiency of parallel calculations is relatively low.

[0034] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.

[0035] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0036] 1) The present invention uses the pre-trained networks GLIP and SAM2 based on the multimodal large model for target detection and semantic segmentation. The network can select the target to be detected with the help of natural language and graphic prompts. The network itself has zero-sample learning ability and does not require re-labeling and training of unknown categories, thereby improving the generalization of the detection module. The present invention is very suitable for the needs of customized operations for different target components in flexible manufacturing scenarios.

[0037] 2) The present invention uses 3DGS to perform three-dimensional reconstruction of the target component based on multi-angle 2D views. The biggest advantage of 3DGS is its fast reconstruction speed, which helps the flexible manufacturing system to quickly obtain 3D models and then apply them to subsequent design and manufacturing processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0039] Figure 1 A schematic flow chart of a method for three-dimensional reconstruction of a target component of a flexible manufacturing system based on a multi-modal large model according to an embodiment of the present invention is shown.

[0040] Figure 2 A schematic structural diagram of a method for three-dimensional reconstruction of a target component of a flexible manufacturing system according to an embodiment of the present invention is shown.

[0041] Figure 3 A schematic diagram of the structure of a SAM2 according to an embodiment of the present invention is shown.

[0042] Figure 4 A schematic diagram of a three-dimensional reconstruction process according to an embodiment of the present invention is shown.

[0043] Figure 5 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] Figure 1 FIG. 4 is a flow chart of a method for 3D reconstruction of a target component of a flexible manufacturing system based on a multi-modal large model according to an embodiment of the present invention. Figure 1 As shown:

[0047] S1. Using a flexible manufacturing system, multiple cameras with different shooting angles are arranged in a target component area of ​​a vehicle. The cameras take 2D images of target components at different viewing angles at the same time. The target components include a vehicle body, a vehicle window, a vehicle frame, and a vehicle tire. The 2D images are preprocessed to generate a target image.

[0048] Figure 2 The structure diagram of the 3D reconstruction method of the target component of the flexible manufacturing system is shown. In this diagram, flexible manufacturing can customize any desired target component. Through the combination of a rich standardized hardware system and a digital software system, the fixed path of traditional non-standard automation is broken, thereby establishing a manufacturing system that can quickly adapt to changes in the internal and external environment. It can meet the needs of intelligent production of multiple varieties and small batches, has the space for expansion and evolution, realizes lean production, and reduces business risks.

[0049] In the process of preprocessing the 2D images taken by the camera to generate the target image, methods such as image denoising, image enhancement, image correction, image segmentation and image registration are usually involved.

[0050] Image denoising During the image acquisition process, noise is generated due to various factors (such as thermal noise of electronic equipment, uneven light, environmental interference, etc.), which will affect subsequent image processing and analysis. The purpose of denoising is to reduce or eliminate these noises and improve the quality of the image.

[0051] Image enhancement is to make the target component features in the image more obvious. The image needs to be enhanced to highlight the areas and details of interest and suppress irrelevant information.

[0052] Image correction is due to the camera's shooting angle, lens distortion, etc. The captured 2D image may have geometric distortion, which needs to be corrected so that the target components in the image can correctly reflect their actual shape and positional relationship.

[0053] Image segmentation is to separate the target component from the background and other irrelevant objects, which makes it easier to extract and analyze the features of the target component.

[0054] Image registration is required when 2D images taken from different perspectives need to be fused to generate a target image so that the target components in different images can be correctly aligned in space.

[0055] like Figure 2 As shown, in this embodiment, multiple cameras are used to shoot components of the car from different perspectives, including tires, frames, windows, etc. These components of the car are the target components of this embodiment, which are shot by the camera and enter preprocessing to generate a target image; wherein, the cameras need to be reasonably arranged to achieve the goal of shooting all the target components in place and saving the use of cameras as much as possible; further, the camera position can be feedback adjusted through the preprocessed target image.

[0056] S2, using a multimodal target detection GLIP model to locate the target image with the assistance of a prompt word to obtain a target area;

[0057] The model training process of the multimodal target detection GLIP model specifically includes:

[0058] Extracting a feature map from the target image using a visual encoder, and extracting a feature vector from the prompt word using a text encoder;

[0059] Extracting target regions from the feature map, each target region corresponds to a target frame, performing feature pooling on the target region to obtain target region features;

[0060] The target region feature is fused with the feature vector through a cross attention mechanism, and the correlation between the target region feature and the feature vector is calculated, and at the same time, the object in the target region feature is analyzed for correlation with the object in the feature vector;

[0061] The multimodal object detection GLIP model is optimized using object detection loss and text-image matching loss.

[0062] The specific formula of the cross attention mechanism is as follows:

[0063]

[0064] Among them, Q represents the query matrix, specifically the target area features, K represents the key matrix, specifically the feature vector extracted from the prompt word, which is used to calculate the similarity with the query matrix, V represents the value matrix, specifically the feature vector extracted from the prompt word, which is used to generate the final attention output, D represents the dimension of the feature vector, and Softmax represents the activation function.

[0065] The specific formula of the target detection loss is as follows:

[0066]

[0067] in, represents the true category label of the i-th target area, represents the predicted category label, represents the true bounding box of the i-th target region, represents the predicted bounding box, represents the cross entropy loss, represents the regression loss, N represents the total number of labels, represents the target detection loss function.

[0068] The prompt word is a natural language description with guiding function, such as "Please find the frame in the picture". Since GLIP uses a large number of image-text pairs for comparative learning, it has strong zero-sample and few-sample migration capabilities. Therefore, the present invention can identify new target types without additional annotation, reducing the development cost of target detection and improving the generalization ability of the network.

[0069] S3, inputting the target area into the semantic segmentation model SAM2 for segmentation, and obtaining a segmentation result;

[0070] In this implementation, if Figure 3 As shown, the semantic segmentation model SAM2 first extracts image features from the target image, and then performs segmentation and result prediction based on the input target area;

[0071] Among them, the semantic segmentation model SAM2 also introduces a memory module, which uses a cross-attention mechanism to fuse the image features, prediction results and prompt words of the historical frames, and provides them to the current frame as a reference.

[0072] S4, inputting the segmentation result into a three-dimensional Gaussian splash module to generate 3D point cloud information, the specific steps include:

[0073] Use SFM reconstruction technology to restore the scene of the input segmentation result and obtain point cloud data;

[0074] The three-dimensional Gaussian splash module uses Gaussian ellipsoid superposition technology to fit the opacity and color of the point cloud data, and at the same time, performs perspective projection and rasterization on the overlapping Gaussian ellipsoids to obtain a projection result image;

[0075] Using the projection result image and the real image corresponding to the projection viewing angle to perform MSE loss optimization, and obtain a loss optimization value;

[0076] The loss optimization value is used to perform gradient descent optimization on the parameters of the Gaussian ellipsoid, thereby changing the position and shape of the ellipsoid.

[0077] The specific formula for MSE loss optimization is as follows:

[0078]

[0079] Where H represents the height of the target image, W represents the width of the target image, C(u,v) represents the color value of the pixel (u,v) in the projection result image, and I(u,v) represents the color value of the pixel (u,v) in the target image.

[0080] S5. Convert the 3D point cloud information into a mesh model using a pixel-level reconstruction algorithm to obtain a three-dimensional reconstruction result of the target component, and obtain a manufacturing model of the target vehicle based on the three-dimensional reconstruction result of the target component.

[0081] like Figure 4 As shown, the SFM reconstruction technology performs scene restoration on the input segmentation results and uses incremental reconstruction to convert 3D point cloud information into a mesh model;

[0082] Among them, incremental reconstruction is the process of gradually adding new images and updating the 3D model. Specifically, it includes: initializing the image, usually starting with two images; selecting two images with sufficient overlapping areas, estimating the relative pose of the two images through the basic matrix or the essential matrix, and aligning the new image with the existing 3D model through image registration. In practical applications, it includes: feature matching, feature matching between new images and existing images; PnP algorithm, estimating the camera pose of the new image through known 3D points and their projection points in the new image.

[0083] Then the initial 3D point cloud is calculated by triangulation method. The triangulation formula is as follows:

[0084]

[0085] Among them, A represents the matrix composed of camera pose and built-in matrix, and They represent the two-dimensional matrix of pixels of the selected image, and X represents the preliminary 3D point cloud matrix calculated by the triangulation method.

[0086] Furthermore, bundle adjustment is a process of global optimization of the entire 3D model and camera pose to minimize the reprojection error. Outlier filtering is the process of removing erroneous 3D points and camera poses generated during the reconstruction process. Reprojection error calculates the reprojection error of each 3D point in all views and removes points with large errors. Geometric consistency checks the geometric consistency of 3D points in different views and removes inconsistent points. Generating high-quality 3D point clouds from 2D segmentation results is a challenge and requires accurate depth information. The registration of multi-view point clouds requires high-precision geometric transformations, otherwise the reconstruction results will be distorted.

[0087] Finally, the 3D point cloud information is converted into a mesh model using a pixel-level reconstruction algorithm, which includes point cloud preprocessing, surface reconstruction, mesh generation and post-processing.

[0088] In summary, the present invention solves the problem of multi-view image consistency and insufficient model generalization ability through image preprocessing and data enhancement technology, improves the accuracy of target detection and semantic segmentation by optimizing prompt words and post-processing technology, and further ensures the generation and precise registration of high-quality 3D point clouds with the help of depth sensors and advanced point cloud registration algorithms; finally, the stability of the system and the ability to process large-scale data are improved through modular design, optimization of calculation processes and the use of efficient distributed storage systems. These measures jointly guarantee the efficiency and accuracy of the generation from 2D images to 3D models. The flexible manufacturing of the present invention can quickly customize the required mechanical parts, including three-dimensional reconstruction technology, which can quickly obtain the three-dimensional model information of the produced components, providing technical support for customized production. At the same time, combined with the semantically sensitive feature extraction capabilities and intelligent algorithms of multimodal large models, intelligent control and management of the production process can be achieved, and the flexibility and reliability of production can be improved.

[0089] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 suitable for implementing an electronic device of an embodiment of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0090] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0091] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0092] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0093] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0096] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: arranges multiple cameras with different shooting angles in the target component area of ​​the flexible manufacturing system, and the cameras take 2D pictures of the target component at different viewing angles at the same time, pre-processes the 2D pictures, and generates a target image; uses the multimodal target detection GLIP model to locate the target image with the assistance of prompt words to obtain the target area; inputs the target area into the semantic segmentation model SAM2 for segmentation to obtain the segmentation result; inputs the segmentation result into the three-dimensional Gaussian splash module to generate 3D point cloud information; uses the pixel-level reconstruction algorithm to convert the 3D point cloud information into a grid model to obtain the three-dimensional reconstruction result of the target component.

[0097] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A three-dimensional reconstruction method for target components of a flexible manufacturing system based on a multi-modal large model, characterized in that: The following steps are involved: S1. Using a flexible manufacturing system, multiple cameras with different shooting angles are arranged in a target component area of ​​a vehicle. The cameras take 2D images of target components at different viewing angles at the same time. The target components include a vehicle body, a vehicle window, a vehicle frame, and a vehicle tire. The 2D images are preprocessed to generate a target image. S2. Using a multimodal target detection GLIP model to locate the target image with the aid of a prompt word to obtain a target area, wherein the model training process of the multimodal target detection GLIP model specifically includes: Extracting a feature map from the target image using a visual encoder, and extracting a feature vector from the prompt word using a text encoder; Extracting target regions from the feature map, each target region corresponds to a target frame, performing feature pooling on the target region to obtain target region features; The target region feature is fused with the feature vector through a cross attention mechanism, and the correlation between the target region feature and the feature vector is calculated, and at the same time, the object in the target region feature is analyzed for correlation with the object in the feature vector; Optimizing the multimodal object detection GLIP model using object detection loss and text-image matching loss; S3, input the target area into the semantic segmentation model SAM2 for segmentation, and obtain the segmentation result, wherein the semantic segmentation model SAM2 first extracts image features of the target image, and then performs segmentation and result prediction based on the input target area; the semantic segmentation model SAM2 also introduces a memory module, and the memory module uses a cross-attention mechanism to fuse the image features, prediction results and prompt words of the historical frames, and provides them to the current frame as a reference; S4, inputting the segmentation result into a three-dimensional Gaussian splash module to generate 3D point cloud information, specifically including: using SFM reconstruction technology to perform scene restoration on the input segmentation result to obtain point cloud data; The three-dimensional Gaussian splash module uses Gaussian ellipsoid superposition technology to fit the opacity and color of the point cloud data, and at the same time, performs perspective projection and rasterization on the overlapping Gaussian ellipsoids to obtain a projection result image; Using the projection result image and the real image corresponding to the projection viewing angle to perform MSE loss optimization, and obtain a loss optimization value; Using the loss optimization value, the parameters of the Gaussian ellipsoid are optimized by gradient descent, thereby changing the position and shape of the ellipsoid; S5. Convert the 3D point cloud information into a mesh model using a pixel-level reconstruction algorithm to obtain a three-dimensional reconstruction result of the target component, and obtain a manufacturing model of the target vehicle based on the three-dimensional reconstruction result of the target component.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The specific formula of the cross attention mechanism is as follows: Among them, Q represents the query matrix, specifically the target area features, K represents the key matrix, specifically the feature vector extracted from the prompt word, which is used to calculate the similarity with the query matrix, V represents the value matrix, specifically the feature vector extracted from the prompt word, which is used to generate the final attention output, D represents the dimension of the feature vector, and Softmax represents the activation function.

3. The three-dimensional reconstruction method according to claim 1, characterized in that: The specific formula of the target detection loss is as follows: in, represents the true category label of the i-th target area, represents the predicted category label, represents the true bounding box of the i-th target region, represents the predicted bounding box, represents the cross entropy loss, represents the regression loss, N represents the total number of labels, represents the target detection loss function.

4. The three-dimensional reconstruction method according to claim 1, characterized in that: The specific formula for MSE loss optimization is as follows: Where H represents the height of the target image, W represents the width of the target image, C(u,v) represents the color value of the pixel (u,v) in the projection result image, and I(u,v) represents the color value of the pixel (u,v) in the target image.

5. A computer program product, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

6. A computing system, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to execute the method according to any one of claims 1 to 4.

Images