Three-dimensional modeling method and device of MEP equipment, storage medium and electronic device

Abstract

The application discloses a three-dimensional modeling method and device of MEP equipment, a storage medium and an electronic device. The three-dimensional modeling method of MEP equipment comprises the following steps: acquiring a two-dimensional image training set of MEP equipment; inputting a to-be-modeled image into a semantic segmentation model trained using the two-dimensional image training set to predict a mask graph; performing depth estimation on the to-be-modeled image to generate an initial point cloud model; and completing semantic labeling of a three-dimensional space by fusing the mask graph and the initial point cloud model to form a three-dimensional semantic point cloud model. The application solves the technical problems of large data volume, high redundancy, much noise, high storage and processing cost, large calculation volume, high calculation cost, easy loss of color information, large data labeling workload and poor training effect.

Description

Technical Field

[0001] This application relates to the field of 3D modeling, and more specifically, to a 3D modeling method, apparatus, storage medium, and electronic device for MEP equipment. Background Technology

[0002] The technical process of automated 3D modeling can be briefly summarized into two main parts: data acquisition and data processing.

[0003] The automated 3D modeling process for MEP equipment mainly uses emerging acquisition devices such as depth cameras and LiDAR to obtain high-precision geometric information in terms of data acquisition. In terms of data processing, it mainly involves directly performing noise reduction and extraction on point clouds, or using deep learning methods based on point clouds and images to extract content related to the MEP equipment from the original 3D or 2D information, and then building a model.

[0004] However, the geometric information of MEP devices suffers from problems such as large data volume, high redundancy, high noise, and high storage and processing costs. In addition, the processing of geometric information of MEP devices is characterized by large computational load, high computational cost, easy loss of color information, large workload of data annotation, and poor training effect.

[0005] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0006] The main objective of this application is to provide a 3D modeling method, apparatus, storage medium, and electronic device for MEP equipment, in order to solve the problems of large data volume, high redundancy, high noise, high storage and processing costs, large computational load, high computational cost, easy loss of color information, large data annotation workload, and poor training effect.

[0007] To achieve the above objectives, according to one aspect of this application, a three-dimensional modeling method for MEP devices is provided.

[0008] The 3D modeling method for MEP devices according to this application includes: acquiring a 2D image training set of the MEP device; inputting the image to be modeled into a semantic segmentation model trained using the 2D image training set to predict a mask image; performing depth estimation on the image to be modeled to generate an initial point cloud model; and completing semantic annotation in 3D space by fusing the mask image and the initial point cloud model to form a 3D semantic point cloud model.

[0009] Furthermore, obtaining the two-dimensional image training set includes: receiving a video of the MEP device captured by a monocular camera; extracting keyframes from the MEP device video; performing distortion correction on all keyframes to obtain several two-dimensional images; and labeling the several two-dimensional images to obtain the two-dimensional image training set.

[0010] Furthermore, several two-dimensional images are labeled to obtain a two-dimensional image training set, including: labeling several two-dimensional images using the labelme software, combining salient annotation and global annotation; saving the labeled two-dimensional images as JSON files; and converting the JSON files into a two-dimensional image training set in VOC format.

[0011] Furthermore, the training of the semantic segmentation model includes: dividing the two-dimensional image training set into a training set and a validation set; inputting the training set into the semantic segmentation model combining MobileNetv2 and DeeplabV3+ for training; and inputting the validation set into the semantic segmentation model combining MobileNetv2 and DeeplabV3+ for validation.

[0012] Furthermore, depth estimation of the image to be modeled and generation of an initial point cloud model include: matching feature points of the image to be modeled by the SfM algorithm, performing sparse reconstruction of the point cloud and calculating the camera pose; based on the calculation results of the SfM algorithm, matching and reconstructing a large number of pixels in the image by the MVS algorithm to obtain the initial point cloud model.

[0013] Furthermore, semantic annotation in 3D space is completed by fusing the mask image and the initial point cloud model to form a 3D semantic point cloud model. This includes: establishing a relationship between the mask image with semantic labels and the initial point cloud model to find corresponding pixels; integrating spatiotemporally inconsistent semantic labels using a Bayesian probability update method to determine a unique label for pixels in the same image; and mapping the unique label of each pixel to its corresponding point in the initial point cloud model to generate a 3D semantic point cloud model.

[0014] Furthermore, establishing an association between the semantically labeled mask images and the initial point cloud model to find corresponding pixels involves: assigning each mask image to the image to be modeled using file information; calculating the projection center of the mask image based on the image projection pose associated with the image to be modeled, and reconstructing the camera parameters; for each point, finding the corresponding mask image based on its original projected image number and camera number; and finding the corresponding pixel in the corresponding mask image based on the 3D point and projection information, thus completing the mapping from one point to multiple pixels in multiple images.

[0015] To achieve the above objectives, according to another aspect of this application, a three-dimensional modeling apparatus for a MEP device is provided.

[0016] The 3D modeling apparatus for a MEP device according to this application includes: an acquisition module for acquiring a 2D image training set of the MEP device; a training prediction module for inputting the image to be modeled into a semantic segmentation model trained using the 2D image training set to predict a mask image; a depth estimation module for performing depth estimation on the image to be modeled to generate an initial point cloud model; and a fusion module for completing semantic annotation in 3D space by fusing the mask image and the initial point cloud model to form a 3D semantic point cloud model.

[0017] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided.

[0018] According to the computer-readable storage medium of this application, the computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the three-dimensional modeling method of the MEP device at runtime.

[0019] To achieve the above objectives, according to another aspect of this application, an electronic device is provided.

[0020] The electronic device according to this application includes: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the three-dimensional modeling method of the MEP device.

[0021] In this embodiment, a two-dimensional image information processing method is adopted. A two-dimensional image training set from a MEP device is acquired; the image to be modeled is input into a semantic segmentation model trained using the two-dimensional image training set to predict a mask image; depth estimation is performed on the image to be modeled to generate an initial point cloud model; semantic annotation in three-dimensional space is completed by fusing the mask image and the initial point cloud model to form a three-dimensional semantic point cloud model. This achieves the goal of using a combination of semantic segmentation, depth estimation, and semantic annotation to build a model instead of processing three-dimensional geometric information to build a model. This significantly reduces data volume, redundancy, and noise, as well as storage and processing costs, and significantly reduces computational load and cost while preserving color information, reducing annotation workload, and improving training effectiveness. This solves the technical problems of large data volume, high redundancy, high noise, high storage and processing costs, large computational load, high computational cost, easy loss of color information, large data annotation workload, and poor training effect. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of the accompanying drawings are used to explain the application and do not constitute an undue limitation of the application. In the drawings:

[0023] Figure 1 This is a flowchart illustrating a three-dimensional modeling method for a MEP device according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the structure of a three-dimensional modeling device for a MEP device according to an embodiment of this application;

[0025] Figure 3 This is a flowchart illustrating a three-dimensional modeling method for a MEP device according to a preferred embodiment of this application;

[0026] Figure 4 This is a schematic diagram of a two-dimensional image taken from a distance according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of a two-dimensional image captured at close range according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of a prominently annotated two-dimensional image according to an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of a globally annotated two-dimensional image according to an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing the invention and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0033] Furthermore, in addition to indicating direction or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in certain situations to indicate a dependency or connection. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0034] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] According to embodiments of the present invention, a three-dimensional modeling method for MEP devices is provided, such as... Figure 1 and 3 As shown, the method includes the following steps S001 to S004:

[0037] Step S001: Obtain the two-dimensional image training set of the MEP device;

[0038] According to an embodiment of the present invention, preferably, obtaining a two-dimensional image training set includes:

[0039] Receive MEP device video captured by a monocular camera;

[0040] Keyframes were extracted from the video of the MEP device;

[0041] Distortion removal is performed on all keyframes to obtain several two-dimensional images;

[0042] Annotate several two-dimensional images to obtain a two-dimensional image training set.

[0043] Specifically, data samples were first obtained by capturing videos of MEP devices using a monocular camera in a typical MEP device environment. During filming, a relatively stable movement speed was maintained to capture a comprehensive view of the MEP devices within the scene; the focus was primarily on capturing the overall scene from a distance, supplemented by close-up, focused shots of specific areas, to enhance the learning of smaller objects with less prominent features. The captured images are as follows: Figure 4 and 5 Then, keyframes are extracted at intervals of 5 or 10 frames per frame. These keyframes are then processed for image distortion correction in "colmap" to obtain several 2D images. In this example, approximately 1000 initial JPG format 2D images are collected. Finally, these 2D images are labeled to obtain a 2D image training set, which is used as input for model training. Using 2D images to replace geometric information can significantly reduce the amount of data, reduce redundancy and noise, and lower storage and processing costs. Furthermore, compared to 3D ensemble information, 2D images can also reduce the amount of annotation required during the labeling process.

[0044] In this embodiment, the monocular camera can be a smartphone, specifically a phone with an RGB camera. Compared to depth cameras and LiDAR scanning devices, this reduces the size and weight of the equipment used in modeling, improving versatility and ease of use. Whether mounted on other devices such as drones or used handheld by staff, it significantly enhances convenience.

[0045] In this example, preferably, several two-dimensional images are labeled to obtain a two-dimensional image training set, including:

[0046] Using the labelme software, several two-dimensional images were annotated using a combination of salient annotation and global annotation.

[0047] Save the annotated 2D image as a JSON file;

[0048] Convert the JSON file into a two-dimensional image training set in VOC format.

[0049] Specifically, polygonal bounding boxes are used to meticulously circle the pixels of different objects and assign them different labels. The coordinates of the polygon corner points are stored in a JSON file with the same name as the image, and then a labeled VOC dataset is created. The annotation process mainly consists of salient annotation (ignoring minor background information) and global annotation (annotating all pixels in the image corresponding to the label). The dataset primarily uses salient annotation, supplemented by global annotation, to reduce the annotation workload while ensuring accuracy. The annotation method is described in the comments. Figure 6 and 7 .

[0050] Step S002: Input the image to be modeled into the semantic segmentation model trained using the two-dimensional image training set to predict the mask image;

[0051] The weights of the iterations with the smallest loss values ​​during training are selected as prediction weights. The RGB images of the modeling site (i.e. the images to be modeled) captured by video frame capture are fed into the trained semantic segmentation convolutional neural network, and the corresponding mask images are output. The same color blocks used in the annotation represent the neural network's prediction of the semantics of the MEP device in the image.

[0052] According to an embodiment of the present invention, preferably, the training of the semantic segmentation model includes:

[0053] The two-dimensional image training set is divided into a training set and a validation set;

[0054] The training set is input into a semantic segmentation model that combines MobileNetv2 and DeeplabV3+ for training;

[0055] The validation set was input into a semantic segmentation model that combines MobileNetv2 and DeeplabV3+ for validation.

[0056] The obtained 2D image training set was fed into a deeplabv3+ neural network based on the VOC extended dataset, using weights trained with MobileNetv2 as the backbone network as pre-training weights for both training and validation. Specifically, the training and validation sets of the 2D image training set were in a 9:1 ratio. MobileNetv2 was used as the backbone feature extraction network; weights trained with MobileNetv2 as the backbone network based on the VOC extended dataset were selected as pre-training weights; and DeeplabV3+ was used as the deep convolutional neural network model. The training set was divided into 8 batches for 50 iterations during the model training freeze phase and divided into 4 batches for 100 iterations during the unfreezing phase.

[0057] The inputs for both training and prediction are two-dimensional images, which reduces the computational load and cost during training and prediction, while improving training performance.

[0058] Step S003: Perform depth estimation on the image to be modeled to generate an initial point cloud model;

[0059] According to an embodiment of the present invention, preferably, performing depth estimation on the image to be modeled to generate an initial point cloud model includes:

[0060] The SfM algorithm is used to match feature points in the captured frame images to be modeled, sparse reconstruction of the point cloud is performed, and the camera pose is calculated.

[0061] Based on the calculation results of the SfM algorithm, the MVS algorithm is used to match and reconstruct a large number of pixels in the image to obtain an initial point cloud model.

[0062] Specifically, in the 3D modeling process, to reduce computational load and cost, 2D images are used to replace geometric information for model training and prediction. This requires depth estimation of the image to be modeled, converting it into an initial point cloud model without semantic meaning. The main workflow uses the SfM and MVS algorithms. In the SfM stage, feature point matching and point cloud fusion are iterated across multiple input images, ultimately generating a relatively sparse feature point cloud. In the MVS stage, a set of images with known camera parameters calculated by the SfM stage is input, and a large number of pixels in the images are matched and reconstructed to obtain the initial point cloud model. Thus, the initial point cloud model serves as the foundational 3D model for 3D modeling, providing a guarantee for subsequent fusion.

[0063] Step S004: Complete the semantic annotation of the three-dimensional space by fusing the mask image with the initial point cloud model to form a three-dimensional semantic point cloud model.

[0064] According to an embodiment of the present invention, preferably, the process of completing semantic annotation in three-dimensional space by fusing a mask image with an initial point cloud model to form a three-dimensional semantic point cloud model includes:

[0065] The mask image with semantic labels is associated with the initial point cloud model to find the corresponding pixels;

[0066] Semantic labels that are inconsistent in time and space are integrated using a Bayesian probability update method to determine a unique label for each pixel in the same image.

[0067] The unique label of each pixel is mapped to its corresponding point in the initial point cloud model to generate a 3D semantic point cloud model.

[0068] Specifically, a one-to-many relationship between 3D points and 2D semantic points is sought between the initial point cloud model and the 2D mask image. Then, unique labels calculated using a Bayesian probability update model correct for semantic errors in recognizing the same object, assigning unique and more accurate labels to the point cloud model. Finally, by mapping the unique labels of each pixel to its corresponding point in the initial point cloud model, a 3D semantic point cloud model is generated. This effectively avoids the loss of color information during processing, while simultaneously achieving 2D-to-3D semantic information mapping and error correction in 2D semantic segmentation. Furthermore, since semantic segmentation only targets key elements such as pipelines and devices in the scene, ignoring other background content, semantic label fusion also enables noise reduction and invalid point filtering in the point cloud model.

[0069] Preferably, establishing an association between the semantically labeled mask and the initial point cloud model to find the corresponding pixels includes: numbering each mask with the image to be modeled using file information; calculating the projection center of the mask based on the image projection pose associated with the image to be modeled, and reconstructing the camera parameters; for each point, finding the corresponding mask based on its original projected image number and camera number; and finding the corresponding pixel in the corresponding mask based on the 3D point and projection information, thus completing the mapping from one point to multiple pixels in multiple images.

[0070] In summary, using a combination of semantic segmentation, depth estimation, and semantic annotation to build a model instead of processing 3D geometric information can significantly reduce the amount of data, redundancy, and noise, as well as storage and processing costs. It can also significantly reduce computational load, reduce computational costs, preserve color information, reduce annotation workload, and improve training results.

[0071] As can be seen from the above description, the present invention achieves the following technical effects:

[0072] In this embodiment, a two-dimensional image information processing method is adopted. A two-dimensional image training set from a MEP device is acquired; the image to be modeled is input into a semantic segmentation model trained using the two-dimensional image training set to predict a mask image; depth estimation is performed on the image to be modeled to generate an initial point cloud model; semantic annotation in three-dimensional space is completed by fusing the mask image and the initial point cloud model to form a three-dimensional semantic point cloud model. This achieves the goal of using a combination of semantic segmentation, depth estimation, and semantic annotation to build a model instead of processing three-dimensional geometric information to build a model. This significantly reduces data volume, redundancy, and noise, as well as storage and processing costs, and significantly reduces computational load and cost while preserving color information, reducing annotation workload, and improving training effectiveness. This solves the technical problems of large data volume, high redundancy, high noise, high storage and processing costs, large computational load, high computational cost, easy loss of color information, large data annotation workload, and poor training effect.

[0073] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0074] According to embodiments of the present invention, an apparatus for implementing the above-described three-dimensional modeling method for MEP equipment is also provided, such as... Figure 2 and 3 As shown, the device includes:

[0075] Module 10 is used to acquire a two-dimensional image training set of the MEP device;

[0076] According to an embodiment of the present invention, preferably, obtaining a two-dimensional image training set includes:

[0077] Receives video footage of the MEP device captured by a monocular camera;

[0078] Keyframes were extracted from the video of the MEP device;

[0079] Distortion removal is performed on all keyframes to obtain several two-dimensional images;

[0080] Annotate several two-dimensional images to obtain a two-dimensional image training set.

[0081] Specifically, data samples were first obtained by capturing videos of MEP devices using a monocular camera in a typical MEP device environment. During filming, a relatively stable movement speed was maintained to capture a comprehensive view of the MEP devices within the scene; the focus was primarily on capturing the overall scene from a distance, supplemented by close-up, focused shots of specific areas, to enhance the learning of smaller objects with less prominent features. The captured images are as follows: Figure 4 and 5 Then, keyframes are extracted at intervals of 5 or 10 frames per frame. These keyframes are then processed for image distortion correction in "colmap" to obtain several 2D images. In this example, approximately 1000 initial JPG format 2D images are collected. Finally, these 2D images are labeled to obtain a 2D image training set, which is used as input for model training. Using 2D images to replace geometric information can significantly reduce the amount of data, reduce redundancy and noise, and lower storage and processing costs. Furthermore, compared to 3D ensemble information, 2D images can also reduce the amount of annotation required during the labeling process.

[0082] In this embodiment, the monocular camera can be a smartphone, specifically a phone with an RGB camera. Compared to depth cameras and LiDAR scanning devices, this reduces the size and weight of the equipment used in modeling, improving versatility and ease of use. Whether mounted on other devices such as drones or used handheld by staff, it significantly enhances convenience.

[0083] In this example, preferably, several two-dimensional images are labeled to obtain a two-dimensional image training set, including:

[0084] Using the labelme software, several two-dimensional images were annotated using a combination of salient annotation and global annotation.

[0085] Save the annotated 2D image as a JSON file;

[0086] Convert the JSON file into a two-dimensional image training set in VOC format.

[0087] Specifically, polygonal bounding boxes are used to meticulously circle the pixels of different objects and assign them different labels. The coordinates of the polygon corner points are stored in a JSON file with the same name as the image, and then a labeled VOC dataset is created. The annotation process mainly consists of salient annotation (ignoring minor background information) and global annotation (annotating all pixels in the image corresponding to the label). The dataset primarily uses salient annotation, supplemented by global annotation, to reduce the annotation workload while ensuring accuracy. The annotation method is described in the comments. Figure 6 and 7 .

[0088] Training prediction module 20 is used to input the image to be modeled into the semantic segmentation model trained using the two-dimensional image training set, and predict the mask image.

[0089] The weights of the iterations with the smallest loss values ​​during training are selected as prediction weights. The RGB images of the modeling site (i.e. the images to be modeled) captured by video frame capture are fed into the trained semantic segmentation convolutional neural network, and the corresponding mask images are output. The same color blocks used in the annotation represent the neural network's prediction of the semantics of the MEP device in the image.

[0090] According to an embodiment of the present invention, preferably, the training of the semantic segmentation model includes:

[0091] The two-dimensional image training set is divided into a training set and a validation set;

[0092] The training set is input into a semantic segmentation model that combines MobileNetv2 and DeeplabV3+ for training;

[0093] The validation set was input into a semantic segmentation model that combines MobileNetv2 and DeeplabV3+ for validation.

[0094] The obtained 2D image training set was fed into a deeplabv3+ neural network based on the VOC extended dataset, using weights trained with MobileNetv2 as the backbone network as pre-training weights for both training and validation. Specifically, the training and validation sets of the 2D image training set were in a 9:1 ratio. MobileNetv2 was used as the backbone feature extraction network; weights trained with MobileNetv2 as the backbone network based on the VOC extended dataset were selected as pre-training weights; and DeeplabV3+ was used as the deep convolutional neural network model. The training set was divided into 8 batches for 50 iterations during the model training freeze phase and divided into 4 batches for 100 iterations during the unfreezing phase.

[0095] The inputs for both training and prediction are two-dimensional images, which reduces the computational load and cost during training and prediction, while improving training performance.

[0096] The depth estimation module 30 is used to perform depth estimation on the image to be modeled and generate an initial point cloud model.

[0097] According to an embodiment of the present invention, preferably, performing depth estimation on the image to be modeled to generate an initial point cloud model includes:

[0098] The SfM algorithm is used to match feature points in the captured frame images to be modeled, sparse reconstruction of the point cloud is performed, and the camera pose is calculated.

[0099] Based on the calculation results of the SfM algorithm, the MVS algorithm is used to match and reconstruct a large number of pixels in the image to obtain an initial point cloud model.

[0100] Specifically, in the 3D modeling process, to reduce computational load and cost, 2D images are used to replace geometric information for model training and prediction. This requires depth estimation of the image to be modeled, converting it into an initial point cloud model without semantic meaning. The main workflow uses the SfM and MVS algorithms. In the SfM stage, feature point matching and point cloud fusion are iterated across multiple input images, ultimately generating a relatively sparse feature point cloud. In the MVS stage, a set of images with known camera parameters calculated by the SfM stage is input, and a large number of pixels in the images are matched and reconstructed to obtain the initial point cloud model. Thus, the initial point cloud model serves as the foundational 3D model for 3D modeling, providing a guarantee for subsequent fusion.

[0101] The fusion module 40 is used to complete the semantic annotation of the three-dimensional space by fusing the mask image with the initial point cloud model, and form a three-dimensional semantic point cloud model.

[0102] According to an embodiment of the present invention, preferably, the process of completing semantic annotation in three-dimensional space by fusing a mask image with an initial point cloud model to form a three-dimensional semantic point cloud model includes:

[0103] The mask image with semantic labels is associated with the initial point cloud model to find the corresponding pixels;

[0104] Semantic labels that are inconsistent in time and space are integrated using a Bayesian probability update method to determine a unique label for each pixel in the same image.

[0105] The unique label of each pixel is mapped to its corresponding point in the initial point cloud model to generate a 3D semantic point cloud model.

[0106] Specifically, a one-to-many relationship between 3D points and 2D semantic points is sought between the initial point cloud model and the 2D mask image. Then, unique labels calculated using a Bayesian probability update model correct for semantic errors in recognizing the same object, assigning unique and more accurate labels to the point cloud model. Finally, by mapping the unique labels of each pixel to its corresponding point in the initial point cloud model, a 3D semantic point cloud model is generated. This effectively avoids the loss of color information during processing, while simultaneously achieving 2D-to-3D semantic information mapping and error correction in 2D semantic segmentation. Furthermore, since semantic segmentation only targets key elements such as pipelines and devices in the scene, ignoring other background content, semantic label fusion also enables noise reduction and invalid point filtering in the point cloud model.

[0107] Preferably, establishing an association between the semantically labeled mask and the initial point cloud model to find the corresponding pixels includes: numbering each mask with the image to be modeled using file information; calculating the projection center of the mask based on the image projection pose associated with the image to be modeled, and reconstructing the camera parameters; for each point, finding the corresponding mask based on its original projected image number and camera number; and finding the corresponding pixel in the corresponding mask based on the 3D point and projection information, thus completing the mapping from one point to multiple pixels in multiple images.

[0108] In summary, using a combination of semantic segmentation, depth estimation, and semantic annotation to build a model instead of processing 3D geometric information can significantly reduce the amount of data, redundancy, and noise, as well as storage and processing costs. It can also significantly reduce computational load, reduce computational costs, preserve color information, reduce annotation workload, and improve training results.

[0109] As can be seen from the above description, the present invention achieves the following technical effects:

[0110] In this embodiment, a two-dimensional image information processing method is adopted. A two-dimensional image training set from a MEP device is acquired; the image to be modeled is input into a semantic segmentation model trained using the two-dimensional image training set to predict a mask image; depth estimation is performed on the image to be modeled to generate an initial point cloud model; semantic annotation in three-dimensional space is completed by fusing the mask image and the initial point cloud model to form a three-dimensional semantic point cloud model. This achieves the goal of using a combination of semantic segmentation, depth estimation, and semantic annotation to build a model instead of processing three-dimensional geometric information to build a model. This significantly reduces data volume, redundancy, and noise, as well as storage and processing costs, and significantly reduces computational load and cost while preserving color information, reducing annotation workload, and improving training effectiveness. This solves the technical problems of large data volume, high redundancy, high noise, high storage and processing costs, large computational load, high computational cost, easy loss of color information, large data annotation workload, and poor training effect.

[0111] Obviously, those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0112] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A three-dimensional modeling method for MEP equipment, characterized in that, include: Obtain the two-dimensional image training set of the MEP device; The image to be modeled is input into the semantic segmentation model trained using the two-dimensional image training set, and a mask image is predicted. Depth estimation is performed on the image to be modeled to generate an initial point cloud model, including: The SfM algorithm is used to match feature points in the captured frame images to be modeled, sparse reconstruction of the point cloud is performed, and the camera pose is calculated. The semantic annotation of the three-dimensional space is completed by fusing the mask image with the initial point cloud model, forming a three-dimensional semantic point cloud model; The mask image with semantic labels is associated with the initial point cloud model to find the corresponding pixels; The semantically labeled mask image is associated with the initial point cloud model to find corresponding pixels, including: Each mask image is numbered and associated with the image to be modeled using the file information; Based on the image projection pose associated with the image to be modeled, the projection center of the mask image is calculated, and the camera parameters are restored. For each point, find the corresponding mask image based on the image number to be modeled and the camera number; Based on the 3D point and projection information, find the corresponding pixel in the corresponding mask image to complete the mapping from one point to multiple pixels in multiple photos.

2. The three-dimensional modeling method according to claim 1, characterized in that, Obtaining the 2D image training set includes: Receives video footage of the MEP device captured by a monocular camera; Keyframes were extracted from the video of the MEP device; Distortion removal is performed on all keyframes to obtain several two-dimensional images; Annotate several two-dimensional images to obtain a two-dimensional image training set.

3. The three-dimensional modeling method according to claim 2, characterized in that, Annotate several two-dimensional images to obtain a two-dimensional image training set, including: Using the labelme software, several two-dimensional images were annotated using a combination of salient annotation and global annotation. Save the annotated 2D image as a JSON file; Convert the JSON file into a two-dimensional image training set in VOC format.

4. The three-dimensional modeling method according to claim 1, characterized in that, The training of the semantic segmentation model includes: The two-dimensional image training set is divided into a training set and a validation set; The training set is input into a semantic segmentation model that combines MobileNetv2 and DeeplabV3+ for training; The validation set was input into a semantic segmentation model that combines MobileNetv2 and DeeplabV3+ for validation.

5. The three-dimensional modeling method according to claim 1, characterized in that, Depth estimation of the image to be modeled, generating an initial point cloud model, includes: The SfM algorithm is used to match feature points in the captured frame images to be modeled, sparse reconstruction of the point cloud is performed, and the camera pose is calculated. Based on the calculation results of the SfM algorithm, the MVS algorithm is used to match and reconstruct a large number of pixels in the image to obtain an initial point cloud model.

6. The three-dimensional modeling method according to claim 1, characterized in that, Semantic annotation in 3D space is achieved by fusing the mask image with the initial point cloud model, forming a 3D semantic point cloud model, including: Semantic labels that are inconsistent in time and space are integrated using a Bayesian probability update method to determine a unique label for each pixel in the same image. The unique label of each pixel is mapped to its corresponding point in the initial point cloud model to generate a 3D semantic point cloud model.

7. A three-dimensional modeling device for MEP equipment, characterized in that, include: The acquisition module is used to acquire the two-dimensional image training set of the MEP device; The training prediction module is used to input the image to be modeled into the semantic segmentation model trained using the two-dimensional image training set, and predict the mask image. The depth estimation module is used to estimate the depth of the image to be modeled and generate an initial point cloud model, including: The SfM algorithm is used to match feature points in the captured frame images to be modeled, sparse reconstruction of the point cloud is performed, and the camera pose is calculated. The fusion module is used to complete the semantic annotation of 3D space by fusing the mask image with the initial point cloud model, forming a 3D semantic point cloud model: The mask image with semantic labels is associated with the initial point cloud model to find the corresponding pixels; The semantically labeled mask image is associated with the initial point cloud model to find corresponding pixels, including: Each mask image is numbered and associated with the image to be modeled using the file information; Based on the image projection pose associated with the image to be modeled, the projection center of the mask image is calculated, and the camera parameters are restored. For each point, find the corresponding mask image based on the image number to be modeled and the camera number; Based on the 3D point and projection information, find the corresponding pixel in the corresponding mask image to complete the mapping from one point to multiple pixels in multiple photos.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the three-dimensional modeling method of the MEP device according to any one of claims 1 to 6 when it is run.

9. An electronic device comprising: The memory and processor are characterized in that the memory stores a computer program, wherein the processor is configured to run the computer program to perform the three-dimensional modeling method for the MEP device according to any one of claims 1 to 6.

Images