A voxel-based dynamic three-dimensional reconstruction method and computer equipment for intelligent workshops
By using semantic segmentation network in the intelligent workshop to extract and process motion targets, generate dense three-dimensional maps and convert them into voxel models, the existing three-dimensional reconstruction methods have solved the problems of low accuracy and high resource consumption in the intelligent workshop, and achieved efficient 3D reconstruction and storage optimization.
Patent Information
- Application Number
- CN202411222911.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-09-03
AI Technical Summary
The existing three-dimensional reconstruction method based on machine vision is disturbed by moving targets in intelligent workshops, resulting in low model accuracy and consumes a large amount of storage resources, making it difficult to effectively apply in edge computing scenarios.
Using the intelligent workshop dynamic three-dimensional reconstruction method based on voxels, the pixel points of the moving target are extracted through the semantic segmentation network MobileNet-V2, the position information is obtained, the dense three-dimensional map is generated and the moving targets are eliminated, and the movement is converted into a voxel-based three-dimensional model to reduce the motion Target interference and reduces storage consumption.
It significantly improves the accuracy and storage efficiency of the three-dimensional reconstruction model, and is suitable for edge computing scenarios in smart workshops.
Smart Images

Figure CN118736162B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a voxel-based dynamic three-dimensional reconstruction method and computer equipment for an intelligent workshop. Background Art
[0002] As an important carrier for promoting enterprise upgrading, improving quality and efficiency, and production management, smart workshops are an important driving force for the development of the manufacturing industry. Three-dimensional reconstruction technology is one of the key technologies in the computer field. It can play an important role in virtual monitoring of smart workshops and the establishment of metaverse models by reconstructing the surrounding scenes to obtain a virtual three-dimensional model of the environment. At the same time, the three-dimensional reconstruction method based on machine vision is the current mainstream three-dimensional reconstruction method because of its objective cost, simple implementation method, and more detailed information about the environment.
[0003] The 3D reconstruction method based on machine vision is a process of generating a 3D model by back-projecting the pixels in the image. The basic process is: first, take target images from different angles, and then generate a 3D model through these images from different angles. However, the current 3D reconstruction method based on machine vision is significantly affected by other targets in the image and cannot accurately reconstruct the 3D model of the environment. In particular, when the targets in the scene move over time, the accuracy of the reconstructed model will be seriously affected by these moving targets, that is, the current 3D reconstruction method will be strictly constrained by the static scene. On the other hand, the reconstruction granularity of the 3D reconstruction method based on machine vision is image pixels, and the computational complexity is very large. The reconstructed 3D model also consumes a lot of memory resources, which is not conducive to the application and promotion of edge computing scenarios such as smart workshops. Summary of the invention
[0004] The present invention solves the problems of motion target interference and memory consumption in existing 3D reconstruction methods, and provides a voxel-based intelligent workshop dynamic 3D reconstruction method and computer equipment. The present invention can reduce the interference of motion targets on the 3D reconstruction process, greatly improve the accuracy of the 3D reconstruction model, and greatly reduce the consumption of storage resources by the 3D reconstruction model.
[0005] The technical solution adopted by the present invention is:
[0006] A voxel-based dynamic three-dimensional reconstruction method for an intelligent workshop comprises the following steps:
[0007] Step S1, obtaining two consecutive frames of real-time workshop scene pictures A and real-time workshop scene pictures B obtained by using an image capture device, and importing the real-time workshop scene pictures A and the real-time workshop scene pictures B into the trained semantic segmentation network MobileNet-V2 for feature extraction and recognition, and extracting pixel points corresponding to the marked moving targets from the real-time workshop scene pictures A and the real-time workshop scene pictures B; wherein the moving targets include one or both of humans and robots;
[0008] Step S2, processing the pixel points marked as corresponding to the moving target in the real-time workshop scene picture A and the real-time workshop scene picture B to obtain the position and posture information of the moving target and the position and posture information of the image capture device;
[0009] Step S3, based on the position information of the image capture device, generating a dense three-dimensional map by back-projecting the real-time workshop scene picture A or the real-time workshop scene picture B;
[0010] Step S4, based on the position information of the moving target, removing the moving target from the dense three-dimensional map to form a static three-dimensional model containing only static targets; wherein the static targets include one or more of a milling machine, a lathe, a drilling machine and a workshop background;
[0011] Step S5, defining voxel grid units, and converting the static three-dimensional model into a voxel-based three-dimensional reconstruction model based on the voxel grid units.
[0012] Furthermore, the specific process of training the semantic segmentation network MobileNet-V2 in step S1 is:
[0013] Step S11, collecting an original workshop scene target data set, and labeling the original workshop scene target data set;
[0014] Step S12, expanding the original workshop scene target data set by means of data enhancement such as rotation, scaling, and translation to obtain an expanded workshop scene target data set;
[0015] Step S13, importing the expanded workshop scene target data set into the semantic segmentation network MobileNet-V2 for training to obtain training weights; retaining the semantic segmentation network MobileNet-V2 with the highest training weight to perform feature extraction and recognition.
[0016] Furthermore, the specific execution process in step S2 is:
[0017] Step S21, resetting the pixel values of the pixel points marked as the moving targets in the real-time workshop scene picture A and the real-time workshop scene picture B to 0, and obtaining the feature map FA1 and the feature map FB1 respectively;
[0018] Step S22, extracting feature points from the feature graph FA1 and the feature graph FB1 respectively, and collecting them to obtain a first feature point set and a second feature point set corresponding to the real-time workshop scene picture A and the real-time workshop scene picture B respectively;
[0019] Step S23, reset the pixel values of all pixels whose pixel values are not 0 in the feature map FA1 to 255, to form an effect map FA2; then extract the edge contour P of the moving object from the effect map FA2. c ;
[0020] Step S24, calculating the relationship between each feature point in the first feature point set and the edge contour P c If the distance D between the feature point and the edge contour P c The distance D is greater than the first preset threshold , then the corresponding feature point is determined to be a dynamic feature point, otherwise it is a static feature point; all the dynamic feature points are collected to form a dynamic feature point set Key D , collect all the static feature points to form a static feature point set Key S ;
[0021] Step S25, matching the feature points in the first feature point set with the feature points in the second feature point set, calculating the basic matrix F through the matched feature points, and calculating the right limit L through the basic matrix F R ; Then calculate the characteristic points in the second characteristic point set and the right limit L R The distance De between the feature point and the right limit L R The distance D e If the feature point is greater than the second preset threshold β, the corresponding feature point is determined to be a dynamic feature point and is added to the dynamic feature point set Key D Otherwise, it is a static feature point, which is imported into the static feature point set Key S ; Among them, the basic matrix F is expressed as: ;
[0022] Wherein, K represents the internal parameters of the image capture device;
[0023] represents the inverse matrix of K;
[0024] represents the inverse rank matrix of K;
[0025] t represents the translation matrix of the pose of the image capture device;
[0026] R represents the rotation matrix of the pose of the image capture device;
[0027] Right limit L R Calculated by the following formula: ;
[0028] Where F represents the basic matrix;
[0029] Represents the three-dimensional coordinates of the feature points;
[0030] represents the i-th feature point in the second feature point set The rank of the transformation of the homogeneous coordinates of ;
[0031] Step S26, respectively calculating the descriptor of each feature point in the first feature point set and the second feature point set, and then calculating the distance D between the descriptors of each pair of feature points in the first feature point set and the second feature point set that match each other. d , if the distance D between the descriptors of a pair of matching feature points d Greater than the third preset threshold , then the corresponding pair of feature points is determined to be a dynamic feature point, and is imported into the dynamic feature point set Key D Otherwise, it is a static feature point, which is imported into the static feature point set Key S ;
[0032] Step S27, based on the dynamic feature point set Key D and the static feature point set Key S The position and posture information of the moving target and the position and posture information of the image capture device are obtained by using an iterative closest point (ICP) algorithm.
[0033] Furthermore, the edge profile P in step S23 c It is obtained through the edge detection algorithm based on the Sobel operator.
[0034] Furthermore, the feature points in the first feature point set in step S24 and the edge contour P c The distance D is calculated by the following formula: ;
[0035] In the formula, Represents the i-th feature point in the first feature point set The minimum Euclidean distance to the edge contour Pc;
[0036] Representing feature points The pixel horizontal coordinate of
[0037] Representing feature points The pixel ordinate of
[0038] Represents the edge profile P c Pixels on The pixel horizontal coordinate of
[0039] Represents the edge profile P c Pixels on The pixel ordinate of
[0040] W G1 Represents the noise term with Gaussian distribution in the distance D calculation process.
[0041] Furthermore, the feature points in the second feature point set in step S25 are related to the right limit L R The distance D e Calculated by the following formula: ;
[0042] Where P H represents the i-th feature point in the second feature point set The homogeneous coordinates of ;
[0043] F represents the fundamental matrix;
[0044] represents the i-th feature point in the second feature point set The rank of the transformation of the homogeneous coordinates of ;
[0045] represents the i-th feature point in the second feature point set To the right limit L R distance;
[0046] represents the i-th feature point in the second feature point set The pixel coordinates of
[0047] W G2 Indicates distance D e The noise term in the calculation process conforms to the Gaussian distribution.
[0048] Furthermore, in step S26, the distance D between the descriptors of a pair of feature points in the first feature point set and the descriptors of a pair of feature points in the second feature point set is d Calculated by the following formula: ;
[0049] In the formula, Represents the i-th feature point in the first feature point set The i-th feature point in the second feature point set matched with it The Hamming distance between the descriptors of ;
[0050] Represents the i-th feature point in the first feature point set Descriptor of ;
[0051] Represents the i-th feature point in the second feature point set Descriptor of ;
[0052] W G3 Indicates distance D d The noise term in the calculation process conforms to the Gaussian distribution.
[0053] Furthermore, the specific execution process in step S4 is:
[0054] Step S41, taking the centroid of each moving target in the dense three-dimensional map as a parent node, and taking the feature points extracted from the corresponding real-time workshop scene picture A or the real-time workshop scene picture B as nodes; wherein the parent node position is calculated by the following formula: ;
[0055] In the formula, Indicates the position of the parent node;
[0056] Representing the three-dimensional coordinates of all feature points on the moving target;
[0057] n represents the total number of feature points on the corresponding moving target;
[0058] The pose matrix represents the moving target. The pose matrix size is 3×4 and is specifically composed of: ;
[0059] t represents a 1×3 translation matrix;
[0060] R represents a 3×3 rotation matrix;
[0061] Step S42, according to the position of the parent node of the corresponding moving target, calculate the distance Dis between all nodes in the current parent node domain and the parent node, and the probability P that the current node is the node of the corresponding moving target. If the probability P that the current node is the node of the corresponding moving target is less than the third preset threshold ɑ, then determine that the current node is the node of the corresponding moving target, and remove the node of the corresponding moving target from the dense three-dimensional map; wherein, the distance D between all nodes in the current parent node domain and the parent node is is And the probability P that the current node is the moving target node is calculated by the following formula: ; ;
[0062] In the formula, Represents the three-dimensional coordinates of the parent node;
[0063] Represents the three-dimensional coordinates of the current node in the domain of the current parent node;
[0064] r represents the radius of the current parent node's domain;
[0065] C represents the noise coefficient that converts distance information into probability information;
[0066] Step S43, repeat the above steps until all the nodes of the moving target are eliminated to obtain a static three-dimensional reconstructed model.
[0067] Furthermore, the specific execution process in step S5 is:
[0068] Step S51, defining a voxel grid unit corresponding to all point clouds in the static three-dimensional model, and calculating the coordinates of the corresponding voxel grid unit for each point cloud coordinate; wherein the coordinates of the voxel grid unit are calculated by the following formula: ; ; ;
[0069] In the formula, Represents the three-dimensional coordinates of the minimum boundary point in the static three-dimensional model; represents the resolution of the voxel grid cell;
[0070] Represents the three-dimensional coordinates of the voxel grid unit;
[0071] Step S52, repeating step S51, calculating the coordinates corresponding to all voxel grid units; then adding the coordinates of all voxel grid units to the corresponding voxel grid, and generating a voxel model, that is, obtaining a voxel-based three-dimensional reconstruction model.
[0072] A computer device, comprising: a processor and a memory; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the computer device executes the aforementioned voxel-based intelligent workshop dynamic three-dimensional reconstruction method.
[0073] The beneficial effects of the present invention are:
[0074] The present invention provides a voxel-based dynamic three-dimensional reconstruction method for an intelligent workshop and a computer device. In the method, after obtaining two consecutive frames of real-time workshop scene pictures A and B obtained by an image capture device, feature extraction and recognition are performed using a semantic segmentation network MobileNet-V2 to extract pixel points corresponding to moving targets, and then the posture information of the moving target and the posture information of the image capture device are obtained; after generating a dense three-dimensional map using the posture information of the image capture device, the moving target is removed from the dense three-dimensional map using the posture information of the moving target, and finally a voxel-based three-dimensional reconstruction model is obtained. The present invention can reduce the interference of moving targets on the three-dimensional reconstruction process, greatly improve the accuracy of the three-dimensional reconstruction model, and at the same time greatly reduce the consumption of storage resources by the three-dimensional reconstruction model. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0076] Figure 1 This is a flow chart of the voxel-based dynamic three-dimensional reconstruction method for an intelligent workshop in Example 1.
[0077] Figure 2 This is the original real-time workshop scene picture.
[0078] Figure 3 This is the effect image after processing the original real-time workshop scene image. DETAILED DESCRIPTION
[0079] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.
[0080] The disclosure below provides many different embodiments or examples to realize different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention.
[0081] The embodiments of the invention are described in detail below with reference to the accompanying drawings.
[0082] Embodiment 1, Figure 1 FIG. 1 is a flow chart of the method for dynamic 3D reconstruction of a smart workshop based on voxels in this embodiment. Figure 1 As shown in , the voxel-based intelligent workshop dynamic 3D reconstruction method includes the following steps:
[0083] Step S1, obtain two consecutive frames of real-time workshop scene pictures A and B obtained by using an image capture device (such as a camera), and import the real-time workshop scene pictures A and B into the trained semantic segmentation network MobileNet-V2 for feature extraction and recognition, and extract pixel points corresponding to the marked moving targets from the real-time workshop scene pictures A and B; wherein the moving targets include one or both of humans and robots. In the smart workshop, except for humans and robots, the rest, such as milling machines, lathes, drilling machines, and workshop backgrounds (workshop layout, tool racks, workbenches, etc.), are all static targets.
[0084] Specifically, the specific process of training the semantic segmentation network MobileNet-V2 in step S1 is:
[0085] Step S11, collecting an original workshop scene target data set, and labeling the original workshop scene target data set.
[0086] Step S12, the original workshop scene target data set is expanded by data enhancement methods such as rotation, scaling, and translation to obtain an expanded workshop scene target data set.
[0087] Step S13, importing the expanded workshop scene target data set into the semantic segmentation network MobileNet-V2 for training to obtain training weights; retaining the semantic segmentation network MobileNet-V2 with the highest training weight to perform feature extraction and recognition.
[0088] For a more detailed training process of the semantic segmentation network MobileNet-V2, please refer to the already disclosed invention (title: "A lightweight semantic segmentation method for targets in smart workshop scenarios", announcement number: CN114494703B, announcement date: 2022-06-28).
[0089] It should be noted that, in addition to the training method of the semantic segmentation network MobileNet-V2 described above, other training methods in the prior art for improving the recognition accuracy of the semantic segmentation network MobileNet-V2 can be used in this embodiment.
[0090] Step S2, processing the pixel points marked as corresponding to the moving target in the real-time workshop scene picture A and the real-time workshop scene picture B, and obtaining the posture information of the moving target and the posture information of the image capture device.
[0091] Specifically, the specific execution process in step S2 is:
[0092] Step S21, the pixel values of the pixel points marked as corresponding to the moving targets in the real-time workshop scene picture A and the real-time workshop scene picture B are uniformly reset to 0, and the feature map FA1 and the feature map FB1 are obtained respectively.
[0093] Step S22 , extracting feature points from the feature map FA1 and the feature map FB1 respectively, and collecting them to obtain a first feature point set and a second feature point set corresponding to the real-time workshop scene picture A and the real-time workshop scene picture B respectively.
[0094] Step S23, reset the pixel values of all pixels whose pixel values are not 0 in the feature map FA1 to 255, to form an effect map FA2; then extract the edge contour P of the moving object from the effect map FA2. c ; Wherein, the edge profile P c It is obtained through the edge detection algorithm based on the Sobel operator, which can be obtained through the following formula: ;
[0095] Where P c Represents the obtained edge contour;
[0096] Func represents the edge detection algorithm based on the Sobel operator;
[0097] FA2 represents the corresponding rendering;
[0098] mode means to retrieve only the outer contour of the trajectory.
[0099] for example Figure 2 It is an original real-time workshop scene picture, which contains two people and other workshop backgrounds. Figure 3 This is the effect image formed after the original real-time workshop scene picture is processed, with only two images.
[0100] Step S24, calculating the relationship between each feature point in the first feature point set and the edge contour P c If the distance D between the feature point and the edge contour P c The distance D is greater than the first preset threshold , then the corresponding feature point is determined to be a dynamic feature point, otherwise it is a static feature point; all the dynamic feature points are collected to form a dynamic feature point set Key D , collect all the static feature points to form a static feature point set Key S ; wherein the feature points in the first feature point set and the edge contour P c The distance D is calculated by the following formula: ;
[0101] In the formula, Represents the i-th feature point in the first feature point set The minimum Euclidean distance to the edge contour Pc;
[0102] Representing feature points The pixel horizontal coordinate of
[0103] Representing feature points The pixel ordinate of
[0104] Represents the edge profile P c Pixels on The pixel horizontal coordinate of
[0105] Represents the edge profile P c Pixels on The pixel ordinate of
[0106] W G1 Represents the noise term with Gaussian distribution in the distance D calculation process.
[0107] Step S25, matching the feature points in the first feature point set with the feature points in the second feature point set, calculating the basic matrix F through the matched feature points, and calculating the right limit L through the basic matrix F R ; Then calculate the characteristic points in the second characteristic point set and the right limit L R The distance De between the feature point and the right limit L R The distance D e If the feature point is greater than the second preset threshold β, the corresponding feature point is determined to be a dynamic feature point and is added to the dynamic feature point set Key D Otherwise, it is a static feature point, which is imported into the static feature point set Key S ; Among them, the basic matrix F is: ;
[0108] Wherein, K represents the internal parameters of the image capture device;
[0109] represents the inverse matrix of K;
[0110] represents the inverse rank matrix of K;
[0111] t represents the translation matrix of the pose of the image capture device;
[0112] R represents the rotation matrix of the pose of the image capture device;
[0113] Right limit L R and the feature points in the second feature point set and the right limit L R The distance D e Calculated by the following formula: ; ;
[0114] Where F represents the basic matrix;
[0115] Indicates L R The three-dimensional coordinates of
[0116] P H represents the i-th feature point in the second feature point set The homogeneous coordinates of ;
[0117] represents the i-th feature point in the second feature point set The rank of the transformation of the homogeneous coordinates of ; represents the i-th feature point in the second feature point set To the right limit L R distance; represents the i-th feature point in the second feature point set The pixel coordinates of
[0118] W G2 Indicates distance D e The noise term in the calculation process conforms to the Gaussian distribution.
[0119] Step S26, respectively calculating the descriptor of each feature point in the first feature point set and the second feature point set, and then calculating the distance D between the descriptors of each pair of feature points in the first feature point set and the second feature point set that match each other. d , if the distance D between the descriptors of a pair of matching feature points d Greater than the third preset threshold , then the corresponding pair of feature points is determined to be a dynamic feature point, and is imported into the dynamic feature point set Key D Otherwise, it is a static feature point, which is imported into the static feature point set Key S ; Wherein, the distance D between the descriptors of a pair of feature points in the first feature point set and the second feature point set matching each other d Calculated by the following formula: ;
[0120] In the formula, Represents the i-th feature point in the first feature point set The i-th feature point in the second feature point set matched with it The Hamming distance between the descriptors of ;
[0121] Represents the i-th feature point in the first feature point set Descriptor of ;
[0122] Represents the i-th feature point in the second feature point set Descriptor of ;
[0123] W G3 Indicates distance D d The noise term in the calculation process conforms to the Gaussian distribution.
[0124] Step S27, based on the dynamic feature point set Key D and the static feature point set Key S The position and posture information of the moving target and the position and posture information of the image capture device are obtained by using an iterative closest point (ICP) algorithm.
[0125] Step S3, based on the position information of the image capture device, generate a dense three-dimensional map by back-projecting the real-time workshop scene picture A or the real-time workshop scene picture B.
[0126] Specifically, after the image capture device obtains the position information, the two-dimensional pixel points in the real-time workshop scene picture A or the real-time workshop scene picture B are projected into the three-dimensional space. The projection process can be expressed by the following formula: ;
[0127] In the formula, Represents all nodes in the currently generated dense three-dimensional map;
[0128] Represents the dense three-dimensional map nodes generated from the start time to the time t-1;
[0129] Represents a node in the currently generated dense 3D map.
[0130] Step S4, based on the posture information of the moving target, remove the moving target from the dense three-dimensional map to form a static three-dimensional model containing only static targets; wherein the static targets include one or more of a milling machine, a lathe, a drilling machine and a workshop background.
[0131] Specifically, the specific execution process in step S4 is:
[0132] Step S41, taking the centroid of each moving target in the dense three-dimensional map as a parent node, and taking the feature points extracted from the corresponding real-time workshop scene picture A or the real-time workshop scene picture B as nodes; wherein the parent node position is calculated by the following formula: ;
[0133] In the formula, Indicates the position of the parent node;
[0134] The pose matrix represents the moving target. The pose matrix size is 3×4 and is specifically composed of: ;
[0135] t represents a 1×3 translation matrix;
[0136] R represents a 3×3 rotation matrix;
[0137] Representing the three-dimensional coordinates of all feature points on the moving target;
[0138] n represents the total number of feature points on the corresponding moving target.
[0139] Step S42, according to the position of the parent node of the corresponding moving target, calculate the distance Dis between all nodes in the current parent node domain and the parent node, and the probability P that the current node is the node of the corresponding moving target. If the probability P that the current node is the node of the corresponding moving target is less than the third preset threshold ɑ, then determine that the current node is the node of the corresponding moving target, and remove the node of the corresponding moving target from the dense three-dimensional map; wherein, the distance D between all nodes in the current parent node domain and the parent node is is And the probability P that the current node is the moving target node is calculated by the following formula: ; ;
[0140] In the formula, Represents the three-dimensional coordinates of the parent node;
[0141] Represents the three-dimensional coordinates of the current node in the domain of the current parent node;
[0142] r represents the radius of the current parent node's domain;
[0143] C represents the noise coefficient that converts distance information into probability information.
[0144] Step S43, repeat the above steps until all the nodes of the moving target are eliminated to obtain a static three-dimensional reconstructed model.
[0145] In this embodiment, there may be only one moving target or multiple moving targets in the real-time workshop scene picture A or the real-time workshop scene picture B. Therefore, in this embodiment, each moving target is processed one by one through the above-designed steps, and only static targets are retained in the end.
[0146] Step S5, defining voxel grid units, and converting the static three-dimensional model into a voxel-based three-dimensional reconstruction model based on the voxel grid units.
[0147] Specifically, the specific execution process in step S5 is:
[0148] Step S51, defining a voxel grid unit corresponding to all point clouds in the static three-dimensional model, and calculating the coordinates of the corresponding voxel grid unit for each point cloud coordinate; wherein the coordinates of the voxel grid unit are calculated by the following formula: ; ; ;
[0149] In the formula, Represents the three-dimensional coordinates of the minimum boundary point in the static three-dimensional model; represents the resolution of the voxel grid cell;
[0150] Represents the 3D coordinates of a voxel grid cell.
[0151] Step S52, repeating step S51, calculating the coordinates corresponding to all voxel grid units; then adding the coordinates of all voxel grid units to the corresponding voxel grid, and generating a voxel model, that is, obtaining a voxel-based three-dimensional reconstruction model.
[0152] The voxel-based dynamic three-dimensional reconstruction method of the intelligent workshop in this embodiment, after obtaining two consecutive frames of real-time workshop scene pictures A and real-time workshop scene pictures B obtained by using an image capture device, uses the semantic segmentation network MobileNet-V2 to perform feature extraction and recognition, extracts the pixel points corresponding to the moving target marked, and then obtains the posture information of the moving target and the posture information of the image capture device; after generating a dense three-dimensional map using the posture information of the image capture device, the posture information of the moving target is used to remove the moving target from the dense three-dimensional map, and finally obtains a voxel-based three-dimensional reconstruction model. The present invention can reduce the interference of moving targets on the three-dimensional reconstruction process, greatly improve the accuracy of the three-dimensional reconstruction model, and at the same time greatly reduce the consumption of storage resources by the three-dimensional reconstruction model.
[0153] Embodiment 2: In this embodiment, a computer device is provided, the computer device comprising: a processor and a memory. The memory is used to store computer program code, and the computer program code comprises computer instructions. When the processor executes the computer instructions, the computer device executes the voxel-based dynamic three-dimensional reconstruction method of the intelligent workshop in the aforementioned embodiment 1, and the specific execution process is as follows:
[0154] Step S1, obtaining two consecutive frames of real-time workshop scene pictures A and real-time workshop scene pictures B obtained by using an image capture device, and importing the real-time workshop scene pictures A and the real-time workshop scene pictures B into the trained semantic segmentation network MobileNet-V2 for feature extraction and recognition, and extracting pixel points corresponding to the marked moving targets from the real-time workshop scene pictures A and the real-time workshop scene pictures B; wherein the moving targets include one or both of humans and robots;
[0155] Step S2, processing the pixel points marked as corresponding to the moving target in the real-time workshop scene picture A and the real-time workshop scene picture B to obtain the position and posture information of the moving target and the position and posture information of the image capture device;
[0156] Step S3, based on the position information of the image capture device, generating a dense three-dimensional map by back-projecting the real-time workshop scene picture A or the real-time workshop scene picture B;
[0157] Step S4, based on the position information of the moving target, removing the moving target from the dense three-dimensional map to form a static three-dimensional model containing only static targets; wherein the static targets include one or more of a milling machine, a lathe, a drilling machine and a workshop background;
[0158] Step S5, defining voxel grid units, and converting the static three-dimensional model into a voxel-based three-dimensional reconstruction model based on the voxel grid units.
[0159] The present invention can reduce the interference of moving targets on the three-dimensional reconstruction process, greatly improve the accuracy of the three-dimensional reconstruction model, and at the same time greatly reduce the consumption of storage resources by the three-dimensional reconstruction model.
Claims
1. A voxel-based dynamic 3D reconstruction method for intelligent workshops. It is characterized in that The following steps are involved: Step S1, obtaining two consecutive frames of real-time workshop scene pictures A and real-time workshop scene pictures B obtained by using an image capture device, and importing the real-time workshop scene pictures A and the real-time workshop scene pictures B into the trained semantic segmentation network MobileNet-V2 for feature extraction and recognition, and extracting pixel points corresponding to the marked moving targets from the real-time workshop scene pictures A and the real-time workshop scene pictures B; wherein the moving targets include one or both of humans and robots; Step S2, processing the pixel points marked as corresponding to the moving target in the real-time workshop scene picture A and the real-time workshop scene picture B to obtain the position and posture information of the moving target and the position and posture information of the image capture device; Step S3, based on the position information of the image capture device, generating a dense three-dimensional map by back-projecting the real-time workshop scene picture A or the real-time workshop scene picture B; Step S4, based on the position information of the moving target, removing the moving target from the dense three-dimensional map to form a static three-dimensional model containing only static targets; wherein the static targets include one or more of a milling machine, a lathe, a drilling machine and a workshop background; Step S5, defining voxel grid units, and converting the static three-dimensional model into a voxel-based three-dimensional reconstruction model based on the voxel grid units; The specific execution process of step S2 is as follows: Step S21, the pixel values of the pixels marked as the moving targets in the real-time workshop scene picture A and the real-time workshop scene picture B are uniformly reset to 0, and feature maps FA and FA are obtained respectively. 1 And feature map FB 1 ; Step S22, respectively from the feature map FA 1 And the characteristic graph FB 1 Extracting feature points respectively, and collecting them to obtain a first feature point set and a second feature point set corresponding to the real-time workshop scene picture A and the real-time workshop scene picture B respectively; Step S23: the feature map FA 1 The pixel values of all pixels whose pixel values are not 0 are reset to 255, forming the effect diagram FA 2 ; Then from the effect diagram FA 2 Extract the edge contour P of the moving target c ; Step S24, calculating the relationship between each feature point in the first feature point set and the edge contour P c If the distance D between the feature point and the edge contour P c The distance D is greater than the first preset threshold The corresponding feature point is determined to be a dynamic feature point, otherwise it is a static feature point; all the dynamic feature points are collected to form a dynamic feature point set Key D , collect all the static feature points to form a static feature point set Key S ; Step S25, matching the feature points in the first feature point set with the feature points in the second feature point set, calculating the basic matrix F through the matched feature points, and calculating the right limit L through the basic matrix F R ; Then calculate the characteristic points in the second characteristic point set and the right limit L R The distance De between the feature point and the right limit L R The distance D e If the feature point is greater than the second preset threshold β, the corresponding feature point is determined to be a dynamic feature point and is added to the dynamic feature point set Key D Otherwise, it is a static feature point, which is imported into the static feature point set Key S ; Among them, the basic matrix F is expressed as: F=K -T ×t×R×K -1 Wherein, K represents the internal parameters of the image capture device; K -1 represents the inverse matrix of K; K -T represents the inverse rank matrix of K; t represents the translation matrix of the pose of the image capture device; R represents the rotation matrix of the pose of the image capture device; Right limit L R Calculated by the following formula: L R =[x,y,z]=F×P H T ; Where F represents the basic matrix; [x, y, z] represents the three-dimensional coordinates of the feature point; P H T represents the i-th feature point in the second feature point set The rank of the transformation of the homogeneous coordinates of ; Step S26, respectively calculating the descriptor of each feature point in the first feature point set and the second feature point set, and then calculating the distance D between the descriptors of each pair of feature points in the first feature point set and the second feature point set that match each other. d , if the distance D between the descriptors of a pair of matching feature points d If the pair of feature points is greater than the third preset threshold θ, the corresponding pair of feature points is determined to be a dynamic feature point and is added to the dynamic feature point set Key D Otherwise, it is a static feature point, which is imported into the static feature point set Key S ; Step S27, based on the dynamic feature point set Key D and the static feature point set Key S Acquire the position and posture information of the moving target and the position and posture information of the image capture device by using an iterative closest point ICP algorithm; The specific execution process in step S4 is: Step S41, taking the centroid of each moving target in the dense three-dimensional map as a parent node, and taking the feature points extracted from the corresponding real-time workshop scene picture A or the real-time workshop scene picture B as nodes; wherein the parent node position is calculated by the following formula: In the formula, Indicates the position of the parent node; (X i ,Y i ,Z i ) represents the three-dimensional coordinates of all feature points on the moving target; n represents the total number of feature points on the corresponding moving target; Obj_Pose represents the pose matrix of the moving target. The pose matrix size is 3×4, and its specific composition is: t represents a 1×3 translation matrix; R represents a 3×3 rotation matrix; Step S42, according to the position of the parent node of the corresponding moving target, calculate the distance Dis between all nodes in the current parent node domain and the parent node, and the probability P that the current node is the node of the corresponding moving target. If the probability P that the current node is the node of the corresponding moving target is less than the third preset threshold ɑ, then determine that the current node is the node of the corresponding moving target, and remove the node of the corresponding moving target from the dense three-dimensional map; wherein, the distance D between all nodes in the current parent node domain and the parent node is is And the probability P that the current node is the moving target node is calculated by the following formula: P=C×e -Dis ; In the formula, (N x ,N y ,N z ) represents the three-dimensional coordinates of the parent node; (x, y, z) represents the three-dimensional coordinates of the current node in the domain of the current parent node; r represents the radius of the current parent node's domain; C represents the noise coefficient that converts distance information into probability information; Step S43, repeat the above steps until all the nodes of the moving target are eliminated to obtain a static three-dimensional reconstructed model.
2. The voxel-based dynamic three-dimensional reconstruction method of an intelligent workshop according to claim 1, It is characterized in that The specific process of training the semantic segmentation network MobileNet-V2 in step S1 is: Step S11, collecting an original workshop scene target data set, and labeling the original workshop scene target data set; Step S12, expanding the original workshop scene target data set by means of data enhancement such as rotation, scaling, and translation to obtain an expanded workshop scene target data set; Step S13, importing the expanded workshop scene target data set into the semantic segmentation network MobileNet-V2 for training to obtain training weights; The semantic segmentation network MobileNet-V2 with the highest training weight is retained to perform feature extraction and recognition.
3. The voxel-based dynamic three-dimensional reconstruction method of an intelligent workshop according to claim 1, It is characterized in that The edge profile P in step S23 c It is obtained through the edge detection algorithm based on the Sobel operator.
4. The voxel-based dynamic three-dimensional reconstruction method of an intelligent workshop according to claim 1, It is characterized in that In step S24, the feature points in the first feature point set and the edge contour P c The distance D is calculated by the following formula: In the formula, D(k p (i) ,P c ) represents the i-th feature point k in the first feature point set p (i) The minimum Euclidean distance to the edge contour Pc; k p (i) (x) represents feature point k p (i) The pixel horizontal coordinate of k p (i) (y) represents feature point k p (i) The pixel ordinate of P c (j) (x) represents the edge profile P c The pixel point P on c (j) The pixel horizontal coordinate of P c (j) (y) represents the edge profile P c The pixel point P on c (j) The pixel ordinate of W G1 Represents the noise term with Gaussian distribution in the distance D calculation process.
5. The voxel-based dynamic three-dimensional reconstruction method of an intelligent workshop according to claim 1, It is characterized in that In step S25, the feature points in the second feature point set and the right limit L R The distance D e Calculated by the following formula: Where P H represents the i-th feature point in the second feature point set Homogeneous coordinates of; F represents the basic matrix; P H T represents the i-th feature point in the second feature point set The rank of the transformation of the homogeneous coordinates of ; represents the i-th feature point in the second feature point set To the right limit L R The distance; (x, y) represents the i-th feature point in the second feature point set The pixel coordinates of W G2 Indicates distance D e The noise term in the calculation process conforms to the Gaussian distribution.
6. The voxel-based dynamic three-dimensional reconstruction method of an intelligent workshop according to claim 1, It is characterized in that The distance D between the descriptors of a pair of feature points in the first feature point set and the second feature point set in step S26 is d Calculated by the following formula: Where D d (k p (i) ,k′ p (i) ) represents the i-th feature point k in the first feature point set p (i) The i-th feature point in the second feature point set matched with it The Hamming distance between the descriptors of ; des(k p (i) ) represents the i-th feature point k in the first feature point set p (i) Descriptor of ; Represents the i-th feature point in the second feature point set Descriptor of ; W G3 Indicates distance D d The noise term in the calculation process conforms to the Gaussian distribution.
7. The voxel-based dynamic three-dimensional reconstruction method of an intelligent workshop according to claim 1, It is characterized in that The specific execution process in step S5 is: Step S51, defining a voxel grid unit corresponding to all point clouds in the static three-dimensional model, and calculating the coordinates of the corresponding voxel grid unit for each point cloud coordinate; wherein the coordinates of the voxel grid unit are calculated by the following formula: In the formula, (x min ,y min ,z min ) represents the three-dimensional coordinates of the minimum boundary point in the static three-dimensional model; (size x ,size y ,size z ) represents the resolution of the voxel grid unit; (vol x ,vol y ,vol z ) represents the three-dimensional coordinates of the voxel grid unit; Step S52, repeating step S51, calculating the coordinates corresponding to all voxel grid units; then adding the coordinates of all voxel grid units to the corresponding voxel grid, and generating a voxel model, that is, obtaining a voxel-based three-dimensional reconstruction model.
8. A computer device, It is characterized in that The computer device includes: a processor and a memory; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the computer device executes the voxel-based dynamic three-dimensional reconstruction method of an intelligent workshop as described in any one of claims 1 to 7.
Citation Information
Patent Citations
A lightweight semantic segmentation method for targets in a smart workshop scene
CN114494703B
Semantic SLAM robustness improvement method based on instance segmentation
CN111581313A
Semantic SLAM method and device based on point clouds, electronic equipment and storage medium
CN113126115A
Three-dimensional reconstruction method of cross-border financial place facing element universe
CN115761177A
Dynamic scene self-positioning and online high-precision three-dimensional reconstruction method
CN117115343A
