Three-dimensional reconstruction method and related apparatus

Abstract

The embodiment of the application provides a three-dimensional reconstruction method and related devices. The method comprises: based on an image frame, extracting point data from point cloud data associated with the image frame and putting the point data into a target point data set; wherein, the associated image frame and the point cloud data are: there is an overlapping surface area between a surface area of a target object corresponding to the image frame and a surface area of the target object corresponding to the point cloud data; the point data put into the target point data set corresponds to the overlapping surface area; in the process of continuously scanning to generate the associated image frame and the point cloud data, point data meeting a specified condition in the point cloud data is added to the target point data set; and a three-dimensional reconstruction model of the target object is generated according to the point data in the target point data set. The three-dimensional reconstruction efficiency can be improved to a certain extent.

Description

Technical Field

[0001] The embodiments described in this application relate to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method and related apparatus. Background Technology

[0002] In the field of industrial 3D scanning and reconstruction, it is typically necessary to extract the point data corresponding to the target object from the point cloud data obtained from scanning, and then generate a 3D reconstructed model of the target object based on the point data. For example, when scanning a workpiece, the scanning scene often includes not only the target object itself, but also workbenches, fixtures, and other environmental clutter. In related technologies, one approach is to arrange coded points or markers on the background plane, and then remove the background point cloud after identifying the background plane based on the coded points; another approach is for the operator to manually select and delete background clutter outside the target object in point cloud processing software after scanning.

[0003] However, the relevant technologies still require a significant amount of manual work, resulting in low efficiency in 3D reconstruction. Summary of the Invention

[0004] In view of this, multiple embodiments of this application aim to provide a three-dimensional reconstruction method and related apparatus, which can improve the efficiency of three-dimensional reconstruction to a certain extent.

[0005] In a first aspect, one embodiment of this application provides a three-dimensional reconstruction method, comprising: extracting point data from point cloud data associated with an image frame and placing it into a target point dataset based on an image frame; wherein the associated image frame and point cloud data are: a surface region of a target object corresponding to the image frame and a surface region of the target object corresponding to the point cloud data that overlaps; the point data placed into the target point dataset corresponds to the overlapping surface region; during the continuous scanning process of generating associated image frames and point cloud data, adding point data from the point cloud data that meets specified conditions to the target point dataset; wherein the specified conditions are used to constrain the point data belonging to the target point dataset, requiring it to satisfy distance constraints and color constraints; the distance constraints are used to limit the distance between the point data and the point data used to characterize the boundary of the target point dataset; the color constraints are used to limit the color of the corresponding pixel of the point data belonging to the target point data in the associated image frame; and generating a three-dimensional reconstruction model of the target object based on the point data in the target point dataset.

[0006] Optionally, based on an image frame, point data is extracted from the point cloud data associated with the image frame and placed into a target point dataset, including: performing object detection on the image frame to obtain a target mask corresponding to the target object; mapping the point cloud data associated with the image frame to the image frame to obtain point data falling within the target mask, and placing the point data into the target point dataset.

[0007] Optionally, performing object detection on the image frame to obtain a target mask corresponding to the target object includes: performing foreground segmentation on the image frame to obtain a two-dimensional foreground mask of the target object in the image frame as the target mask.

[0008] Optionally, the point data in the target point dataset is subjected to gridding processing to obtain a topological grid; wherein the topological grid has an open boundary; during the continuous scanning process to generate associated image frames and point cloud data, point data in the point cloud data that meet specified conditions are added to the target point dataset, including: filtering out point data within a specified distance from the open boundary from the generated point cloud data and adding them to a candidate point dataset; if the point data in the candidate point dataset meets the specified conditions, the point data is added to the target point dataset.

[0009] Optionally, from the generated point cloud data, point data within a specified distance of the open boundary are selected and added to the candidate point dataset, including: calculating the spatial Euclidean distance between the point data in the generated point cloud data and the point data included in the open boundary; if the spatial Euclidean distance is less than the specified distance, the point data of the point cloud data are added to the candidate point dataset.

[0010] Optionally, during the continuous scanning process to generate associated image frames and point cloud data, point data that meet specified conditions in the point cloud data are added to the target point dataset, including: calculating the spatial Euclidean distance between the point data in the generated point cloud data and the point data used to characterize the boundary of the target point dataset; calculating the color Euclidean distance between the color value of the corresponding pixel of the point data in the associated image frame and the color value of the point data used to characterize the boundary of the target point dataset; and identifying the point data that meet the specified conditions based on the spatial Euclidean distance and the color Euclidean distance.

[0011] Optionally, the specified conditions include: the joint distance generated based on the spatial Euclidean distance and the color Euclidean distance is less than a specified joint constraint threshold; or, the spatial Euclidean distance is less than a specified distance threshold, and the color Euclidean distance is less than a specified color threshold.

[0012] Optionally, the joint distance is the sum of the product of the spatial Euclidean distance and the first weight, and the product of the color Euclidean distance and the second weight; wherein, the second weight is dynamically adjusted based on the geometric continuity evaluation value of the point data in the generated point cloud data and the point data used to characterize the boundary of the target point dataset in the corresponding surface region.

[0013] Optionally, the geometric continuity evaluation value is characterized by the angle between the normal vector of the surface region where the point data is located in the generated point cloud data and the normal vector of the surface region where the point data is located, which is used to characterize the boundary of the target point dataset; wherein, the second weight decreases as the angle decreases.

[0014] Secondly, one embodiment of this application provides a three-dimensional reconstruction apparatus, comprising: an extraction module, configured to extract point data from point cloud data associated with an image frame and place it into a target point dataset; wherein the associated image frame and point cloud data are: a surface region of a target object corresponding to the image frame and a surface region of the target object corresponding to the point cloud data that overlaps; the point data placed into the target point dataset corresponds to the overlapping surface region; a filtering module, configured to add point data that meets specified conditions from the point cloud data to the target point dataset during the continuous scanning process of generating associated image frames and point cloud data; wherein the specified conditions are used to constrain the point data belonging to the target point dataset to satisfy distance constraints and color constraints; the distance constraint is used to limit the distance between the point data and the point data used to characterize the boundary of the target point dataset; the color constraint is used to limit the color of the corresponding pixel of the point data belonging to the target point data in the associated image frame; and a generation module, configured to generate a three-dimensional reconstruction model of the target object based on the point data in the target point dataset.

[0015] Thirdly, one embodiment of this application also provides an electronic device, the electronic device including a memory and a processor, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the method as described above.

[0016] Fourthly, one embodiment of this application also provides a computer-readable storage medium storing at least one computer program that, when executed by a processor, can implement the method described above.

[0017] Fifthly, one embodiment of this application also provides a computer program product for implementing the method as described above.

[0018] In several embodiments provided in this application, multiple examples extract point data corresponding to overlapping surface areas from point cloud data associated with image frames and place them into a target point dataset. During continuous scanning, point data that meets specified conditions is continuously added to the target point dataset based on distance and color constraints. A three-dimensional reconstruction model of the target object is generated based on the point data in the target point dataset. This achieves continuous filtering of point data belonging to the target object during the three-dimensional reconstruction process and automatic removal of background point data in the scene where the target object is located, thereby improving the efficiency of three-dimensional reconstruction. Attached Figure Description

[0019] Figure 1 A schematic diagram of a three-dimensional reconstruction system provided in one embodiment of this application.

[0020] Figure 2 A schematic diagram of the operation of a three-dimensional reconstruction system provided in one embodiment of this application.

[0021] Figure 3 A flowchart of a method for displaying video images provided in one embodiment of this application.

[0022] Figure 4 A schematic diagram of a measuring device provided for one embodiment of this application.

[0023] Figure 5 A schematic diagram of an electronic device provided according to one embodiment of this application. Detailed Implementation

[0024] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0025] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0026] In related technologies, particularly in industrial 3D scanning and reconstruction, it is typically necessary to extract point data corresponding to the target object from the point cloud data acquired during the scanning process, and then generate a 3D reconstructed model of the target object based on the extracted point data. Taking workpiece scanning as an example, the scanning scene often includes not only the workpiece itself but also workbenches, fixtures, and other environmental clutter. Therefore, before generating the 3D reconstructed model, it is usually necessary to separate the point data corresponding to the target object from the overall point cloud data.

[0027] Specifically, one existing method involves pre-arranging coded points or markers on the background plane and identifying these points during the scanning process to determine the location of the background plane, thereby removing the corresponding point cloud data. Another existing method involves operators manually selecting and deleting point cloud data outside the target object using point cloud processing software after scanning. The former method can remove some background point cloud data to a certain extent, but it usually requires pre-arrangement of the scanning environment and is highly dependent on the physical environment. While the latter method can process background noise after scanning, it typically requires significant manual effort from operators and the process is cumbersome.

[0028] It is evident that current technologies still heavily rely on background plane recognition based on coded points or manual selection and deletion after scanning, requiring significant operator effort and resulting in low 3D reconstruction efficiency. Therefore, further improvements to these technologies are necessary.

[0029] Please see Figure 1 and Figure 2 In various embodiments of this application, the 3D reconstruction system may include a variety of electronic devices. Specifically, the 3D reconstruction system may include a tracker and a measuring device. Both the tracker and the measuring device may integrate computing chips and memory, enabling them to possess certain data processing capabilities. In some embodiments, the electronic devices of the 3D reconstruction system may further include a host computer, which can receive data provided by the tracker and the measuring device and perform data processing. The host computer may be a desktop computer, laptop computer, tablet computer, workstation, or server, etc. In some embodiments, the 3D reconstruction system may not include a tracker, and the measuring device may be a handheld scanner.

[0030] Measurement equipment can include, but is not limited to, optical and non-optical measurement equipment. Optical measurement equipment can include, but is not limited to, visible light scanners, structured light scanners, laser scanners, and light pens. Non-optical measurement equipment can include, but is not limited to, ultrasonic scanners and X-ray scanners. For example, a laser scanner can measure the distance to an object's surface by emitting a laser beam and detecting changes in the laser's reflection time or phase, thus generating high-precision scan data. Specifically, this scan data can be three-dimensional point cloud data. A structured light scanner can project a structured light pattern (such as stripes or a dot matrix) onto an object's surface and generate scan data by detecting the light reflected from the object's surface.

[0031] This application does not specify the particular type and principle of the measuring equipment.

[0032] The measuring device possesses tracking features, which can serve as its positioning characteristics. Multiple tracking features can be present and deployed at various locations on the measuring device, allowing for tracking of the device's spatial pose. In some embodiments, the tracking features may include, but are not limited to, marker points, coded points, stereo targets, geometric features of objects, and other features that can be acquired and identified by the tracker. The marker points can be reflective, in which case the tracker emits light and receives the reflected light. In some embodiments, the marker points can be luminescent, in which case the tracker can directly receive the light emitted by the marker points.

[0033] A tracker can be used based on stereo vision tracking principles to output tracking information corresponding to the position and orientation of a measuring device in space. This tracking information can be used to determine pose information representing the spatial orientation of the measuring device. Each tracker includes a camera. The number of cameras in a tracker can be one or more. Preferably, the tracker is a binocular tracker or a multi-view tracker. The number of cameras in different trackers can be the same or different. The tracker can form tracking information from images captured by the cameras. Specifically, the camera can continuously capture multiple image frames, with small time intervals between the multiple image frames, resulting in corresponding differences between the multiple image frames as the measuring device moves in space. The tracking information can include frame information. Each frame information can include image frames captured by multiple cameras of the corresponding tracker at the same time. It can be understood that each frame information can include at least one image frame.

[0034] The 3D reconstruction method can be applied to the aforementioned 3D reconstruction system. Specifically, it can be applied to one or more electronic devices within the system. Those skilled in the art can deploy the electronic devices that execute the 3D reconstruction method according to the specific circumstances.

[0035] Please see Figure 3 This application provides a three-dimensional reconstruction method. This three-dimensional reconstruction method is applied to an electronic device within a three-dimensional reconstruction system. The three-dimensional reconstruction method may include the following steps.

[0036] Step S110: Based on the image frame, extract point data from the point cloud data associated with the image frame and put it into the target point dataset; wherein, the associated image frame and point cloud data are: the surface area of ​​the target object corresponding to the image frame and the surface area of ​​the target object corresponding to the point cloud data have overlapping surface areas; the point data put into the target point dataset corresponds to the overlapping surface areas.

[0037] Step S120: During the continuous scanning process of generating associated image frames and point cloud data, point data that meet specified conditions in the point cloud data are added to the target point dataset; wherein, the specified conditions are used to constrain the point data belonging to the target point dataset, and require the satisfying of distance constraints and color constraints; the distance constraints are used to limit the distance between the point data and the point data used to characterize the boundary of the target point dataset; the color constraints are used to limit the color of the corresponding pixel in the associated image frame of the point data belonging to the target point data.

[0038] Step S130: Generate a three-dimensional reconstruction model of the target object based on the point data in the target point dataset.

[0039] In this embodiment, the electronic device can acquire image frames formed during the scanning process and point cloud data associated with the image frames. The image frames can be color image frames, such as RGB image frames, and the pixels in the image frames can carry color information representing the surface of the target object. The point cloud data can be used to represent the spatial point information of the target object's surface.

[0040] In this embodiment, the associated image frame and point cloud data can be understood as acquisition results corresponding to the same scanning acquisition process, such that there is an overlapping surface area between the surface area of ​​the target object corresponding to the image frame and the surface area of ​​the target object corresponding to the point cloud data. The electronic device can extract point data corresponding to the overlapping surface area from the point cloud data associated with the image frame based on the image frame, and put the point data into a target point dataset. The target point dataset can be understood as a data set used to store point data corresponding to the target object. Thus, the electronic device can first extract a portion of the point data corresponding to the target object from the point cloud data as the data basis for subsequent continuous expansion.

[0041] In this embodiment, the electronic device can also continuously judge the point data in the subsequently generated point cloud data during the continuous scanning and generation of associated image frames and point cloud data, and add the point data that meets the specified conditions to the target point dataset. Specifically, the specified conditions are used to constrain the point data belonging to the target point dataset to meet distance constraints and color constraints. The point data used to characterize the boundary of the target point dataset can be understood as point data located outside the target point dataset and capable of reflecting the edge range of the target point dataset. The distance constraint is used to limit the distance between the point data and the point data used to characterize the boundary of the target point dataset, thereby constraining the newly added point data to maintain continuity with the target point dataset in spatial location. The color constraint is used to limit the color of the corresponding pixel of the point data belonging to the target point data in the associated image frame, thereby constraining the newly added point data to maintain consistency with the target point dataset in color representation. Since the image frame is a color image frame, the electronic device can use the color value of the corresponding pixel of the point data in the image frame, such as the red, green, and blue color components, to determine whether the point data matches the color characteristics of the target object surface. Specifically, during continuous scanning of a workpiece, if certain point data in the subsequent point cloud data are not only spatially close to point data near the boundary of the target point dataset, but also have corresponding pixel color values ​​similar to the boundary color, the electronic device can identify these point data as still belonging to the target object and add them to the target point dataset. Conversely, for point data corresponding to workbenches, fixtures, or other environmental noise, if the distance between them and the boundary point data does not meet the distance constraint, or if the color of their corresponding pixel in the color image frame does not meet the color constraint, they can be excluded from the target point dataset. In this way, the electronic device can continuously filter point data belonging to the target object during continuous scanning and automatically remove background point data in the scene where the target object is located.

[0042] In this embodiment, after obtaining the updated target point dataset, the electronic device can generate a 3D reconstruction model of the target object based on the point data in the target point dataset. The 3D reconstruction model can be understood as a 3D model that digitally represents the outer surface morphology of the target object, which can be obtained by further fitting, stitching, or reconstructing the point data in the target point dataset. Since the point data in the target point dataset is continuously filtered during the scanning process by combining image frames and point cloud data, the point data used to generate the 3D reconstruction model can more effectively represent the target object itself, while reducing interference from background point data. Thus, the electronic device can gradually extract and expand the target object's point data without requiring extensive manual selection and deletion of background noise by the operator, and ultimately generate a 3D reconstruction model of the target object.

[0043] In some implementations, the electronic device can perform object detection on the image frame to obtain a target mask corresponding to the target object; map the point cloud data associated with the image frame to the image frame to obtain point data falling within the target mask, and put the point data into the target point dataset.

[0044] In this embodiment, the electronic device can perform object detection on the image frame to obtain a target mask corresponding to the target object, and then extract point data from the point cloud data associated with the image frame based on the target mask. The target mask can be understood as a region mask used to characterize the target area of ​​the target object in the image frame. Thus, the electronic device can first determine the two-dimensional region range corresponding to the target object in the image frame, and then extract the point data corresponding to the target object from the point cloud data accordingly. Specifically, the electronic device can perform object detection on the image frame to identify the target object in the image frame and obtain a target mask corresponding to the target object. Since the image frame can be a color image frame, the pixels in the image frame can carry color information characterizing the surface of the target object. The electronic device can combine the appearance features, color features, contour features, or texture features of the target object in the image frame to identify the target object and output a target mask covering the target object region. In some embodiments, each position in the target mask can correspond to whether the corresponding pixel position in the image frame belongs to the target object. Thus, the electronic device can determine the region range corresponding to the target object in the image frame.

[0045] In this embodiment, after obtaining the target mask, the electronic device can also map the point cloud data associated with the image frame to the image frame. Mapping the point cloud data associated with the image frame to the image frame can be understood as converting the spatial position corresponding to each point data in the point cloud data into the pixel position in the image frame. Specifically, since the image frame and the point cloud data are associated data, the image frame and the point cloud data can correspond to the same scanning acquisition process. In this case, the electronic device can, based on the calibration relationship between the imaging device corresponding to the image frame and the measuring device corresponding to the point cloud data, transform the point data in the point cloud data from three-dimensional spatial coordinates to the imaging coordinate system corresponding to the image frame, and further project it onto the image plane of the image frame to obtain the corresponding pixel position of each point data in the image frame. That is, the electronic device can map each point data in the point cloud data to the corresponding pixel position in the image frame, thereby determining whether the point data falls within the target mask.

[0046] In this embodiment, after mapping point cloud data to an image frame, the electronic device can determine the point data falling within the target mask and add the point data to the target point dataset. If the pixel position obtained by mapping a certain point data is within the area covered by the target mask, it can be determined that the point data corresponds to the target object, and the point data can be added to the target point dataset; if the pixel position obtained by mapping a certain point data is outside the target mask, the point data can be left unadded to the target point dataset.

[0047] For example, when scanning a workpiece, the electronic device can first identify the target mask corresponding to the workpiece in the image frame, then map the scanned point cloud data point by point onto the image frame, and filter out the point data that falls within the target mask corresponding to the workpiece, using it as the initial point data in the target point dataset. In this way, the electronic device can perform targeted filtering of the point cloud data based on the two-dimensional detection results of the target object in the image frame, thereby more accurately extracting the point data corresponding to the target object from the point cloud data, serving as the data foundation for subsequent continuous expansion.

[0048] In some implementations, the electronic device can perform foreground segmentation on the image frame to obtain a two-dimensional foreground mask of the target object in the image frame as the target mask.

[0049] In this embodiment, the electronic device can perform foreground segmentation on the image frame to obtain a two-dimensional foreground mask of the target object in the image frame, and use the two-dimensional foreground mask as the target mask. The two-dimensional foreground mask can be understood as a region mask used to represent the foreground region of the target object in a two-dimensional image plane. Thus, the electronic device can first distinguish the foreground region corresponding to the target object from the background region in the image frame, and then use the two-dimensional foreground mask to extract point data corresponding to the target object from the associated point cloud data.

[0050] Specifically, foreground segmentation can be used to identify foreground pixels belonging to the target object from the image frame and distinguish background pixels that do not belong to the target object. Since the image frame can be a color image frame, the pixels in the image frame can carry color information representing the surface of the target object. The electronic device can perform foreground segmentation on the target object based on features such as color distribution, contour boundaries, texture variations, and region continuity in the image frame to obtain a two-dimensional foreground mask covering the target object region. In some embodiments, each position in the two-dimensional foreground mask can be used to characterize whether the corresponding pixel position in the image frame belongs to the target object.

[0051] In some implementations, the electronic device can perform foreground segmentation processing on the image frame using an image segmentation model to output the two-dimensional foreground mask. For example, the electronic device can input the image frame into a pre-trained image segmentation network, which identifies the target object region and background region in the image frame and outputs a two-dimensional foreground mask of the target object in the image frame. Alternatively, the electronic device can combine thresholding, edge segmentation, or region growing to extract the target object region from the image frame. In this application, the specific implementation method of foreground segmentation is not limited, as long as a two-dimensional foreground mask representing the foreground region of the target object can be obtained.

[0052] In this embodiment, after obtaining the two-dimensional foreground mask, the electronic device can use the two-dimensional foreground mask as the target mask for subsequent filtering of point data falling within the target mask.

[0053] In some implementations, the electronic device can further perform gridding processing on the point data in the target point dataset to obtain a topological grid; wherein the topological grid has an open boundary; the electronic device can filter out point data within a specified distance from the open boundary from the generated point cloud data and put them into a candidate point dataset; if the point data in the candidate point dataset meets the specified conditions, the point data is put into the target point dataset.

[0054] In this embodiment, the electronic device can perform meshing processing on the point data in the target point dataset to obtain a topological mesh, and use the topological mesh to locally filter the subsequently generated point cloud data. The topological mesh can be understood as a mesh structure constructed based on the point data in the target point dataset, used to characterize the local surface connectivity of the target object. The topological mesh has open boundaries. Open boundaries can be understood as unclosed edge contours in the topological mesh, which can be used to characterize the edge range that the current target point dataset can continue to expand outwards. In this way, the electronic device can further organize the target point dataset from discrete point data into a structured surface with boundary indication, thereby providing a boundary basis for subsequent filtering of point data from the point cloud data.

[0055] In some implementations, the electronic device can locally connect the point data in the target point dataset based on the spatial adjacency relationships between the point data to form a grid cell representing the surface morphology of the target object. For example, the electronic device can connect adjacent point data using a triangular mesh construction method to form a topological mesh covering a local surface of the target object. In this application, the specific implementation method of the meshing process is not limited, as long as it can form a topological mesh with open boundaries based on the point data in the target point dataset. In this way, the electronic device can use the topological mesh to more intuitively represent the surface area currently determined to belong to the target object, as well as the edge regions that still need to be expanded.

[0056] In this embodiment, the electronic device can also filter out point data within a specified distance from the open boundary from the generated point cloud data during the continuous scanning process of generating associated image frames and point cloud data, and place the point data into a candidate point dataset. The candidate point dataset can be understood as a collection of data used to store point data that may belong to the target object and await further judgment. The specified distance can be understood as a local filtering range set around the open boundary. Since the open boundary is used to characterize the edge range where the current topological mesh can still be expanded, the electronic device can prioritize point data near the open boundary without directly judging each point in all newly generated point cloud data. Thus, the electronic device can first filter out local point data from the point cloud data that are more likely to belong to the continuous extension of the target object, narrowing the subsequent judgment range.

[0057] In this embodiment, after adding point data to the candidate point dataset, the electronic device can further determine whether the point data in the candidate point dataset meets the specified conditions. If the specified conditions are met, the point data is added to the target point dataset. That is, the electronic device can first perform local pre-screening of the point cloud data based on the open boundary, and then further identify the pre-screened point data to determine whether these point data should be incorporated into the target point dataset. Since the point data entering the candidate point dataset is already located near the open boundary, point data that meets the specified conditions can usually be added to the target point dataset as a continuous extension of the target object's edge region; correspondingly, point data that does not meet the specified conditions can be excluded from the target point dataset. In this way, the electronic device can gradually expand the point data belonging to the target object during continuous scanning, guided by the open boundary, and can reduce computational power consumption to a certain extent.

[0058] In some implementations, the electronic device can calculate the spatial Euclidean distance between the point data in the generated point cloud data and the point data included in the open boundary; if the spatial Euclidean distance is less than the specified distance, the point data of the point cloud data is placed into the candidate point dataset.

[0059] In this embodiment, the electronic device can calculate the spatial Euclidean distance between point data in the generated point cloud data and point data included in the open boundary, and add the point data in the point cloud data to the candidate point dataset if the spatial Euclidean distance is less than a specified distance. Spatial Euclidean distance can be understood as a distance metric used to characterize the spatial proximity of two points. Thus, the electronic device can filter point data from the generated point cloud data that is more likely to belong to a continuous extension of the target object based on the spatial proximity relationship between the point data and the open boundary.

[0060] In this embodiment, the electronic device can calculate the spatial Euclidean distance between each point in the generated point cloud data and the point data included in the open boundary. Since the open boundary characterizes the unclosed edge contours in the current topological mesh, point data located near the open boundary are generally more likely to correspond to subsequent extensions of the target object's edge region. In some embodiments, the electronic device can calculate the distance between a point and multiple point data included in the open boundary separately, and use the smaller spatial Euclidean distance as the spatial Euclidean distance between that point and the open boundary.

[0061] In this embodiment, if the spatial Euclidean distance between a point data and the open boundary is less than the specified distance, the electronic device can add the point data to the candidate point dataset; if the spatial Euclidean distance is greater than or equal to the specified distance, the point data can be excluded from the candidate point dataset. For example, when continuously scanning a workpiece, if some point data in the subsequently generated point cloud data are located near the current edge of the workpiece, the spatial Euclidean distance between these point data and the open boundary is usually small, and they can be added to the candidate point dataset; while point data corresponding to workbenches, fixtures, or other environmental noise points that are far from the edge of the workpiece are usually not added to the candidate point dataset. In this way, the electronic device can first perform local pre-screening through spatial Euclidean distance to narrow down the subsequent judgment range.

[0062] In some implementations, the electronic device can calculate the spatial Euclidean distance between point data in the generated point cloud data and point data used to characterize the boundary of the target point dataset; calculate the color Euclidean distance between the color value of the corresponding pixel of the point data in the generated point cloud data in the associated image frame and the color value of the point data used to characterize the boundary of the target point dataset; and identify point data that meet the specified conditions based on the spatial Euclidean distance and the color Euclidean distance.

[0063] In this embodiment, the electronic device can calculate the spatial Euclidean distance between point data in the generated point cloud data and point data used to characterize the boundary of the target point dataset, and calculate the color Euclidean distance between the color value of the corresponding pixel of the point data in the generated point cloud data in the associated image frame and the color value of the point data used to characterize the boundary of the target point dataset. Based on the spatial Euclidean distance and the color Euclidean distance, it identifies point data that meets the specified conditions. The spatial Euclidean distance can be understood as a distance metric used to characterize the degree of proximity of two points in space, and the color Euclidean distance can be understood as a distance metric used to characterize the degree of difference in the corresponding color values ​​of two points. In this way, the electronic device can jointly judge the point data in the generated point cloud data from two dimensions: spatial location and color representation.

[0064] In this embodiment, the electronic device can calculate the spatial Euclidean distance between point data in the generated point cloud data and point data used to characterize the boundary of the target point dataset. Since the point data characterizing the boundary of the target point dataset reflects the edge range of the current target point dataset, point data spatially closer to these boundary point data are generally more likely to belong to a continuous extension of the target object's edge region. In some embodiments, the electronic device can calculate the distance between a point data and multiple boundary point data separately, and determine the corresponding spatial Euclidean distance based on the calculation results. Thus, the electronic device can use the spatial Euclidean distance to characterize the spatial continuity between new point data and the current target object's edge region.

[0065] In this embodiment, the electronic device can also calculate the color Euclidean distance between the color value of the corresponding pixel in the associated image frame of the generated point cloud data and the color value of the point data used to characterize the boundary of the target point dataset. Since the image frame can be a color image frame, the pixels in the image frame can carry color information characterizing the surface of the target object. For example, the color value can include red, green, and blue color components. The electronic device can obtain the color value corresponding to the point data based on the corresponding pixel position of the point data in the associated image frame, and then calculate the distance with the color value corresponding to the boundary point data to obtain the color Euclidean distance. In this way, the electronic device can use the color Euclidean distance to characterize the consistency of the color representation between the new point data and the edge region of the current target object.

[0066] In this embodiment, the electronic device can identify point data that meets the specified conditions based on the spatial Euclidean distance and the color Euclidean distance. That is, the electronic device does not only identify point data based on the spatial positional relationship between the point data and the boundary point data, but also considers the difference between the color value corresponding to the point data and the color value corresponding to the boundary point data to comprehensively determine whether the point data should belong to the target point dataset. For example, when continuously scanning a workpiece, if a point data in the subsequently generated point cloud data is not only spatially close to point data near the workpiece boundary, but also has a color value of its corresponding pixel in the associated image frame that is close to the color value corresponding to the boundary point data, then the electronic device can identify that the point data meets the specified conditions. Correspondingly, for point data corresponding to workbench, fixtures, or other environmental noise, if although it is spatially close to the boundary area, its color appearance is significantly different from the target object boundary area, then the electronic device can identify that the point data does not meet the specified conditions. In this way, the electronic device can more accurately identify point data that should be added to the target point dataset.

[0067] In some implementations, the specified conditions include: the joint distance generated based on the spatial Euclidean distance and the color Euclidean distance is less than a specified joint constraint threshold; or, the spatial Euclidean distance is less than a specified distance threshold, and the color Euclidean distance is less than a specified color threshold.

[0068] In this embodiment, the joint distance can be understood as a distance measure used to comprehensively characterize the overall difference between point data and point data used to characterize the boundary of the target point dataset in two dimensions: spatial location and color representation. Thus, the electronic device can identify point data that meets the specified conditions using a single comprehensive criterion, or using criteria constrained by both spatial and color constraints. Specifically, the electronic device can generate a joint distance based on the spatial Euclidean distance and the color Euclidean distance, and compare the joint distance with a specified joint constraint threshold. If the joint distance is less than the specified joint constraint threshold, the corresponding point data can be identified as meeting the specified conditions; if the joint distance is greater than or equal to the specified joint constraint threshold, the corresponding point data can be identified as not meeting the specified conditions. Since the joint distance comprehensively reflects the spatial proximity and color similarity between point data and boundary point data, the electronic device can use the joint distance to make a unified judgment on point data. This facilitates a comprehensive identification of whether point data should be added to the target point dataset.

[0069] In some implementations, the electronic device may not generate the joint distance, but instead compare the spatial Euclidean distance with a specified distance threshold and the color Euclidean distance with a specified color threshold. If the spatial Euclidean distance is less than the specified distance threshold and the color Euclidean distance is less than the specified color threshold, the corresponding point data can be identified as meeting the specified conditions; if either of them does not meet the corresponding threshold requirement, the corresponding point data can be identified as not meeting the specified conditions. In this way, the electronic device can constrain the point data from both spatial location and color representation aspects.

[0070] For example, during continuous scanning of a workpiece, if a subsequently generated point data is spatially close to point data near the workpiece boundary, and the color value of its corresponding pixel in the associated image frame is also close to the color of the boundary, then regardless of whether the method of joint distance being less than a specified joint constraint threshold or the method of spatial Euclidean distance being less than a specified distance threshold and color Euclidean distance being less than a specified color threshold is used, the electronic device can identify that the point data meets the specified conditions. Conversely, if a point data is spatially close to the workpiece boundary region, but its color appearance differs significantly from the workpiece boundary region, such as point data corresponding to the worktable or fixture surface, then the electronic device can identify that the point data does not meet the specified conditions. In this way, the continuous extension of the target object and background point data can be distinguished more accurately.

[0071] In some implementations, the joint distance is the sum of the product of the spatial Euclidean distance and the first weight, and the product of the color Euclidean distance and the second weight; wherein the second weight is dynamically adjusted based on the geometric continuity evaluation value of the point data in the generated point cloud data and the point data used to characterize the boundary of the target point dataset in the corresponding surface region.

[0072] In this embodiment, the first weight can be understood as a weight parameter used to characterize the influence of spatial Euclidean distance in the joint distance, and the second weight can be understood as a weight parameter used to characterize the influence of color Euclidean distance in the joint distance. Thus, the electronic device can use the joint distance to uniformly represent the differences between point data and point data used to characterize the boundary of the target point dataset in both spatial location and color representation dimensions, thereby facilitating a comprehensive judgment on whether the point data meets the specified conditions.

[0073] In this embodiment, the electronic device can first determine the spatial Euclidean distance between the point data in the generated point cloud data and the point data used to characterize the boundary of the target point dataset, and the color Euclidean distance between the color value of the corresponding pixel in the associated image frame of the point data in the generated point cloud data and the color value of the point data used to characterize the boundary of the target point dataset. Then, the spatial Euclidean distance is multiplied by a first weight, and the color Euclidean distance is multiplied by a second weight. The products are then summed to obtain the joint distance. Since the joint distance reflects both spatial location differences and color differences, the electronic device can avoid identifying point data based solely on a single distance. This allows for a more comprehensive characterization of the proximity between the point data in the generated point cloud data and the boundary region of the current target point dataset.

[0074] In this embodiment, the second weight can be dynamically adjusted based on the geometric continuity evaluation value of the point data in the generated point cloud data and the point data used to characterize the boundary of the target point dataset in the corresponding surface region. The geometric continuity evaluation value can be understood as an evaluation quantity used to characterize the degree of continuity in the local morphology of the surface region corresponding to the point data in the generated point cloud data and the boundary point data. The electronic device can dynamically determine the degree of participation of the color Euclidean distance in the joint distance based on the geometric continuity evaluation value. That is, when there are differences in the geometric continuity of the surface region corresponding to different point data, the electronic device does not need to always introduce the color Euclidean distance with a fixed intensity, but can adaptively adjust the second weight according to the geometric continuity evaluation result. In this way, the joint distance can better conform to the actual continuous changes of the target object surface.

[0075] For example, during continuous scanning of a workpiece, if a subsequently generated point data point maintains good continuity in local morphology with the point data point near the boundary of the target point dataset, but the colors of the corresponding pixels in the image frame differ, this color difference may originate from stickers, markings, coating changes, or local texture variations on the workpiece surface, and does not necessarily indicate that the point data point has detached from the target object surface. In this case, the electronic device can dynamically adjust the second weight based on the geometric continuity evaluation value to avoid the color Euclidean distance having an excessive influence on the joint distance. Correspondingly, if the geometric continuity of the surface region corresponding to a point data point and the boundary point data is poor, the electronic device can also adjust the second weight accordingly based on the geometric continuity evaluation value to match the role of the color Euclidean distance in the joint distance with the actual situation of the current surface region. In this way, the electronic device can utilize color information to assist in identifying point data while reducing misjudgments caused by local color changes on the target object surface.

[0076] In some implementations, the geometric continuity evaluation value is characterized by the angle between the normal vector of the local surface where the point data is located in the generated point cloud data and the normal vector of the local surface where the point data is located, which is used to characterize the boundary of the target point dataset; wherein the second weight decreases as the angle decreases.

[0077] In this embodiment, the normal vector can be understood as a vector used to characterize the local orientation of the corresponding surface region. The included angle can be used to characterize the degree of continuity in local morphology between the surface region containing the point data in the generated point cloud data and the surface region containing the point data used to characterize the boundary of the target point dataset. In this way, the electronic device can transform the geometric continuity between surface regions into a calculable evaluation criterion, thereby providing a basis for the dynamic adjustment of the second weight.

[0078] In this embodiment, the electronic device can determine the normal vector of the surface region where the point data is located in the generated point cloud data, and the normal vector of the surface region where the point data is located, which characterizes the boundary of the target point dataset, and calculate the angle between the two. In some embodiments, the electronic device can perform local fitting on the corresponding surface region based on multiple point data in the neighborhood of each point data to obtain the corresponding normal vector. For example, a local plane can be fitted based on the neighborhood point data, and the normal direction of the local plane can be used as the normal vector of the corresponding surface region. If the angle is small, it can characterize that the two corresponding surface regions are smoother and more continuous in local morphology; if the angle is large, it can characterize that the changes in local morphology of the two corresponding surface regions are more obvious. In this way, the electronic device can quantify the geometric continuity evaluation value through the angle.

[0079] In this embodiment, the second weight decreases as the included angle decreases. That is, when the surface region containing the point data in the generated point cloud data is geometrically more continuous with the surface region containing the point data used to characterize the boundary of the target point dataset, the electronic device can reduce the influence of the color Euclidean distance in the joint distance. Correspondingly, when the included angle is large, the electronic device can maintain a higher level of the second weight so that the color Euclidean distance plays a more significant constraining role in the joint distance. In this way, the electronic device can adapt the second weight to the geometric continuity of the surface region.

[0080] For example, during continuous scanning of a workpiece, if a subsequently generated point data differs in color from point data near the boundary of the target point dataset in the corresponding pixel color within the image frame, but the angle between the normal vectors of their respective surface regions is small, this usually indicates that the surface regions corresponding to these point data still maintain good continuity in local morphology. In this case, the color difference may originate from stickers, markings, coating changes, or texture variations on the workpiece surface, and does not necessarily indicate that the point data has deviated from the target object surface. The electronic device can reduce the second weight to decrease the influence of color Euclidean distance on the joint distance. Conversely, if the angle between the normal vector of a point data and the surface region containing the boundary point data is large, it indicates that the surface region corresponding to that point data is geometrically discontinuous with the current target object boundary region. The electronic device can maintain a higher second weight to enhance the constraint effect of color Euclidean distance on the joint distance. Thus, the electronic device can avoid mistakenly excluding point data belonging to the target object when the local morphology of the target object surface is continuous but the color changes.

[0081] Please see Figure 4 This application provides a three-dimensional reconstruction device. The three-dimensional reconstruction device includes: an extraction module, a filtering module, and a generation module.

[0082] An extraction module is used to extract point data from point cloud data associated with an image frame and put it into a target point dataset based on the image frame; wherein, the associated image frame and point cloud data are: the surface area of ​​the target object corresponding to the image frame and the surface area of ​​the target object corresponding to the point cloud data have overlapping surface areas; the point data put into the target point dataset corresponds to the overlapping surface areas.

[0083] A filtering module is used to add point data that meets specified conditions from the point cloud data to the target point dataset during the continuous scanning process of generating associated image frames and point cloud data. The specified conditions constrain the point data belonging to the target point dataset by requiring it to satisfy distance and color constraints. The distance constraint limits the distance between the point data and the point data used to characterize the boundary of the target point dataset. The color constraint limits the color of the corresponding pixel in the associated image frame for the point data belonging to the target point data.

[0084] The generation module is used to generate a three-dimensional reconstruction model of the target object based on the point data in the target point dataset.

[0085] In this embodiment, the specific functions and effects achieved by the three-dimensional reconstruction device can be explained by referring to other embodiments of this application, and will not be repeated here.

[0086] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method as described above.

[0087] This application also provides a computer program product containing instructions that, when executed by a processor, implement the method as described above.

[0088] Please see Figure 5 Embodiments of this application may provide an electronic device, the electronic device comprising: a memory, and one or more processors communicatively connected to the memory; the memory storing instructions executable by the one or more processors, the instructions being executed by the one or more processors to cause the one or more processors to implement the method as described above.

[0089] In some embodiments, the electronic device may include a processor, a storage medium, and a communication interface connected to a system bus. The storage medium may store related computer programs.

[0090] It is understood that the specific examples in this document are only intended to help those skilled in the art better understand the embodiments of this application, and are not intended to limit the scope of the invention.

[0091] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0092] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and the implementation methods in this application are not limited in this respect.

[0093] Unless otherwise stated, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0094] It is understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0095] It is understood that the memory in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Specifically, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0096] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method implementations, and will not be repeated here.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0100] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0101] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] The above description is merely a specific embodiment of this application, but the scope of protection of this invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: Based on an image frame, point data is extracted from the point cloud data associated with the image frame and placed into a target point dataset; wherein, the associated image frame and point cloud data are: the surface area of ​​the target object corresponding to the image frame and the surface area of ​​the target object corresponding to the point cloud data have overlapping surface areas; the point data placed into the target point dataset corresponds to the overlapping surface areas; During the continuous scanning process of generating associated image frames and point cloud data, point data that meets specified conditions in the point cloud data are added to the target point dataset. The specified conditions constrain the point data belonging to the target point dataset by requiring it to satisfy distance and color constraints. The distance constraint limits the distance between the point data and the point data used to characterize the boundary of the target point dataset. The color constraint limits the color of the corresponding pixel in the associated image frame for the point data belonging to the target point data. A three-dimensional reconstruction model of the target object is generated based on the point data in the target point dataset.

2. The method according to claim 1, characterized in that, Based on an image frame, point data is extracted from the point cloud data associated with the image frame and placed into the target point dataset, including: Perform object detection on the image frame to obtain the target mask corresponding to the target object; The point cloud data associated with the image frame is mapped to the image frame to obtain point data falling within the target mask, and the point data is placed into the target point dataset.

3. The method according to claim 2, characterized in that, Perform object detection on the image frame to obtain a target mask corresponding to the target object, including: The image frame is segmented to obtain a two-dimensional foreground mask of the target object in the image frame, which is then used as the target mask.

4. The method according to claim 1, characterized in that, The method further includes: performing gridding processing on the point data in the target point dataset to obtain a topological grid; wherein the topological grid has open boundaries; During the continuous scanning process of generating associated image frames and point cloud data, point data that meet specified conditions in the point cloud data are added to the target point dataset, including: From the generated point cloud data, select point data that are within a specified distance from the open boundary and add them to the candidate point dataset; If the point data in the candidate point dataset meets the specified conditions, the point data is added to the target point dataset.

5. The method according to claim 4, characterized in that, From the generated point cloud data, point data within a specified distance of the open boundary are selected and added to the candidate point dataset, including: Calculate the spatial Euclidean distance between the point data in the generated point cloud data and the point data included in the open boundary; If the spatial Euclidean distance is less than the specified distance, the point data of the point cloud data is added to the candidate point dataset.

6. The method according to claim 1, characterized in that, During the continuous scanning process of generating associated image frames and point cloud data, point data that meet specified conditions in the point cloud data are added to the target point dataset, including: Calculate the spatial Euclidean distance between the point data in the generated point cloud data and the point data used to characterize the boundary of the target point dataset; The color Euclidean distance between the color value of the pixel corresponding to the point data in the generated point cloud data in the associated image frame and the color value of the point data used to characterize the boundary of the target point dataset is calculated. Based on the spatial Euclidean distance and the color Euclidean distance, identify point data that meet the specified conditions.

7. The method according to claim 6, characterized in that, The specified conditions include: the joint distance generated based on the spatial Euclidean distance and the color Euclidean distance is less than a specified joint constraint threshold; or... The spatial Euclidean distance is less than a specified distance threshold, and the color Euclidean distance is less than a specified color threshold.

8. The method according to claim 7, characterized in that, The joint distance is the sum of the product of the spatial Euclidean distance and the first weight, and the product of the color Euclidean distance and the second weight; The second weight is dynamically adjusted based on the geometric continuity evaluation value of the point data in the generated point cloud data and the point data used to characterize the boundary of the target point dataset in the corresponding surface region.

9. The method according to claim 8, characterized in that, The geometric continuity evaluation value is characterized by the angle between the normal vector of the surface region where the point data is located in the generated point cloud data and the normal vector of the surface region where the point data is located, which is used to characterize the boundary of the target point dataset. The second weight decreases as the included angle decreases.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, is capable of implementing the method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, The computer program product is used to implement the method as described in any one of claims 1 to 9.

Images