3D point cloud data processing system

Abstract

To provide a three-dimensional point cloud data processing system capable of generating point cloud data with reduced computational complexity.SOLUTION: A three-dimensional point cloud data processing system 10 includes: an imaging unit 22 having one or more cameras for acquiring one or more pieces of luminance information 32 by imaging an object 100; and a controller 12 for calculating point cloud data of the object 100 from the one or more pieces of luminance information 32. The controller 12 calculates at least one bounding box comprising a combination of one or more rectangles surrounding the object 100 from at least one of the one or more pieces of luminance information 32, and calculates point cloud data based on the at least one bounding box and the one or more pieces of luminance information 32.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention discloses a three-dimensional point cloud data processing system that generates three-dimensional point cloud data representing the three-dimensional shape of an object from image data obtained by capturing an image of the object. [Background technology]

[0002] In recent years, a technology has been proposed for generating three-dimensional point cloud data (hereinafter abbreviated as "point cloud data") that indicates the three-dimensional shape of an object based on image data obtained by capturing an image of the object. Such point cloud data can be used, for example, in machine tools to measure the three-dimensional shape of an object to be machined, and the results of this shape measurement can be used for detecting interference between a tool and the object to be machined, generating a tool path, measuring machining accuracy, determining the presence or absence of an object, etc.

[0003] Incidentally, when generating point cloud data, point data obtained by extracting points representing the surface of an object from image data (hereinafter, the point data extracted from image data will be referred to as "primary point data") is converted into a world coordinate system (a unique coordinate system per system; in the case of a machine tool, this may be, for example, the machine coordinate system or the workpiece coordinate system). However, the primary point data extracted from image data also contains many points unrelated to the object, i.e., noise. If there is a large amount of such noise, it is impossible to accurately grasp the shape of the object, and the data size becomes extremely large. For this reason, many techniques have been proposed to remove noise from point cloud data.

[0004] For example, Patent Document 1 discloses a technique in which design shape data of an object is stored in advance, this design shape data is compared with point cloud data, and noise is removed from the point cloud data.

[0005] Patent Document 2 proposes a technique for creating an XZ cross section from point cloud data, estimating the stratum surface from the XZ cross section, and removing points below the stratum as noise. Patent Document 3 discloses a technique for rotating point cloud data, determining the stratum surface from the rotated data, and removing point clouds with a large standard deviation from the stratum surface as noise. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-144686 [Patent Document 2] International Publication No. 2015 / 061400 [Patent Document 3] Japanese Patent Application Laid-Open No. 2005-024370 Summary of the Invention [Problem to be solved by the invention]

[0007] However, Patent Document 1 requires that design shape data be available in advance, and it is difficult to generate point cloud data for an object for which design shape data is unavailable. Furthermore, Patent Documents 2 and 3 have the problem that the amount of calculation required for estimating the geological layer and calculating the standard deviation is large, and the noise removal process takes a long time.

[0008] Therefore, this specification discloses a three-dimensional point cloud data processing system that can generate point cloud data with a smaller amount of calculation. [Means for solving the problem]

[0009] The three-dimensional point cloud data processing system disclosed in this specification comprises one or more imagers that capture an image of an object and acquire one or more pieces of brightness information, and a controller that calculates three-dimensional point cloud data of the object from the one or more pieces of brightness information, wherein the controller is configured to calculate at least one BB (bounding box) that surrounds the object and is made up of one or more rectangles from at least one of the one or more pieces of brightness information, and to calculate the three-dimensional point cloud data based on the at least one BB and the one or more pieces of brightness information.

[0010] In this case, the one or more pieces of brightness information may include first brightness information obtained by imaging the object from a first direction, and second brightness information obtained by imaging the object from a second direction different from the first direction.

[0011] The controller may be configured to calculate, from the second luminance information, a second BB surrounding the object and a second BB extrusion space obtained by extruding the second BB in the second direction, and to remove, as noise, points outside the second BB extrusion space from among the points calculated from the first luminance information.

[0012] Furthermore, the one or more pieces of luminance information may include the first luminance information obtained by capturing an image of the object from a first direction, and the controller may be configured to calculate a first BB surrounding the object based on the first luminance information, remove pixel values ​​located outside the first BB from among the multiple pixel values ​​constituting the first luminance information as noise, and calculate the three-dimensional point cloud data based on the first luminance information after the removal.

[0013] The controller may also be configured to function as an AI that has been trained in advance by machine learning, with at least one BB and one or more pieces of luminance information as inputs, and the presence or absence of a valid point in each of the multiple pixels that make up the one or more pieces of luminance information as output.

[0014] The controller may also be configured to function as an AI that has been trained in advance by machine learning, with at least one BB and one or more pieces of brightness information as input, and the presence or absence of a valid point in each of a plurality of lattice spaces formed by dividing a three-dimensional space into a three-dimensional matrix as output.

[0015] Furthermore, the one or more pieces of brightness information may further include one or more additional pieces of brightness information, and the controller may be configured to calculate the three-dimensional point cloud data based on one or more additional BBs calculated from the one or more pieces of additional brightness information, the second BB, and the first brightness information. [Effects of the Invention]

[0016] According to the technology disclosed in this specification, a BB, which is a combination of one or more rectangles, has a simple shape and requires only a small amount of calculation to calculate. By generating point cloud data based on such a BB and brightness information, point cloud data can be generated with a smaller amount of calculation. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a three-dimensional point cloud data processing system. [Figure 2] FIG. 1 is a block diagram showing the configuration of a three-dimensional point cloud data processing system. [Figure 3] FIG. 1 is a diagram illustrating a general flow of creating a three-dimensional model. [Figure 4] FIG. 10 is a schematic diagram illustrating high-precision noise removal. [Figure 5] FIG. 1 is a diagram illustrating an example of an object. [Figure 6] FIG. 10 is a diagram illustrating an example of BB. [Figure 7] FIG. 10 is a diagram showing another manner in which BB is calculated. [Figure 8] FIG. 10 is a diagram showing another example of BB. [Figure 9] FIG. 10 is a schematic diagram illustrating two-dimensional noise removal using BB. [Figure 10]FIG. 10 is a schematic diagram illustrating a BB extrusion space. [Figure 11] FIG. 10 is a schematic diagram illustrating three-dimensional noise removal. [Figure 12] FIG. 10 is a schematic diagram illustrating three-dimensional noise removal. [Figure 13] FIG. 10 is a diagram illustrating coordinate transformation in the BB extrusion space. [Figure 14] 10 is a flowchart showing the first half of the CAD data generation process. [Figure 15] 10 is a flowchart showing the second half of the flow of the CAD data generation process. [Figure 16] 10 is a flowchart showing another example of the first half of the flow of the CAD data generation process. [Figure 17] FIG. 1 is a diagram illustrating an example of the configuration of an AI. [Figure 18] 10 is a flowchart showing another example of the first half of the flow of the CAD data generation process. DETAILED DESCRIPTION OF THE INVENTION

[0018] The configuration of the three-dimensional point cloud data processing system 10 will be described below with reference to the drawings. Fig. 1 is a schematic diagram showing the configuration of the three-dimensional point cloud data processing system 10. In the example of Fig. 1, the three-dimensional point cloud data processing system 10 is used in combination with a machine tool 80.

[0019] Machine tool 80 performs predetermined processing on a workpiece to manufacture a processed product. The type of machine tool 80 is not particularly limited, and machine tool 80 may be, for example, a metal processing machine that performs metal processing (e.g., cutting, pressing, etc.) on a metal workpiece. In FIG. 1, machine tool 80 is a lathe having a spindle 81 and a tool rest. In the following description, the workpiece handled by machine tool 80 will be treated as object 100 for shape measurement. However, object 100 for shape measurement is not limited to a workpiece, and may also be other members, such as a tool or a jig.

[0020] The machine tool 80 has a numerical control device 82. The numerical control device 82 analyzes a machining program (also referred to as an "NC program") and commands the tool path for the workpiece, the steps of the work required for machining, and other numerical information consisting of numbers and codes to operate the machine tool 80. The numerical control device 82 performs various functions based on CAD data 44 indicating the shape of the workpiece (i.e., the object 100), such as checking for interference between the tool and the workpiece, generating a tool path, determining machining accuracy, determining whether the shape of the object 100 matches a reference shape, and determining whether the object 100 is located at a predetermined position. The numerical control device 82 is physically a computer having a processor and memory. The machine tool 80 also has an operation panel 84 that presents information to the operator and accepts commands from the operator. The numerical control device 82 can communicate with the controller 12 of the 3D point cloud data processing system 10 via wired or wireless communication.

[0021] The three-dimensional point cloud data processing system 10 generates point cloud data 40 and CAD data 44 that indicate the shape of an object 100, and includes a controller 12 and an imaging unit 22. The imaging unit 22 captures an image of the object 100. Image data 30 captured by the imaging unit 22 is transmitted to the controller 12. The image data 30 is information in which a brightness value is recorded for each of a plurality of pixels, i.e., brightness information 32.

[0022] The controller 12 controls the imaging process by the imaging unit 22, and generates point cloud data 40 (not shown in FIG. 1) and CAD data 44 based on the obtained image data 30. The generated CAD data 44 is transmitted to the numerical control device 82.

[0023] Next, a more detailed description will be given of the configuration of the three-dimensional point cloud data processing system 10. Fig. 2 is a block diagram showing the configuration of the three-dimensional point cloud data processing system 10. As shown in Fig. 2 and as described above, the three-dimensional point cloud data processing system 10 has a controller 12 and an imaging unit 22.

[0024] The controller 12 is physically a computer having a processor 14, memory 16, a communication I / F 18, and a UI device 20. This "computer" also includes a microcontroller in which a computer system is incorporated into a single integrated circuit. The processor 14 refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0025] The memory 16 stores various data and includes both a main memory device that the processor 14 directly accesses and an auxiliary memory device that the processor 14 accesses via an input / output channel. The main memory device is, for example, a semiconductor memory. The auxiliary memory device is, for example, a semiconductor memory or a magnetic storage device.

[0026] The communication I / F 18 transmits and receives data to and from other external electronic devices via wired or wireless communication. The communication I / F 18 communicates with, for example, a numerical control device 82 and an imaging unit 22. The UI device 20 presents various information to an operator and receives instructions from the operator. The UI device 20 includes, for example, output devices such as a display and a speaker, and input devices such as a keyboard, a mouse, a microphone, and a touch panel. While the UI device 20 is described as a component of the controller 12 in this example, some or all of the UI device 20 may be configured as a device completely separate from the controller 12. For example, some functions of the UI device 20 may be implemented by a user interface of an information terminal (e.g., a smartphone) owned by the operator and capable of communicating with the controller 12. Furthermore, the controller 12 does not need to be a single computer, but may be configured as a combination of multiple physically separated computers.

[0027] The imaging unit 22 captures an image of an object 100 present in a machining chamber of the machine tool 80. As shown in FIG. 2, the imaging unit 22 has a camera 24 and a light source 26. The camera 24 captures an image of the object 100 and generates image data 30. The image data 30 obtained by the camera 24 is sent to the controller 12. The number of cameras 24 may be one or more.

[0028] The light source 26 illuminates the object 100. The light source 26 may be a light that simply illuminates the object 100, or may be a projector that illuminates the object 100 with a predetermined pattern of light.

[0029] Next, the principle of generating point cloud data 40 of the object 100 by the three-dimensional point cloud data processing system 10 will be described with reference to FIG. 3. When generating point cloud data 40 of the object 100, the controller 12 drives the imaging unit 22 to acquire image data 30 of the object 100. At this time, the relative orientation of the object 100 with respect to the camera 24 is changed and images are captured multiple times so that image data 30 can be obtained for all surfaces of the object 100. Note that in order to change the relative orientation of the object 100 with respect to the camera 24, the object 100 may be moved or the camera 24 may be moved. In this example, the orientation of the object 100 with respect to the camera 24 is changed by driving the main shaft 81 that holds the object 100.

[0030] The controller 12 generates point cloud data 40 of the object 100 based on the obtained image data 30 (i.e., brightness information 32). In the example of Fig. 3, point cloud data 40a, 40b of the object 100 are generated from image data 30a obtained by capturing an image of the object 100 from a first direction and image data 30b obtained by capturing an image of the object 100 from a second direction different from the first direction, and further, CAD data 44 of the object 100 is generated based on these.

[0031] When generating the point cloud data 40, the presence or absence of points (hereinafter referred to as "valid points") that constitute the outer surface of the object 100 is identified for each pixel that constitutes the image data 30, and if a valid point is present, its coordinates are identified. Here, the identified coordinates are coordinates in a camera coordinate system that is based on the optical axis of the camera 24. Then, by calculating these valid points, the point cloud data 40 in the camera coordinate system is obtained.

[0032] Once point cloud data 40a, 40b in the camera coordinate system are obtained for each of the plurality of image data 30a, 30b, CAD data 44 representing the shape of the target object 100 is generated based on the point cloud data. When generating the CAD data 44, the data calculated from each of the image data 30a, 30b are combined, the point cloud data is converted to mesh data, and the mesh data is converted to solid data. The data combination may be performed before or after the conversion to mesh data. That is, the plurality of point cloud data 40a, 40b may be converted to a world coordinate system and combined, and one mesh data may be calculated from the combined point cloud data 40. Alternatively, the plurality of point cloud data 40a, 40b may be converted to mesh data, and then the mesh data may be converted to a world coordinate system and combined.

[0033] The point cloud data 40 is calculated by a light section method, a phase shift method, a trigonometry method, a stereo method, or a combination of these. When using the light section method or the phase shift method, the light source 26 irradiates the object 100 with a pattern light suitable for these methods. When using the trigonometry method, the imaging unit 22 simultaneously images one object 100 using two or more cameras 24 arranged at a distance from each other.

[0034] However, point cloud data 40 simply extracted from image data 30 contains many points unrelated to the object 100, known as noise. Numerous noise removal techniques have been proposed to remove such noise. FIG. 4 is a conceptual diagram illustrating an example of a high-precision noise removal technique. This high-precision noise removal technique divides the space containing the point cloud data 40 into multiple lattice spaces 46 arranged in a three-dimensional matrix, and calculates the point density for each lattice space 46. For lattice spaces 46 with a density below a certain level, points within the lattice space 46 are considered noise and are removed. In the example of FIG. 4, lattice space 46a has a high point density, so the points within lattice space 46a are determined to be valid points. On the other hand, lattice space 46b has a low point density, so the points within lattice space 46b are removed as noise.

[0035] While such high-precision noise removal techniques can remove noise with high precision, they have the problem of requiring a large amount of calculation, which makes it time-consuming to obtain the final point cloud data 40. Therefore, in this example, before applying the high-precision noise removal technique, a bounding box 50 (hereinafter referred to as "BB50"), which will be described later, is used to perform rough noise removal in advance. Hereinafter, this noise removal technique using BB50 will be referred to as "BB-type noise removal."

[0036] First, we will explain the BB50 used in the BB-type noise removal. Fig. 5 is a perspective view of an object 100, and Fig. 6 is a schematic diagram of image data 30 obtained by capturing an image of this object 100. BB50 is a figure that surrounds the object 100 and is formed by combining one or more rectangles. In the example of Fig. 6, a rectangle S formed by a line along the rightmost edge of the object 100, a line along the topmost edge of the object 100, a line along the leftmost edge of the object 100, and a line along the bottommost edge of the object 100 is offset outward by a predetermined amount, and a figure is extracted as BB50.

[0037] BB50 may also be configured by combining two or more rectangles. For example, as shown in Fig. 7, after extracting a first rectangle BX1 that surrounds the entire object 100, a corner region BX2 where the object 100 does not exist (i.e., where there is no brightness) is obtained within the first rectangle BX1. Then, as shown in Fig. 8, the first rectangle BX1 may be divided into multiple rectangles BX3, BX4, and BX5 based on the region BX2. These three rectangles BX3, BX4, and BX5 may then be obtained as the final BB50.

[0038] When calculating BB50, one or more image processing processes may be performed on the image data 30 to clearly identify the area in which the object 100 exists. For example, the image data 30 may be subjected to edge enhancement, solarization, binarization, morphology processing, blob processing, or a combination thereof. For example, if edge enhancement and solarization are performed on the image data 30, and then binarization and morphology processing are performed, the object in the image data 30 will appear as a continuous black block. The controller 12 extracts a circumscribing rectangle surrounding the continuous black block as the BB50. If multiple black blocks exist in one image data 30, a circumscribing rectangle is calculated for each of the multiple blocks, and the largest of the multiple rectangles is extracted as the BB50. Furthermore, if the background of the object 100 interferes with the calculation of BB50, background image data may be stored in advance, and the background may be removed from the image data 30 using a subtraction method before calculating the BB50.

[0039] Since the BB50 is a frame indicating the range in which the object 100 exists, points outside this BB50 can be eliminated as noise unrelated to the object 100. Therefore, in this example, the BB50 of the object 100 is identified from the image data 30, and points calculated from the pixel information or brightness information 32 outside this BB50 are eliminated as noise.

[0040] This noise removal process can be broadly divided into two-dimensional noise removal processes that are completed using only one image data 30, and three-dimensional noise removal processes that are performed using two or more image data 30 captured from different directions.

[0041] First, two-dimensional noise removal processing will be described with reference to FIG. 9. In this case, the controller 12 calculates a first BB 50f surrounding the object 100 captured in one piece of image data 30 (hereinafter referred to as "first image data 30f" or "first luminance information 32f") from the image data 30. Then, the controller 12 removes, as noise, pixel values ​​(i.e., luminance values ​​of the pixels) located outside the first BB 50f or points calculated from the first luminance information 32f. The controller 12 performs this processing on all captured image data 30. This two-dimensional noise removal may be performed before or after calculating the point cloud data 40. In this case, the point cloud data 40 may be expressed in the camera coordinate system or may be converted to the world coordinate system. In the example of FIG. 9, the black points or pixel values ​​are located outside the first BB 50f when viewed from the first imaging direction (i.e., the z1 direction), and therefore these black points or pixel values ​​are removed as noise.

[0042] Here, the first BB50f is a simple shape formed by straight lines parallel to the camera coordinate system. The first BB50f, which has such a simple shape, can be calculated much more easily than the outer shape of the object 100 itself. Furthermore, the amount of calculation required to determine whether a point or pixel is inside the first BB50f can be significantly reduced compared to the amount of calculation required to determine whether a point or pixel is inside the outer shape of the object 100. As a result, when noise is removed using the first BB50f, the amount of calculation can be significantly reduced compared to when noise is removed using the outer shape of the object 100 itself.

[0043] However, because the first BB 50f itself is only two-dimensional information, three-dimensional noise removal cannot be performed using the first BB 50f alone. Therefore, in this example, three-dimensional noise removal is performed by further utilizing second luminance information 32s, which is captured in a direction different from that of the first luminance information 32f. This will be described with reference to FIGS. 10 and 11. The controller 12 calculates point cloud data 40 from one piece of image data 30 (hereinafter referred to as "first image data 30f" or "first luminance information 32f"). The point cloud data 40 here may be point cloud data expressed in a camera coordinate system or point cloud data expressed in a world coordinate system. Furthermore, the controller 12 calculates second BBs 50s surrounding the object 100 and the second BB extrusion space 52s from another image (hereinafter referred to as "second image data 30s" or "second luminance information 32s") captured of the object 100 from a direction different from that of the first image data 30f. FIG. 10 is a schematic diagram showing the second BBs 50s and the second BB extrusion space 52s.

[0044] Hereinafter, the camera coordinate system in the first luminance information 32f will be referred to as the first camera coordinate system (x1, y1, z1), and the camera coordinate system in the second luminance information 32s will be referred to as the second camera coordinate system (x2, y2, z2). As shown in FIG. 9, the first BB 50f is a two-dimensional figure parallel to the x1-y1 plane of the first camera coordinate system. Also, as shown in FIG. 10, the second BB 50s is a two-dimensional figure parallel to the x2-y2 plane of the second camera coordinate system. Furthermore, the second BB extrusion space 52s has a three-dimensional shape elongated in the z2 direction.

[0045] Once the second BB extrusion space 52s is calculated, the controller 12 superimposes the second BB extrusion space 52s on the point cloud data 40 calculated from the first luminance information 32f. If this point cloud data 40 is expressed in the first camera coordinate system, the second BB extrusion space 52s is coordinate-converted to the first camera coordinate system and then superimposed on the point cloud data 40. If the point cloud data 40 is converted to the world coordinate system, the second BB extrusion space 52s is coordinate-converted to the world coordinate system and then superimposed on the point cloud data 40.

[0046] FIG. 13 shows how the second BB extrusion space 52s is coordinate-transformed into the first camera coordinate system. For ease of explanation, the second camera coordinate system is rotated 90 degrees relative to the first camera coordinate system in FIG. 13. As shown in FIG. 13, the second BB extrusion space 52s has a shape in the second camera coordinate system that has a dimension a in the x2 direction, a dimension b in the y2 direction, and an infinite size in the z2 direction. When the second BB extrusion space 52s is coordinate-transformed into the first camera coordinate system, the second BB extrusion space 52s has a shape that has a dimension a in the z1 direction, a dimension b in the y1 direction, and an infinite size in the x1 direction.

[0047] FIG. 11 shows a state in which a second BB extrusion space 52s is superimposed on point cloud data 40 obtained from the first luminance information 32f. In FIG. 11, the point cloud data 40 obtained from the first luminance information 32f has been subjected to two-dimensional noise removal using the first BB 50f in advance. The controller 12 considers points located outside the first BB extrusion space 52f in the point cloud data 40 obtained from the first luminance information 32f to be noise and deletes them. In the example of FIG. 11, points painted in black in the point cloud are removed as noise.

[0048] In this way, by overlaying the second BB extrusion space 52s calculated from the second luminance information 32s on the point cloud data 40 obtained from the first luminance information 32f, noise contained in the point cloud can be removed three-dimensionally. Meanwhile, the second BB extrusion space 52s has a simple shape obtained by simply extruding the second BB 50s in one direction. Therefore, the calculations for calculating the second BB extrusion space 52s and the calculations for determining whether a point is located outside the second BB extrusion space 52s can both be performed with a relatively small amount of calculation. As a result, this example allows noise to be removed three-dimensionally in a short time.

[0049] Here, only the second BB extrusion space 52s is superimposed on the point cloud data 40 obtained from the first luminance information 32f. However, an additional BB extrusion space may be calculated based on additional luminance information captured from another direction, and this additional BB extrusion space may be superimposed on the point cloud data 40 obtained from the first luminance information 32f in addition to the second BB extrusion space 52s. For example, as shown in FIG. 12, a third BB extrusion space 52t may be calculated from third luminance information captured from a third imaging direction and superimposed on the point cloud data 40 obtained from the first luminance information 32f. Then, points outside the third BB extrusion space 52t may be removed as noise from the point cloud. In the example of FIG. 12, by additionally superimposing the third BB extrusion space 52t, the crossed points are newly removed as noise.

[0050] The number of such additional luminance information pieces is not particularly limited. However, theoretically, if the imaging direction of the first luminance information piece 32f, which is the target of noise removal, the imaging direction of the second luminance information piece 32s, and the imaging direction of one additional luminance information piece are orthogonal to each other, it is possible to efficiently remove noise three-dimensionally with a small amount of data.

[0051] Next, a process for generating CAD data 44 of an object 100 using BB-type noise removal will be described with reference to FIGS. 14 and 15. When generating CAD data 44, first, the 3D point cloud data processing system 10 is initialized (S10). This initialization involves, for example, setting the operating parameters of the camera 24 and the light source 26, and aligning the camera coordinate system with the world coordinate system. Next, the operator sets the object 100 in a state where it can be imaged by the camera 24 (S12). In this example, the object 100 is attached to the spindle 81 of the machine tool 80.

[0052] When this state is reached, controller 12 adjusts the posture of object 100 (S14). Specifically, controller 12 instructs machine tool 80 to place object 100 in a predetermined posture. In response to this instruction, machine tool 80 rotates main shaft 81 and adjusts the posture of object 100 so that camera 24 can capture an image of object 100 from any direction.

[0053] Once the posture of the object 100 has been adjusted, the camera 24 captures an image of the object 100 (S16). Image data 30 obtained by capturing the image is sent from the camera 24 to the controller 12, where it is stored in the memory 16 as brightness information 32 linked to information about the imaging position or imaging direction. The controller 12 generates point cloud data 40 from the obtained brightness information 32 (S18). Specifically, the controller 12 determines whether or not a valid point exists for each pixel, and if a valid point exists, the z-direction position of the valid point in the camera coordinate system, using, for example, a light section method, a phase shift method, a trigonometry method, or a combination of these. The set of points thus obtained becomes the point cloud data 40. When the light section method or the phase shift method is used, the light source 26 irradiates the object 100 with a pattern light suitable for these methods.

[0054] Next, the controller 12 checks whether or not imaging of all necessary surfaces of the object 100 has been completed (S20). The number of surfaces to be imaged is often two to four of the top, bottom, right side, and left side of the object 100, but more surfaces may be imaged to improve accuracy. Furthermore, the surfaces to be imaged do not need to be orthogonal to each other, and may be surfaces at any angle.

[0055] If there are any surfaces that have not been imaged, the process returns to step S14, and the posture of the object 100 is changed, and then the object 100 is imaged and point cloud data 40 is generated (S16, S18). On the other hand, if the imaging of all surfaces has been completed, the controller 12 removes noise from the obtained point cloud data 40 (S22 to S28).

[0056] Specifically, the controller 12 first calculates the BB50 for each of the obtained image data 30 (S22). Next, a background noise removal process is performed on the point cloud data 40 obtained for each image data 30 (S24). In the background noise removal process, background image data captured without the object 100 present is acquired in advance, and a background area of ​​the image data 30 is identified as the background area based on a comparison between this background image data and the actual image data 30. Then, points in the background area are deleted all at once from the point cloud data 40 calculated from the image data. In step S24, this background noise removal process is performed on all of the obtained multiple image data 30. Note that this background noise removal process may be omitted.

[0057] Next, the controller 12 applies BB noise removal to remove noise contained in the point cloud data 40 (S26). Specifically, the controller 12 removes, as noise, points located outside the first BB 50f as viewed from the first imaging direction from the point cloud data 40f obtained from the first luminance information 32f. Note that this point cloud data 40f is point cloud data after the background noise removal process described above has been performed.

[0058] The controller 12 also extrudes the second BB 50s calculated from the second luminance information 32s in the second imaging direction to calculate a second BB extrusion space 52s. The controller 12 then performs coordinate transformation on the second BB extrusion space 52s and superimposes it on the point cloud data 40f. The controller 12 then removes points from the point cloud data 40 that are outside the second BB extrusion space 52s as noise. The same process is then performed using an additional BB extrusion space.

[0059] The controller 12 performs two-dimensional noise removal and three-dimensional noise removal using the BB50 on all of the luminance information 32 by sequentially changing the assignment of the first luminance information 32f, the second luminance information 32s, and the additional luminance information. For example, in the first noise removal, noise removal is performed using the luminance information obtained in the first imaging as "first luminance information 32f," the luminance information obtained in the second imaging as "second luminance information 32s," and the luminance information obtained in the third imaging as "additional luminance information," and in the second noise removal, noise removal is performed using the luminance information obtained in the second imaging as "first luminance information 32f," the luminance information obtained in the third imaging as "second luminance information 32s," and the luminance information obtained in the first imaging as "additional luminance information."

[0060] Once BB-type noise removal is complete, the noise contained in the point cloud data 40 has been significantly reduced, and the number of points constituting the point cloud data 40, and therefore the data size, have also been significantly reduced. In this state, the controller 12 applies high-precision noise removal to the point cloud data 40 (S28). As described above and as shown in FIG. 4, high-precision noise removal involves overlaying multiple lattice spaces 46 on the point cloud data 40 and removing noise from the point density of each lattice space 46. This high-precision noise removal requires a large amount of calculation and tends to take a long time. However, in this example, the background noise removal and BB-type noise removal have already been performed to remove the majority of noise and reduce the amount of data, so the processing time for high-precision noise removal can be significantly shortened.

[0061] Once high-precision noise removal is complete, the controller 12 thins out the points constituting the point cloud data 40 to a level necessary for generating CAD data 44 (S30). Next, the controller 12 generates mesh data from the plurality of point cloud data 40 obtained for each of the brightness information 32, and further converts this into a solid model, i.e., CAD data 44 (S30-S42). The mesh data may be generated after combining the plurality of point cloud data 40 obtained for each of the brightness information 32 (S34, S36). Alternatively, each of the plurality of point cloud data 40 may be converted into mesh data, and the resulting plurality of mesh data may be combined (S38, S40). In either case, a single mesh data representing the three-dimensional shape of the object 100, and further CAD data 44, are obtained (S42). Note that instead of meshing the point cloud data 40, it is also possible to convert it into voxels and then convert the voxelized data into CAD data 44. The generated CAD data 44 is transmitted to the machine tool 80 via the communication I / F 18.

[0062] When combining the point cloud data 40 or mesh data, each point cloud data 40 or mesh data must be converted into a world coordinate system. For this coordinate conversion, the imaging direction of the object 100 must be accurately obtained. The imaging direction may be determined from machine tool position information such as the rotation angle of the spindle 81 sent from the machine tool 80 (i.e., the holding device for the object 100), or may be determined using a plane equation. In the plane equation, a reference mark is provided on the object 100 or on a reference object whose position and orientation are invariant with respect to the object 100. Then, the orientation of the object 100 is determined from the position and shape of the reference mark in the captured image of the object 100.

[0063] Next, another example of the process of generating CAD data 44 will be described. In Fig. 14 and Fig. 15, BB noise removal is applied after generating all of the point cloud data 40. However, BB noise removal may be applied during the process of generating the point cloud data 40.

[0064] For example, steps S50 to S60 in Fig. 16 may be executed instead of steps S14 to S26 in Fig. 14. In this case, the controller 12 captures images of the object 100 while changing the posture of the object 100 (S50 to S54). Once images of all necessary surfaces have been captured, the controller 12 calculates the BB 50 and the BB extrusion space 52 for each of the plurality of luminance information 32 (S56).

[0065] Next, the controller 12 overlays the first BB 50f on the first luminance information 32f and deletes pixel values ​​located outside the first BB 50f (S58). That is, before generating point cloud data 40f from the first luminance information 32f, unnecessary information is deleted. This significantly reduces the amount of calculation required to subsequently generate point cloud data 40f. The controller 12 performs this two-dimensional noise removal on all luminance information 32 by changing the allocation of the first luminance information 32f. That is, in the first two-dimensional noise removal, the luminance information obtained in the first imaging is used as the "first luminance information 32f," and in the second two-dimensional noise removal, the luminance information obtained in the second imaging is used as the "first luminance information 32f."

[0066] Once two-dimensional noise removal is complete for all of the luminance information 32, the controller 12 generates point cloud data 40 and performs three-dimensional noise removal (S60). That is, the controller 12 determines the presence or absence of a valid point and the z-direction position of the valid point in the camera coordinate system for each pixel based on the luminance information 32. At this time, the controller 12 further determines whether each valid point is outside the BB extrusion space 52 (i.e., the second BB extrusion space 52s or the additional BB extrusion space) obtained from luminance information 32 (i.e., the second luminance information 32s or the additional luminance information) having an imaging angle different from the target luminance information 32 (i.e., the first luminance information 32f). If a valid point is outside the BB extrusion space 52 having a different imaging angle, the controller 12 determines that the valid point is noise and removes the valid point. On the other hand, if a valid point is inside the BB extrusion space 52 having a different imaging angle, the controller 12 stores the point as one of the points constituting the point cloud data 40. When the three-dimensional noise removal is complete, the process proceeds to step S28 in FIG.

[0067] In this way, by using the BB 50 to remove noise two-dimensionally before generating the point cloud data 40, i.e., in the state of the brightness information 32, it is possible to significantly reduce the calculation time required to generate the point cloud data 40. Furthermore, by using the BB extrusion space 52 to remove noise three-dimensionally during the process of generating the point cloud data 40, it is possible to reduce the amount of data in the point cloud data 40. As a result, it is possible to reduce the amount and calculation time required for subsequent high-precision noise removal and for generating CAD data.

[0068] Next, another example of the process for generating CAD data 44 will be described. In the above example, the controller 12 performs noise removal by individually calculating each point or pixel. However, noise removal may be performed by a pre-trained AI. Specifically, the AI ​​is a supervised learning device that receives the first BB 50f, the second BB 50s, and the first luminance information 32f as input and outputs the point cloud data 40 generated from the first luminance information 32f. Such an AI can be implemented as a function of the controller 12. For example, as shown in FIG. 17, the AI ​​60 may be a three-layer neural network or a four-layer or more deep neural network having one input layer 62, one or more intermediate layers 64, and one output layer 66. In the example of FIG. 17, the first imaging position, the second imaging position, the first luminance information 32f, the first BB 50f, the second BB 50s, and the second BB extrusion space 52s are input values. The output value is the existence probability of a valid point in each of the multiple pixels constituting the first luminance information 32f, or the existence probability of a valid point in each of the multiple lattice spaces 46 when the three-dimensional space is divided into multiple lattice spaces 46. When the existence probability of a valid point for each pixel is output, the controller 12 calculates the z-direction position of a point existing in each pixel for which the existence probability is equal to or greater than a certain level, and generates point cloud data 40. Furthermore, when the existence probability of a valid point for each lattice space 46 is output, the controller 12 generates point cloud data 40 by assuming that a point exists in the lattice space 46 for which the existence probability is equal to or greater than a certain level.

[0069] When noise removal is performed using such AI 60, for example, steps S70 to S80 in FIG. 18 are executed instead of steps S14 to S26 in FIG. 14. That is, the controller 12 captures images of the object 100 while changing the posture of the object 100 (S70, S72), and calculates the BB 50 and the BB extrusion space 52 from the obtained brightness information 32 (S74). Once the brightness information 32 for all surfaces has been obtained, the controller 12 inputs the obtained brightness information 32, the imaging position when each piece of brightness information 32 was obtained, and the BB 50 and the BB extrusion space 52 for each piece of brightness information 32 to the AI ​​60 (S78). Thereafter, point cloud data 40 is generated based on the output values ​​output from the AI ​​60 (S80). Thereafter, the process proceeds to step S28 in FIG. 15.

[0070] As is clear from the above explanation, by constructing an AI60 that takes the first luminance information 32f and the second BB50s as input and outputs the probability of existence of a valid point for each pixel or lattice space 46, the generation of point cloud data 40 and the noise removal process can be simplified.

[0071] Note that the configurations described so far are all examples, and other configurations may be changed as long as point cloud data 40 is generated based on at least one or more pieces of brightness information 32 and one or more BBs 50. For example, in this example, object 100 is attached to machine tool 80 and the posture of object 100 is changed using the functions of machine tool 80, but object 100 may be held by a device unrelated to machine tool 80. Also, instead of moving object 100, camera 24 may be moved. Furthermore, in the description so far, both two-dimensional noise removal using BBs 50 and three-dimensional noise removal using BB extrusion space 52 are performed, but only one of them may be performed. [Explanation of symbols]

[0072] 10 3D point cloud data processing system, 12 controller, 14 processor, 16 memory, 18 communication I / F, 20 UI device, 22 imaging unit, 24 camera, 26 light source, 30 image data, 32 brightness information, 40 point cloud data, 44 CAD data, 46 lattice space, 50 BB, 52 BB extrusion space, 62 input layer, 64 intermediate layer, 66 output layer, 80 machine tool, 81 spindle, 82 numerical control device, 84 operation panel, 100 object.

Claims

1. One or more imagers that capture an image of an object and acquire one or more pieces of luminance information; a controller that calculates three-dimensional point cloud data of the object from the one or more pieces of luminance information; wherein the controller Calculating at least one BB surrounding the object and formed by combining one or more rectangles from at least one of the one or more pieces of luminance information; Calculating the three-dimensional point cloud data based on the at least one BB and the one or more pieces of luminance information; The one or more pieces of luminance information include first luminance information obtained by capturing an image of the object from a first direction, and second luminance information obtained by capturing an image of the object from a second direction different from the first direction. A three-dimensional point cloud data processing system.

2. 2. The three-dimensional point cloud data processing system according to claim 1, The controller Calculating a second BB surrounding the object and a second BB extrusion space obtained by extruding the second BB in the second direction from the second luminance information; Among the points calculated from the first luminance information, points outside the second BB extrusion space are removed as noise. A three-dimensional point cloud data processing system configured as follows.

3. One or more imagers that capture an image of an object and acquire one or more pieces of luminance information; a controller that calculates three-dimensional point cloud data of the object from the one or more pieces of luminance information; wherein the controller Calculating at least one BB surrounding the object and formed by combining one or more rectangles from at least one of the one or more pieces of luminance information; Calculating the three-dimensional point cloud data based on the at least one BB and the one or more pieces of luminance information; the one or more pieces of luminance information include first luminance information obtained by capturing an image of the object from a first direction, The controller Calculating a first BB surrounding the object based on the first luminance information; removing, as noise, pixel values ​​located outside the first BB from among the plurality of pixel values ​​constituting the first luminance information, and calculating the three-dimensional point cloud data based on the first luminance information after the noise removal; A three-dimensional point cloud data processing system configured as follows.

4. One or more imagers that capture an image of an object and acquire one or more pieces of luminance information; a controller that calculates three-dimensional point cloud data of the object from the one or more pieces of luminance information; wherein the controller Calculating at least one BB surrounding the object and formed by combining one or more rectangles from at least one of the one or more pieces of luminance information; Calculating the three-dimensional point cloud data based on the at least one BB and the one or more pieces of luminance information; The controller is configured to function as an AI that has been machine-learned in advance, with at least one BB and one or more pieces of luminance information as inputs, and the presence or absence of a valid point in each of a plurality of pixels that constitute the one or more pieces of luminance information as output. A three-dimensional point cloud data processing system.

5. One or more imagers that capture an image of an object and acquire one or more pieces of luminance information; a controller that calculates three-dimensional point cloud data of the object from the one or more pieces of luminance information; wherein the controller Calculating at least one BB surrounding the object and formed by combining one or more rectangles from at least one of the one or more pieces of luminance information; Calculating the three-dimensional point cloud data based on the at least one BB and the one or more pieces of luminance information; The controller is configured to function as an AI that has been machine-learned in advance, with at least one BB and one or more pieces of luminance information as input, and with the presence or absence of a valid point in each of a plurality of lattice spaces formed by dividing a three-dimensional space into a three-dimensional matrix as output. A three-dimensional point cloud data processing system.

6. 3. The three-dimensional point cloud data processing system according to claim 2, the one or more pieces of luminance information further include one or more pieces of additional luminance information; The three-dimensional point cloud data processing system is characterized in that the controller is configured to calculate the three-dimensional point cloud data based on one or more additional BBs calculated from the one or more additional brightness information, the second BB, and the first brightness information.

Images