Data processing device, robot control system, and program

By dividing the distance image data into multiple segments and converting it into point group data, using different filter coefficients to process it, the problem of poor processing of coordinate information with large noise components and small noise components is solved, and high-precision Point group data processing.

CN118974768AInactive Publication Date: 2024-11-15KYOCERA CORP +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202380031513.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-30
Filing Date
2023-03-28
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When processing point group data in the measurement space, it is difficult to effectively process coordinate information with large noise components and information with small noise components, resulting in poor processing effects.

Method used

By dividing the distance image data into multiple distance image segment data, converting them into point group data, and using different filter coefficients for noise removal processing, the point group data is appropriately processed and the accuracy is improved.

Benefits of technology

High-precision processing of point group data is realized, and the error deletion of coordinate information with large noise components and the problem of unelimination of information with small noise components is reduced, thereby improving the overall processing effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118974768A_ABST
    Figure CN118974768A_ABST
Patent Text Reader

Abstract

A data processing apparatus includes a dividing unit, a converting unit, and a point group processing unit. The dividing unit divides at least a part of distance image data including distance information to a measurement point in a measurement space into a plurality of first distance image segment data in accordance with the distance information. The conversion unit individually converts each of the plurality of pieces of first distance image segment data into first point group segment data, which is point group data including coordinate information of measurement points in a measurement space. The point group processing unit individually processes each of a plurality of pieces of first point group segment data obtained from each of the plurality of pieces of first distance image segment data by the conversion unit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to data processing technology. Background Art

[0002] Patent Document 1 describes a technique related to processing of point cloud data.

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-13660 Summary of the invention

[0006] A data processing device, a robot control system, and a program are disclosed. In one embodiment, the data processing device includes a segmentation unit, a transformation unit, and a point group processing unit. The segmentation unit divides at least a portion of the range image data including the distance information to the measurement point in the measurement space into a plurality of first range image segmentation data according to the distance information. The transformation unit transforms the plurality of first range image segmentation data into first point group segmentation data, respectively, wherein the first point group segmentation data is point group data including the coordinate information of the measurement point in the measurement space. The point group processing unit processes the plurality of first point group segmentation data obtained from the plurality of first range image segmentation data in the transformation unit respectively and separately.

[0007] In one embodiment, the data processing device includes a transformation unit, a segmentation unit, and a point group processing unit. The transformation unit transforms at least a portion of the distance image data including the distance information to the measurement point in the measurement space into first point group data including the coordinate information of the measurement point in the measurement space. The segmentation unit divides at least a portion of the first point group data into a plurality of first point group data according to the coordinate value in the depth direction of the measurement space. The point group processing unit processes the plurality of first point group segment data separately.

[0008] In one embodiment, the robot control system includes the segmentation unit, the transformation unit, the point group processing unit, and the robot control unit of the data processing device. The robot control unit controls the robot based on the plurality of first point group segment data processed by the point group processing unit or the integrated point group data integrating the plurality of first point group segment data.

[0009] In one embodiment, the program causes the computer device to execute: dividing at least a portion of the range image data including distance information to the measurement point in the measurement space into a plurality of first range image segmentation data according to the distance information. In addition, the program causes the computer device to execute: transforming the plurality of first range image segmentation data into point group data including coordinate information of the measurement point in the measurement space, i.e., first point group segmentation data. In addition, the program causes the computer device to execute: processing the plurality of first point group segmentation data obtained from the plurality of first range image segmentation data separately.

[0010] In one embodiment, the program causes the computer device to execute: transforming at least a portion of the distance image data including distance information to the measurement point in the measurement space into first point group data including coordinate information of the measurement point in the measurement space. In addition, the program causes the computer device to execute: dividing at least a portion of the first point group data into a plurality of first point group segment data according to the coordinate value in the depth direction of the measurement space. In addition, the program causes the computer device to execute: processing the plurality of first point group segment data separately.

[0011] In one embodiment, the data processing device includes a determination unit, a conversion unit, and a point group processing unit. The determination unit determines, from the range image data including the distance information to the measurement point in the measurement space, first partial range image data including the distance information related to the first object in the measurement space and second partial range image data including the distance information related to the second object in the measurement space. The conversion unit converts the first partial range image data into the first point group data and converts the second partial range image data into the second point group data. The point group processing unit performs different processing on the first point group data and the second point group data.

[0012] In one embodiment, the data processing device includes a transformation unit, a determination unit, and a point group processing unit. The transformation unit transforms at least a portion of the distance image data including the distance information to the measurement point in the measurement space into point group data including the coordinate information of the measurement point in the measurement space. The determination unit determines, in the point group data, first point group data including the coordinate information related to the first object in the measurement space and second point group data including the coordinate information related to the second object in the measurement space. The point group processing unit performs different processing on the first point group data and the second point group data.

[0013] In one embodiment, the robot control system includes the determination unit, the transformation unit, the point group processing unit, and the robot control unit of the data processing device. The robot control unit controls the robot based on the first point group data and the second point group data processed by the point group processing unit, or based on the integrated point group data obtained by integrating the first point group data and the second point group data.

[0014] In one embodiment, the program causes the computer device to execute the following processing: determining, in the range image data including the distance information to the measurement point in the measurement space, first partial range image data including the distance information related to the first object in the measurement space and second partial range image data including the distance information related to the second object in the measurement space. In addition, the program causes the computer device to execute: transforming the first partial range image data into first point group data and transforming the second partial range image data into second point group data. In addition, the program causes the computer device to execute: performing different processing on the first point group data and the second point group data.

[0015] In one embodiment, the program causes the computer device to execute: transforming at least a portion of the distance image data including distance information to the measurement point in the measurement space into point group data including coordinate information of the measurement point in the measurement space. In addition, the program causes the computer device to execute: determining, in the point group data, first point group data including coordinate information related to a first object in the measurement space and second point group data including coordinate information related to a second object in the measurement space. In addition, the program causes the computer device to execute: performing different processing on the first point group data and the second point group data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram showing an example of a measurement system.

[0017] Figure 2 This is a schematic diagram showing an example of a data processing device.

[0018] Figure 3 This is a schematic diagram showing an example of a sensor device.

[0019] Figure 4 This is a schematic diagram for explaining an example of distance information included in distance image data.

[0020] Figure 5 This is a schematic diagram for explaining an example of an orthogonal coordinate system related to point group data.

[0021] Figure 6 This is a schematic diagram showing an example of point cloud data.

[0022] Figure 7 This is a schematic diagram showing an example of a control unit of a data processing device.

[0023] Figure 8 This is a schematic diagram showing an example of point group generation processing.

[0024] Fig. 9FIG. 4 is a schematic diagram showing an example of a plurality of distance divisions used in the segmentation process.

[0025] Fig.10 This is a schematic diagram showing an example of point group generation processing.

[0026] Fig.11 This is a schematic diagram showing an example of a control unit of a data processing device.

[0027] Fig.12 This is a schematic diagram showing an example of point group generation processing.

[0028] Fig.13 This is a schematic diagram showing an example of point group generation processing.

[0029] Fig.14 This is a schematic diagram showing an example of point group generation processing.

[0030] Fig.15 This is a schematic diagram showing an example of point group generation processing.

[0031] Fig.16 This is a schematic diagram showing an example of a control unit of a data processing device.

[0032] Fig.17 This is a schematic diagram showing an example of point group generation processing.

[0033] Fig.18 This is a schematic diagram showing an example of a control unit of a data processing device.

[0034] Fig.19 This is a schematic diagram showing an example of point group generation processing.

[0035] Fig. 20 This is a schematic diagram showing an example of point group generation processing.

[0036] Fig.21 This is a schematic diagram showing an example of a control unit of a data processing device.

[0037] Fig. 22 This is a schematic diagram showing an example of point group generation processing.

[0038] Fig.23 This is a schematic diagram showing an example of point group generation processing.

[0039] Fig.24 This is a schematic diagram showing an example of point group generation processing.

[0040] Fig.25 This is a schematic diagram showing an example of point group generation processing.

[0041] Fig.26 This is a schematic diagram showing an example of a robot control system. DETAILED DESCRIPTION

[0042] Implementation method 1.

[0043] Figure 1 1 is a schematic diagram showing an example of the structure of a measurement system 1 according to Embodiment 1. The measurement system 1 is, for example, a system capable of contactlessly measuring the three-dimensional position of a measurement point in a measurement space that is a space to be measured. At least one object exists in the measurement space. The measurement point is, for example, located on the surface of an object in the measurement space.

[0044] like Figure 1 As shown, the measurement system 1 includes, for example, a sensor device 2 and a data processing device 3. The sensor device 2 obtains, for example, distance information to a measurement point in the measurement space and color information of the measurement point in the measurement space. The data processing device 3 obtains the three-dimensional position of the measurement point in the measurement space based on the distance information and color information obtained by the sensor device 2.

[0045] <Configuration Example of Data Processing Device>

[0046] Figure 2 2 is a schematic diagram showing an example of the structure of the data processing device 3. Figure 2 As shown, the data processing device 3 includes, for example, a control unit 30, a first interface 31, a second interface 32, and a storage unit 33. The data processing device 3 may also be called a data processing circuit. The data processing device 3 is, for example, a type of computer device.

[0047] The first interface 31 can communicate with the sensor device 2. The first interface 31 may also be called, for example, an interface circuit. In addition, the first interface 31 may also be called, for example, a communication unit or a communication circuit.

[0048] The first interface 31 can communicate with the sensor device 2 by wire or by wireless communication. The first interface 31 can also communicate with the sensor device 2 according to at least one communication standard. The at least one communication standard according to which the first interface 31 is based can include, for example, at least one of USB (Universal Serial Bus), I2C (Inter-Integrated Circuit), CSI (Clocked Serial Interface), SPI (Serial Peripheral Interface), WiFi, and Ethernet. In addition, the first interface 31 can also communicate with the sensor device 2 through the Internet.

[0049] The second interface 32 can communicate with an external device (also referred to as an external device) of the measurement system 1. The second interface 32 can communicate with at least one external device. The second interface 32 can also be called an interface circuit, for example. In addition, the second interface 32 can also be called a communication unit or a communication circuit, for example.

[0050] The second interface 32 can communicate with the external device by wire or by wireless communication. The second interface 32 can also communicate with the external device according to at least one communication standard. The at least one communication standard according to which the second interface 32 is based can include, for example, at least one of USB, I2C, CSI, SPI, WiFi, and Ethernet. In addition, the second interface 32 can also communicate with the external device through the Internet. The at least one external device that the second interface 32 can communicate with can include a display device, and can also include a server such as a cloud server. The display device that the second interface 32 can communicate with can also be a NEDs (near-eye displays) such as a HUD (heads-up display) and an HMD (head-mounted device). In addition, at least one external device can also include a portable device. The portable device can be a portable phone such as a smart phone, a tablet terminal, a personal computer, or a wearable device. The wearable device that the second interface 32 can communicate with can be a wristband type or a watch type equipped on the arm, a headband type or a glasses type equipped on the head, or a clothing type equipped on the body.

[0051] The control unit 30 can centrally manage the operation of the data processing device 3 by controlling other components of the data processing device 3. The control unit 30 may also be referred to as a control circuit. As described in further detail below, the control unit 30 includes at least one processor to provide control and processing capabilities for executing various functions.

[0052] According to various embodiments, at least one processor may also be implemented as a single integrated circuit (IC) or as a plurality of communicatively connected integrated circuits ICs and / or discrete circuits. At least one processor may be implemented according to various known techniques.

[0053] In one embodiment, the processor includes one or more circuits or units configured to perform one or more data calculation procedures or processes by executing instructions stored in an associated memory. In other embodiments, the processor may also be firmware (e.g., discrete logic components) configured to perform one or more data calculation procedures or processes.

[0054] According to various embodiments, the processor may include one or more processors, controllers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processing devices, programmable logic devices, field programmable gate arrays, or any combination of these devices or structures, or other known devices and structures, to perform the functions described below.

[0055] The control unit 30 may include, for example, a CPU (Central Processing Unit) as a processor. The storage unit 33 may include a non-temporary recording medium such as a ROM (Read Only Memory) and a RAM (Random Access Memory) that can be read by the CPU of the control unit 30. The storage unit 33 may store, for example, a program 330 for controlling the data processing device 3. Various functions of the control unit 30 are realized by, for example, the CPU of the control unit 30 executing the program 330 in the storage unit 33.

[0056] In addition, the structure of the control unit 30 is not limited to the above example. For example, the control unit 30 may also include a plurality of CPUs. In addition, the control unit 30 may also include at least one DSP (Digital Signal Processor). In addition, all functions of the control unit 30 or a part of the functions of the control unit 30 may also be implemented by a hardware circuit that does not require software in the implementation of its functions. In addition, the storage unit 33 may also include a non-temporary recording medium that can be read by a computer other than ROM and RAM. The storage unit 33 may also include, for example, a small hard disk drive and an SSD (Solid State Drive).

[0057] <Configuration Example of Sensor Device>

[0058] Figure 3 2 is a schematic diagram showing an example of the structure of the sensor device 2. Figure 3 As shown, the sensor device 2 includes, for example, a control unit 20, an interface 21, a distance measurement sensor 22, a color camera 23, and a storage unit 24. The sensor device 2 may also be referred to as a sensor circuit, for example.

[0059] The interface 21 can communicate with the first interface 31 of the data processing device 3. The interface 21 may be called an interface circuit, for example. In addition, the interface 21 may be called a communication unit or a communication circuit. The interface 21 has the same structure as the first interface 31, for example.

[0060] The control unit 20 can manage the operation of the sensor device 2 in a unified manner by controlling other components of the sensor device 2. The control unit 20 can also be referred to as a control circuit, for example. As described in further detail below, the control unit 20 includes at least one processor in order to provide control and processing capabilities for performing various functions. The above description related to the at least one processor of the control unit 30 can also be applied to the at least one processor of the control unit 20.

[0061] The control unit 20 may include, for example, a CPU as a processor. The storage unit 24 may include a non-temporary recording medium such as a ROM and a RAM that can be read by the CPU of the control unit 20. The storage unit 24 may store, for example, a program 240 for controlling the sensor device 2. The various functions of the control unit 20 are realized by, for example, the CPU of the control unit 20 executing the program 240 in the storage unit 24.

[0062] In addition, the structure of the control unit 20 is not limited to the above example. For example, the control unit 20 may also include a plurality of CPUs. In addition, the control unit 20 may also include at least one DSP. In addition, all functions of the control unit 20 or a part of the functions of the control unit 20 may also be implemented by a hardware circuit that does not require software in the implementation of its functions. In addition, similar to the storage unit 33, the storage unit 24 may also include a non-temporary recording medium that can be read by a computer other than ROM and RAM.

[0063] The distance measurement sensor 22 is a sensor for acquiring distance information to a measurement point in the measurement space, and includes, for example, a stereo camera 220 and an irradiation unit 225. The control unit 20 acquires distance information to a measurement point in the measurement space based on the detection result of the distance measurement sensor 22.

[0064] The irradiation unit 225 irradiates an infrared pattern to the measurement space, for example, under the control of the control unit 20. The stereo camera 220 photographs the measurement space when the irradiation unit 225 irradiates the infrared pattern, under the control of the control unit 20. The stereo camera 220 can photograph the infrared pattern (also referred to as the projection pattern) projected onto the object together with the object in the measurement space. The stereo camera 220 includes a left camera and a right camera. The left camera and the right camera each include, for example, an image sensor and a lens. In each of the left camera and the right camera, light is incident on the image sensor through the lens. The image sensor includes, for example, a plurality of sensor units (also referred to as a pixel unit) arranged in a matrix. The left camera and the right camera each generate and output image data 250 representing an image of the measurement space.

[0065] The color camera 23 is controlled by the control unit 20 to perform color photography of the measurement space when the irradiation unit 225 does not irradiate the infrared pattern. The color camera 23 includes, for example, an image sensor and a lens. In the color camera 23, light enters the image sensor through the lens. The image sensor includes, for example, a plurality of sensor units arranged in a matrix. The color camera 23 generates and outputs color image data 260 representing a color image of the measurement space. A plurality of measurement points are respectively photographed in a plurality of pixels constituting the color image represented by the color image data 260.

[0066] The color image data 260 is data that two-dimensionally represents the color information of the measurement point in the measurement space. The color image data 260 includes the pixel values ​​of the plurality of pixels constituting the color image represented by it. The pixel value of a certain pixel is the color information of the measurement point photographed in the certain pixel. The pixel value of a certain pixel can also be called the color information of the measurement point corresponding to the certain pixel. The color information of the measurement point is information indicating the color of the measurement point. The color of the measurement point is represented by, for example, three primary colors. The pixel value, i.e., the color information, includes, for example, an R component (red component), a G component (green component), and a B component (blue component). Such color image data 260 is also referred to as RGB image data. In addition, the color camera 23 is also referred to as an RGB camera 23. In the color image data 260, for each of the plurality of pixels constituting the color image represented by it, the pixel position information indicating the position of the pixel and the pixel value of the pixel (in other words, color information) are associated with each other.

[0067] The color image data 260 generated by the color camera 23 is input to the control unit 20 . The control unit 20 performs, for example, filtering processing on the input color image data 260 , and inputs the resultant to the interface 21 .

[0068] The control unit 20 generates distance image data 270 representing a distance image based on the image data 250 output from the two cameras of the stereo camera 220. The plurality of pixels constituting the distance image respectively correspond to the plurality of measurement points respectively photographed in the plurality of pixels constituting the image photographed by the camera of one side of the stereo camera 220. The distance image data 270 includes the pixel values ​​of the plurality of pixels constituting the distance image. The pixel value of a certain pixel included in the distance image is the distance information representing the distance to the measurement point corresponding to the certain pixel. The distance image data 270 can be said to be data representing the distance information to the measurement point in the measurement space in two dimensions. In the distance image data 270, for each of the plurality of pixels constituting the distance image represented by the distance image, the pixel position information representing the position of the pixel and the distance information which is the pixel value of the pixel are mutually associated.

[0069] The control unit 20 inputs the generated distance image data 270 and the filtered color image data 260 to the interface 21 as an image data set 280 . The interface 21 transmits the input image data set 280 to the data processing device 3 .

[0070] The number of the plurality of measurement points corresponding to the plurality of color information included in the color image data 260 is consistent with the number of the plurality of measurement points corresponding to the plurality of distance information included in the distance image data 270. Furthermore, the color information of a pixel at a certain pixel position included in the color image data 260 and the distance information of a pixel at the same pixel position as the certain pixel position included in the distance image data 270 correspond to the same measurement point. That is, the color information of a certain measurement point included in the color image data 260 and the distance information to the certain measurement point included in the distance image data 270 correspond to the pixel at the same pixel position.

[0071] In the following description, a certain color information included in the color image data 260 and the distance information of the pixel at the same pixel position as the pixel position corresponding to the certain color information included in the distance image data 270 are expressed as corresponding to each other. In other words, the color information of a certain measurement point and the distance information to the certain measurement point are expressed as corresponding to each other.

[0072] Figure 4 2 is a schematic diagram for explaining an example of distance information included in the distance image data 270. Figure 4 , a measurement point on the surface of a certain object 900 in the measurement space 1000 is shown as an example of a measurement point 1001 in the measurement space 1000. Hereinafter, when simply referring to distance information, it refers to the distance information included in the distance image data 270.

[0073] The distance information is, for example, information indicating the distance from the stereo camera 220 to the measurement point 1001. The distance information may also be, for example, information indicating the distance from the stereo camera 220 to the measurement point 1001 along a specific direction. In this case, the distance information may also be, for example, information indicating the distance d0 (also referred to as the specific direction distance d0) from the stereo camera 220 to the measurement point 1001 in a direction perpendicular to the reference straight line 990 connecting the two cameras of the stereo camera 220. The specific direction distance d0 may be referred to as the distance from the reference straight line 990 to the measurement point 1001 in a direction perpendicular to the reference straight line 990. The reference straight line 990 may be, for example, a line connecting the centers of the imaging planes of the image sensors of the two cameras of the stereo camera 220. The specific direction distance d0 may be calculated, for example, based on the principle of triangulation using the parallax between the two cameras of the stereo camera 220 (also referred to as the inter-camera parallax), the distance between the two cameras (also referred to as the inter-camera distance), and the focal lengths of the two cameras. The inter-camera distance and the focal length are known values. The control unit 20 determines the inter-camera parallax based on the image data 250 output from the two cameras included in the stereo camera 220. Then, the control unit 20 can calculate the specific direction distance d0 using the determined inter-camera parallax and the known inter-camera distance and focal length.

[0074] The specific direction distance d0 can be referred to as the depth of the measurement point 1001 in the measurement space 1000 when viewed from the stereo camera 220. The distance information representing the specific direction distance d0 is also referred to as depth information. In addition, the distance image data 270 including the distance information representing the specific direction distance d0 is also referred to as depth image data. In addition, the distance image represented by the distance image data 270 including the distance information representing the specific direction distance d0 is also referred to as a depth image.

[0075] Alternatively, the distance information may be a straight-line distance d1 from the stereo camera 220 to the measurement point 1001 (see Figure 4 The straight-line distance d1 may be, for example, the straight-line distance from the middle point of the two cameras of the stereo camera 220 to the measurement point 1001, or the straight-line distance from one of the two cameras to the measurement point 1001. The straight-line distance d1 may be calculated based on, for example, the specific direction distance d0 and the performance of the stereo camera 220.

[0076] <Operation Example of Data Processing Device>

[0077] The control unit 30 of the data processing device 3 involved in the first embodiment is capable of transforming at least a part of the distance image data 270 output from the sensor device 2 into point group data including coordinate information of measurement points in the measurement space. Point group data is data representing a point group (also called a measurement point group) as a collection of multiple measurement points in the measurement space. Point group data can also be called data representing an object in the measurement space using multiple points. Point group data is also data representing the surface shape of an object in the measurement space. The point group data includes coordinate information representing the position of a point on the surface of the object in the measurement space. The distance between two measurement points included in the point group is, for example, the actual distance in the measurement space.

[0078] The control unit 30 can also transform at least a portion of the distance image data 270 together with at least a portion of the color image data 260 into point group data. In other words, the control unit 30 can also transform at least a portion of the image data set 280 composed of the distance image data 270 and the color image data 260 into point group data. The control unit 30 transforms at least a portion of the distance image data 270 together with at least a portion of the color image data 260 including a plurality of color information corresponding to a plurality of distance information included in the at least a portion into point group data. In this case, the point group data includes coordinate information and color information of the measurement point in the measurement space. The point group data is also data representing the surface shape and color of an object in the measurement space. In addition, the point group data includes coordinate information representing the position of a point on the surface of an object in the measurement space and color information representing the color of the point.

[0079] Hereinafter, the data of the object transformed into the point group data is sometimes referred to as the point group transformation target data. The point group transformation target data is at least a part of the range image data 270, or at least a part of the image data set 280. The point group transformation target data includes at least a part of the range image data.

[0080] When the point group transformation target data is transformed into the point group data, the number of measurement points increases by the complementation process or the like. Therefore, the number of the plurality of measurement points constituting the point group represented by the point group data is greater than the number of the plurality of measurement points corresponding to the plurality of distance information included in the point group transformation target data. For example, when the range image data 270 is entirely transformed into the point group data, the number of the plurality of measurement points constituting the point group represented by the point group data is greater than the number of the plurality of measurement points corresponding to the plurality of distance information included in the range image data 270. Furthermore, when the range image data 270 and the color image data 260 are all transformed into the point group data (in other words, when the image data set 280 is entirely transformed into the point group data), the number of the plurality of measurement points constituting the point group represented by the point group data is greater than the number of the plurality of measurement points corresponding to the plurality of color information included in the color image data 260.

[0081] The positions of the plurality of measurement points constituting the point group are represented by coordinate information in a three-dimensional orthogonal coordinate system 1100 set to include the measurement space. Figure 5 1 is a schematic diagram showing an example of a three-dimensional orthogonal coordinate system 1100 (also referred to simply as coordinate system 1100). Figure 5 , a field of view 220a of the stereo camera 220 is schematically shown. The measurement space 1000 is included in the field of view 220a.

[0082] The origin 1101 of the coordinate system 1100 is set, for example, at the stereo camera 220. The origin 1101 can be set, for example, at the middle point of the two cameras of the stereo camera 220, or can be set at other positions. The z-axis of the coordinate system 1100 is set, for example, to be parallel to the optical axis direction of the stereo camera 220, and the positive side is facing the inner side of the field of view 220a (in other words, the inner side of the measurement space 1000 when viewed from the stereo camera 220). The z-coordinate value of the coordinate system 1100 can be called the coordinate value in the depth direction of the measurement space 1000. The closer the z-coordinate value is to the inner side of the measurement space 1000, the larger its value. The x-axis of the coordinate system 1100 is set to be perpendicular to the z-axis of the coordinate system 1100, and the y-axis of the coordinate system 1100 is set to be perpendicular to the x-axis and z-axis of the coordinate system 1100.

[0083] Figure 6 2 is a schematic diagram showing an example of point group data in the case where the point group transformation target data is at least a part of the image data set 280. Figure 6As shown, the point group data includes, for example, coordinate information 800 and color information 810 in a coordinate system 1100 about a measurement point in a measurement space. In the point group data, for each measurement point, the coordinate information 800 of the measurement point and the color information 810 of the measurement point are associated with each other. In the point group data, for each point constituting the point group, the coordinate information 800 of the point and the color information 810 of the point are associated with each other. In the point group data, it can be said that for a plurality of points on the surface of an object in the measurement space, the coordinate information 800 indicating the position of the point and the color information 810 indicating the color of the point are respectively included. The coordinate information 800 includes an x-coordinate value, a y-coordinate value, and a z-coordinate value of the coordinate system 1100. The color information 810 includes an R component, a G component, and a B component. The control unit 30 generates a plurality of coordinate information 800 included in the point group data based on at least a portion of the distance image data 270 included in the point group transformation target data and performance parameters of the stereo camera 220 such as the size, angle of view, and focal length of the image sensor. Furthermore, the control unit 30 generates a plurality of color information 810 included in the point group data based on at least a portion of the color image data 260 included in the point group transformation target data and performance parameters of the stereo camera 220 such as the size, viewing angle, and focal length of the image sensor. Figure 6 Such point group data is also called colored point group data.

[0084] In addition, when the point group transformation target data is at least a part of the distance image data 270, the point group data does not include the color information 810. In the point group data, information other than the color information 810 may be associated with the coordinate information 800. For example, vector information indicating the normal vector of the certain measurement point may be associated with the coordinate information 800 of the certain measurement point, and curvature information indicating the curvature of the certain measurement point may be associated with the coordinate information 800 of the certain measurement point.

[0085] As described above, the point group data is generated based on at least a portion of the distance image data 270. Regarding the distance information included in the point group transformation target data, the larger the distance (e.g., the specific direction distance d0) represented by the distance information is, the larger the error tends to be. This is because the angular resolution of the stereo camera 220 and the spatial resolution of the infrared pattern emitted by the irradiation unit 225 decrease as they are away from the distance measurement sensor 22. Therefore, regarding the coordinate information 800 included in the point group data generated based on at least a portion of the distance image data 270, the larger the z coordinate value included therein is, the larger the noise component tends to be.

[0086] When processing is performed with a common processing parameter for a plurality of coordinate information included in such point group data, it may not be possible to properly perform processing on the point group data. For example, consider the case where noise removal processing is performed with a common processing parameter for a plurality of coordinate information included in the point group data. In this case, if the processing parameters are adjusted to improve the noise removal effect, the noise component can be properly removed for coordinate information with a large noise component, but the noise removal effect becomes too large for coordinate information with a small noise component, and it is possible that the required information is removed from the coordinate information. On the other hand, if the noise removal effect is reduced, the noise component can be properly removed for coordinate information with a small noise component, but the noise removal effect is insufficient for coordinate information with a large noise component, and it is possible that the noise component cannot be sufficiently removed from the coordinate information.

[0087] Therefore, the control unit 30 divides the point group transformation object data into a plurality of segmented data according to the distance information, and transforms each segmented data into point group data separately. Then, the control unit 30 processes each point group data obtained separately. Thus, the control unit 30 can appropriately process each point group data according to the size of the z coordinate value included therein, and can improve the accuracy of each point group data. The operation of the control unit 30 is described in detail below.

[0088] Figure 7 300, a data acquisition unit 300, a segmentation unit 301, a conversion unit 302, a point group processing unit 303, and a synthesis unit 304 as functional blocks. In addition, all functions of the data acquisition unit 300 or a part of the functions of the data acquisition unit 300 may be realized by a hardware circuit that does not require software in the realization of the functions. The same applies to the conversion unit 302, the point group processing unit 303, and the synthesis unit 304.

[0089] The data acquisition unit 300 performs an acquisition process of acquiring at least the distance image data 270 in the image data set 280 from the sensor device 2 via the first interface 31. Here, in the acquisition process, the data acquisition unit 300 acquires, for example, the distance image data 270 and the color image data 260 from the sensor device 2. In other words, the data acquisition unit 300 acquires the image data set 280 from the sensor device 2.

[0090] In the acquisition process, the data acquisition unit 300 transmits, for example, photographing start instruction information notifying an instruction to start photographing to the sensor device 2 via the first interface 31. In the sensor device 2, when the control unit 20 receives the photographing start instruction information from the interface 21, it controls the distance measurement sensor 22 and the color camera 23 so that the stereo camera 220 and the color camera 23 respectively perform photographing. Next, the control unit 20 generates the distance image data 270 based on the image data 250 from the stereo camera 220, and processes the color image data 260 from the color camera 23. Then, the control unit 20 transmits the image data set 280 composed of the distance image data 270 and the color image data 260 to the data processing device 3 via the interface 21. In the data processing device 3, the image data set 280 received from the sensor device 2 by the first interface 31 is input to the control unit 30. Thus, in the acquisition process, the data acquisition unit 300 can acquire the image data set 280 according to the transmission of the photographing start instruction information.

[0091] The segmentation unit 301 performs a segmentation process for segmenting at least a part of at least the range image data 270 acquired in the acquisition process into a plurality of segment data. Here, in the segmentation process, the segmentation unit 301 segments the image data set 280 acquired in the acquisition process into a plurality of segment data, for example.

[0092] In the segmentation process, the segmentation unit 301 divides the distance image data 270 and the color image data 260 included in the image data set 280 into a plurality of segmented data. In the segmentation process, the distance image data 270 is divided into a plurality of distance image segmented data, and the color image data 260 is divided into a plurality of color image segmented data. As described later, one distance image segmented data and one color image segmented data constitute one segmented data set. In the segmentation process, the segmentation unit 301 divides the image data set 280 into a plurality of segmented data sets. The segmented data set may also be referred to as a type of segmented data.

[0093] The conversion unit 302 performs a conversion process for converting each segment data obtained by the segmentation process into point group data. In the conversion process, for example, the conversion unit 302 converts each of the segment data sets obtained by the segmentation process into point group data. The segment data set is point group conversion target data.

[0094] The point group processing unit 303 performs point group processing for performing predetermined processing on each of the plurality of point group data obtained by the conversion processing. The integration unit 304 performs integration processing for integrating the plurality of point group data that have undergone the point group processing.

[0095] Figure 82 is a schematic diagram showing an example of a point cloud generation process for generating point cloud data based on data acquired by the sensor device 2. The point cloud generation process is executed by the control unit 30. Figure 8 In the point group generation process of the example, one point group data is generated based on the image data set 280.

[0096] like Figure 8 As shown, in the point cloud generation process, first, an acquisition process is performed to acquire the image data set 280 from the sensor device 2 .

[0097] Next, a segmentation process is performed on the image data set 280 acquired in the acquisition process to generate a plurality of segmented data sets 285. In the segmentation process, the range image data 270 included in the acquired image data set 280 is segmented into a plurality of range image segmented data 275 based on the distance information. In addition, in the segmentation process, the color image data 260 included in the acquired image data set 280 is segmented into a plurality of color image segmented data 265 based on the distance information corresponding to the color information included therein.

[0098] In the segmentation process, for example, the range of possible distances indicated by the distance information is divided into N distance divisions (N is an integer greater than or equal to 2) from the first distance division to the Nth distance division. Then, in the segmentation process, the distance image data 270 is divided into N distance image segment data 275 corresponding to the N distance divisions, and the color image data 260 is divided into N color image segment data 265 corresponding to the N distance divisions.

[0099] Fig. 9 It is a schematic diagram showing an example from the first distance division to the Nth distance division. The nth distance division (n is a variable, an integer satisfying 1≤n≤N) is, for example, a range greater than distance An-1 and less than distance An. However, the Nth distance division is exceptionally set to a range greater than distance AN-1 and less than distance AN. Distance An is greater than distance An-1. Distance A0 is, for example, the minimum value (for example, zero) of the range that the distance represented by the distance information can take. Distance AN is, for example, the maximum value of the range that the distance represented by the distance information can take. For example, the first distance division is a range greater than distance A0 and less than distance A1, and the second distance division is a range greater than distance A1 and less than distance A2.

[0100] The segmentation unit 301 includes the distance information indicating the distance included in the n-th distance division and the pixel position information corresponding to the distance information in the distance image segmentation data 275 corresponding to the n-th distance division. Thus, the plurality of distance information included in one distance image segmentation data 275 indicates the distances close to each other. In addition, the segmentation unit 301 includes the color information corresponding to the n-th distance division in the color image data 260 and the pixel position information corresponding to the color information in the color image segmentation data 265 corresponding to the n-th distance division. Here, the color information corresponding to the n-th distance division included in the color image data 260 of a certain image data set 280 refers to the color information corresponding to the distance information indicating the distance included in the n-th distance division in the distance image data 270 of the certain image data set 280. The color information corresponding to the n-th distance division included in the color image data 260 is the color information corresponding to the same measurement point as the distance information indicating the distance included in the n-th distance division in the distance image data 270.

[0101] The segmentation unit 301 processes one distance image segmentation data 275 and one color image segmentation data 265 corresponding to the same distance division as one segmentation data set 285 corresponding to the distance division. In other words, the distance image segmentation data 275 and the color image segmentation data 265 corresponding to the n-th distance division constitute one segmentation data set 285 corresponding to the n-th distance division. In the segmentation process, N segmentation data sets 285 corresponding to the N distance divisions are generated. In the segmentation process, it can also be said that the image data set 280 is divided into N segmentation data sets 285 according to the distance information. The plurality of distance image segmentation data 276 and the plurality of color image segmentation data 265 constitute a plurality of segmentation data sets 285 each consisting of one distance image segmentation data and one color image segmentation data of the same distance division.

[0102] In addition, the N distance divisions may include multiple distance divisions with different division lengths or multiple distance divisions with the same division lengths. For example, the length of the first distance division may be the same as or different from the length of the second distance division. Fig. 9 The examples are different and not continuous.

[0103] In the point cloud generation process, after the segmentation process is executed, a transformation process of transforming the segmented data set 285 into point cloud data is executed. The transformation unit 302 performs a transformation process of transforming each of the N segmented data sets 285 obtained by the segmentation process into point cloud data 290 (also referred to as point cloud segmented data 290). Figure 8 In the example of , the segmented data set 285 becomes the point group transformation target data.

[0104] In each point group segment data 290, as described above Figure 6 As shown, the plurality of coordinate information 800 and the plurality of color information 810 respectively corresponding to the plurality of coordinate information 800 are included. When the segmented data set 285 is converted into the point group segmented data 290, the number of measurement points increases by the complementation process or the like. Therefore, the number of the plurality of measurement points constituting the point group represented by the point group segmented data 290 is greater than the number of the plurality of measurement points respectively corresponding to the plurality of distance information included in the range image segmented data 275, and greater than the number of the plurality of measurement points respectively corresponding to the plurality of color information included in the color image segmented data 265. Since the plurality of distance information included in one range image segmented data 275 indicates mutually close distances, the z coordinate values ​​of the plurality of coordinate information 800 included in one point group segmented data 290 generated based on one range image segmented data 275 become mutually close values.

[0105] For example, the transformation unit 302 may perform the transformation processing for at least two of the N segmented data sets 285 in parallel. In other words, the transformation unit 302 may also perform at least two transformation processings of the plurality of segmented data sets 285 into the plurality of point group segmented data 290 in parallel. Thus, the processing time for obtaining the point group segmented data 290 can be shortened. Figure 8 As shown, the transformation unit 302 may also perform N transformation processes in parallel to transform the N segmented data sets 285 into N point group segmented data 290, respectively. In addition, the transformation unit 302 may also perform multiple transformation processes that are part of the N transformation processes in parallel. In addition, the transformation unit 302 may also perform at least two transformation processes of the N transformation processes in series. These parallel processes and serial processes may also be referred to as transformation processes performed by the transformation unit 302, which will be described later. In addition, these parallel processes and serial processes may also be referred to as point group processes performed by the transformation unit 352, which will be described later.

[0106] Thereafter, point cloud segment data 290 obtained by transforming the segment data set 285 corresponding to the n-th distance division is set as point cloud segment data 290 corresponding to the n-th distance division. The transforming unit 302 generates N point cloud segment data 290 corresponding to the first to N-th distance divisions, respectively.

[0107] The z coordinate values ​​of the plurality of coordinate information 800 included in the point group segmented data 290 tend to represent larger values ​​as the distance division corresponding to the point group segmented data 290 is larger. In other words, the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segmented data 290 corresponding to the (n+1)th distance division tend to be larger than the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segmented data 290 corresponding to the nth distance division.

[0108] In the point group generation process, after the transformation process is performed, point group processing is performed to perform a given process on the point group segmented data 290. The point group processing unit 303 performs point group processing on each of the N generated point group segmented data 290 separately. In the point group processing, for example, noise removal processing is performed on the point group segmented data 290. Specifically, noise removal processing is performed on the multiple coordinate information 800 included in the point group segmented data 290. As a noise removal process, for example, a low-pass filter can also be used. The low-pass filter can also be a low-pass filter based on wavelet transform. In addition, as a noise removal process, a bilateral filter can be used, and a filter that removes statistical noise can also be used. In addition, noise removal processing is not performed on the multiple color information 810 included in the point group segmented data 290.

[0109] The point group processing unit 303 may, for example, make different processing parameters between the processes for the N point group segmented data 290. For example, the point group processing unit 303 may make different processing parameters between the noise removal processes for the N point group segmented data 290. In this case, the point group processing unit 303 may make different filter coefficients, for example, between the noise removal processes for the N point group segmented data 290. Thus, for example, different noise removal effects can be achieved between the noise removal processes for the N point group segmented data 290.

[0110] Here, as described above, with respect to the coordinate information 800 included in the point group data, the larger the z coordinate value included therein, the larger the noise component tends to be. On the other hand, the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segmented data 290 tend to represent larger values ​​as the distance division corresponding to the point group segmented data 290 is larger. Thus, with respect to the coordinate information 800 included in the point group segmented data 290, the larger the distance division corresponding to the point group segmented data 290 is, the larger the noise component tends to be. In other words, in the coordinate information 800 included in the point group segmented data 290 corresponding to the (n+1)th distance division, the noise component tends to be larger than that of the coordinate information 800 included in the point group segmented data 290 corresponding to the nth distance division.

[0111] Therefore, the point group processing unit 303 may make the filter coefficients different between the noise removal processes for the N point group segmented data 290, so that the larger the distance division corresponding to the point group segmented data 290, the greater the noise removal effect of the noise removal process for the point group segmented data 290. For example, the point group processing unit 303 may make the noise removal effect for the coordinate information 800 included in the point group segmented data 290 corresponding to the (n+1)th distance division greater than the noise removal effect for the coordinate information 800 included in the point group segmented data 290 corresponding to the nth distance division. Thus, the noise removal effect can be increased for the coordinate information 800 included in the point group segmented data 290 with a large corresponding distance division and having a large noise component, and the noise removal effect can be reduced for the coordinate information 800 included in the point group segmented data 290 with a small corresponding distance division and having a small noise component. Therefore, the noise removal process can be appropriately performed on each point group segmented data 290. As a result, the accuracy of each point group segment data 290 can be improved.

[0112] Furthermore, the point cloud processing unit 303 may make processing parameters different between the processing of at least two point cloud segment data 290 among the N point cloud segment data 290. In this case, appropriate processing can be performed on each of the at least two point cloud segment data 290.

[0113] For example, the point group processing unit 303 may also perform point group processing on at least two point group segment data 290 of the N point group segment data 290 in parallel. In this case, the processing time for the at least two point group segment data 290 can be shortened. Figure 8 As shown, the point group processing unit 303 may perform point group processing on N point group segmented data 290 in parallel, and may also perform point group processing on a plurality of point group segmented data 290 which are a part of the N point group segmented data 290 in parallel. In addition, the point group processing unit 303 may also perform point group processing on at least two point group segmented data 290 of the N point group segmented data 290 in series. These parallel processing and serial processing may also be referred to as point group processing to be performed by the point group processing unit 303, which will be described later. In addition, these parallel processing and serial processing may also be referred to as point group processing to be performed by the point group processing unit 353, which will be described later.

[0114] In the point group generation process, after the point group process is executed, an integration process is executed to integrate the N point group segment data 290. In the integration process, the integration unit 304 integrates the N point group segment data 290 that have been subjected to the point group process to generate one point group data 295 (also referred to as integrated point group data 295). In the integration process, the integration unit 304 may, for example, simply aggregate the N point group segment data 290 that have been subjected to the point group process and output them as one integrated point group data 295.

[0115] As described above, the data processing device 3 generates the point group data 295 based on the distance image data 270 and the color image data 260 obtained by the sensor device 2 in the point group generation process. It can also be said that the data processing device 3 transforms the distance image data 270 and the color image data 260 into the point group data 295. In the point group data 295, a plurality of coordinate information 800 and a plurality of color information 810 are associated one to one. The point group data 295 is also called colored point group data.

[0116] The control unit 30 may also send the point group data (in the above example, the comprehensive point group data 295) finally obtained in the point group generation process to an external device through the second interface 32. The external device may also perform processing based on the received point group data. The display device included in the external device may display the received point group data. In addition, the external device may, for example, generate polygon data based on the received point group data, or may generate CAD (computer-aided design) data based on the generated polygon data. In addition, the control unit 30 may generate polygon data based on the point group data generated in the point group generation process, or may generate CAD data based on the generated polygon data.

[0117] In addition, when the measurement system 1 includes a plurality of sensor devices 2 that capture the measurement space from different angles, the control unit 30 of the data processing device 3 may generate a plurality of point group data 295 based on a plurality of image data sets 280 respectively output by the plurality of sensor devices 2 during the point group generation process. Furthermore, during the point group generation process, the control unit 30 may synthesize the generated plurality of point group data 295, for example, to generate one point group data as a map.

[0118] In the above example, the range image data 270 and the color image data 260 are converted into point cloud data together, but only the range image data 270 may be converted into point cloud data. In this case, the sensor device 2 may or may not include the color camera 23 .

[0119] Fig.10 2 is a schematic diagram showing an example of a point cloud generation process for generating point cloud data based only on the range image data 270. Fig.10 In the example of , in the acquisition process, the data acquisition unit 300 acquires the distance image data 270 from the sensor device 2. Next, in the segmentation process, the segmentation unit 301 divides the distance image data 270 acquired in the acquisition process into a plurality of distance image segment data 275 based on the distance information. Here, for example, as described above, the distance image data 270 is divided into N distance image segment data 275. In addition, the data acquisition unit 300 may acquire both the distance image data 270 and the color image data 260 from the sensor device 2, and the segmentation unit 301 may divide only the distance image data 270 of the acquired distance image data 270 and the color image data 260 into a plurality of segment data.

[0120] After the segmentation process, the conversion unit 302 performs a conversion process on each of the N range image segment data 275 obtained by the segmentation process to convert the range image segment data 275 into point group data 291 (also referred to as point group segment data 291). Fig.10 In the example of , the range image segment data 275 is the point group transformation target data.

[0121] The point group data 291 includes the coordinate information 800 but does not include the color information 810. Similar to the above-mentioned point group segmented data 290, the number of the plurality of measurement points constituting the point group represented by the point group segmented data 291 is greater than the number of the plurality of measurement points corresponding to the plurality of distance information included in the range image segmented data 275. In addition, the z coordinate values ​​of the plurality of coordinate information 800 included in one point group segmented data 291 are close to each other.

[0122] Thereafter, the point group segmented data 291 obtained by transforming the distance image segmented data 275 corresponding to the n-th distance division is set as the point group segmented data 291 corresponding to the n-th distance division. Similar to the point group segmented data 290, the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segmented data 291 tend to represent larger values ​​as the distance division corresponding to the point group segmented data 291 is larger.

[0123] After executing the transformation process, the point group processing unit 303 performs the same point group processing as described above on each of the N point group segmented data 291 generated. The point group processing unit 303 may make processing parameters different between at least two of the N point group segmented data 291, similarly to the point group segmented data 290. For example, the point group processing unit 303 may make the filter coefficients different between the noise removal processes for the N point group segmented data 291, so that the larger the distance division corresponding to the point group segmented data 291, the greater the noise removal effect of the noise removal process for the point group segmented data 291. As a result, the noise removal effect can be increased for the coordinate information 800 with a tendency to have a large noise component included in the corresponding point group segmented data 291 with a large distance division, and the noise removal effect can be reduced for the coordinate information 800 with a tendency to have a small noise component included in the corresponding point group segmented data 291 with a small distance division. Therefore, it is possible to appropriately perform noise removal processing on each point group segment data 291 .

[0124] After the point group processing is executed, the integration unit 304 integrates the N point group segment data 291 that have been processed to generate one point group data 296 (also referred to as integrated point group data 296 ). The integrated point group data 296 includes the coordinate information 800 , but does not include the color information 810 .

[0125] As described above, in the data processing device 3, the distance image data 270 is divided into a plurality of distance image segmented data 275 according to the distance information. Then, the plurality of distance image segmented data 275 are individually transformed into point group data. Thus, each point group data can be properly processed according to the distance division corresponding thereto. The distance division corresponding to the point group data corresponds to the size of the z coordinate value included in the point group data, so each point group data can be properly processed according to the size of the z coordinate value included therein. As a result, the accuracy of each point group data can be improved. Therefore, the accuracy of the comprehensive point group data can be improved.

[0126] In addition, Figure 8 In the example of , not only the distance image data 270 but also the color image data 260 is divided into a plurality of color image segment data 265 according to the distance information. Then, the distance image segment data 275 is transformed into point group data together with the color image segment data 265. Thus, point group data in which the coordinate information 800 and the color information 810 are appropriately associated can be obtained. In addition, in the point group data, the coordinate information 800 and the color information 810 are associated, so even when the point group data is subjected to point group processing, the corresponding relationship between the coordinate information 800 and the color information 810 can be appropriately maintained.

[0127] In addition, it can be understood from the above description that the more the number of divisions of the range image data 270 and the color image data 260 is, that is, the larger the value of N is, the more detailed the point group processing (such as noise removal processing) can be performed. Therefore, the larger the value of N is, the more the accuracy of the comprehensive point group data can be improved. The value of N can be set according to the accuracy required by the user for the comprehensive point group data. In addition, the value of N can be determined based on the measurement environment, and can also be set based on the characteristics of the lens of the stereo camera 220 and the color camera 23.

[0128] Implementation method 2.

[0129] Fig.11 This is a schematic diagram showing an example of the configuration of the control unit 30 of the data processing device 3 included in the measurement system 1 according to the second embodiment. Fig.12 1 is a schematic diagram showing an example of point group generation processing according to Embodiment 2. The measurement system 1 according to Embodiment 2 has the same structure as the measurement system 1 according to Embodiment 1. The measurement system 1 according to Embodiment 2 is different from the measurement system 1 according to Embodiment 1 only in the operation of the control unit 30 of the data processing device 3. The control unit 30 according to Embodiment 2 generates point group data representing a specific object in the measurement space based on the data obtained by the sensor device 2, for example. In Embodiment 2, it is possible to obtain point group data representing a specific object in the measurement space with high accuracy.

[0130] like Fig.11 As shown in FIG. 1 , the control unit 30 according to the second embodiment includes, as a functional block, a determination unit 305 in addition to the above-mentioned data acquisition unit 300, segmentation unit 301, conversion unit 302, point group processing unit 303, and integration unit 304. In addition, all functions of the determination unit 305 or a part of the functions of the determination unit 305 may be realized by a hardware circuit that does not require software in the realization of its functions.

[0131] The determination unit 305 performs a determination process of determining data related to a specific object in the measurement space from at least the range image data 270 acquired by the data acquisition unit 300. In the determination process, the determination unit 305 determines data related to a specific object in the image data set 280, for example. Specifically, Fig.12As shown, the determination unit 305 determines the partial distance image data 271 including the distance information related to the specific object in the distance image data 270 during the determination process. In addition, the determination unit 305 determines the partial color image data 261 including the color information related to the specific object in the color image data 260 during the determination process. The partial distance image data 271 includes the distance information to each measurement point on the surface of the specific object and the pixel position information corresponding to the distance information. The partial color image data 261 includes the color information of each measurement point on the surface of the specific object and the pixel position information corresponding to the color information. Hereinafter, the partial distance image data 271 and the partial color image data 261 determined in the determination process are sometimes collectively referred to as the partial image data set 281.

[0132] The specific object includes at least one object in the measurement space. The specific object can also be determined, for example, based on the purpose of using the measurement system 1 (in other words, the system). For example, when the measurement system 1 is used for the inspection of piping in industry, the piping in the measurement space can also be set as a specific object. In addition, when the measurement system 1 is used to control a robot that holds an object, the object held by the robot can also be set as a specific object in the measurement space. In addition, when the measurement system 1 is used to identify vehicles on the road, the vehicle in the measurement space can also be set as a specific object. The specific object can include multiple objects of the same type, or multiple objects of different types.

[0133] In the data processing device 3 involved in the second embodiment, the storage unit 33 stores, for example, feature information (also referred to as feature quantity) representing the features of a specific object. The determination unit 305 can determine the partial range image data 271 and the partial color image data 261 based on the feature information in the storage unit 33. The feature information may include, for example, information representing the color of the specific object, or may include information representing the shape of the specific object. In addition, when the position of the specific object is fixed, the feature information may also include information representing the position where the specific object is captured in the color image data 260.

[0134] In the point group generation process involved in implementation mode 2, if Fig.12As shown, a segmentation process is performed after the determination process. In the segmentation process, the partial image data set 281 is divided into a plurality of segmented data according to the distance information. As in the case of dividing the distance image data 270, the segmentation unit 301 divides the partial distance image data 271 into N distance image segmented data 276 according to the distance information. In addition, as in the case of dividing the color image data 260, the segmentation unit 301 divides the partial color image data 261 into N color image segmented data 266 according to the distance information. The segmentation unit 301 processes one distance image segmented data 276 and one color image segmented data 266 corresponding to the same distance division as one segmented data set 286 corresponding to the distance division. In the segmentation process, N segmented data sets 286 corresponding to the N distance divisions are generated. The plurality of distance information included in one distance image segmented data 276 represent mutually close distances.

[0135] After the segmentation process, the transformation unit 302 performs a transformation process on each of the N segmented data sets 286 obtained by the segmentation process to transform the segmented data sets 286 into point group data 292 (also referred to as point group segmented data 292), similarly to the case of transforming the segmented data set 285 into the point group data 290. Fig.12 In the example of , the segmented data set 286 becomes the point group transformation target data. Thereafter, the point group segmented data 292 obtained by performing the transformation process on the segmented data set 286 corresponding to the n-th distance division is set as the point group segmented data 292 corresponding to the n-th distance division.

[0136] The conversion unit 302 generates N point group segment data 292 corresponding to the N distance divisions. The z coordinate values ​​of the plurality of coordinate information 800 included in one point group segment data 292 are close to each other. Also, similarly to the point group segment data 290, the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segment data 292 tend to represent larger values ​​as the distance division corresponding to the point group segment data 292 is larger.

[0137] After the transformation process is executed, the point group processing unit 303 performs the same point group processing as described above on each of the N point group segmented data 292 generated. As with the point group segmented data 290, the point group processing unit 303 may, for example, make the processing parameters different between at least two of the N point group segmented data 292. For example, the point group processing unit 303 may also make the filter coefficients different between the noise removal processes for the N point group segmented data 292, so that the larger the distance division corresponding to the point group segmented data 292, the greater the noise removal effect of the noise removal process for the point group segmented data 292. In this way, the noise removal process can be appropriately performed on each point group segmented data 292.

[0138] The point group processing unit 303 may, for example, execute point group processing on at least two of the N point group segment data 292 in parallel. Alternatively, the point group processing unit 303 may execute point group processing on at least two of the N point group segment data 292 in series.

[0139] After the point group processing is executed, the integration unit 304 integrates the N point group segment data 292 that have been processed by the point group to generate one point group data 297 (also referred to as integrated point group data 297). The integrated point group data 297 is data that represents a specific object in the measurement space using a plurality of points. The integrated point group data 297 is also data that represents the surface shape and color of the specific object.

[0140] exist Fig.12 In the example shown in FIG. 1 , a part of the range image data 270 is converted into point group data together with a part of the color image data 260 . However, only a part of the range image data 270 may be converted into point group data. Fig.13 1 is a schematic diagram showing an example of a point cloud generation process for generating point cloud data based on only a part of the range image data 270 .

[0141] exist Fig.13 In the example, Fig.10 In the acquisition process, the data acquisition unit 300 acquires the distance image data 270 from the sensor device 2. Next, in the determination process, the identification unit 305 identifies the partial distance image data 271 including the distance information related to the specific object in the distance image data 270. Next, in the segmentation process, the segmentation unit 301 divides the partial distance image data 271 into N distance image segment data 276 according to the distance information in the same manner as described above.

[0142] After segmentation, Fig.10 Similarly to the example of , the conversion unit 302 performs a conversion process on each of the N distance image segment data 276 obtained by the segmentation process to convert the distance image segment data 276 into point group data 293 (also referred to as point group segment data 293). Fig.13 In the example of , the distance image segment data 276 becomes the point group transformation target data. The point group data 293 includes the coordinate information 800 but does not include the color information 810. The z coordinate values ​​of the plurality of coordinate information 800 included in one point group segment data 293 are close to each other.

[0143] Thereafter, the point group segmented data 293 obtained by transforming the distance image segmented data 276 corresponding to the n-th distance division is set as the point group segmented data 293 corresponding to the n-th distance division. Similar to the point group segmented data 290, the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segmented data 293 tend to represent larger values ​​as the distance division corresponding to the point group segmented data 293 is larger.

[0144] After executing the transformation process, the point group processing unit 303 performs the same point group processing as described above on each of the N point group segmented data 293 generated. After executing the point group processing, the integration unit 304 integrates the N point group segmented data 293 that have been subjected to the point group processing in the integration process to generate one point group data 298 (also referred to as integrated point group data 298). The integrated point group data 298 includes coordinate information 800, but does not include color information 810. The integrated point group data 298, like the integrated point group data 297, is data that represents a specific object in the measurement space using multiple points. The integrated point group data 298 can also be referred to as data that represents the surface shape of a specific object.

[0145] As described above, in the second embodiment, not all of the range image data 270 but a part thereof is divided into a plurality of range image segment data 276 according to the distance information. Then, the plurality of range image segment data 276 are individually converted into point group data. Thus, as in the first embodiment, each point group data can be appropriately processed according to the distance division corresponding thereto. As a result, for example, point group data with high accuracy that represents a specific object in the measurement space can be obtained.

[0146] In addition, in the point group generation process performed by the control unit 30, the above-mentioned segmentation process, transformation process, and point group process may be performed separately for each of the multiple specific objects. For example, the above-mentioned segmentation process, transformation process, and point group process may be performed separately for each of the multiple partial image data sets 281 corresponding to each of the multiple specific objects. The partial image data set 281 corresponding to a certain specific object is a partial image data set 281 composed of partial distance image data 271 including distance information related to the certain specific object and partial color image data 261 including color information related to the certain specific object. In addition, the control unit 30 may perform the above-mentioned segmentation process, transformation process, and point group process separately for each of the multiple partial distance image data 271 corresponding to each of the multiple specific objects. The partial distance image data 271 corresponding to a certain specific object is the partial distance image data 271 including distance information related to the certain specific object.

[0147] As an example of a case where segmentation processing, transformation processing, and point group processing are performed separately for multiple specific objects, point group processing is described below in the case where segmentation processing, transformation processing, and point group processing are performed separately for a first specific object and a second specific object. In addition, segmentation processing, transformation processing, and point group processing may be performed separately for three or more specific objects.

[0148] Fig.14 1 is a schematic diagram showing an example of point group generation processing in which segmentation processing, transformation processing, and point group processing are performed separately for two partial image data sets 281 corresponding to the first specific object and the second specific object, respectively. By performing segmentation processing, transformation processing, and point group processing on the partial image data set 281 corresponding to the first specific object, a plurality of point group segmented data 292 representing the first specific object is generated. Similarly, by performing segmentation processing, transformation processing, and point group processing on the partial image data set 281 corresponding to the second specific object, a plurality of point group segmented data 292 representing the second specific object is generated. Since the plurality of point group segmented data 292 representing the first specific object and the plurality of point group segmented data 292 representing the second specific object are generated separately, for example, the accuracy of the plurality of point group segmented data 292 representing the first specific object and the accuracy of the plurality of point group segmented data 292 representing the second specific object can be made different from each other.

[0149] The first specific object and the second specific object may be determined, for example, according to the purpose of using the measurement system 1. The first specific object may be, for example, an object that should be paid attention to in a certain process in the purpose of using the measurement system 1 (referred to as the object of attention). In addition, the second specific object may be, for example, an object other than the object of attention in the measurement space (sometimes referred to as the object of non-attention). For example, when the measurement system 1 is used for the inspection of piping in industry, the piping as the object of attention may be set as the first specific object, and the objects other than this (i.e., the object of non-attention) may be set as the second specific body. In addition, when the measurement system 1 is used for the control of a robot that holds an object, the object held by the robot as the object of attention may be set as the first specific object, and the objects other than this may be set as the second specific object. In addition, when the measurement system 1 is used for the identification of vehicles on the road, the vehicle as the object of attention may be set as the first specific object, and the road and the like other than this may be set as the second specific object. The object of attention and the object of non-attention may be one object, or may include multiple objects of the same kind, or may include multiple objects of different kinds.

[0150] Hereinafter, the segmentation processing, transformation processing, and point group processing related to the first specific object are sometimes referred to as the first segmentation processing, the first transformation processing, and the first point group processing, respectively. In addition, the segmentation processing, transformation processing, and point group processing related to the second specific object are sometimes referred to as the second segmentation processing, the second transformation processing, and the second point group processing, respectively.

[0151] exist Fig.14 In the example of , in the determination process, in the image data set 280 acquired by the acquisition process, the partial image data set 281 (also referred to as the partial image data set 281a) corresponding to the first specific object and the partial image data set 281 (also referred to as the partial image data set 281b) corresponding to the second specific object are determined. For example, in a case where the first specific object is the object of interest and the second specific object is the object of no interest, in the determination process, it can be said that the image data set 280 is divided into the partial image data set 281a corresponding to the first specific object and the partial image data set 281b corresponding to the second specific object. Therefore, in this case, the determination process can be said to be a segmentation process in which the image data set 280 is divided into a plurality of segmented data, and the partial image data set 281a and the partial image data set 281b become segmented data, respectively.

[0152] The storage unit 33 of the data processing device 3 stores, for example, first feature information indicating features of a first specific object and second feature information indicating features of a second specific object. The determination unit 305 can determine the partial image data set 281a based on the first feature information in the storage unit 33. In addition, the determination unit 305 can determine the partial image data set 281b based on the second feature information in the storage unit 33.

[0153] Hereinafter, the partial range image data 271 and the partial color image data 261 included in the partial image data set 281a are sometimes referred to as partial range image data 271a and partial color image data 261a, respectively. Also, the partial range image data 271 and the partial color image data 261 included in the partial image data set 281b are sometimes referred to as partial range image data 271b and partial color image data 261b, respectively.

[0154] After the determination process, the segmentation unit 301 performs segmentation processes for the partial image data sets 281a and 281b, respectively. Thus, in the first segmentation process, the partial distance image data 271a of the partial image data set 281a is divided into a plurality of distance image segmented data 276a according to the distance information, and the partial color image data 261a of the partial image data set 281a is divided into a plurality of color image segmented data 266a according to the distance information. In addition, in the second segmentation process, the partial distance image data 271b of the partial image data set 281b is divided into a plurality of distance image segmented data 276b according to the distance information, and the partial color image data 261b of the partial image data set 281a is divided into a plurality of color image segmented data 266b according to the distance information. The segmentation unit 301 processes one distance image segmented data 276a and one color image segmented data 266a corresponding to the same distance division as one segmented data set 286a corresponding to the distance division. Furthermore, the segmentation unit 301 handles one piece of distance image segment data 276 b and one piece of color image segment data 266 b corresponding to the same distance division as one piece of segment data set 286 b corresponding to the distance division.

[0155] In the segmentation process for the partial image data set 281a, i.e., the first segmentation process, for example, the partial image data set 281a may be divided into N segmented data sets 286a as described above. In addition, in the segmentation process for the partial image data set 281b, i.e., the second segmentation process, for example, the partial image data set 281b may be divided into M (M is an integer greater than or equal to 2) segmented data sets 286. N=M may be satisfied, or N≠M may be satisfied.

[0156] In the second segmentation process, the range of the distance represented by the distance information is divided into M distance divisions from the first distance division to the Mth distance division. The M distance divisions are set in the same way as the above-mentioned N distance divisions. The mth distance division (m is a variable, an integer satisfying 1≤m≤M) is, for example, a range greater than the distance Bm-1 and less than the distance Bm. However, the Mth distance division is exceptionally set to a range greater than the distance BM-1 and less than the distance BM. The distance Bm is greater than the distance Bm-1. The distance B0 is, for example, the minimum value (for example, zero) of the range of the distance represented by the distance information. The distance BM is, for example, the maximum value of the range of the distance represented by the distance information.

[0157] In the second segmentation process, similarly to the first segmentation process, the partial range image data 271b is divided into M range image segment data 276b corresponding to the M range divisions, and the partial color image data 261b is divided into M color image segment data 266b corresponding to the M range divisions. The plurality of distance information included in one range image segment data 276b indicates mutually close distances. The first segmentation process and the second segmentation process may be performed in parallel or in series.

[0158] In addition, the M distance divisions may include a plurality of distance divisions with different division lengths, or may include a plurality of distance divisions with the same division lengths. In addition, the M distance divisions may also be discontinuous.

[0159] After the first segmentation process, the transformation unit 302 transforms each of the N segmentation data sets 286a obtained in the first segmentation process into point group data 292a (also referred to as point group segmentation data 292a). Here, the segmentation data set 286a becomes the point group transformation target data. Thereafter, the point group segmentation data 292a obtained by performing the transformation process on the segmentation data set 286a corresponding to the n-th distance division is set as the point group segmentation data 292a corresponding to the n-th distance division.

[0160] After the second segmentation process, the transformation unit 302 transforms each of the M segmentation data sets 286b obtained in the second segmentation process into point group data 292b (also referred to as point group segmentation data 292b) in the second transformation process. Here, the segmentation data set 286b becomes the point group transformation object data. Thereafter, the point group segmentation data 292b obtained by performing the transformation process on the segmentation data set 286b corresponding to the m-th distance division is set as the point group segmentation data 292b corresponding to the m-th distance division. The first transformation process and the second transformation process may be performed in parallel or in series.

[0161] After executing the first transformation process, the point group processing unit 303 performs the first point group processing on each of the N point group segment data 292a generated. In addition, after executing the second transformation process, the point group processing unit 303 performs the second point group processing on each of the M point group segment data 292a generated. The first point group processing and the second point group processing can be executed in parallel or in series.

[0162] After the execution of the first point group processing and the second point group processing, the integration unit 304 integrates the N point group segment data 292a that have been processed by the first point group and the M point group segment data 292a that have been processed by the second point group to generate one point group data 297ab (also referred to as integrated point group data 297ab). The integrated point group data 297ab is data that represents the first specific object and the second specific object in the measurement space using a plurality of points. The integrated point group data 297ab is also data that represents the surface shape and color of the first specific object and the second specific object.

[0163] Fig.15 1 is a schematic diagram showing an example of point group generation processing in which segmentation processing, transformation processing, and point group processing are performed separately on two partial distance image data 271 corresponding to a first specific object and a second specific object, respectively. By performing segmentation processing, transformation processing, and point group processing on the partial distance image data 271 corresponding to the first specific object, a plurality of point group segmentation data 293 representing the first specific object is generated. Similarly, by performing segmentation processing, transformation processing, and point group processing on the partial distance image data 271 corresponding to the second specific object, a plurality of point group segmentation data 293 representing the second specific object is generated. By separately generating a plurality of point group segmentation data 293 representing the first specific object and a plurality of point group segmentation data 293 representing the second specific object, for example, the accuracy of the plurality of point group segmentation data 293 representing the first specific object and the accuracy of the plurality of point group segmentation data 293 representing the second specific object can be made different from each other.

[0164] exist Fig.15 In the example of , in the acquisition process, the data acquisition unit 300 acquires the distance image data 270 from the sensor device 2. Next, in the determination process, the identification unit 305 identifies the partial distance image data 271a corresponding to the first specific object and the partial distance image data 271b corresponding to the second specific object in the distance image data 270.

[0165] Next, the segmentation unit 301 divides the partial range image data 271a into N range image segment data 276a according to the range information in the first segmentation process, as described above. Also, the segmentation unit 301 divides the partial range image data 271b into M range image segment data 276b according to the range information in the second segmentation process, as described above.

[0166] After the first segmentation process, the conversion unit 302 converts the N range image segmentation data 276a obtained in the first segmentation process into point group data 293a (also referred to as point group segmentation data 293a) in the first conversion process. Here, the range image segmentation data 276a is the point group conversion target data. Thereafter, the point group segmentation data 293a obtained by performing the conversion process on the range image segmentation data 276a corresponding to the n-th range division is set as the point group segmentation data 293a corresponding to the n-th range division.

[0167] After the second segmentation process, the transformation unit 302 transforms the M range image segmentation data 276b obtained in the second segmentation process into point group data 293b (also referred to as point group segmentation data 293b) in the second transformation process. Here, the range image segmentation data 276b is the point group transformation target data. Thereafter, the point group segmentation data 293b obtained by performing the transformation process on the range image segmentation data 276b corresponding to the m-th distance segment is set as the point group segmentation data 293b corresponding to the m-th distance segment.

[0168] After the first transformation, the point group processing unit 303 performs first point group processing on each of the N point group segment data 293a generated. After the second transformation, the point group processing unit 303 performs second point group processing on each of the M point group segment data 293b generated.

[0169] After the first point group processing and the second point group processing, the integration unit 304 integrates the N point group segment data 293a that have been processed by the first point group and the M point group segment data 293b that have been processed by the second point group to generate one point group data 298ab (also referred to as integrated point group data 298ab). The integrated point group data 298ab is data that represents the first specific object and the second specific object in the measurement space using a plurality of points. The integrated point group data 298ab can also be referred to as data that represents the surface shapes of the first specific object and the second specific object.

[0170] As mentioned above, in Fig.14 In the example of FIG. 15, the partial distance image data 271a corresponding to the first specific object and the partial distance image data 271b corresponding to the second specific object are segmented separately. Thus, the point group segmentation data with the accuracy corresponding to the first specific object and the point group segmentation data with the accuracy corresponding to the second specific object can be generated.

[0171] For example, consider a case where the first specific object is an object of interest and the second specific object is an object of non-interest. In this case, for example, by setting N>M, the number of multiple distance image segmentation data 276a corresponding to the object of interest (i.e., the number of divisions) is greater than the number of multiple distance image segmentation data 276b corresponding to the object of non-interest (i.e., the number of divisions). As a result, the object of interest can be processed with a more detailed point group (e.g., noise removal processing) than the object of non-interest, so that the accuracy of the multiple point group segmentation data representing the object of interest can be higher than the accuracy of the multiple point group segmentation data representing the object of non-interest. Furthermore, the number of divisions of the multiple distance image segmentation data 276b corresponding to the object of non-interest is reduced, so the processing amount for the object of non-interest can be reduced.

[0172] Thus, in the first specific object and the second specific object, the number of divisions of the specific object for generating point group data requiring high precision may be greater than the number of divisions of the specific object for generating point group data not requiring high precision.

[0173] As another example, consider the case where the first specific object is a high object and the second specific object is a low object. Here, a high object refers to an object with a larger size in the z-axis direction, and a low object refers to an object with a smaller size in the z-axis direction. In this case, for example, by setting N>M, the number of divisions of the multiple distance image segmentation data 276a corresponding to the high object is greater than the number of divisions of the multiple distance image segmentation data 276b corresponding to the low object. As a result, for objects with a higher height, point group processing can be performed more finely, and for objects with a lower height, point group processing can be performed more roughly. Therefore, both the accuracy of the multiple point group segmentation data representing the high object and the accuracy of the multiple point group segmentation data representing the low object can be appropriately set respectively.

[0174] In addition, Fig.14 In the example, point group processing is performed on the N point group segmented data representing the first specific object, and point group processing is performed on the M point group segmented data representing the second specific object, respectively. Therefore, the accuracy of the N point group segmented data representing the first specific object can be improved, and the accuracy of the M point group segmented data representing the second specific object can be improved.

[0175] In addition, at least one of the N point group segment data representing the first specific object may not be subjected to point group processing. In addition, at least one of the M point group segment data representing the second specific object may not be subjected to point group processing.

[0176] For example, consider a case where the first specific object is an object of interest and the second specific object is an object of non-interest. In this case, point group processing may not be performed on all of the M point group segment data representing the object of non-interest, or may not be performed on a portion of the M point group segment data. Thus, the amount of processing for the object of non-interest that does not require high-precision point group data can be reduced.

[0177] In the integrated processing, N point group segmented data representing the first specific object that have been subjected to point group processing respectively and M point group segmented data representing the second specific object including at least one point group segmented data that have not been subjected to point group processing may be integrated to generate integrated point group data. Similarly, in the integrated processing, M point group segmented data representing the second specific object that have been subjected to point group processing respectively and N point group segmented data representing the first specific object including at least one point group segmented data that have not been subjected to point group processing may be integrated to generate integrated point group data.

[0178] For example, consider a case where the first specific object is an object of interest and the second specific object is an object of no interest. In this case, in the integrated processing, N point group segmented data representing the object of interest that have been subjected to point group processing and M point group segmented data representing the object of no interest, including at least one point group segmented data that has not been subjected to point group processing, may be integrated. In this way, the accuracy of the data related to the object of interest in the integrated point group data can be improved, and the processing amount related to the object of no interest can be reduced.

[0179] In addition, the point group processing unit 303 may also perform a thinning-out process to reduce the number of data of the point group segmented data. In other words, the point group processing unit 303 may also perform a thinning-out process to reduce the number of multiple points constituting the point group represented by the point group data. The thinning-out process may be performed on each of the N point group segmented data representing the first specific object, or may be performed on a portion of the N point group segmented data. In addition, the thinning-out process may be performed on each of the M point group segmented data representing the second specific object, or may be performed on a portion of the M point group segmented data.

[0180] For example, consider a case where the first specific object is the object of interest and the second specific object is the object of no interest. In this case, instead of performing thinning processing on the N point group segment data representing the object of interest, at least one point group segment data among the M point group segment data representing the object of no interest may be thinned out. In this way, the density of the point group representing the object of interest can be increased, and the density of the point group representing the object of no interest can be reduced.

[0181] Furthermore, for example, when there are a first target object and a second target object whose required precisions for point cloud data are different from each other, the first specific object may be set as the first target object and the second specific object may be set as the second target object.

[0182] In addition, the integration unit 304 may integrate the N point group segment data representing the first specific object to generate one integrated point group data, and integrate the M point group segment data representing the second specific object to generate one integrated point group data. In this way, the integrated point group data representing the first specific object and the integrated point group data representing the second specific object are separately generated.

[0183] Implementation method 3.

[0184] Fig.16 1 is a schematic diagram showing an example of a configuration of a control unit 30 of a data processing device 3 included in a measurement system 1 according to a third embodiment. Fig.17 1 is a schematic diagram showing an example of point group generation processing according to Embodiment 3. The measurement system 1 according to Embodiment 3 has the same structure as the measurement system 1 according to Embodiments 1 and 2. The measurement system 1 according to Embodiment 3 is different from the measurement system 1 according to Embodiments 1 and 2 only in the operation of the control unit 30 of the data processing device 3.

[0185] The control unit 30 involved in the third embodiment, for example, performs a transformation process of transforming at least a part of the distance image data 270 obtained by the sensor device 2 into point group data. Next, the control unit 30 performs a segmentation process of dividing the point group data into a plurality of point group segmented data according to the z coordinate value. Then, the control unit 30 processes the plurality of point group segmented data separately. In this way, a plurality of point group data with high accuracy can be obtained. Hereinafter, the segmentation process involved in the third embodiment is sometimes distinguished from the segmentation process of the first and second embodiments and referred to as the point group segmentation process. In addition, hereafter, if it is only referred to as the segmentation process, it means the segmentation process involved in the first and second embodiments.

[0186] like Fig.16 As shown in FIG. 1 , the control unit 30 according to the third embodiment includes the above-mentioned data acquisition unit 300, the conversion unit 352, the segmentation unit 351, the point group processing unit 353, and the synthesis unit 354 as functional blocks. In addition, all functions of the conversion unit 352 or part of the functions of the conversion unit 352 may be realized by a hardware circuit that does not require software in the realization of its functions. The same applies to the segmentation unit 351, the point group processing unit 353, and the synthesis unit 354.

[0187] The conversion unit 352, like the conversion unit 302, performs conversion of the image data set 280 or the distance image data 270 acquired by the data acquisition unit 300 into the point cloud data 360 (see Fig.17 ). In the point group data 360 based on the image data set 280, as in the above-mentioned point group segmented data 290, etc., a plurality of coordinate information 800 and a plurality of color information 810 are associated with each other. On the other hand, in the point group data 360 based on the distance image data 270, as in the above-mentioned point group segmented data 291, etc., the coordinate information 800 is included, but the color information 810 is not included.

[0188] The dividing unit 351 divides the point cloud data 360 obtained by the transformation process into a plurality of point cloud segment data 370 (see Fig.17 ) is performed. The point group processing unit 353 performs point group processing for each of the plurality of point group segmented data 370 generated in the point group segmentation processing. The point group processing unit 353 performs point group processing on each of the point group segmented data 370 in the same manner as the above-mentioned point group processing unit 303. The synthesis unit 354 synthesizes the plurality of point group segmented data 370 that have been subjected to point group processing in the same manner as the above-mentioned synthesis unit 304 to generate synthesis point group data 380 (see Fig.17 ) comprehensive processing.

[0189] In the point group generation process involved in implementation mode 3, if Fig.17 As shown, the acquisition process, the transformation process and the point group segmentation process are performed in sequence. Then, after performing a plurality of point group processes corresponding to the plurality of point group segmentation data 370, the integrated process is performed. At least two of the plurality of point group processes may be performed in parallel. In addition, at least two of the plurality of point group processes may be performed serially in sequence.

[0190] In the point group segmentation process, for example, the range of the z coordinate value included in the coordinate information 800 is divided into P coordinate divisions from the first coordinate division to the P-th coordinate division (P is an integer greater than or equal to 2). Moreover, in the point group segmentation process, the point group data 360 is divided into P point group segment data 370 corresponding to the P coordinate divisions, respectively.

[0191] The P coordinate divisions are set, for example, in the same manner as the N distance divisions of the above-mentioned segmentation processing. The p-th coordinate division (p is a variable, an integer satisfying 1≤p≤P) is, for example, a range greater than the coordinate value Cp-1 and less than the coordinate value Cp. However, the P-th coordinate division is exceptionally set to a range greater than the coordinate value CP-1 and less than the coordinate value CP. The coordinate value Cp is greater than the coordinate value Cp-1. The coordinate value C0 is, for example, the minimum value (for example, zero) of the range that the z-coordinate value can take. The coordinate value CP is, for example, the maximum value of the range that the z-coordinate value can take. For example, the first coordinate division is a range greater than the coordinate value C0 and less than the coordinate value C1, and the second coordinate division is a range greater than the coordinate value C1 and less than the coordinate value C2.

[0192] Regarding the point group data 360 based on the image data set 280, the segmentation unit 351 includes the coordinate information 800 including the z coordinate value included in the p-th coordinate division in the point group data 360 and the color information 810 associated with the coordinate information 800 in the point group segment data 370 corresponding to the p-th coordinate division. Thus, the multiple z coordinate values ​​included in one point group segment data 370 represent values ​​close to each other. In addition, regarding the point group data 360 based on the distance image data 270, the segmentation unit 351 includes the coordinate information 800 including the z coordinate value included in the p-th coordinate division in the point group segment data 370 corresponding to the p-th coordinate division.

[0193] The z coordinate values ​​of the plurality of coordinate information 800 included in the point group segment data 370 tend to represent larger values ​​as the coordinate division corresponding to the point group segment data 370 is larger. In other words, the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segment data 370 corresponding to the (p+1)th coordinate division tend to be larger than the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segment data 370 corresponding to the pth coordinate division.

[0194] In addition, the P coordinate divisions may include multiple coordinate divisions with different division lengths or multiple coordinate divisions with the same division lengths. For example, the length of the first coordinate division may be the same as or different from the length of the second coordinate division. In addition, the P coordinate divisions may also be different from the above example and not continuous.

[0195] The point cloud processing unit 353 performs point cloud processing on each of the generated P point cloud segment data 370. For example, the point cloud processing unit 353 may make processing parameters different between the processing of at least two of the P point cloud segment data 370, as described above.

[0196] Here, as described above, with respect to the coordinate information 800 included in the point group data, the larger the z coordinate value included therein, the larger the noise component tends to be. On the other hand, the z coordinate values ​​of the plurality of coordinate information 800 included in the point group segmented data 370 tend to represent larger values ​​as the coordinate division corresponding to the point group segmented data 370 is larger. Thus, with respect to the coordinate information 800 included in the point group segmented data 370, the larger the coordinate division corresponding to the point group segmented data 370 is, the larger the noise component tends to be. In other words, in the coordinate information 800 included in the point group segmented data 370 corresponding to the (p+1)th coordinate division, the noise component tends to be larger than that of the coordinate information 800 included in the point group segmented data 370 corresponding to the pth coordinate division.

[0197] Therefore, the point group processing unit 353 may also make the filter coefficients different between the noise removal processes for the P point group segmented data 370, so that the larger the coordinate division corresponding to the point group segmented data 370, the greater the noise removal effect of the noise removal process for the point group segmented data 370. For example, the point group processing unit 353 may make the noise removal effect for the coordinate information 800 included in the point group segmented data 370 corresponding to the (p+1)th coordinate division greater than the noise removal effect for the coordinate information 800 included in the point group segmented data 370 corresponding to the pth coordinate division. Thus, the noise removal effect can be increased for the coordinate information 800 included in the point group segmented data 370 with a larger corresponding coordinate division and having a larger noise component, and the noise removal effect can be reduced for the coordinate information 800 included in the point group data 370 with a smaller corresponding coordinate division and having a smaller noise component. In other words, the noise removal effect can be increased for the coordinate information 800 that is included in the point group segmented data 370 with a large z coordinate value and has a tendency to have a large noise component, and the noise removal effect can be reduced for the coordinate information 800 that is included in the point group segmented data 370 with a small z coordinate value and has a tendency to have a small noise component. Therefore, the noise removal process can be appropriately performed on each point group segmented data 370. As a result, the accuracy of each point group segmented data 370 can be improved.

[0198] As described above, in Embodiment 3, the point group data 360 is divided into a plurality of point group segmented data 370 according to the z coordinate value. Thus, each point group segmented data 370 can be appropriately processed according to the coordinate division corresponding thereto. That is, each point group segmented data 370 can be appropriately processed according to the size of the z coordinate value included therein. As a result, the accuracy of each point group segmented data 370 can be improved. Therefore, the accuracy of the integrated point group data 380 can be improved. The more the number of divisions of the point group data 360 is, that is, the larger the value of P is, the more detailed the point group processing (for example, noise removal processing) can be performed. Therefore, by setting the value of P to be larger, the accuracy of the integrated point group data 380 can be improved. The value of P can also be set according to the accuracy sought by the user for the integrated point group data 380. In addition, the value of P can be determined based on the measurement environment, or can be set based on the specifics of the lenses of the stereo camera 220 and the color camera 23.

[0199] Furthermore, when the image data set 280 is converted into the point group data 360, in other words, when the range image data 270 and the color image data 260 are converted into the point group data 360 together, the point group data 360 in which the coordinate information 800 and the color information 810 are appropriately associated can be divided into a plurality of point group segmented data 370. Therefore, the point group segmented data 370 in which the coordinate information 800 and the color information 810 are appropriately associated can be obtained. Furthermore, in the point group segmented data 370, the coordinate information 800 and the color information 810 are appropriately associated, so that even when the point group segmented data 370 is subjected to point group processing, the correspondence relationship between the coordinate information 800 and the color information 810 can be appropriately maintained. In addition, in the conversion process, a part of the image data set 280 or the range image data 270 can be converted into the point group data 360.

[0200] Implementation method 4.

[0201] Fig.18 1 is a schematic diagram showing an example of a configuration of a control unit 30 of a data processing device 3 included in a measurement system 1 according to a fourth embodiment. Fig.19 1 is a schematic diagram showing an example of point group generation processing according to Embodiment 4. The measurement system 1 according to Embodiment 4 has the same structure as the measurement system 1 according to Embodiment 3 described above. The measurement system 1 according to Embodiment 4 is different from the measurement system 1 according to Embodiment 3 only in the operation of the control unit 30 of the data processing device 3. The control unit 30 according to Embodiment 4 generates point group data representing a specific object in the measurement space based on the data obtained by the sensor device 2, for example, as in Embodiment 2.

[0202] like Fig.18As shown in FIG. 1 , the control unit 30 according to the fourth embodiment includes, as a functional block, a determination unit 355 in addition to the above-mentioned data acquisition unit 300, conversion unit 352, segmentation unit 351, point group processing unit 353, and integration unit 354. In addition, all functions of the determination unit 355 or a part of the functions of the determination unit 355 may be realized by a hardware circuit that does not require software when realizing its functions.

[0203] The determination unit 355 performs determination processing for determining data related to the specific object in the measurement space in the image data set 280 acquired by the data acquisition unit 300, similarly to the determination unit 305. In the determination processing, the determination unit 355 determines the partial range image data 271 including the distance information related to the specific object in the range image data 270, similarly to the determination unit 305. In addition, in the determination processing, the determination unit 355 determines the partial color image data 261 including the color information related to the specific object in the color image data 260, similarly to the determination unit 305. The partial range image data 271 and the partial color image data 261 determined by the determination processing constitute the partial image data set 281.

[0204] In the determination process, the determination unit 355 may also be combined with the above-mentioned Fig.13 Similarly to the example of , the partial range image data 271 including the distance information related to the specific object is determined from the range image data 270 acquired by the acquisition process.

[0205] In the conversion process, the conversion unit 352 converts the partial image data set 281 or the partial range image data 271 determined by the determination process into the point cloud data 361 (see Fig.19 ).

[0206] In the point group segmentation process, the segmentation unit 351 divides the point group data 361 into P point group segment data 371 according to the z coordinate value, similarly to the third embodiment. The point group processing unit 353 performs point group processing for each of the P point group segment data 371 generated in the point group segmentation process, similarly to the third embodiment. In the integration process, the integration unit 354 integrates the P point group segment data 371 that have been subjected to the point group processing, similarly to the third embodiment, to generate integrated point group data 381.

[0207] In the point group generation process involved in Implementation 4, if Fig.19 As shown, the acquisition process, the determination process, the conversion process, and the point group segmentation process are sequentially performed. Then, after performing a plurality of point group processes corresponding to the plurality of point group segment data 371, the integration process is performed.

[0208] As described above, in the fourth embodiment, not all of the range image data 270 but a part of it is converted into the point group data 361. Moreover, the point group data 361 is divided into a plurality of point group segment data 371 according to the z coordinate value. Thus, as in the third embodiment, each point group segment data 371 can be appropriately processed according to the coordinate division corresponding thereto. In other words, each point group segment data 371 can be appropriately processed according to the size of the z coordinate value included therein. As a result, for example, it is possible to obtain point group data with high accuracy that represents a specific object in the measurement space.

[0209] In addition, in the point group generation process, the above-mentioned transformation process, point group segmentation process, and point group process may be performed separately for a plurality of specific objects. For example, the control unit 30 may perform transformation process, point group segmentation process, and point group process separately for a plurality of partial image data sets 281 corresponding to a plurality of specific objects. In addition, the control unit 30 may perform transformation process, point group segmentation process, and point group process separately for a plurality of partial range image data 271 corresponding to a plurality of specific objects.

[0210] Fig. 20 1 is a schematic diagram showing an example of point group generation processing in which transformation processing, point group segmentation processing, and point group processing are performed separately for a first specific object and a second specific object, respectively. Hereinafter, similarly to Embodiment 2, the transformation processing, point group segmentation processing, and point group processing for the first specific object are sometimes referred to as first transformation processing, first point group segmentation processing, and first point group processing, respectively. In addition, the transformation processing, point group segmentation processing, and point group processing for the second specific object are sometimes referred to as second transformation processing, second point group segmentation processing, and second point group processing, respectively.

[0211] exist Fig. 20 In the example of , in the determination process, similarly to Embodiment 2, in the image data set 280 acquired by the acquisition process, the partial image data set 281a corresponding to the first specific object and the partial image data set 281b corresponding to the second specific object are determined. In the determination process, similarly to Embodiment 2, in the distance image data 270 acquired by the acquisition process, the partial distance image data 271a corresponding to the first specific object and the partial distance image data 271b corresponding to the second specific object are determined.

[0212] After the determination process, the transformation unit 352 transforms the partial image data set 281a or the partial range image data 271a into point group data 361a in the first transformation process. The point group data 361a is point group data representing the first specific object with a plurality of points. In addition, the transformation unit 352 transforms the partial image data set 281b or the partial range image data 271b into point group data 361b in the second transformation process. The point group data 361b is point group data representing the second specific object with a plurality of points. The first transformation process and the second transformation process may be performed in parallel or in series.

[0213] After the first transformation process, the first point group segmentation process is performed, and after the second transformation process, the second point group segmentation process is performed. In the first point group segmentation process, for example, the point group data 361a can be divided into P point group segmented data 371a as described above. In addition, in the second point group segmentation process, for example, the point group data 361b can also be divided into Q (Q is an integer greater than 2) point group segmented data 371b. It can be P = Q or P ≠ Q.

[0214] In the second point group segmentation process, for example, the range of the z coordinate value included in the coordinate information 800 is divided into Q coordinate divisions from the first coordinate division to the Qth coordinate division. In addition, in the second point group segmentation process, the point group data 361b is divided into Q point group segment data 371b corresponding to the Q coordinate divisions, respectively.

[0215] The Q coordinate divisions are set in the same way as the P coordinate divisions. The qth coordinate division (q is a variable, an integer satisfying 1≤q≤Q) is, for example, a range greater than the coordinate value Dq-1 and less than the coordinate value Dq. However, the Qth coordinate division is exceptionally set to a range greater than the coordinate value DQ-1 and less than the coordinate value DQ. The coordinate value Dq is greater than the coordinate value Dq-1. The coordinate value D0 is, for example, the minimum value (e.g., zero) of the range that the z coordinate value can take. The coordinate value DQ is, for example, the maximum value of the range that the z coordinate value can take.

[0216] In the second point group segmentation process, similarly to the first point group segmentation process, the point group data 361b is segmented into Q point group segment data 371b corresponding to the Q coordinate segments. The first point group segmentation process and the second point group segmentation process may be executed in parallel or in series.

[0217] In addition, the Q coordinate divisions may include a plurality of coordinate divisions with different division lengths, or may include a plurality of coordinate divisions with the same division lengths. In addition, the Q coordinate divisions may also be discontinuous.

[0218] After the first point group segmentation process, the point group processing unit 353 performs the first point group process on the generated P point group segment data 371a separately. In addition, after the second point group segmentation process, the point group processing unit 353 performs the second point group process on the generated Q point group segment data 371b separately. The first point group process and the second point group process may be performed in parallel or in series.

[0219] After the first point group processing and the second point group processing are executed, the integration unit 354 integrates the P point group segment data 371a representing the first specific object that has been processed by the first point group and the Q point group segment data 371b representing the second specific object that has been processed by the second point group to generate integrated point group data 381ab. The integrated point group data 381ab is data that represents the first specific object and the second specific object in the measurement space using multiple points.

[0220] As mentioned above, in Fig. 20 In the example of , the point group data 361a representing the first specific object and the point group data 361b representing the second specific object are segmented separately. Thus, the point group segmented data 371a with the accuracy corresponding to the first specific object and the point group segmented data 371b with the accuracy corresponding to the second specific object can be generated.

[0221] For example, consider a case where the first specific object is an object of interest and the second specific object is an object of no interest. In this case, for example, by setting P>Q, the number of divisions of the plurality of point group segmented data 371a representing the object of interest is greater than the number of divisions of the plurality of point group segmented data 371b representing the object of no interest. As a result, the object of interest can be subjected to more detailed point group processing (e.g., noise removal processing) than the object of no interest. Therefore, the accuracy of the plurality of point group segmented data 371a representing the object of interest can be made higher than the accuracy of the plurality of point group segmented data 371b representing the object of no interest. Furthermore, since the number of divisions of the plurality of point group segmented data 371b representing the object of no interest is small, the amount of processing for the object of no interest can be reduced.

[0222] Thus, in the first specific object and the second specific object, the number of divisions of the specific object for generating point group data requiring high precision may be greater than the number of divisions of the specific object for generating point group data not requiring high precision.

[0223] In addition, Fig. 20In the example, point group processing is performed separately for the P point group segmented data 371a representing the first specific object, and point group processing is performed separately for the Q point group segmented data 371b representing the second specific object, thereby improving the accuracy of the P point group segmented data 371a representing the first specific object, and improving the accuracy of the Q point group segmented data 371b representing the second specific object.

[0224] In addition, at least one point group segment data 371a of the P point group segment data 371a representing the first specific object may not be subjected to point group processing. In addition, at least one point group segment data 371b of the Q point group segment data 371b representing the second specific object may not be subjected to point group processing.

[0225] In the integrated processing, the P point group segmented data 371a representing the first specific object that have been subjected to point group processing and the Q point group segmented data 371a representing the second specific object including at least one point group segmented data 371b that have not been subjected to point group processing may be integrated to generate integrated point group data 381ab. Similarly, in the integrated processing, the Q point group segmented data 371b representing the second specific object that have been subjected to point group processing and the P point group segmented data 371a representing the first specific object including at least one point group segmented data 371a that have not been subjected to point group processing may be integrated to generate integrated point group data 381ab.

[0226] In addition, the point group processing unit 353 may also perform thinning processing to reduce the number of data of the point group segment data 371a or the point group segment data 371b. The thinning processing may also be performed on at least one of the P point group segment data 371a representing the first specific object. In addition, the thinning processing may also be performed on at least one of the Q point group segment data 371b representing the second specific object.

[0227] Alternatively, the integration unit 354 may integrate P point group segment data 371a to generate one integrated point group data, and integrate Q point group segment data 371b to generate one integrated point group data, thereby separately generating integrated point group data representing the first specific object and integrated point group data representing the second specific object.

[0228] exist Figures 18 to 20 In the example of , data related to a specific object is determined in the image data set 280 or the distance image data 270, and the determined data is transformed into point group data. In contrast, the following describes an example in which the image data set 280 or the distance image data 270 is transformed into point group data, and data related to a specific object is determined in the point group data. Fig.21 : is a schematic diagram showing an example of the configuration of the control unit 30 in this case. Fig. 22 Yes means Fig.21 Schematic diagram of an example of the point cloud generation process executed by the control unit 30 shown.

[0229] exist Fig.21 In the example of FIG. 3 , the control unit 30 includes a determination unit 356 as a functional block in addition to the above-mentioned data acquisition unit 300, conversion unit 352, segmentation unit 351, point group processing unit 353, and integration unit 354. In addition, all functions of the determination unit 356 or a part of the functions of the determination unit 356 may be realized by a hardware circuit that does not require software in the realization of its functions.

[0230] exist Fig.21 In the example, the transformation unit 352 and Fig.17 Similarly to the example of , the image data set 280 or the distance image data 270 acquired in the acquisition process of the data acquisition unit 300 is converted into the point group data 360. The determination unit 356 determines the partial point group data 365 (see ) including the coordinate information 800 related to the specific object in the point group data 360 acquired by the conversion process based on the feature information in the storage unit 33. Fig. 22 ). The partial point group data 365 includes coordinate information 800 of a plurality of points representing a specific object in the point group represented by the point group data 360. In addition, the partial point group data 365 included in the point group data 360 based on the image data set 280 includes not only the coordinate information 800 of a plurality of points representing a specific object, but also a plurality of color information 810 corresponding to the coordinate information 800 of the plurality of points.

[0231] In the point cloud segmentation process, the segmentation unit 351 divides the partial point cloud data 365 determined in the determination process into P point cloud segment data 372 (see FIG. 1 ) according to the z coordinate value in the same manner as described above. Fig. 22 The point group processing unit 353 performs point group processing on each of the P point group segment data 372 generated by the point group segmentation processing. In the integration processing, the integration unit 354 integrates the P point group segment data 372 that have undergone point group processing to generate integrated point group data 382 (see Fig. 22 ).

[0232] In this way, even when not all but part of the point group data 360 is divided into a plurality of point group segment data 372, each point group segment data 372 can be appropriately processed according to the corresponding coordinate division. Thus, for example, high-precision point group data representing a specific object in the measurement space can be obtained.

[0233] In the point cloud generation process, the point cloud segmentation process and the point cloud process may be performed separately for each of the plurality of specific objects. For example, the control unit 30 may perform the point cloud segmentation process and the point cloud process separately for each of the plurality of partial point cloud data 365 representing the plurality of specific objects.

[0234] Fig.23 This is a schematic diagram showing an example of point group generation processing when point group segmentation processing and point group processing are performed separately for a first specific object and a second specific object, respectively. Hereinafter, similarly to the above, the point group segmentation processing and point group processing for the first specific object are sometimes referred to as the first point group segmentation processing and the first point group processing, respectively. In addition, the point group segmentation processing and point group processing for the second specific object are sometimes referred to as the second point group segmentation processing and the second point group processing, respectively.

[0235] exist Fig.23 In the example of , in the determination process, the determination unit 356 determines the partial point group data 365a representing the first specific object and the partial point group data 365b representing the second specific object in the point group data 360 obtained by the transformation process. The determination unit 356 determines the partial point group data 365a based on the first feature information in the storage unit 33. In addition, the determination unit 356 determines the partial point group data 365b based on the second feature information in the storage unit 33.

[0236] After the determination process, the first point group segmentation process and the second point group segmentation process are performed. In the first point group segmentation process, for example, the partial point group data 365a is segmented into P point group segmented data 372a according to the z coordinate value. In the second point group segmentation process, for example, the partial point group data 365b is divided into Q point group segmented data 372b according to the z coordinate value. The first point group segmentation process and the second point group segmentation process can be performed in parallel or in series.

[0237] After the first point group segmentation process, the point group processing unit 353 performs the first point group process on the generated P point group segment data 372a separately. In addition, after the second point group segmentation process, the point group processing unit 353 performs the second point group process on the generated Q point group segment data 372b separately. The first point group process and the second point group process may be performed in parallel or in series.

[0238] After the first point group processing and the second point group processing are executed, the integration unit 354 integrates the P point group segment data 372a representing the first specific object that has been processed by the first point group and the Q point group segment data 372b representing the second specific object that has been processed by the second point group to generate integrated point group data 382ab. The integrated point group data 382ab is data that represents the first specific object and the second specific object in the measurement space using multiple points.

[0239] As mentioned above, in Fig.23 In the example of FIG. 1 , the partial point group data 365a representing the first specific object and the partial point group data 365b representing the second specific object are separately segmented. Fig. 20 Similarly to the example of , point cloud segment data 372 a with accuracy corresponding to the first specific object and point cloud segment data 372 b with accuracy corresponding to the second specific object can be generated.

[0240] In addition, Fig. 20 Similarly, at least one of the P point group segment data 372a representing the first specific object may not be subjected to point group processing. In addition, at least one of the Q point group segment data 372b representing the second specific object may not be subjected to point group processing.

[0241] In addition, Fig. 20 Similarly, in the integrated processing, the P point group segment data 372a representing the first specific object, which have been subjected to point group processing, and the Q point group segment data 372a representing the second specific object, including at least one point group segment data 372b that have not been subjected to point group processing, may be integrated to generate integrated point group data 382ab. Similarly, in the integrated processing, the Q point group segment data 372b representing the second specific object, which have been subjected to point group processing, and the P point group segment data 372a representing the first specific object, including at least one point group segment data 372a that have not been subjected to point group processing, may be integrated to generate integrated point group data 382ab.

[0242] In addition, Fig. 20 Similarly to the example of , the point group processing unit 353 may also perform thinning processing to reduce the number of data of the point group segmented data 372a or the point group segmented data 372b. The thinning processing may be performed on at least one of the P point group segmented data 372a. In addition, the thinning processing may also be performed on at least one of the Q point group segmented data 372b representing the second specific object.

[0243] Alternatively, the integration unit 354 may integrate P point group segment data 372a to generate one integrated point group data, and integrate Q point group segment data 372b to generate one integrated point group data, thereby separately generating integrated point group data representing the first specific object and integrated point group data representing the second specific object.

[0244] exist Figures 18 to 20 In the determination process of the example, data related to a specific object is determined in the image data set 280 or the distance image data 270. On the other hand, in Figure 21 to Figure 23 In the determination process of the example, data related to the specific object is determined in the point group data based on the image data set 280 or the distance image data 270. As described above, when the image data set 280 or the distance image data 270 is converted into the point group data, the number of measurement points increases, so the data amount of the point group data is larger than the data amount of the image data set 280 or the distance image data 270. Figures 18 to 20 As in the example of the identification process, by identifying data related to a specific object in the image data set 280 or the distance image data 270, which has a smaller data volume than the point group data, the load of the identification process can be reduced.

[0245] In Embodiments 1 and 2, the segmentation process is performed on the image data set 280 or the distance image data 270, which has a relatively small amount of data. On the other hand, in Embodiments 3 and 4, the segmentation process is performed on the point group data, which has a relatively large amount of data. As in Embodiments 1 and 2, by performing the segmentation process on the image data set 280 or the distance image data 270, which has a relatively small amount of data, the load of the segmentation process can be reduced.

[0246] As described above, the measurement system 1 involved in multiple embodiments has been described, but the measurement system 1 is not limited to the above examples. For example, the sensor device 2 and the data processing device 3 may be housed in one housing or in different housings. In addition, in the sensor device 2, at least one of the distance measurement sensor 22 and the color camera 23 may be housed in a housing different from other structures. In addition, the distance measurement sensor 22 and the color camera 23 may be housed in the same housing or in different housings.

[0247] In addition, the control unit 20 of the sensor device 2 may also function as the control unit 30 of the data processing device 3. In other words, the control unit 20 may also include the above-mentioned data acquisition unit, determination unit, division unit, conversion unit, point group processing unit, and integration unit included in the control unit 30. In this case, the first interface 31, the second interface 32, and the storage unit 33 of the data processing device 3 are not required. For example, the interface 21 of the sensor device 2 may communicate with an external device, and the sensor device 2 may also function as the data processing device 3. In addition, in the measurement system 1, instead of the control unit 20 generating the distance image data 270 based on the image data 250, the control unit 30 may generate the distance image data 270 based on the image data 250. In this case, for example, the control unit 30 of the data processing device 3 may also function as the control unit 20 of the sensor device 2. In this case, the sensor device 2 may also include the distance measurement sensor 22, the color camera 23, and the interface 21, and the image data 250 output by the distance measurement sensor 22 and the color image data 260 output by the color camera 23 are input to the data processing device 3 through the interface 21. The control unit 30 of the data processing device 3 may generate the distance image data 270 based on the image data 250 , or perform filtering processing on the color image data 260 .

[0248] In addition, if Fig.14 , Fig.15 , Fig. 20 , Fig.23 In that case, when a plurality of point group segmentation data are generated for a plurality of specific objects, each point group segmentation data of at least one non-interested object among the plurality of specific objects may not be used in the integrated processing and may not be integrated with other data. In this case, the point group segmentation data not used in the integrated processing may be discarded or not discarded.

[0249] In addition, Fig. 20 And in the point cloud generation process of 23, the segmentation process may not be performed. Fig.24 It means that the segmentation process is not performed. Fig. 20 A schematic diagram of an example of point group generation processing. Fig.25 It means that the segmentation process is not performed. Fig.23 A schematic diagram of an example of point group generation processing. Fig.24 In the example of 25 and 25 , the control unit 30 does not include the conversion unit 352 .

[0250] exist Fig.24In the point group generation process, the point group processing unit 353 performs the first point group processing on the point group data 361a representing the first specific object and performs the second point group processing on the point group data 361b representing the second specific object. Then, the integration unit 354 integrates the point group data 361a that has been subjected to the first point group processing and the point group data 361b that has been subjected to the second point group processing to generate integrated point group data 385.

[0251] exist Fig.25 In the point group generation process of , the point group processing unit 353 performs the first point group processing on the partial point group data 365a (also referred to as point group data 365a) representing the first specific object, and performs the second point group processing on the partial point group data 365b (also referred to as point group data 365b) representing the second specific object. Then, the integration unit 354 integrates the partial point group data 365a that has been subjected to the first point group processing and the partial point group data 365b that has been subjected to the second point group processing to generate integrated point group data 386.

[0252] exist Fig.24 as well as Fig.25 In one example, the first point group processing and the second point group processing may be different from each other. For example, consider a case where the first specific object is an object of interest and the second specific object is an object of no interest. In this case, the second point group processing includes a thinning-out process that reduces the number of data in the point group data 361b or the point group data 365b, and the first point group processing may not include the thinning-out process. Thus, the density of the point group representing the object of interest can be increased, and the density of the point group representing the object of no interest can be reduced. In addition, the first point group processing may include a filtering process, and the second point group processing may not include a filtering process. Thus, the accuracy of the point group data representing the object of interest can be improved, and the processing load of the point group data representing the object of no interest can be reduced. In addition, the first point group processing may include a first filter process having a filter coefficient that increases the noise removal effect, and the second point group processing may include a second filter process having a filter coefficient that decreases the noise removal effect. In this case, the processing parameters of the filtering process are different between the first point group processing and the second point group processing.

[0253] In addition, the first point group processing and the second point group processing may be the same as each other. In addition, the first point group processing may not be performed on at least one of the point group data 361a and 365a representing the first specific object. In addition, the second point group processing may not be performed on at least one of the point group data 361b and 365b representing the second specific object. In addition, the point group data 361a and 365a representing the first specific object and the point group data 361b and 365b representing the second specific object may not be integrated.

[0254] Later, sometimes Fig.24Point group data representing a specific object that has not been segmented, such as the point group data 361a, 361b, 365a, and 365b of FIG. 25, is called non-SG point group data. Fig.24 In the example of 25, the integration unit 354 integrates a plurality of non-SG point group data to generate integrated point group data.

[0255] Furthermore, in each of the above-mentioned embodiments, the control unit 30 is provided with a synthesis unit for synthesizing a plurality of point group segment data or a plurality of non-SG point group data, but the synthesis unit may not be provided in the control unit 30. In this case, the control unit 30 may transmit a plurality of point group segment data or non-SG point group data that has been subjected to point group processing by the point group processing unit to an external device via the second interface 32. Furthermore, the external device may synthesize a plurality of point group segment data or non-SG point group data to generate integrated point group data.

[0256] <Example of using the measurement system>

[0257] The measuring system 1 can be used for various purposes. For example, as described above, the measuring system 1 can also be used for the inspection of piping in industry. In this case, for example, the position and shape of the piping can be identified based on the integrated point group data generated by the data processing device 3, the multiple point group segmented data representing the piping, or the non-SG point group data representing the piping. In addition, the measuring system 1 can also be used for the identification of vehicles on the road. In this case, for example, the position and shape of the vehicle can be identified based on the integrated point group data generated by the data processing device 3, the multiple point group segmented data representing the vehicle, or the non-SG point group data representing the vehicle. In addition, the measuring system 1 can also be used for the control of a robot. An example of a robot control system 500 having the measuring system 1 is described below.

[0258] Fig.26 is a schematic diagram showing an example of a robot control system 500 for controlling a robot 600. Fig.26 As shown, the robot control system 500 includes, for example, a sensor device 2, a data processing device 3, and a robot control unit 510 that controls a robot 600. The robot 600 and the robot control system 500 that controls the robot 600 constitute a robot system.

[0259] The robot control unit 510 is a type of computer device, and has, for example, the same structure as the data processing device 3. The robot control unit 510 can communicate with, for example, the second interface 32 of the data processing device 3. The robot control unit 510 controls the robot 600 based on, for example, the integrated point group data output from the second interface 32. In addition, all functions of the robot control unit 510 or part of the functions of the robot control unit 510 may be realized by a hardware circuit that does not require software in the realization of its functions.

[0260] The robot 600, for example, moves a work object 610 from a work start platform 620 to a work target platform 630. The robot control unit 510 controls the robot 600 so that the work object 610 moves from the work start platform 620 to the work target platform 630. The robot 600, for example, includes an arm 601 and an end effector 602 that holds the work object 610. The robot 600 moves the arm 601 while the end effector 602 holds the work object 610, thereby moving the work object 610 from the work start platform 620 to the work target platform 630. The robot control unit 510 can control the arm 601 and the end effector 602.

[0261] The robot 600, the work object 610, the work start platform 620, and the work target platform 630 are located in the measurement space 1000 of the measurement system 1. The stereo camera 220 and the color camera 23 of the sensor device 2 photograph the work object 610, the work start platform 620, and the work target platform 630. The data processing device 3 outputs, for example, integrated point group data representing the work object 610, the work start platform 620, and the work target platform 630 to the robot control unit 510. The stereo camera 220 and the color camera 23 perform photographing at regular intervals, for example, and the data processing device 3 sequentially outputs the integrated point group data to the robot control unit 510. In addition, when there is an obstacle that hinders the movement of the robot 600 in the measurement space 1000, the integrated point group data also represents the obstacle.

[0262] The robot control unit 510 can identify objects in the measurement space 1000 based on the comprehensive point group data from the data processing device 3. For example, the robot control unit 510 performs recognition processing to identify the work object 610, the work start platform 620, the work target platform 630, and obstacles. In the recognition processing, the robot control unit 510 can, for example, recognize the positions and postures of the work object 610, the work start platform 620, the work target platform 630, and the obstacles. The robot control unit 510 controls the arm 601 and the end effector 602 based on the result of the recognition processing. As a result, the robot 600 can appropriately hold the work object 610 and appropriately move the held work object 610 from the work start platform 620 to the work target platform 630.

[0263] If there is an obstacle in the measurement space 1000, the robot control unit 510 determines, for example, a motion path of the arm 601 that avoids the obstacle based on the result of the recognition process. Then, the robot control unit 510 controls the arm 601 to move along the determined motion path.

[0264] For example, consider a data processing device 3 such as Fig.14 , Fig.15 , Fig. 20 , Fig.23 , Fig.24 , Fig.25 In the example of , the integrated point group data representing the object of interest and the object of non-interest is output. In this case, when the robot 600 performs a process of holding the work object 610, the work object 610 and the work start platform 620 may be set as the object of interest, and the work target platform 630 may be set as the object of non-interest. In addition, when the process of moving the arm 601 is performed, the work target platform 630 may be set as the object of interest, and the work object 610 and the work start platform 620 may be set as the object of non-interest. In this way, by setting the object of interest and the object of non-interest according to the process, the robot 600 can be appropriately controlled. In addition, when the stereo camera 220 and the color camera 23 of the sensor device 2 also shoot the arm 601 of the robot 600, the arm 601 may be used as the second object of non-interest. In this case, the plurality of point group segmented data or non-SG point group data about the second object of non-interest may not be used in the integrated process and may not be integrated with other data. Thus, the robot control unit 510 can control the robot 600 based on the integrated point cloud data that does not include data indicating the arm 601. Therefore, even when the stereo camera 220 and the color camera 23 capture the arm 601, the robot control unit 510 does not recognize the arm 601 as an obstacle, for example.

[0265] When the data processing device 3 does not include a synthesis unit and outputs a plurality of point group segment data or a plurality of non-SG point group data, the robot control unit 510 may also control the robot 600 based on the plurality of point group segment data or the plurality of non-SG point group data. In this case, the robot control unit 510 may, for example, synthesize a plurality of point group data or non-SG point group data to generate integrated point group data, and control the robot 600 based on the generated integrated point group data.

[0266] The robot control unit 510 may also be implemented by a cloud server. In this case, a communication unit connected to the Internet may also be provided in the robot 600, and the robot control unit 510 controls the robot 600 via the Internet. In addition, the data processing device 3 may also be implemented by a cloud server. In this case, the data processing device 3 may also communicate with the robot control unit 510 via the Internet. In addition, the control unit 30 of the data processing device 3 may also function as the robot control unit 510. In this case, the data processing device 3 functions as the robot control system 500. When the control unit 30 functions as the robot control unit 510, the integrated unit may also be included in the robot control unit 510. In addition, the robot control system 500 may also not include the sensor device 2.

[0267] As described above, the measurement system, sensor device, data processing device, and robot control system are described in detail, but the above description is illustrative in all aspects, and the present disclosure is not limited thereto. In addition, the above various examples can be combined and applied as long as they are not contradictory to each other. Moreover, countless examples that are not illustrated can be understood as being conceivable without departing from the scope of the present disclosure.

[0268] -Description of Reference Numerals-

[0269] 3 Data processing device

[0270] 260 color image data

[0271] 261, 261a, 261b Partial color image data

[0272] 265, 266, 266a, 266b color image segmentation data

[0273] 270 distance image data

[0274] 271, 271a, 271b Partial range image data

[0275] 275, 276, 276a, 276b Range image segmentation data

[0276] 285, 286, 286a, 286b Segmented Datasets

[0277] 290, 291, 292, 292a, 292b, 293, 293a, 293b, 370, 371, 371a, 371b, 372, 372a, 372b point group segment data

[0278] 295, 296, 297, 297ab, 298, 298ab, 380, 381, 381ab, 382, ​​382ab comprehensive point group data

[0279] 301, 351 Division

[0280] 302, 352 Transformation Unit

[0281] 303, 353 Point Group Processing Unit

[0282] 304, 354 General Affairs Department

[0283] 330 Program

[0284] 360, 361, 361a, 361b point group data

[0285] 365, 365a, 365b Partial point group data

[0286] 500 Robot Control System

[0287] 600 Robots

[0288] 1000 Measurement Space

[0289] 1001 measuring points.

Claims

1. A data processing device comprising: a segmentation unit that divides at least a portion of the range image data including distance information to a measurement point in a measurement space into a plurality of first range image segment data according to the distance information; A transformation unit transforms the plurality of first range image segment data into first point group segment data individually, wherein: The first point group segment data is point group data including coordinate information of measurement points in the measurement space; as well as The point group processing unit individually processes the plurality of first point group segment data obtained from the plurality of first range image segment data in the conversion unit.

2. The data processing device according to claim 1, wherein: The segmentation section divides first partial range image data including the distance information related to a first object in the measurement space in the range image data into the plurality of first range image segment data.

3. The data processing device according to claim 2, wherein: The segmentation unit divides the second portion of the range image data including the distance information related to the second object in the measurement space in the range image data into a plurality of second range image segment data according to the distance information, The conversion unit converts each of the plurality of second range image segment data individually into second point group segment data, wherein the second point group segment data is point group data including coordinate information of measurement points in the measurement space.

4. The data processing device according to claim 3, wherein: The dividing unit makes the number of divisions of the plurality of first range image segment data and the number of divisions of the plurality of second range image segment data different from each other.

5. The data processing device according to any one of claims 1 to 4, wherein: The conversion unit executes at least two of a plurality of conversion processes for converting the plurality of first range image segmented data into the plurality of first point group segmented data in parallel.

6. The data processing device according to any one of claims 1 to 5, wherein: The segmentation unit divides at least a portion of the color image data including the color information of the measurement point in the measurement space into a plurality of color image segment data according to the distance information corresponding to the color information. The plurality of first distance image segmentation data and the plurality of color image segmentation data constitute a plurality of segmentation data sets, wherein each set of the plurality of segmentation data sets is constituted by a first distance image segmentation data and a color image segmentation data divided by the same distance, The conversion unit converts each of the plurality of segment data sets individually into the first point group segment data including coordinate information and color information of measurement points in the measurement space.

7. A data processing device comprising: a conversion unit that converts at least a portion of the range image data including distance information to a measurement point in a measurement space into first point group data including coordinate information of the measurement point in the measurement space; a segmentation unit, which divides at least a portion of the first point group data into a plurality of first point group segment data according to the coordinate values ​​in the depth direction of the measurement space; as well as The point group processing unit processes each of the plurality of first point group segment data individually.

8. The data processing device according to claim 7, wherein: The conversion unit converts first partial range image data including the distance information related to the first object in the measurement space in the range image data into the first point group data, The dividing unit divides the first point group data into the plurality of first point group segment data.

9. The data processing device according to claim 8, wherein: The conversion unit converts the second partial range image data including the distance information related to the second object in the measurement space in the range image data into second point group data including coordinate information of the measurement points in the measurement space, The segmentation unit divides the second point group data into a plurality of second point group segment data according to coordinate values ​​in a depth direction of the measurement space.

10. The data processing device according to claim 7, wherein: The dividing unit divides first partial point group data including the coordinate information related to the first object in the measurement space in the first point group data into the plurality of first point group segment data.

11. The data processing device according to claim 10, wherein: The segmentation unit divides second partial point group data including the coordinate information related to the second object in the measurement space in the first point group data into a plurality of second point group segment data according to coordinate values ​​in a depth direction of the measurement space.

12. The data processing device according to claim 9 or 11, wherein: The dividing unit makes the number of divisions of the plurality of first point group segment data and the number of divisions of the plurality of second point group segment data different from each other.

13. The data processing device according to any one of claims 7 to 12, wherein: The conversion unit converts at least a part of a data set including color image data including color information of measurement points in the measurement space and the distance image data into the first point group data including coordinate information and color information of the measurement points in the measurement space.

14. The data processing apparatus according to claim 3, claim 4, claim 9 or claim 11, wherein: The point group processing unit processes each of the plurality of second point group segment data obtained in the conversion unit individually.

15. The data processing apparatus according to claim 3, claim 4, claim 9, claim 11 or claim 14, wherein: The data processing device further includes: an integrating unit that integrates the plurality of first point group segment data and the plurality of second point group segment data.

16. The data processing apparatus according to claim 15, wherein: the point group processing unit does not process at least one of the plurality of second point group segment data obtained in the transformation unit, The integrating unit integrates the plurality of first point group segment data processed by the point group processing unit and the plurality of second point group segment data including the at least one unprocessed second point group segment data.

17. The data processing device according to any one of claims 1 to 16, wherein: The point group processing unit performs processing on at least two first point group segment data of the plurality of first point group segment data in parallel.

18. The data processing device according to any one of claims 1 to 17, wherein: The point cloud processing unit makes a processing parameter different between processes on at least two first point cloud segment data among the plurality of first point cloud segment data.

19. A robot control system comprising: The dividing unit, the transforming unit, and the point group processing unit included in the data processing device according to any one of claims 1 to 18; and The robot control unit controls the robot based on the plurality of first point group segment data processed by the point group processing unit or integrated point group data obtained by integrating the plurality of first point group segment data.

20. The robot control system according to claim 19, wherein: The robot control unit recognizes an object in the measurement space based on the plurality of first point group segment data or the integrated point group data.

21. A program causing a computer device to execute the following processing: dividing at least a portion of the range image data including distance information to the measurement point in the measurement space into a plurality of first range image segment data according to the distance information, The plurality of first range image segmentation data are transformed into first point group segmentation data respectively and individually, wherein: The first point group segment data is point group data including coordinate information of measurement points in the measurement space. The plurality of first point group segmented data respectively obtained from the plurality of first range image segmented data are processed individually.

22. A program causing a computer device to execute the following processing: transforming at least a portion of the range image data including distance information to the measurement points in the measurement space into first point group data including coordinate information of the measurement points in the measurement space, dividing at least a portion of the first point group data into a plurality of first point group segment data according to the coordinate values ​​in the depth direction of the measurement space, The plurality of first point group segment data are processed separately.

Citation Information

Patent Citations

  • Point group data processing device, point group data processing system, point group data processing method, and point group data processing program

    JP2012013660A