Information processing apparatus, information processing method, imaging apparatus, and information processing system

Abstract

To provide an information processing device, an information processing method, an imaging device, and an information processing system capable of easily correcting depth information.SOLUTION: The information processing apparatus 10 includes a controller 11. The control unit 11 acquires depth information in a predetermined space and distortion information of the imaging device 20 that generates the depth information. The control unit 11 corrects the depth information based on the distortion information to generate corrected depth information.SELECTED DRAWING: Figure 1

Description

Cross-reference to related applications

[0001] This application claims priority to Japanese Patent Application No. 2021-121955 (filed July 26, 2021), the entire disclosure of which is incorporated herein by reference. [Technical Field]

[0002] The present disclosure relates to an information processing device, an information processing method, an imaging device, and an information processing system. [Background technology]

[0003] Conventionally, a system for correcting depth information is known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2018-534699 Summary of the Invention

[0005] An information processing device according to an embodiment of the present disclosure includes a control unit that acquires depth information in a predetermined space and distortion information of an image capturing device that generates the depth information, and corrects the depth information based on the distortion information to generate corrected depth information.

[0006] An information processing method according to an embodiment of the present disclosure includes acquiring depth information in a predetermined space and distortion information of an image capturing device that generates the depth information, and correcting the depth information based on the distortion information to generate corrected depth information.

[0007] An imaging device according to an embodiment of the present disclosure includes at least two image sensors that capture an image of a predetermined space, an optical system that causes the image sensors to form an image of the predetermined space, a storage unit, and a control unit. The storage unit stores, as distortion information, information regarding enlargement, reduction, or distortion of images captured by each image sensor due to at least one of characteristics of the optical system and errors in the placement of the image sensors. The control unit generates depth information of the predetermined space based on images of the predetermined space captured by each image sensor, and outputs the depth information and the distortion information.

[0008] An information processing system according to an embodiment of the present disclosure includes an information processing device and an imaging device. The imaging device includes at least two imaging elements for capturing an image of a predetermined space, an optical system for forming an image of the predetermined space on the imaging elements, a storage unit, and a control unit. The storage unit stores, as distortion information, information regarding enlargement, reduction, or distortion of images captured by each imaging element due to at least one of characteristics of the optical system and errors in the placement of the imaging elements. The control unit generates depth information of the predetermined space based on images captured by each imaging element and outputs the depth information and the distortion information to the information processing device. The information processing device acquires the depth information and the distortion information from the imaging device and corrects the depth information based on the distortion information to generate corrected depth information. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a robot control system according to an embodiment. [Figure 2] FIG. 1 is a schematic diagram illustrating an example of the configuration of a robot control system according to an embodiment. [Figure 3] FIG. 10 is a schematic diagram showing an example of a configuration for capturing an image of a depth measurement point using an imaging device. [Figure 4] FIG. 10 is a diagram illustrating an example of the positional relationship between an imaging device and a measurement point. [Figure 5]FIG. 10 compares a distorted image with an undistorted image. [Figure 6] 10 is a graph showing an example of a depth calculation result and an example of a true depth value. [Figure 7] 10 is a graph showing an example of a depth correction value. [Figure 8] 1 is a flowchart illustrating an example of a procedure of an information processing method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] There is a demand for easily correcting depth information. According to the information processing device, information processing method, imaging device, and information processing system according to the present disclosure, depth information can be easily corrected.

[0011] (Overview of Robot Control System 1) As illustrated in FIGS. 1 and 2, a robot control system 1 according to one embodiment includes a robot 40, a robot controller 10, and an imaging device 20. The robot 40 operates in a predetermined workspace. The imaging device 20 generates depth information of the workspace in which the robot 40 operates. The imaging device 20 generates the depth information of the workspace based on an (X, Y, Z) coordinate system. The robot controller 10 operates the robot 40 based on the depth information generated by the imaging device 20. The robot controller 10 operates the robot 40 based on an (X_RB, Y_RB, Z_RB) coordinate system.

[0012] The (X_RB,Y_RB,Z_RB) coordinate system may be set as the same coordinate system as the (X,Y,Z) coordinate system, or may be set as a different coordinate system. When the (X_RB,Y_RB,Z_RB) coordinate system is set as a coordinate system different from the (X,Y,Z) coordinate system, the robot controller 10 converts the depth information generated in the (X,Y,Z) coordinate system by the imaging device 20 into depth information in the (X_RB,Y_RB,Z_RB) coordinate system for use.

[0013] The number of robots 40 and robot controllers 10 is not limited to one, but may be two or more. The number of imaging devices 20 per workspace may be one, or two or more. Each component will be described in detail below.

[0014] <Robot Controller 10> The robot controller 10 includes a control unit 11 and a storage unit 12. The robot controller 10 is also referred to as an information processing device. Note that in the present invention, the information processing device is not limited to the robot controller 10, and may be another component of the robot control system 1. The information processing device may be, for example, an imaging device 20.

[0015] The control unit 11 may be configured to include at least one processor to realize various functions of the robot controller 10. The processor may execute programs that realize various functions of the robot controller 10. The processor may be realized as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be realized as multiple integrated circuits and discrete circuits that are connected to each other in a communicative manner. The processor may be configured to include a CPU (Central Processing Unit). The processor may be configured to include a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The processor may be realized based on various other known technologies.

[0016] The robot controller 10 further includes a storage unit 12. The storage unit 12 may include an electromagnetic storage medium such as a magnetic disk, or may include a memory such as a semiconductor memory or a magnetic memory. The storage unit 12 may be configured as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage unit 12 stores various information and programs executed by the control unit 11. The storage unit 12 may function as a work memory for the control unit 11. The control unit 11 may be configured to include at least a part of the storage unit 12.

[0017] The robot controller 10 may further include a communication device configured to be able to communicate with the imaging device 20 and the robot 40 via wired or wireless communication. The communication device may be configured to be able to communicate using a communication method based on various communication standards. The communication device can be configured using known communication technology. Detailed descriptions of the hardware of the communication device will be omitted. The functions of the communication device may be realized by a single interface, or may be realized by separate interfaces for each connection destination. The control unit 11 may be configured to be able to communicate with the imaging device 20 and the robot 40. The control unit 11 may be configured to include a communication device.

[0018] <Robot 40> As illustrated in FIG. 2, the robot 40 includes an arm 42 and an end effector 44 attached to the arm 42. The arm 42 may be configured as, for example, a six- or seven-axis vertical articulated robot. The arm 42 may be configured as a three- or four-axis horizontal articulated robot or a SCARA robot. The arm 42 may be configured as a two- or three-axis Cartesian robot. The arm 42 may be configured as a parallel link robot or the like. The number of axes constituting the arm 42 is not limited to those illustrated.

[0019] The end effector 44 may include, for example, a gripping hand configured to grip a workpiece. The gripping hand may have multiple fingers. The gripping hand may have two or more fingers. The fingers of the gripping hand may have one or more joints. The end effector 44 may include a suction hand configured to pick up a workpiece by suction. The end effector 44 may include a scooping hand configured to scoop up a workpiece. The end effector 44 may include a tool such as a drill and be configured to perform various processes, such as drilling a hole in a workpiece. The end effector 44 is not limited to these examples and may be configured to perform various other operations.

[0020] The robot 40 can control the position of the end effector 44 by operating the arm 42. The end effector 44 may have an axis that serves as a reference for the direction in which it acts on a workpiece. If the end effector 44 has an axis, the robot 40 can control the direction of the axis of the end effector 44 by operating the arm 42. The robot 40 controls the start and end of the operation of the end effector 44 acting on the workpiece. The robot 40 can move or process the workpiece by controlling the operation of the end effector 44 while controlling the position of the end effector 44 or the direction of the axis of the end effector 44.

[0021] The robot 40 may further include sensors that detect the state of each component of the robot 40. The sensors may detect information about the actual position or posture of each component of the robot 40, or about the speed or acceleration of each component of the robot 40. The sensors may detect forces acting on each component of the robot 40. The sensors may detect the current flowing through or the torque of a motor that drives each component of the robot 40. The sensors can detect information obtained as a result of the actual operation of the robot 40. The robot controller 10 can grasp the result of the actual operation of the robot 40 by obtaining the detection results of the sensors.

[0022] The robot 40 further includes, but is not required to include, a mark 46 attached to the end effector 44. The robot controller 10 recognizes the position of the end effector 44 based on an image of the mark 46 captured by the imaging device 20. The robot controller 10 can calibrate the robot 40 using the position of the end effector 44 determined based on the detection results of the sensor and the result of recognizing the position of the end effector 44 using the mark 46.

[0023] <Imaging device 20> As shown in FIG. 1, the imaging device 20 includes an imaging element 21, a control unit 22, and a storage unit 23. As will be described later, the control unit 22 calculates the distance from the imaging element 21 to a measurement point 52 (see FIG. 3) based on an image of a workspace including the measurement point 52 captured by the imaging element 21. The distance to the measurement point 52 is also referred to as depth. The control unit 22 generates depth information including the calculation results of the distance (depth) to each measurement point 52 included in the workspace, and outputs the depth information to the robot controller 10. The imaging device 20 may be configured as a 3D stereo camera. Note that the distance (depth) information may be, for example, the distance from the imaging element 21 along the Z-axis direction, or may be a distance taking into account each component of an xyz coordinate system originating from the imaging element 21.

[0024] The control unit 22 may be configured to include at least one processor. The processor may execute programs that realize various functions of the imaging device 20. The storage unit 23 may be configured to include an electromagnetic storage medium such as a magnetic disk, or may be configured to include a memory such as a semiconductor memory or a magnetic memory. The storage unit 23 may be configured as an HDD or SSD. The storage unit 23 stores various information, programs executed by the control unit 22, etc. The storage unit 23 may function as a work memory for the control unit 22. The control unit 22 may be configured to include at least a part of the storage unit 23.

[0025] As shown in FIGS. 3 and 4, the imaging element 21 includes a left imaging element 21L, a right imaging element 21R, a left optical system 24L, and a right optical system 24R. In FIGS. 3 and 4, an (X, Y, Z) coordinate system is established with the imaging device 20 as the reference. The left imaging element 21L and the right imaging element 21R are positioned side by side in the X-axis direction. The left optical system 24L has an optical system center 26L and an optical axis 25L that passes through the optical system center 26L and extends in the Z-axis direction. The right optical system 24R has an optical system center 26R and an optical axis 25R that passes through the optical system center 26L and extends in the Z-axis direction. The left imaging element 21L and the right imaging element 21R capture images of an object 50 located on the negative side of the Z-axis, formed by the left optical system 24L and the right optical system 24R, respectively. The left imaging element 21L captures an image of the object 50 formed by the left optical system 24L and a measurement point 52 included in the object 50. The image captured by the left imaging element 21L is shown as a left captured image 51L in Fig. 4. The right imaging element 21R captures an image of the object 50 formed by the right optical system 24R and a measurement point 52 included in the object 50. The image captured by the right imaging element 21R is shown as a right captured image 51R in Fig. 4.

[0026] The left captured image 51L includes a left image 52L of the measurement point 52, which is a representation of the measurement point 52. The right captured image 51R includes a right image 52R of the measurement point 52, which is a representation of the measurement point 52. Furthermore, a virtual measurement point image 52V is depicted in the left captured image 51L and the right captured image 51R. The virtual measurement point image 52V represents the virtual measurement point 52 that appears in the captured image when the imaging element 21 captures a virtual measurement point 52 located at infinity from the imaging device 20.

[0027] The virtual measurement point image 52V is located on the optical axis 25L of the left optical system 24L and the optical axis 25R of the right optical system 24R. The virtual measurement point image 52V is located at the center of the left captured image 51L and the right captured image 51R. In FIG. 4, the virtual measurement point image 52V represents a virtual measurement point 52 that appears in a captured image when the image sensor 21 captures a virtual measurement point 52 located at infinity along a line extending from the midpoint between the two image sensors 21 in the negative direction of the Z axis. In this case, the virtual measurement point image 52V is located at the center of the captured image in the X-axis direction. If the measurement point 52 is shifted from the midpoint between the two image sensors 21 in the positive or negative direction of the X axis, the virtual measurement point image 52V will be shifted from the center of the captured image in the X-axis direction.

[0028] In other words, an actual measurement point 52 located a finite distance from the imaging device 20 is imaged at a position shifted from the virtual measurement point image 52V, such as a left image 52L of the measurement point 52 in the left captured image 51L and a right image 52R of the measurement point 52 in the right captured image 51R. The positions of the left image 52L and the right image 52R of the measurement point 52 are determined as follows. First, an incident point 27L is assumed between a dashed line connecting the optical system center 26L of the left optical system 24L and the optical system center 26R of the right optical system 24R, and a dashed line connecting the measurement point 52 and the virtual measurement point image 52V in the left captured image 51L. Also, an incident point 27R is assumed between a dashed line connecting the measurement point 52 and the virtual measurement point image 52V in the right captured image 51R. A left image 52L of the measurement point 52 in the left captured image 51L is located at the intersection of a dashed line extending from the incident point 27L in the positive direction of the Z axis and a dashed line connecting the virtual measurement point images 52V in the left captured image 51L and the right captured image 51R. A right image 52R of the measurement point 52 in the right captured image 51R is located at the intersection of a dashed line extending from the incident point 27R in the positive direction of the Z axis and a dashed line connecting the virtual measurement point images 52V in the left captured image 51L and the right captured image 51R.

[0029] The control unit 22 of the imaging device 20 can calculate the distance from the imaging device 20 to the measurement point 52 based on the difference between the X coordinate of the left image 52L of the measurement point 52 captured in the left captured image 51L and the X coordinate of the right image 52R of the measurement point 52 captured in the right captured image 51R, and the X coordinate of the virtual measurement point image 52V. The control unit 22 may calculate the distance from the imaging device 20 to the measurement point 52 further based on the center-to-center distance between the two imaging elements 21 and the focal length of the optical system that forms an image on each imaging element 21. In FIG. 4, the distance (depth) from the imaging device 20 to the measurement point 52 is represented by D. An example of calculating the depth will be described below.

[0030] The parameters used to calculate depth are described below. The distance between the center of the left captured image 51L and the center of the right captured image 51R is represented by T. The difference between the X coordinate of the virtual measurement point image 52V in the left captured image 51L and the X coordinate of the left image 52L of the measurement point 52 is represented by XL. The sign of XL is positive when the left image 52L of the measurement point 52 is located in the positive direction of the X axis relative to the virtual measurement point image 52V in the left captured image 51L. The difference between the X coordinate of the virtual measurement point image 52V in the right imaging element 21R and the X coordinate of the right image 52R of the measurement point 52 is represented by XR. The sign of XR is positive when the right image 52R of the measurement point 52 is located in the negative direction of the X axis relative to the virtual measurement point image 52V in the right captured image 51R. That is, the signs of XL and XR are positive when the left image 52L and the right image 52R of the measurement point 52 deviate in directions approaching each other from the virtual measurement point image 52V.

[0031] The focal length of the left optical system 24L and the right optical system 24R is represented by F. The left imaging element 21L and the right imaging element 21R are disposed so that the focal points of the left optical system 24L and the right optical system 24R are located on the imaging planes of the left imaging element 24L and the right optical system 24R. The focal length (F) of the left optical system 24L and the right optical system 24R corresponds to the distance from the left optical system 24L and the right optical system 24R to the imaging planes of the left imaging element 21L and the right imaging element 21R.

[0032] Specifically, the control unit 22 can calculate the depth by operating as follows.

[0033] The control unit 22 detects the X coordinate of the left image 52L of the measurement point 52 that appears in the left captured image 51L. The control unit 22 calculates the difference XL between the X coordinate of the left image 52L of the measurement point 52 and the X coordinate of the virtual measurement point image 52V. The control unit 22 detects the X coordinate of the right image 52R of the measurement point 52 that appears in the right captured image 51R. The control unit 22 calculates the difference XR between the X coordinate of the right image 52R of the measurement point 52 and the X coordinate of the virtual measurement point image 52V.

[0034] Here, it is assumed that a first triangle exists in the XZ plane, with the measurement point 52, the virtual measurement point image 52V in the left captured image 51L, and the virtual measurement point image 52V in the right captured image 51R as vertices. Also, it is assumed that a second triangle exists in the XZ plane, with the measurement point 52, the assumed incident point 27L in the left optical system 24L, and the assumed incident point 27R in the right optical system 24R as vertices. The Z coordinate of the optical system center 26L of the left optical system 24L is the same as the Z coordinate of the optical system center 26R of the right optical system 24R. Also, the Z coordinate of the virtual measurement point image 52V in the left captured image 51L is the same as the Z coordinate of the virtual measurement point image 52V in the right captured image 51R. Therefore, the first triangle and the second triangle are similar to each other.

[0035] The distance between the two virtual measurement point images 52V of the first triangle is T. The distance between the incident points 27L and 27R of the second triangle is T-(XL+XR). Based on the similarity between the first triangle and the second triangle, the following equation (1) holds: T / {T-(XL+XR)}=D / (DF) (1)

[0036] Based on equation (1), equation (2) for calculating D is derived as follows: D=T×F / (XL+XR) (2)

[0037] In equation (2), the larger XL+XR is, the smaller the calculated value of D is. On the other hand, when XL+XR=0 holds, D is infinite. For example, when the left image 52L of the left captured image 51L coincides with the virtual measurement point image 52V and the right image 52R of the right captured image 51R coincides with the virtual measurement point image 52V, D is calculated to be infinite. In fact, since it is defined that the left image 52L and the right image 52R coincide with the virtual measurement point image 52V when the measurement point 52 is located at infinity, it can be said that D can be correctly calculated using equation (2).

[0038] As described above, the control unit 22 can calculate the depth based on two captured images of the measurement points 52 taken by the two image sensors 21. The control unit 22 calculates the distances to the multiple measurement points 52 included in the captured images of the workspace of the robot 40, generates depth information indicating the distance (depth) to each measurement point 52, and outputs this information to the robot controller 10. The depth information can be expressed by a function that takes the X coordinate and the Y coordinate as arguments in the (X, Y, Z) coordinate system of the image capture device 20. The depth information can also be expressed as a two-dimensional map in which depth values ​​are plotted on the XY plane of the image capture device 20.

[0039] (Effect of image distortion on depth information) As described above, the imaging device 20 generates depth information based on the position of the image of the measurement point 52 captured in two captured images of the workspace captured by the left imaging element 21L and the right imaging element 21R. Here, the captured images may be captured as images that are enlarged, reduced, or distorted relative to the actual workspace. Enlargement, reduction, or distortion of the captured image relative to the actual workspace is also referred to as distortion of the captured image. If distortion occurs in the captured image, the position of the image of the measurement point 52 captured in the captured image may be shifted. As a result, distortion of the captured image causes an error in the depth calculation result. Furthermore, distortion of the captured image causes an error in the depth information that represents the depth calculation result.

[0040] 5, a distorted image 51Q is assumed as a captured image having distortion. On the other hand, a non-distorted image 51P is assumed as a captured image without distortion. The distorted image 51Q is assumed to be reduced in size relative to the non-distorted image 51P. The reduction ratios of the distorted image 51Q in the X-axis and Y-axis directions are assumed to be smaller than the reduction ratios in the other directions.

[0041] The measurement point image 52Q in the distorted image 51Q is closer to the virtual measurement point image 52V than the measurement point image 52P in the undistorted image 51P. That is, X_DIST, which represents the distance from the measurement point image 52Q to the virtual measurement point image 52V in the undistorted image 51Q, is shorter than XL or XR, which represent the distance from the measurement point image 52P to the virtual measurement point image 52V in the distorted image 51P. Therefore, the value of XL+XR in the above-described equation (2) for calculating the depth (D) becomes smaller. As a result, the result of the depth (D) calculated by the control unit 22 of the imaging device 20 based on the distorted image 51Q becomes larger than the calculation result based on the undistorted image 51P. That is, distortion of the captured image may cause an error in the calculation result of the depth (D).

[0042] (Depth information correction) The control unit 11 of the robot controller 10 acquires depth information from the imaging device 20. The control unit 11 further acquires information related to distortion of the imaging device 20. The information related to distortion of the imaging device 20 is also referred to as distortion information. The distortion information may be, for example, optical and geometric parameters acquired during manufacturing inspection of the imaging device 20. The distortion information represents distortion in the left captured image 51L and the right captured image 51R. As described above, the error in the calculation result of the depth (D) is determined by the distortion information. Therefore, the control unit 11 can correct the error in the depth (D) represented by the depth information acquired from the imaging device 20 based on the distortion information. A specific example of a correction method will be described below.

[0043] The control unit 11 acquires distortion information of the imaging device 20. The control unit 11 may acquire distortion information for each of the left imaging element 21L and the right imaging element 21R as the distortion information. The control unit 11 may acquire the distortion information of the imaging device 20 from an external device such as cloud storage. The control unit 11 may acquire the distortion information from the imaging device 20. In this case, the imaging device 20 may store the distortion information in the storage unit 23. The control unit 11 may acquire the distortion information from the storage unit 23 of the imaging device 20. The imaging device 20 may store address information in the storage unit 23 that specifies where the distortion information of the imaging device 20 itself is stored. The address information may include, for example, an IP address or a URL (Uniform Resource Locator) for accessing an external device such as cloud storage. The control unit 11 may acquire the address information from the imaging device 20 and acquire the distortion information by accessing the external device specified in the address information.

[0044] The distortion information may include distortion of the left optical system 24L and the right optical system 24R. The distortion of each optical system may include distortion that occurs in the captured image due to the characteristics of each optical system. The distortion information may include distortion of the imaging planes of the left imaging element 21L and the right imaging element 21R. The distortion information may include distortion that occurs in the captured image due to errors in the positioning of the left optical system 24L or the right optical system 24R or the left imaging element 21L or the right imaging element 21R.

[0045] The characteristics of each optical system may be, for example, the curvature or size of a curved lens, etc. The error in the placement of the image sensor 21 may be, for example, an error in the planar position of the image sensor 21 when it is mounted, or a manufacturing error such as the tilt of the optical axis.

[0046] For example, as shown in FIG. 6, the depth information is represented as a graph of depth values ​​calculated at each X coordinate when the Y coordinate in the captured image is fixed at a predetermined value. Here, the fixed Y coordinate value may be the Y coordinate of the center of the captured image, or an arbitrary Y coordinate within the captured image. In FIG. 6, the horizontal axis represents the X coordinate. The vertical axis represents the depth value (D) at each X coordinate. The solid line represents depth information composed of depth values ​​calculated by the imaging device 20. On the other hand, the dashed line represents depth information composed of true depth values. The difference between the depth calculated by the imaging device 20 and the true depth is caused by distortion of the imaging device 20.

[0047] Here, the control unit 11 can estimate errors in the depth values ​​at each X coordinate based on the distortion information. Specifically, for a measurement point 52 located at (X1, Y1), the control unit 11 estimates errors in the positions of the left image 52L and the right image 52R of the measurement point 52 in the captured image caused by distortion based on the distortion information. The control unit 11 can calculate a correction value for the depth value of the measurement point 52 located at (X1, Y1) based on the estimated errors in the positions of the left image 52L and the right image 52R of the measurement point 52. The correction value for the depth value is represented as D_corr. For example, if XL and XR are each smaller by ΔXerr / 2 due to an implementation error of the image sensor 21, the correction value (D_corr) for the depth value of the measurement point 52 located at (X1, Y1) can be expressed, for example, by the following equation (3): D_corr={D 2 / (F·T)}×ΔXerr (3)

[0048] In equation (3), D is the depth value before correction. F and T are the focal length of the image capture device 20 and the center-to-center distance between the two image capture elements 21, and are parameters determined as specifications of the image capture device 20. ΔXerr can be obtained, for example, as a value estimated based on distortion information. ΔXerr can also be obtained, for example, as the distortion information itself.

[0049] The control unit 11 can calculate the correction value (D_corr) of the depth value by substituting the estimated result of the position error of the left image 52L and the right image 52R at each measurement point 52 in the captured image into equation (3). The correction value (D_corr) of the depth value calculated by the control unit 11 can be expressed, for example, as the graph shown in FIG. 7. In FIG. 7, the horizontal axis represents the X coordinate, and the vertical axis represents the correction value (D_corr) of the depth value at each X coordinate. The equation for calculating the correction value (D_corr) of the depth value is not limited to the above-mentioned equation (3), and may be expressed by various other mathematical equations.

[0050] As described above, the control unit 11 can estimate a correction value (D_corr) for the depth value based on the distortion information. The control unit 11 can estimate the correction value for each measurement point 52, correct the depth value for each measurement point 52, and bring the depth value for each measurement point 52 closer to the true value. The control unit 11 can generate corrected depth information representing the corrected depth value by correcting the depth value for each measurement point 52 in the depth information. The control unit 11 may control the robot 40 based on the corrected depth information. By correcting the depth value, for example, the positional accuracy of the robot 40 relative to an object 50 located in the workspace can be improved.

[0051] (Example of information processing procedure) The control unit 11 of the robot controller 10 may execute an information processing method including the steps of the flowchart illustrated in Fig. 8. The information processing method may be realized as an information processing program executed by a processor constituting the control unit 11. The information processing program may be stored in a non-transitory computer-readable medium.

[0052] The control unit 11 acquires depth information from the imaging device 20 (step S1). The control unit 11 acquires distortion information from the imaging device 20 that generated the depth information (step S2). The control unit 11 corrects the depth information based on the distortion information and generates corrected depth information (step S3). After executing the procedure of step S3, the control unit 11 ends execution of the procedure of the flowchart in FIG. 8. The control unit 11 may control the robot 40 based on the corrected depth information.

[0053] The control unit 11 may further acquire color information from the imaging device 20. The control unit 11 may acquire an image of the workspace captured by the imaging device 20 as color information. In other words, the captured image may have color information. The control unit 11 may then detect the location of an object 50 or the like located in the workspace based on the corrected depth information and the color information. As a result, detection accuracy can be improved compared to when the location of the object 50 or the like is detected based on depth information. In this case, for example, the control unit 11 may generate integrated information in which the corrected depth information and the color information are integrated. As will be described later, correction may be performed based on the corrected depth information.

[0054] When controlling the robot 40, the control unit 11 can also convert the working space expressed in the (X, Y, Z) coordinate system into the robot's configuration coordinate system. The robot's configuration coordinate system refers to a coordinate system configured by parameters that indicate the robot's movements, for example.

[0055] (summary) As described above, the robot controller 10 according to this embodiment can correct depth information acquired from the imaging device 20 based on distortion information from the imaging device 20. If the robot controller 10 were to acquire a captured image from the imaging device 20 and correct the distortion of the captured image itself, the amount of communication between the robot controller 10 and the imaging device 20 and the computational load on the robot controller 10 would increase. The robot controller 10 according to this embodiment can reduce the amount of communication between the robot controller 10 and the imaging device 20 by acquiring depth information from the imaging device 20 with a data volume smaller than that of the initial captured image. Furthermore, by estimating a depth correction value based on distortion information, the robot controller 10 can reduce the computational load compared to correcting the distortion of the captured image itself and calculating the depth based on the corrected captured image. From the above, the robot controller 10 according to this embodiment can easily correct depth information. Furthermore, only the depth information can be easily corrected without changing the accuracy of the imaging device 20 itself.

[0056] Furthermore, the robot controller 10 according to this embodiment can correct the depth information, thereby improving the positional accuracy of the robot 40 relative to the object 50 located in the workspace of the robot 40.

[0057] (Other embodiments) Other embodiments are described below.

[0058] <Application to spaces other than the working space of the robot 40> The depth information can be acquired in various spaces, not limited to the workspace of the robot 40. The robot control system 1 and the robot controller 10 may be replaced by an information processing system and an information processing device, respectively, that process depth information of various spaces. The various spaces from which the depth information is acquired are also referred to as predetermined spaces.

[0059] The space where depth information is acquired may include, for example, a space where an AGV (Automatic Guided Vehicle) equipped with a 3D stereo camera drives to perform the operation of pressing a door open / close switch. The depth information acquired in this space is used to improve the accuracy of measuring the distance that needs to be traveled from the current location.

[0060] The space in which depth information is acquired may include, for example, a space in which a VR (Virtual Reality) or 3D game device equipped with a 3D stereo camera is operated. In this space, the measurement results of the distance to a controller, marker, etc. held by a player of the VR or 3D game device are acquired as depth information. The depth information acquired in this space is used to improve the accuracy of the distance to the controller, marker, etc. Improving the distance accuracy improves the accuracy of matching between the position of an object (e.g., a punching ball) that exists in the space in which the player is located and the position of the player's hand in the virtual space.

[0061] <Generating point cloud data> The control unit 11 of the robot controller 10 may acquire depth information from the imaging device 20 in the form of point cloud data including coordinate information of measurement points in a measurement space. In other words, the depth information may have the form of point cloud data. In other words, the point cloud data may have depth information. Point cloud data is data representing a point cloud (also called a measurement point cloud), which is a collection of multiple measurement points in a measurement space. Point cloud data can also be said to be data representing an object in the measurement space using multiple points. Point cloud data is also data representing the surface shape of an object in the measurement space. The point cloud data includes coordinate information representing the positions of points on the surface of the object in the measurement space. The distance between two measurement points included in the point cloud is, for example, the actual distance in the measurement space. Depth information in the form of point cloud data can have a lower data density and a smaller data volume than depth information based on the initial data captured by the imaging element 21. As a result, the load of calculation processing when correcting the depth information can be further reduced. The conversion of the initial format of the depth information into the point cloud data format or the generation of the point cloud data including the depth information may be performed by the control unit 22 of the image capturing device 20.

[0062] Furthermore, when point cloud data containing depth information is generated, color information may be integrated into this point cloud data, and then the depth information and color information may be corrected. Even in this case, since the point cloud data has a smaller data capacity than the initial captured image, it is possible to reduce the computational overhead when correcting depth information, etc., and ultimately reduce the amount of communication between the robot controller 10 and the image capture device 20. In this case, the integration of the depth information and color information may be performed by the control unit 22 of the image capture device 20.

[0063] <Peripheral illumination correction for captured images> In the above-described embodiment, the imaging device 20 outputs depth information of a predetermined space to an information processing device. The imaging device 20 may output an image of the predetermined space, such as an RGB (Red, Green, Blue) image or a monochrome image, directly to the information processing device. The information processing device may correct the image of the predetermined space based on the corrected depth information. For example, the information processing device may correct color information of the image of the predetermined space based on the corrected depth information. Color information refers to, for example, hue, saturation, luminance, or brightness.

[0064] The information processing device may correct, for example, the brightness or luminance of the image based on the corrected depth information. The information processing device may correct, for example, the peripheral illumination of the image based on the corrected depth information. Note that peripheral illumination refers to the brightness of the light at the periphery of the lens of the imaging device 20. Furthermore, since the brightness of the light at the periphery of the lens is reflected in the brightness of the periphery or corners of the captured image, peripheral illumination can also refer to, for example, the brightness of the periphery or corners of the image. If the imaging device 20 has a lens, the captured image may appear darker at the periphery than at the center due to, for example, a difference in luminous flux density between the center and periphery of the lens caused by lens distortion of the imaging device 20. However, even if the peripheral illumination is low and the periphery or corners of the image are dark, the information processing device can correct the peripheral illumination or color information of the periphery of the image based on the corrected depth information, thereby generating image data that does not interfere with robot control.

[0065] The information processing device may correct peripheral illumination or color information of an image based on the magnitude of a depth correction value corresponding to each pixel of the image. For example, the larger the depth correction value, the more peripheral illumination or color information corresponding to that correction value may be corrected.

[0066] The information processing device may correct peripheral illumination or color information of the image, or may perform the correction when integrating depth information with color information of an RGB image or the like.

[0067] The information processing device may also generate corrected depth information by correcting depth information in a specified space based on distortion information from the imaging device 20, and correct the peripheral light intensity or color information of the image based on the generated corrected depth information.

[0068] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined or divided into one.

[0069] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0070] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein. [Explanation of symbols]

[0071] 1. Robot Control System 10 Robot controller (11: control unit, 12: memory unit) 20 imaging device (21: imaging element, 21L: left imaging element, 21R: right imaging element, 22: memory unit, 24L: left optical system, 24R: right optical system, 25L, 25R: optical axis, 26L, 26R: optical system center, 27L, 27R: incident point) 40 Robot (42: Arm, 44: Hand, 46: Mark) 50 objects 51L, 51R Left captured image, Right captured image 51P undistorted image 51Q Distortion Image 52 measurement points 52L, 52R, 52P, 52Q measurement point image 52V Virtual measurement point image

Claims

[Claim 1] A control unit is provided, The control unit acquiring depth information in a predetermined space and distortion information of an imaging device that generates the depth information; correcting the depth information based on the distortion information to generate corrected depth information; Information processing device.

Images