Imaging method, imaging system, manufacturing system, and method for manufacturing product

Abstract

The present invention discloses an imaging method, an imaging system, a manufacturing system, and a method for manufacturing a product. The imaging system includes a plurality of cameras and a controller, wherein the controller detects synchronization deviation of image capture timing of the plurality of cameras by using images captured by the plurality of cameras respectively.

Description

Technical Field

[0001] The present disclosure relates to an imaging method and an imaging system that synchronize image capture timings of a plurality of cameras and capture an image of an object through the plurality of cameras, and to a manufacturing system and method for manufacturing a product using the imaging method or the imaging system. Background Art

[0002] In recent years, in production lines of industrial products and the like, assembly operations are performed by assembly manufacturing devices including robot devices instead of manual assembly operations.

[0003] In such a manufacturing system, in some cases, a camera and an image processing device for two-dimensional or three-dimensional measurement of a workpiece are used to perform measurement or inspection of the workpiece required for assembly operation. For example, in the case where information in the depth direction is required for measurement or inspection, a method of performing three-dimensional measurement of an object using the principle of triangulation by using a stereo camera including two or more cameras is used. In this three-dimensional measurement, the difference (i.e., parallax) in the position of an object between multiple images captured by multiple cameras is calculated, and three-dimensional information is obtained by converting this parallax into a depth amount.

[0004] In such three-dimensional measurement, in some cases, it is necessary to measure the workpiece with high accuracy while the camera or the workpiece is relatively moving or relatively vibrating. In this case, when there is a difference in image capture time between multiple cameras constituting a stereo camera, the position of the object changes in the image captured by the camera, so the parallax cannot be accurately calculated. In this case, three-dimensional information such as depth cannot be accurately obtained. Therefore, it is necessary to accurately synchronize the image capture timing so that there is no time difference between the multiple cameras.

[0005] As disclosed in Japanese Patent Laid-Open No. 2011-239379, conventionally, a configuration in which one of the cameras includes a communication section for controlling another camera to match the image capture timing of a plurality of cameras constituting a stereo camera is known. This configuration allows an image capture instruction to be output from one camera to another camera at an arbitrary timing, thereby reducing the image capture time difference between the cameras. In addition, as disclosed in Japanese Patent Laid-Open No. 2018-007031, a configuration in which each of the plurality of cameras constituting a stereo camera has a function of storing image capture time and images in association with each other and an image captured almost simultaneously by one camera and another camera for measurement is selected based on the image capture time is also known. In addition, as disclosed in Japanese Patent Laid-Open No. 2014-175931, a configuration in which synchronized image capture is performed by a plurality of cameras by using a stroboscopic light is also known.

[0006] In the configuration of Japanese Patent Laid-Open No. 2011-239379, each of the plurality of cameras constituting the stereo camera needs to have a communication section for transmitting a trigger signal. As a result, the size and cost of each camera increase, and the stereo camera cannot be installed in an environment where the distance between the cameras is large or where it is difficult to provide wiring between the cameras.

[0007] In addition, in the configuration of Japanese Patent Laid-Open No. 2018-007031, each camera needs to have a function of storing the image capture time and the image in association with each other. As a result, the system becomes more complicated and expensive, and in the case where the image capture start time or image capture cycle of the camera is different, the image capture time cannot be accurately matched between the cameras. Summary of the invention

[0008] In view of the above problems, an embodiment of the present disclosure detects a synchronization deviation of image capturing timings of a plurality of cameras.

[0009] According to an embodiment of the present disclosure, an imaging system includes a plurality of cameras and a controller, wherein the controller detects a synchronization deviation of image capturing timings of the plurality of cameras by using images respectively captured by the plurality of cameras.

[0010] According to the above configuration, it is possible to detect a synchronization deviation of the image capturing timings of a plurality of cameras.

[0011] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A and 1B is an explanatory diagram illustrating the configuration of an imaging system including a plurality of cameras.

[0013] Figure 2 It is a graphic Figure 1A and 1B An illustration of the device configuration of a monocular camera.

[0014] Figure 3 It is a graphic Figure 2 A state transition diagram showing the state changes of an image sensor.

[0015] Figure 4 is a flow chart illustrating a method for detecting synchronization deviation in the first embodiment.

[0016] Figure 5 This is an explanatory diagram showing the output of the monocular camera of the first embodiment in a time line.

[0017] Figure 6 is a flowchart illustrating a method of synchronization in the first embodiment.

[0018] Figure 7 is an explanatory diagram illustrating a configuration for synchronized image capturing in the second embodiment.

[0019] Fig. 8A and 8B : is an explanatory diagram illustrating an image captured by using a rolling shutter in the second embodiment.

[0020] Figure 8C is an explanatory diagram illustrating an image captured by using a global shutter.

[0021] Fig. 9A and 9B : is an explanatory diagram illustrating the configuration of a plurality of cameras including lights disposed outside the field of view in the third embodiment.

[0022] Fig. 10A and 10B : is an explanatory diagram illustrating the configuration of a plurality of cameras including a diffusion plate in the third embodiment.

[0023] Figures 11A to 11C : is an explanatory diagram illustrating the configuration of an imaging system including a recursive reflective mark in the fourth embodiment.

[0024] Fig.12 is a flowchart illustrating a method of synchronization in the fifth embodiment.

[0025] Fig.13 is an explanatory diagram illustrating the configuration of an imaging system including three cameras in the sixth embodiment.

[0026] Fig.14 is a flowchart illustrating a method of camera switching in the sixth embodiment.

[0027] Fig.15 is an explanatory diagram illustrating an example of a control system used as a controller of an imaging system. DETAILED DESCRIPTION

[0028] The embodiments of the present disclosure will be described below with reference to the accompanying drawings. It should be noted that the configuration described below is only an example, and for example, those skilled in the art can appropriately modify its details within the spirit of the present disclosure. In addition, the numerical values ​​mentioned in the embodiments are only examples of reference values.

[0029] First embodiment

[0030] Figure 1A The configuration of an imaging system including a stereo camera composed of a plurality of cameras in this embodiment is illustrated. Figure 1AIn the imaging system of the present invention, a stereo camera 1 is connected to an image processing device 2. The connection cable between the stereo camera 1 and the image processing device 2 constitutes a communication interface between the two, and includes a power line, a communication line for transmitting captured image data, an input and output line for communication control, etc.: IO line, etc. Among these, the communication interface can be based on a standard such as a universal serial bus: USB, for example. The stereo camera 1 includes monocular cameras 101 and 102, which are arranged so that their respective imaging optical axes are separated from each other by an appropriate baseline length. The captured image data obtained by the monocular cameras 101 and 102 of the stereo camera 1 can be transmitted to the image processing device 2 via the above-mentioned communication interface. In addition, image capture parameters as setting information for image capture can be controlled according to a setting command or the like received from the image processing device 2 connected to the stereo camera 1 via the above-mentioned communication interface. These image capture parameters include exposure time, gain, image size, etc. In addition, the stereo camera 1 can control the light emission timing of the synchronization deviation detection lamps 104 to 107 provided as light emitting members for detecting synchronization deviation (miss-synchronization) in the stereo camera 1 via the IO line of the above-mentioned communication interface.

[0031] In the present embodiment, a stereo camera 1 is used to measure three-dimensional information of an object. For example, the stereo camera 1 is deployed in a production line for manufacturing products together with a manufacturing device such as a robot device and a robot controller that controls the robot device. In this configuration, the robot controller can control the robot device based on the result of three-dimensional measurement of an object (such as a workpiece) obtained by the stereo camera 1. It should be noted that the stereo camera 1 of the present embodiment is merely an example of a component constituting an imaging system including a plurality of cameras. For example, the image capture control described below can be performed by any imaging system that performs synchronous image capture by a plurality of cameras for some purpose. Examples of imaging systems consisting of a plurality of cameras to which the image capture control of the present embodiment can be applied include multi-viewpoint cameras for capturing free viewpoint moving images.

[0032] As mentioned above, Figure 1A The stereo camera 1 includes monocular cameras 101 and 102, which are arranged so that their imaging optical axes are separated from each other by a predetermined baseline length. In the present embodiment, an illumination board 103 is disposed on the front side of the monocular cameras 101 and 102, and synchronization deviation detection lamps 104 to 107 are provided on the illumination board 103. The synchronization deviation detection lamps 104 to 107 constitute an illumination device that radiates illumination light at the same light emission timing, under which the monocular cameras 101 and 102 can capture images.

[0033] In the present embodiment, the synchronous deviation detection lights 104 to 107 are disposed outside the common field of view 108 of the monocular cameras 101 and 102. According to this configuration, by synchronizing the deviation detection lights 104 to 107, image capture can be performed without narrowing the common field of view 108, which is a spatial area in which three-dimensional measurement can be performed.

[0034] Furthermore, the synchronization deviation detection lights 104 and 105 are disposed within the individual field of view of the monocular camera 101, and the synchronization deviation detection lights 106 and 107 are disposed within the individual field of view of the monocular camera 102. According to this configuration, light emitted from the synchronization deviation detection lights 104 to 107 is necessarily incident on the monocular cameras 101 and 102.

[0035] Figure 1B The figure shows the layout of the synchronization deviation detection lamp. Figure 1B In FIG. 1 , the upper side and the lower side of the figure correspond to the upper side and the lower side of the stereo camera 1, respectively. Figure 1B As shown in , the synchronization deviation detection lamps 104 and 106 are deployed on the upper side of the monocular cameras 101 and 102, and the synchronization deviation detection lamps 105 and 107 are deployed on the lower side of the monocular cameras 101 and 102. The synchronization deviation detection lamps 104 and 106 are connected to the driving power supply line in parallel, and their light sources are synchronously driven so as to emit light at the same timing without delay relative to each other. Similarly, the synchronization deviation detection lamps 105 and 107 are connected to the driving power supply line in parallel, and their light sources are synchronously driven so as to emit light at the same timing without delay relative to each other.

[0036] A lamp not shown for three-dimensional measurement is disposed on the front surface of the lighting board 103. As this lamp for three-dimensional measurement, a lamp corresponding to the measurement method, such as a pattern floodlight, can be provided. In such a configuration as described above, it is not necessary to provide a drive board dedicated to the synchronization deviation detection lamps 104 to 107, so the size of the stereo camera 1 can be reduced.

[0037] Figure 1A The image processing device 2 shown in the figure may be composed of hardware such as a central processing unit: CPU or field programmable gate array: FPGA that performs calculations, a memory portion composed of a read-only memory: ROM and a random access memory: RAM, and an interface portion: I / F portion that communicates with the outside. Figure 1A , the image processing device 2 is shown as functional blocks 201 to 205. Note that the image processing device 2 of the present embodiment also has an image capture control function of the stereo camera 1 to be described later, and can also be conceptually regarded as a controller that performs image capture control via image processing.

[0038] Here, we will refer to Fig.15Describe the composition Figure 1A An example of a specific hardware configuration of a control system of the image processing device 2. Fig.15 In the configuration, Figure 1A Each functional block constituting the image processing apparatus 2 shown in FIG. 1 is realized by a CPU 1601 and its peripheral hardware or software executed by the CPU 1601. The storage section for image processing or image capture control is constituted by a storage area of ​​a ROM 1602, a RAM 1603, an external storage device 1606 such as a hard disk drive: HDD, and the like.

[0039] Fig.15 The control system includes a CPU 1601 as a main control unit, and a ROM 1602 and a RAM 1603 as storage devices. The ROM 1602 can store control programs and constant information for the CPU 1601 to implement the control processing process of this embodiment. In addition, the RAM 1603 is used as a work area of ​​the CPU 1601 when executing the control process. In addition, Fig.15 The control system is connected to an external storage device 1606. The external storage device 1606 is not necessarily required to implement the embodiments of the present disclosure, and may be constituted by a HDD, a solid state drive: SSD, an external storage device of another system installed on a network, or the like.

[0040] The control program for CPU 1601 to implement the control of the present embodiment is stored in a storage part such as an electrically erasable programmable read-only memory: EEPROM area of ​​an external storage device 1606 or ROM 1602. In this case, the control program for CPU 1601 to implement the control process of the present embodiment can be supplied to each of the above-mentioned storage parts via a network interface 1607, and updated to a new or different program. Alternatively, the control program for CPU 1601 to implement the control process to be described later can be supplied to each of the above-mentioned storage parts via storage media such as various disks, optical disks and flash memories and their drive devices, and can therefore be updated. Various storage media, storage parts and storage devices storing the control program for CPU 1601 to implement the control process of the present embodiment are used as computer-readable recording media for storing the control process of the present embodiment.

[0041] The network interface 1607 may be provided according to, for example, a wired communication standard such as IEEE 802.3 or a wireless communication standard such as IEEE 802.11 or IEEE 802.15. The CPU 1601 may communicate with another device 1104 via the network interface 1607 and the network 1608. For example, in the case where the stereo camera 1 is connected to the network 1608, for example, the stereo camera 1 is used as another device 1104. In the case where the stereo camera 1 is connected to the CPU 1601 according to a standard different from the network connection, the interface 1605 is used. The interface 1605 may also be used to connect to other peripheral devices.

[0042] In addition, when necessary, Fig.15 A user interface device 400: UI device 400 is provided in the control system of the present invention. The user interface device 400 may be, for example, a graphical user interface device: GUI device composed of a liquid crystal display: LCD, a keyboard, a pointing device (such as a mouse, a joystick, or a jog dial), etc. The user interface device 400 may be used to notify captured images, notify the progress and results of synchronization processing and three-dimensional measurement processing of the camera to be described later, and set image capture parameters and control constants related to synchronization.

[0043] like Figure 1A As shown in FIG. 1 , the image processing device 2 includes functional blocks of a camera controller 201, an illumination controller 202, a synchronization deviation amount calculation section 203, a synchronization controller 204, and a three-dimensional measurement section 205. For example, these functional blocks as controllers may be implemented by hardware blocks in an FPGA, or by, for example, the above-mentioned CPU 1601 loading and executing a program stored in a ROM 1602 or the like.

[0044] The following will describe Figure 1A The outline of the functional blocks 201 to 205 of the image processing device 2 is shown. The camera controller 201 controls the image capturing operation of the monocular cameras 101 and 102. The details of this control will be described when describing the internal configuration of the monocular cameras 101 and 102. Here, the outline of each part will be described.

[0045] In the present embodiment, when capturing an image, first, power is supplied to the monocular cameras 101 and 102, and the camera controller 201 transmits an initialization instruction. When the initialization of the monocular cameras 101 and 102 is completed, an instruction to change the image capturing parameters is transmitted to the monocular cameras 101 and 102. In some cases, the image capturing parameters include, for example, exposure time, gain, image size, and focal length, depending on the optical system.

[0046] When the adjustment of the image capture parameters is completed, the camera controller 201 transmits a moving image output start instruction to cause the monocular cameras 101 and 102 to output a moving image. In addition, at this time, the lighting controller 202 determines the driving conditions of the lights for three-dimensional measurement and the synchronization deviation detection lights 104 to 107 according to the image capture conditions. The camera controller 201 has a function of obtaining still image data by cutting a still image from moving image data when receiving an instruction to obtain an image from another functional block.

[0047] In addition, when the camera controller 201 stops Figure 2 When the power supply of the image sensor 302 shown in is stopped, the output of the moving image is stopped. Then, the above-mentioned processing can be performed again by performing the above-mentioned power supply and initialization, so the output of the moving image can be restarted. In this way, the camera controller 201 controls the timing of starting the output of the moving image.

[0048] The lighting controller 202 controls the lighting timing of the synchronous deviation detection lamps 104 to 107. This lighting control is performed by transmitting a pulse width modulation signal: PWM signal to the synchronous deviation detection lamps 104 to 107 via the IO line, for example. As described above, the drive power supply lines of the synchronous deviation detection lamps 104 and 106 are electrically connected to each other in the lighting board 103, so that synchronous lighting can be performed. Similarly, the synchronous deviation detection lamps 105 and 107 can also emit light synchronously. It should be noted that, for example, the delay time between the time point when the light on / off command is output from the lighting controller 202 and the time point when the illumination light source of interest is actually turned on / off is very short. For example, the response time of the illumination light source to the drive control is sufficiently shorter than the image capture control time of the monocular cameras 101 and 102. Examples of the image capture control time include one frame time of these cameras, such as 1 / 24 second, 1 / 30 second, and 1 / 60 second. That is, the control speed or control time of the synchronous deviation detection lamps 104 to 107 is sufficiently higher or shorter than the image capture control speed or image capture control time of the moving image.

[0049] The synchronization deviation amount calculation section 203 calculates the synchronization deviation amount between the image capturing timings of the monocular cameras 101 and 102. The details of the method for calculating the synchronization deviation amount will be described later.

[0050] When the synchronization deviation amount calculation section 203 detects a synchronization deviation, the synchronization controller 204 synchronizes the image capturing timings of the monocular cameras 101 and 102. The details of this synchronization will be described later.

[0051] The three-dimensional measurement section 205 performs three-dimensional measurement by using images captured by the monocular cameras 101 and 102 of the stereo camera 1. The three-dimensional measurement section 205 for images captured by the monocular cameras 101 and 102 can calculate a distance using the principle of triangulation using the parallax amount obtained by the stereo matching process and the internal and external parameters obtained by the stereo camera calibration.

[0052] In the above-mentioned stereo matching process, for example, the image captured by the monocular camera 101 is set as a reference image, and pixels corresponding to pixels in the reference image, that is, pixels corresponding to the same portion of the object as the pixels in the reference image, are determined in the image captured by the monocular camera 102 by matching. As an example of this stereo matching process, block matching methods such as the sum of absolute differences: SAD and the sum of square differences: SSD are known. Well-known matching processing methods such as these can also be used in the present embodiment.

[0053] The above-mentioned intrinsic parameters, extrinsic parameters, etc. are conceptually equivalent to those used in image processing libraries (such as OpenCV) related to (digital) cameras. The intrinsic parameters indicate optical characteristics (such as the focal length and distortion characteristics of a lens), and the extrinsic parameters indicate the relative position and orientation of two cameras in a stereo camera. The intrinsic parameters and extrinsic parameters can be pre-calculated using an optimization method by capturing a calibration map whose shape is already known. The intrinsic parameters and extrinsic parameters pre-calculated for the monocular cameras 101 and 102 are stored, for example, in a ROM in the image processing device 2.

[0054] It is to be noted that, although the description will be given assuming that the image processing device 2 is a device separate from the stereo camera 1 in the present embodiment, the image processing device 2 may be incorporated in the stereo camera 1 like a so-called smart camera. According to this configuration, wiring between the stereo camera 1 and the image processing device 2 is unnecessary, and thus the man-hours for installing the system can be greatly reduced.

[0055] Next, the internal configuration of the monocular cameras 101 and 102 will be described. The monocular cameras 101 and 102 are relatively small and inexpensive cameras, such as web cameras or module cameras for mobile phones. In the present embodiment, the monocular cameras 101 and 102 are each a product that can be purchased as a separate unit, and the stereo camera 1 is formed by merging these cameras in a single housing or frame. The monocular cameras 101 and 102 are positioned relative to each other by the above-mentioned housing or frame so as to be separated from each other by a predetermined baseline length. In the present embodiment, the monocular cameras 101 and 102 do not require a function of synchronizing with an external synchronization signal such as a genlock function, or a timestamp function of outputting an image capture time. According to the present embodiment, the stereo camera 1 can be formed by units of the monocular cameras 101 and 102 that can be easily obtained and are relatively inexpensive.

[0056] Figure 2 An example of the internal configuration of the monocular camera 101 is illustrated. The configuration of the monocular camera 102 is similar to that of the monocular camera 101. The monocular camera 101 has a structure in which a light collecting section 301, an image sensor 302, a sensor controller 303, an image format changing section 304, and a power supply controller 305 are integrated.

[0057] The light collecting section 301 is a lens, and constitutes an imaging optical system for causing the collected light to be incident on the image sensor 302 .

[0058] The image sensor 302 is an image sensor such as a charge coupled device: CCD or a complementary metal oxide semiconductor: CMOS. The image transmitted to the sensor controller 303 is, for example, a so-called RAW image format compliant with the Mobile Industry Processor Interface Camera Serial Interface-2: MIPI CSI-2. It should be noted that the standard and output image format of the image sensor 302 are not limited to these and can be arbitrarily selected by those skilled in the art.

[0059] Here, the outline of the functional blocks 303 to 305 of the above-mentioned monocular camera 101 will be described. The sensor controller 303, the image format change section 304, and the power supply controller 305 are composed of an electronic circuit including an FPGA, a memory section composed of ROM and RAM, and an interface section for communicating with the outside: an I / F section. These blocks and the image sensor 302 are electrically connected to each other inside the monocular camera 101.

[0060] The sensor controller 303 controls the state transition of the image sensor 302 by communicating with the camera controller 201 in the image processing device 2. Here, Figure 3 FIG. 3 shows the transition of the operation state of the image sensor 302 of the present embodiment. Figure 3 As shown in FIG. 4 , the operating state of the image sensor 302 includes four states: a power-off state 401, an initialization state 402, an image capture parameter adjustment state 403, and a moving image output state 404. These states will be described below.

[0061] The power-off state 401 is a state in which power is not supplied to the image sensor 302. When receiving a power supply instruction from the camera controller 201 in the image processing apparatus 2, the sensor controller 303 supplies power to the image sensor 302. When power is supplied to the image sensor 302, the image sensor 302 transitions to the initialized state 402.

[0062] The initialization state 402 is a state in which the image sensor 302 is initialized. First, the sensor controller 303 supplies a clock signal to the image sensor 302. When receiving an initialization start instruction from the camera controller 201 in the image processing apparatus 2, the sensor controller 303 transmits an initialization signal to the image sensor 302. When the initialization is completed in this way, the sensor controller 303 and the image sensor 302 become able to communicate with each other, and the operation state transitions to the image capturing parameter adjustment state 403.

[0063] In addition, the image capture parameter adjustment state 403 is a state in which the sensor controller 303 can adjust the image capture parameters of the image sensor 302. Examples of image capture parameters include exposure time, gain, and image size. In this state, when the sensor controller 303 receives an instruction to change the above-mentioned image capture parameters from the camera controller 201 of the image processing device 2, the sensor controller 303 transmits, for example, a control command to the image sensor 302 to rewrite the register value storing the image capture parameters.

[0064] When the sensor controller 303 receives a moving image output start instruction from the camera controller 201 of the image processing apparatus 2 , the sensor controller 303 transmits a moving image output start signal to the image sensor 302 to switch the image sensor 302 to the moving image output state 404 .

[0065] Figure 3 The moving image output state 404 is a state in which the image sensor 302 continuously outputs moving image data to the image format changing section 304. In this state, when the sensor controller 303 receives a moving image output stop instruction from the camera controller 201 in the image processing apparatus 2, the sensor controller 303 stops the power supply to the image sensor 302 to stop the output of the moving image. As a result, the image sensor 302 of the present embodiment transitions to the power-off state 401.

[0066] The monocular camera 101 or 102 has been transformed to Figure 3 After the power-off state 401, the camera controller 201 in the image processing device 2 can make the image sensor 302 change state again via the sensor controller 303 to output the moving image again. As described above, the image processing device 2 can control the moving image output start timing or the moving image output end timing of the monocular cameras 101 and 102.

[0067] A configuration has been described above in which the monocular camera 101 or 102 can only transition to the power-off state 401 from the moving image output state 404. However, in the case where the image sensor 302 has a state transition function from the moving image output state 404 to the initialization state 402, the state of the monocular camera 101 or 102 can be switched from the moving image output state 404 to the initialization state 402. In the case where the image sensor 302 has a state transition function like this, for example, there is a possibility that the moving image output timing of the monocular cameras 101 and 102 can be changed for synchronization without turning off the power supply.

[0068] Furthermore, a configuration in which the image sensor 302 of the monocular camera 101 or 102 has an image capture mode change function in which the output of a moving image is started when the moving image mode is selected and the output of a moving image is stopped when the still image mode is selected can also be considered. In such a camera configuration, there is a possibility that the moving image output timing of the monocular cameras 101 and 102 can be changed for synchronization without turning off the power supply by switching the image capture mode between the moving image mode and the still image mode.

[0069] In the present embodiment, the control of resetting the monocular cameras 101 and 102 and changing the moving image output start timing to synchronize the monocular cameras 101 and 102 by switching the operation state from the moving image output state 404 to the power-off state 401 will be described. However, the moving image output timing may be changed to synchronize the monocular cameras 101 and 102 by, for example, using a different state transition function or switching the image capturing mode as described above.

[0070] exist Figure 2 In the embodiment, the control interface between the image sensor 302 and the sensor controller 303 may be composed of an IO terminal and an integrated circuit bus: I2C. In addition, the image format changing section 304 of the present embodiment has a function of changing the format of the image received from the image sensor 302 from the RAW image format to the image format for transmission to the image processing device 2. Examples of formats supported by the image format changing section 304 include formats compliant with the USB video class: UVC. It is to be noted that the image format changing section 304 may support image formats other than UVC.

[0071] In addition, the power controller 305 has a function of supplying power to the sensor controller 303 and the image format changing section 304 when receiving a power supply command from the image processing device 2. According to this power supply command, the states of the monocular cameras 101 and 102 can be switched from the power-off state 401 to the initialization state 402. The power supply to the image sensor 302 is controlled via the sensor controller 303 described above.

[0072] Method for calculating synchronization deviation

[0073] Here, a method for calculating the amount of synchronization deviation between the monocular cameras 101 and 102 of the stereo camera 1 in the present embodiment will be described. Figure 4 illustrates a process for detecting synchronization deviation, and Figure 5 An example of images output from the monocular cameras 101 and 102 is illustrated in a time line.

[0074] Figure 4 Steps S10 to S15 correspond to the light emission image capturing step for synchronization deviation detection in the present embodiment, and step S16 corresponds to the image processing step for synchronization deviation detection in the present embodiment.

[0075] exist Figure 4 In step S10, the sensor controller 303 switches the monocular camera 101 to the moving image output state 404 based on the moving image output start instruction transmitted from the camera controller 201 of the image processing device 2. Then, the synchronization deviation amount calculation part 203 instructs the camera controller 201 to transmit the moving image output start instruction to the monocular camera 101. This makes it possible to obtain an image captured by the monocular camera 101 at an arbitrary timing.

[0076] In step S11, the sensor controller 303 switches the monocular camera 102 to the moving image output state 404 based on the moving image output start instruction transmitted from the camera controller 201 of the image processing device 2. Then, the synchronization deviation amount calculation part 203 instructs the camera controller 201 to transmit the moving image output start instruction to the monocular camera 102. This makes it possible to obtain an image captured by the monocular camera 102 at an arbitrary timing.

[0077] In step S12, control of the synchronization deviation detection lights 104 to 107 is started. As described above, the control for turning on / off the synchronization deviation detection lights 104 to 107 is preferably performed at a speed higher than the frame rate of the moving image output from the monocular cameras 101 and 102. It should be noted that in the present embodiment, for the sake of simplicity of description, a case will be described where the cycle of the power-on / power-off control of the lights is equal to the frame rate. First, in step S12, the synchronization deviation amount calculation part 203 instructs the lighting controller 202 to flash the synchronization deviation detection lights 104 and 106 simultaneously after a certain time Δt [ms] has passed. Then, similarly, after another certain time Δt [ms] has passed, the lighting controller 202 flashes the synchronization deviation detection lights 105 and 107 simultaneously. Then, similarly, after another certain time Δt [ms] has passed, the lighting controller 202 flashes the synchronization deviation detection lights 104 and 106 simultaneously. Then, similarly, after another certain time Δt [ms] has passed, the lighting controller 202 flashes the synchronization deviation detection lamps 105 and 107 simultaneously. This process of alternately flashing the pair of synchronization deviation detection lamps 104 and 106 and the pair of synchronization deviation detection lamps 105 and 107 at intervals of the certain time Δt [ms] is repeatedly performed during the process of obtaining images from the monocular cameras 101 and 102 in steps S13 and S14 until step S15.

[0078] In step S13, an image captured by the monocular camera 101 is obtained. At this time, the synchronization deviation amount calculation part 203 transmits an image acquisition instruction to the camera controller 201, cuts out a still image from the moving image data transmitted from the monocular camera 101, and thus obtains its still image data. In step S14, an image of the monocular camera 102 is obtained. At this time, the synchronization deviation amount calculation part 203 transmits an image acquisition instruction to the camera controller 201, cuts out a still image from the moving image data transmitted from the monocular camera 102, and thus obtains its still image data.

[0079] In step S15, the synchronization deviation amount calculation section 203 instructs the lighting controller 202 to turn off the synchronization deviation detection lamps 104 to 107. Then, in step S16, the synchronization deviation amount calculation section 203 calculates the synchronization deviation amount from the images obtained in steps S13 and S14, for example, as follows.

[0080] exist Figure 5 In FIG. 6 , 601A to 606A represent images output from the monocular camera 101 at various time points. Figure 5 , time points after Δt from time point t1 are sequentially represented by t2, t3, t4, ... That is, time point t2 is t1+Δt, time point t3 is t2+Δt, and time point t4 is t3+Δt.

[0081] exist Figure 5 , image 601A is an image captured at time point t1 when the synchronization deviation detection lights 104 and 106 are turned on. Image 602A is an image captured at time point t2 when the synchronization deviation detection lights 104 to 107 are turned on. Image 603A is an image captured at time point t3 when the synchronization deviation detection lights 105 and 107 are turned on. Image 604A is an image captured at time point t4 when the synchronization deviation detection lights 104 to 107 are turned off. After this time point, images 605A, 606A, ... similar to images 601A to 604A are repeatedly output. At the same time, images 601B to 606B are images output from the monocular camera 102 at various time points similar to the monocular camera 101.

[0082] At time point t12, all the synchronous deviation detection lamps 104 to 107 are turned on, and at time point t23, only the synchronous deviation detection lamps 105 and 107 are turned on. In addition, at time point t34, all the synchronous deviation detection lamps 104 to 107 are turned on, and at time point t45, only the synchronous deviation detection lamps 104 and 106 are turned on. This lighting pattern is repeated. In this example, the intervals between time points t12, t23, t34, and t45 are also Δt, and the difference between time point t1 and time point t12 is Δt / 2.

[0083] For example, here, it is assumed that the images obtained in steps S13 and S14 are images 603A and 603B obtained at time point t3. In this case, the time point at which image 603A is actually captured is between time point t23 and time point t34, and the time point at which image 603B is actually captured is between time point t12 and time point t23. In this example, the detected synchronization deviation amount between the monocular cameras 101 and 102 is equal to or less than Δt. Therefore, for example, the synchronization deviation amount between the monocular cameras 101 and 102 can be calculated by image processing that searches for similar image patterns (such as brightness patterns) between the images captured by the monocular cameras 101 and 102, respectively.

[0084] In the present embodiment, the case where the interval Δt of the on and off switching of the synchronization deviation detection lamps 104 to 107 is equal to the image capturing interval Δt has been described. If the lighting switching interval is shortened, the synchronization deviation can be detected with higher accuracy. For example, if the lighting switching interval is Δt / 2, the synchronization deviation amount can be detected with a higher resolution than Δt.

[0085] However, if the lighting switching interval is too short, the synchronization deviation cannot be accurately detected in some cases. For example, a case where the power on and off of the lights are switched between four levels as in the present embodiment will be described, the four levels including a level where all the upper and lower lights are turned on, a level where only the upper light is turned on, a level where only the lower light is turned on, and a level where all the upper and lower lights are turned off. In this case, when the lighting condition is switched at an interval shorter than Δt / 4, the monocular cameras 101 and 102 capture images of the same lighting condition with a synchronization deviation amount of Δt. As described above, the lighting switching interval is preferably longer than a value obtained by dividing the image capture interval Δt by the number of levels of the lighting condition.

[0086] It is to be noted that the interval Δt of the switching of power on and power off of the synchronization deviation detection lamps 104 to 107 can be obtained in advance by actual measurement. For example, before calculating the synchronization deviation amount, the synchronization deviation detection lamps 104 to 107 are turned on only once after the monocular cameras 101 and 102 are switched to the moving image output state. Then, the interval Δt can be determined based on the difference between the time point at which the image output from the monocular camera 101 is switched from an image with all lamps turned off as in the image 604A to an image with the lamps turned on as in the image 602A, and the time point at which the image output from the monocular camera 101 is switched from an image with all lamps turned off to an image with the lamps turned on.

[0087] In addition, generally, the distortion of the captured image is large in the peripheral portion of the image, so the peripheral portion is often not used for image processing (such as three-dimensional measurement). In addition, the portion that does not correspond to the common field of view of the monocular cameras 101 and 102 is not used in the three-dimensional measurement to be described later. That is, in the present embodiment, the synchronization deviation detection lights 104 to 107 are arranged to affect only the above-mentioned portion that will not be used in the image processing. Therefore, according to the above-mentioned lighting control, the synchronization deviation can be always detected even during the three-dimensional measurement to be described later.

[0088] In addition, although four synchronization deviation detection lights 104 to 107 are used in the present embodiment, a configuration in which only one synchronization deviation detection light is provided may be adopted, as long as the synchronization deviation detection light is deployed at a position that affects both the monocular cameras 101 and 102 at the same time. Alternatively, a change in the light emission mode (such as emission color or emission pattern) of the synchronization deviation detection light may be adopted so that the configuration of each synchronization deviation detection light can be identified. In this case, for example, the synchronization deviation detection light that is turned on can be identified from the captured image. By utilizing this, for example, the number of synchronization deviation detection lights can be increased. According to this configuration, there is a possibility that the synchronization deviation amount can be calculated with higher accuracy.

[0089] synchronous

[0090] Here, we will refer to Figure 6 An example of a method of synchronizing the monocular cameras 101 and 102 based on the synchronization deviation amount detected as described above is described. Figure 6 The diagram illustrates the synchronization process.

[0091] exist Figure 6 In step S20, the synchronization deviation amount is calculated as described above. In step S21, it is determined whether the calculated synchronization deviation amount is equal to or less than a predetermined value. In the case where the calculated synchronization deviation amount is equal to or less than the predetermined value, that is, in the case where the result of step S21 is "yes", Figure 6 The three-dimensional measurement process is performed after the process shown in . In addition, in the case where the calculated synchronization deviation amount is larger than the predetermined value, that is, in the case where the result of step S21 is "No", the process proceeds to step S22.

[0092] In step S22, as a process for synchronization, the power supply of the monocular camera 102 is turned off. This synchronization control utilizes Figure 3 The state transition patterns of the monocular cameras 101 and 102 shown in FIG.

[0093] After step S22, in step S23, one of the unsynchronized monocular cameras, such as monocular camera 102, is switched to Figure 3 As a result, the monocular camera 102 is shown in the moving image output state 404. Figure 3 The described transition from the power-off state 401 through the initialization state 402 and the image capture parameter adjustment state 403 to the moving image output state 404. After step S23, the processing proceeds to step S20. At this time, the time elapsed before the monocular camera 102 reaches the moving image output state 404 is determined by a somewhat random process. Therefore, by repeating the processing of the above steps S20 to S23 multiple times, the synchronization deviation amount eventually becomes equal to or less than a predetermined value. Therefore, the difference between the time points at which two images to be used for image processing such as three-dimensional measurement are respectively captured is less than the difference between the time points at which two images for image processing for detecting synchronization deviation are respectively captured.

[0094] As described above, in the present embodiment, by using a pattern in which the synchronization deviation detection light is turned on at predetermined intervals, that is, by using an illumination light pattern, the amount of synchronization deviation between the monocular cameras 101 and 102 can be detected from a brightness pattern in an image captured by the monocular cameras 101 and 102. Then, in a case where the amount of synchronization deviation is greater than a predetermined value, the monocular cameras 101 and 102 are synchronized by relatively random processing by turning off and initializing one of the monocular cameras 101 and 102.

[0095] In the present embodiment, the synchronization deviation detection lamps 104 to 107 are disposed at positions that do not affect the three-dimensional measurement, so that the synchronization deviation detection can also be performed during the three-dimensional measurement. In addition, for example, the above-mentioned synchronization processing can be performed when the stereo camera 1 does not perform the measurement processing. In the present embodiment, according to this control, the synchronization between the monocular cameras 101 and 102 constituting the stereo camera 1 can be performed without extending the measurement time for the three-dimensional measurement.

[0096] Second embodiment

[0097] In the above-described first embodiment, the synchronization deviation detection lamps 104 to 107 are included in the stereo camera 1. However, in order to simplify the configuration of the stereo camera 1 and reduce cost, size and weight, a configuration in which synchronization deviation detection is performed by using external lamps as in the present embodiment may also be considered.

[0098] In the following description, parts of the configuration of the hardware and control system that are different from those of the first embodiment will be illustrated and described, and a detailed description of parts that are similar to those of the first embodiment will be omitted, assuming that the parts can be configured in a manner similar to that described above and can have similar effects.

[0099] In this embodiment, if Figure 7 As shown in FIG. 1 , the external light 130 is disposed within the common field of view 108 of the monocular cameras 101 and 102. In this embodiment, the external light 130 is used as a synchronization deviation detection light. The external light 130 may be a light dedicated to synchronization deviation detection or a light also used for other image processing, etc.

[0100] In addition, also in this embodiment, the control time of the external light 130 is sufficiently shorter than one frame time (such as 1 / 24 second, 1 / 30 second, or 1 / 60 second) in the moving image capture control of the monocular cameras 101 and 102. That is, the control speed or control time of the external light 130 is sufficiently higher or shorter than the image capture control speed or image capture control time of the moving image. Also in this embodiment, the control process for synchronization deviation detection and synchronization is similar to the reference Figure 4 and 6 The control process described is largely the same.

[0101] However, the method for synchronization deviation detection is slightly different between the case where a rolling shutter is used in the monocular cameras 101 and 102 and the case where a global shutter is used in the monocular cameras 101 and 102. Figures 8A to 8C The synchronization deviation detection method of this embodiment is described.

[0102] Fig. 8A An example of an image captured by using the external light 130 in the case of using a rolling shutter in the monocular cameras 101 and 102 is illustrated. Figure 8BThe diagram shows instructions for Fig. 8A A diagram showing an example of average values ​​of brightness at positions in the height direction in an image captured under the condition of FIG.

[0103] As mentioned above, the synchronization deviation detection control can be carried out with Figure 4 The first embodiment shown in FIG. is performed in substantially the same manner. Figure 4 In step S12, the external light 130 is turned on for a certain time. This illumination for a certain time is shorter than one frame time of the monocular cameras 101 and 102. The external light 130 is turned off after a certain time after being turned on. In this embodiment, Figure 4 Step S15 is unnecessary.

[0104] In step S13, an image captured by the monocular camera 101 is obtained. In this step, the image captured by the monocular camera 101 is obtained. Fig. 8A Subsequently, in step S14, an image captured by the monocular camera 102 is obtained. In this step, an image similar to the image 701A shown in FIG. Fig. 8A In the case where a rolling shutter is used in the monocular cameras 101 and 102, the light from the external light 130 appears in a linear shape in the captured image, such as Fig. 8A This is because when using a rolling shutter, the exposure of each pixel or each image capture line of the camera is turned on sequentially.

[0105] In step S16, Fig. 8A The synchronization deviation amount is calculated as follows for the captured image shown in . Here, for example, the frame rate (fps) of the monocular cameras 101 and 102 is represented by F, and the height and width of the image are represented by H and W, respectively.

[0106] The average value of the brightness of the pixels at the same height in the image 701A obtained by the monocular camera 101 is calculated for each height in the image 701A, and the height in the image 701A where the brightness is the highest is represented by H1. Here, the brightness is represented by 256 levels from 0 to 255. In the case where the average value of the brightness of the pixels at the same height in the image 701B obtained by the monocular camera 102 is calculated for each height in the image 701B, and the height in the image 701B where the brightness is the highest is represented by H2, the synchronization deviation amount can be calculated by (H2-H1) / (F×H).

[0107] That is, in the present embodiment, when a rolling shutter is used in the monocular cameras 101 and 102, the amount of synchronization deviation between the monocular cameras 101 and 102 is calculated based on the position of the light image of the external light 130 serving as a synchronization deviation detection light in the captured image.

[0108] It should be noted that in the above description, when a rolling shutter is used in the monocular cameras 101 and 102, the average value of the brightness of pixels at the same height in the obtained image is obtained for each height in the obtained image, and the height in the image where the brightness is the highest is used. However, the median point of the brightness in the height direction in the image may be used after calculating the average value of the brightness of pixels at the same height for each height in the image.

[0109] Furthermore, in the present embodiment, in the case where a global shutter is used in the monocular cameras 101 and 102 , synchronization deviation detection can be performed by the following calculation. Figure 8C An image captured when a global shutter is used in the monocular cameras 101 and 102 is illustrated.

[0110] The synchronization deviation detection process in this example is similar to Figure 4 In step S12, the external light 130 is turned on for a certain time shorter than one frame time and then turned off. Also in this example, Figure 4 Step S15 is unnecessary. In step S13, an image captured by the monocular camera 101 is obtained. In step S13, an image such as the image 702A is obtained. In step S14, an image captured by the monocular camera 102 is obtained. In step S14, an image such as the image 702B is obtained.

[0111] In the case where the global shutter is used in the monocular cameras 101 and 102, control is performed so that all pixels or image capture lines are exposed at the same time. Therefore, in the case where the relationship between the lighting time of the external light 130 and the image capture timing is different between the monocular cameras 101 and 102, that is, in the case where the monocular cameras 101 and 102 are not synchronized, the light amount in the pixel is different, such as Figure 8C as shown in .

[0112] In the case where a global shutter is used in the monocular cameras 101 and 102, in step S16, the synchronization deviation amount is calculated as follows. Here, the frame rate (fps) of the monocular cameras 101 and 102 is represented by F, and the average brightness of all pixels in each image captured by the monocular camera 101 or 102 when the external light 130 is always turned on is represented by Lmax. In addition, the average brightness of all pixels in each image captured by the monocular camera 101 or 102 when the external light 130 is always turned off is represented by Lmin. In the case of 8-bit quantization, Lmax is preferably less than 255. Then, in step S12, when the time t when the external light 130 is turned on is 1 / F, the average brightness of all pixels in the image 702A is La, and the average brightness of all pixels in the image 702B is Lb, the synchronization deviation amount can be calculated by (La-Lb) / (Lmax-Lmin).

[0113] That is, in the present embodiment, when a global shutter is used in the monocular cameras 101 and 102, the amount of synchronization deviation between the monocular cameras 101 and 102 is calculated based on the ratio of the brightness or density between the images captured thereby of the external light 130 serving as a synchronization deviation detection light.

[0114] As described above, according to the present embodiment, the amount of synchronization deviation between the monocular cameras 101 and 102 can be calculated based on the position of the light image of the synchronization deviation detection light and the brightness or average brightness of the image.

[0115] Third embodiment

[0116] In the first embodiment, the synchronization deviation detection lights 104 and 105 are disposed in the individual field of view of the monocular camera 101, and the synchronization deviation detection lights 106 and 107 are disposed in the individual field of view of the monocular camera 102. However, since the distance between the monocular cameras 101 and 102 and the synchronization deviation detection lights 104 to 107 is short, the individual field of view of the monocular cameras 101 and 102 excluding the common field of view 108 is not so large. Therefore, there is a possibility that the operation of finely adjusting the installation positions of the synchronization deviation detection lights 104 to 107 is difficult.

[0117] In addition, since the synchronization deviation detection lights 104 and 105 are deployed in a separate field of view of the monocular camera 101 and the synchronization deviation detection lights 106 and 107 are deployed in a separate field of view of the monocular camera 102, there is a problem of a narrow measurement range when, for example, measurement is performed through two-dimensional image processing using only an image captured by the monocular camera 101.

[0118] In the present embodiment, taking the above into consideration, different arrangements of the synchronization deviation detection lamps 104 to 107 will be discussed.

[0119] In the following description, parts of the configuration of the hardware and control system that are different from those of the first embodiment will be illustrated and described, and detailed descriptions of parts similar to those of the first embodiment will be omitted, assuming that the parts can be configured in a similar manner to that described above and can have similar effects.

[0120] In this embodiment, if Fig. 9A As shown in FIG. 1 , the synchronization deviation detection lamps 111 to 114 are disposed at positions outside the field of view of the monocular cameras 101 and 102 on the lighting board 103. In this case, as shown in FIG. Fig. 9B As shown in , the synchronization deviation detection lights 111 and 113 are disposed on the upper side of the monocular cameras 101 and 102, and the synchronization deviation detection lights 112 and 114 are disposed on the lower side of the monocular cameras 101 and 102. The driving power lines of the synchronization deviation detection lights 111 and 113 are electrically connected to each other, and the synchronization deviation detection lights 111 and 113 can be turned on and off at the same time. Similarly, the driving power lines of the synchronization deviation detection lights 112 and 114 are electrically connected to each other, and the synchronization deviation detection lights 112 and 114 can be turned on and off at the same time.

[0121] like Fig. 9B As shown in FIG. 1 , the lens of the monocular camera 101 is included in the irradiation ranges 115 and 116 of the synchronization deviation detection lights 111 and 112 , and the lens of the monocular camera 102 is included in the irradiation ranges 117 and 118 of the synchronization deviation detection lights 113 and 114 .

[0122] According to such a setting of the layout of the lights and the irradiation range, when strong light is radiated from the synchronization deviation detection lights 111 to 114 to the lens, scattering of light (i.e., so-called stray light) occurs in the lens barrels and the like of the monocular cameras 101 and 102. In the case where the images of the monocular cameras 101 and 102 are captured in a state where the synchronization deviation detection lights 111 to 114 are turned on, an image in which lens glare occurs can be captured. That is, for example, in the case where an image is captured by the monocular camera 101 when the synchronization deviation detection light 111 is turned on, an image in which only the upper right portion thereof where the synchronization deviation detection light 111 is located is bright can be captured.

[0123] As described above, in the present embodiment, the synchronization deviation detection lamps 111 to 114 are disposed outside the field of view of the monocular cameras 101 and 102, while the irradiation range thereof covers a part of the incident opening of the imaging optical system of the monocular cameras 101 and 102. As a result, an image of incident light from each lamp can be captured by using stray light in the lens barrel.

[0124] In this embodiment, the brightness of a partial area of ​​the image is controlled by turning on / off the synchronous deviation detection lamps 111 to 114. By using this, the brightness of the partial area of ​​the image can be controlled by the method described in the first embodiment. Figure 4The synchronization deviation amount is detected by the method of step S16 shown in FIG.

[0125] In this embodiment, since the synchronization deviation detection lights 111 to 114 are outside the field of view of the monocular cameras 101 and 102, the monocular cameras 101 and 102 can capture images without blocking any part of their field of view. In addition, the positions of the synchronization deviation detection lights 111 to 114 do not need to be finely adjusted, as long as the synchronization deviation detection lights 111 to 114 are deployed outside the field of view and at a position close to the field of view.

[0126] Alternatively, as a modification example of the present embodiment, the following configuration may be considered. Fig. 10A and 10B As shown in , the synchronization deviation detection lamps 121 to 124 are disposed on the front side of the stereo camera 1, and the diffusion plate 125 is disposed on the camera side of the lighting board 103 (that is, on the rear surface of the lighting board 103). The synchronization deviation detection lamps 121 to 124 radiate light onto the diffusion plate 125. In this case, the size and shape of the diffusion plate 125 are set so that the irradiation range of the diffused light covers the lenses of the monocular cameras 101 and 102. According to this configuration as well, by emitting light from the synchronization deviation detection lamps 121 to 124 to cause stray light in the lens barrels of the monocular cameras 101 and 102 as described above, an image in which lens glare occurs can be captured, and synchronization can be performed by using a method similar to the above method.

[0127] As described above, according to the configuration in which the synchronization deviation detection lamps 121 to 124 are disposed on the camera and the diffusion plate 125 is used, the optical path length from the synchronization deviation detection lamps 121 to 124 to the monocular cameras 101 and 102 can be doubled. Therefore, even in the case where the lighting board 103 is close to the monocular cameras 101 and 102, the irradiation range of the diffused light from the diffusion plate 125 can be wide. As a result, the lighting board 103 can be disposed at a position close to the monocular cameras 101 and 102, and thus the size of the structure around the stereo camera 1 can be reduced.

[0128] Fourth embodiment

[0129] In the above-described first and third embodiments, the synchronization deviation detection lamp is incorporated in the unit of the stereo camera 1. Therefore, the size of the unit of the stereo camera 1 may be large and the cost thereof may be high.

[0130] In this embodiment, a configuration example in which the size and weight of a unit part of the stereo camera 1 can be relatively reduced by using a retro-reflective material will be described. In addition, in this embodiment, a configuration that is advantageous in the case of using the stereo camera 1 as a visual system of a robot device will be described. This robot device is deployed together with the stereo camera 1 in a production line or manufacturing system for manufacturing products such as industrial products from workpieces. The stereo camera 1 can be used, for example, to perform three-dimensional measurement of a workpiece manipulated by the robot device. For example, the operation of the robot device can be controlled based on three-dimensional information including the depth of the mounting part of the workpiece, etc., obtained by the three-dimensional measurement of the stereo camera 1.

[0131] In the following description, parts of the configuration of the hardware and control system that are different from those of the first embodiment will be illustrated and described, and detailed descriptions of parts similar to those of the first embodiment will be omitted, assuming that the parts can be configured in a similar manner to that described above and can have similar effects.

[0132] In this embodiment, if Fig.11A and 11B As shown in FIG. 1 , the synchronization deviation detection lamps 131 to 134 are disposed on the front surface of the lighting board 103. The irradiation direction of the synchronization deviation detection lamps 131 to 134 is opposite to the direction directed toward the monocular cameras 101 and 102. That is, in the present embodiment, the synchronization deviation detection lamps 131 to 134 radiate illumination light toward the measurement target object. In addition, as Fig. 11B As shown in FIG. 1 , the synchronization deviation detection lamps 131 and 132 are disposed near the monocular camera 101, and the synchronization deviation detection lamps 133 and 134 are disposed near the monocular camera 102. Fig. 11B As shown in , the synchronization deviation detection lights 131 and 133 are disposed on the upper side of the monocular cameras 101 and 102, and the synchronization deviation detection lights 132 and 134 are disposed on the lower side of the monocular cameras 101 and 102. The driving power lines of the synchronization deviation detection lights 131 and 133 are electrically connected to each other, and the synchronization deviation detection lights 131 and 133 can be turned on / off at the same time. The driving power lines of the synchronization deviation detection lights 132 and 134 are electrically connected to each other, and the synchronization deviation detection lights 132 and 134 can be turned on / off at the same time.

[0133] like Fig. 11C As shown in , the stereo camera 1 of the present embodiment is attached to the robot hand 4 of the robot apparatus. According to this configuration, for example, the position and orientation of the robot hand 4 of the robot apparatus can be controlled based on the result of three-dimensional measurement of the measurement target object obtained by the stereo camera 1.

[0134] The robot hand 4 includes fingers 1401 and 1402 as gripping devices, and the fingers 1401 and 1402 can grip the measurement target object. In addition, in the present embodiment, the retro-reflective marker 501 is attached to the distal end portion of the finger 1401, and the retro-reflective marker 502 is attached to the middle portion of the finger 1401. Similarly, the retro-reflective marker 503 is attached to the distal end portion of the finger 1402, and the retro-reflective marker 504 is attached to the middle portion of the finger 1402. The retro-reflective markers 501 to 504 can be formed of a plastic material or the like, and these can be attached by any method such as screw bonding and bonding.

[0135] The relative positional relationship between the retro-reflective markers 501 to 504 and the stereo camera 1 determined based on the attachment positions of the retro-reflective markers 501 to 504 and the installation position of the stereo camera 1 is determined so that the retro-reflective markers 501 to 504 are included in the common field of view 108 of the stereo camera 1 .

[0136] The synchronization deviation detection lamps 131 to 134 are preferably composed of illumination light sources having high directivity. For example, the illumination light from the synchronization deviation detection lamp 131 is configured to illuminate only the retro-reflective mark 501 and its vicinity. Similarly, the illumination light from the synchronization deviation detection lamps 132, 133, and 134 is configured to illuminate only the retro-reflective marks 502, 503, and 504 and their vicinity, respectively.

[0137] As described above, if the synchronization deviation detection lights 131 to 134 are disposed in the vicinity of the monocular cameras 101 and 102 , the reflected light from the retro-reflective markers 501 to 504 is incident on the monocular cameras 101 and 102 .

[0138] For example, in the case where the monocular cameras 101 and 102 capture images when the synchronous deviation detection lights 131 and 133 are turned on, an image where only the vicinity of the retro-reflective markers 501 and 503 is bright can be obtained. As described above, the brightness of a partial area of ​​the image can be controlled, so that the image can be captured by using the method described in the first embodiment. Figure 4 The synchronization deviation amount is detected by the method of step S16 shown in FIG.

[0139] According to the above configuration, it is not necessary to provide a lamp that radiates light toward the monocular cameras 101 and 102 while ensuring a certain optical path length. In order to provide a lamp that emits light directly incident on the monocular cameras 101 and 102, the distance between the lighting board 103 and the monocular cameras 101 and 102 tends to be long. However, according to the present embodiment, it is not necessary to adopt such a lamp that emits light in such a direction, so the unit of the stereo camera 1 can be more miniaturized.

[0140] It is to be noted that the lighting board 103 may be provided with an opening for ensuring the field of view of the monocular cameras 101 and 102, or may be formed of a transparent material, etc. This also applies to other embodiments described in this specification.

[0141] Furthermore, the synchronization deviation detection lamps 131 to 134 can also be used as lamps for three-dimensional measurement of the stereo camera 1. As a result, the monocular cameras 101 and 102 can be synchronized without providing dedicated lamps, and thus the stereo camera 1 can be manufactured at a relatively low cost.

[0142] In the present embodiment, the stereo camera 1 is deployed on the robot hand 4, so the relative positional relationship between the stereo camera 1 and the robot hand 4 does not change. That is, the retro-reflective markers 501 to 504 are always in the common field of view of the stereo camera 1. It should be noted that a configuration in which a retro-reflective marker is provided on an object such as a workpiece can also be considered. In this configuration, in the case where there are multiple parts whose images are to be captured, it is necessary to provide a retro-reflective material for each part whose image is to be captured. In contrast, according to the configuration in which the retro-reflective material is provided on the robot hand 4, the man-hours for preparing to deploy the retro-reflective material are reduced, so the manufacturing system can be installed very easily.

[0143] It should be noted that, although four retro-reflective markers 501 to 504 are provided in the present embodiment, the number of retro-reflective markers can be arbitrarily selected and can be increased or decreased according to the required image capture specifications. For example, one retro-reflective marker can be attached in an area including almost all directions in which the common visual field range 108 extends, and synchronization can be performed by treating the reflected light from the retro-reflective marker in a manner similar to the illumination light from the synchronization deviation detection lamp of the second embodiment, by using the method described in the second embodiment.

[0144] Fifth embodiment

[0145] In the above-mentioned first to fourth embodiments, especially in the first embodiment, control using random processing has been described in which a monocular camera is repeatedly switched to Figure 3 The power-off state 401 shown in FIG. 4 is switched to the initialization state 402, the image capture parameter adjustment state 403, and then the moving image output state 404 repeatedly until the synchronization deviation amount becomes equal to or less than the predetermined value. The time taken until the synchronization is completed varies with the probability event. In addition, there is a possibility that the synchronization cannot be quickly stabilized and takes a long time (which is actually problematic).

[0146] That is, according to the above control, it may take a long time to complete the synchronization or the synchronization may not be completed, especially in the case where the allowable range of the predetermined synchronization deviation amount is narrow. That is, it is difficult to estimate the time required for synchronization. In addition, as another problem, in the case where the difference in exposure cycle between cameras is large, there is a possibility that the synchronization deviation occurs again when a long time has passed after the synchronization is performed once and the synchronization deviation amount increases as time passes.

[0147] In the present embodiment, a configuration of detecting synchronization deviation during three-dimensional measurement and selecting an image pair with the smallest synchronization deviation from images continuously output from the monocular cameras 101 and 102 in a moving image output state is considered to solve the above problem.

[0148] In the following description, parts of the configuration of the hardware and control system that are different from those of the first embodiment will be illustrated and described, and detailed descriptions of parts similar to those of the first embodiment will be omitted, assuming that the parts can be configured in a similar manner to that described above and can have similar effects.

[0149] In this embodiment, a method of synchronization performed by selecting an image will be mainly described, assuming that the configuration of the imaging system including multiple cameras, each component of the stereo camera 1, and each component of the image processing device 2 are basically the same as those in the first to fourth embodiments.

[0150] The synchronization deviation amount is calculated by a method substantially the same as in the first to fourth embodiments. It should be noted that in the present embodiment, in the moving image output state of the monocular camera 102, images of a plurality of continuous frames including a target frame are stored on the memory without being deleted from a storage portion such as a memory or an image memory. The number of the plurality of continuous frames changes according to the synchronization deviation amount, and the plurality of continuous frames includes at least one frame before the target frame and at least one frame after the target frame.

[0151] Synchronous methods

[0152] Fig.12 The figure shows the synchronization process performed by selecting an image in this embodiment. Fig.12 In step S30, the synchronization deviation amount is calculated. In this step, the synchronization deviation amount is calculated by any of the methods described in the above embodiments. It should be noted that in order to calculate the synchronization deviation amount, among the images captured by the monocular camera 102 during the moving image output state, the images before and after the target frame are stored in the memory.

[0153] In step S31, it is determined whether the amount of synchronization deviation of the monocular camera 102 relative to the monocular camera 101, which indicates the degree to which the monocular camera 102 is ahead in time relative to the monocular camera 101, is +1 / 2 frame or more. For example, in a case where the frame rate of the monocular cameras 101 and 102 is 25fps (i.e., 0.04 seconds per frame), it is determined whether the amount of synchronization deviation of the monocular camera 102 relative to the monocular camera 101 is +0.02 or more. In the case where the amount of synchronization deviation is +1 / 2 frame or more (i.e., in the case where the result of step S31 is "yes"), the processing proceeds to step S35. In addition, in the case where the amount of synchronization deviation is not equal to or greater than +1 / 2 frame (i.e., in the case where the result of step S31 is "no"), the processing proceeds to step S32.

[0154] In step S32, it is determined whether the synchronization deviation amount of the monocular camera 102 relative to the monocular camera 101 is -1 / 2 frame or more negative. For example, when the frame rate of the monocular cameras 101 and 102 is 25fps, it is determined whether the synchronization deviation amount of the monocular camera 102 relative to the monocular camera 101 is -0.02 or more negative. In the case where the synchronization deviation amount is -1 / 2 frame or more negative (that is, in the case where the result of step S32 is "yes"), the processing proceeds to step S34. In addition, in the case where the synchronization deviation amount is not equal to or more negative than -1 / 2 frame (that is, in the case where the result of step S32 is "no"), the processing proceeds to step S33.

[0155] In step S33, an image of a target frame captured by the monocular camera 101 and an image of a target frame captured by the monocular camera 102 are selected as a synchronized image pair. In step S34, an image of a target frame captured by the monocular camera 101 and an image of a next frame captured by the monocular camera 102 are selected as a synchronized image pair. In step S35, an image of a target frame captured by the monocular camera 101 and an image of a previous frame captured by the monocular camera 102 and stored on a memory are selected as a synchronized image pair.

[0156] Regarding steps S33, S34, and S35, the time difference between the time point at which the image for detecting the synchronization deviation among the multiple images captured by the monocular camera 101 is captured and the time point at which the image selected for synchronization (i.e., one of the image pair) captured by the monocular camera 101 is captured is set to Δf1. In addition, the time difference between the time point at which the image for detecting the synchronization deviation among the multiple images captured by the monocular camera 102 is captured and the time point at which the image captured by the monocular camera 102 and selected for synchronization (i.e., the other of the image pair) is captured is set to Δf2. In step S33, Δf1 may be equal to Δf2. However, in steps S34 and S35, Δf1 may be different from Δf2. For example, in steps S34 and S35, there is a difference of one frame between Δf1 and Δf2.

[0157] In the above description, a synchronization method applicable to the case where the synchronization deviation amount between the monocular cameras 101 and 102 is 1 frame or less has been described. However, in the case where the synchronization deviation amount between the monocular cameras 101 and 102 is greater than 1 frame, for example, the frame is shifted by using the integer part of the synchronization deviation amount [frame]. Then, the image to be used can be selected in a manner similar to the above-described manner based on the image of the frame to which the frame has been shifted by using the decimal part of the synchronization deviation amount.

[0158] For example, it is assumed that the monocular camera 102 is ahead of the monocular camera 101 by 2.4 frames in time. In this case, an image 2 frames before the image of the target frame captured by the monocular camera 102 is set as a reference, and determination is performed in step S31 by using 0.4 frames as the synchronization deviation amount. Then, since the synchronization deviation amount is not equal to or positively greater than 1 / 2 frame, the processing proceeds to the determination of step S32. In the determination of step S32, since the synchronization deviation amount is not equal to or negatively greater than -1 / 2 frame, the processing proceeds to step S33, and an image pair is selected by using the image 2 frames before the image of the target frame as a reference. In this case, as a result, the image of the target frame captured by the monocular camera 101 and the image 2 frames before the image of the target frame captured by the monocular camera 102 are selected as an image pair. In this case, there is a difference of 2 frames between Δf1 and Δf2.

[0159] By setting an appropriate image pair, the time difference between the time points at which two images to be used for image processing (such as three-dimensional measurement) are respectively captured becomes smaller than the time difference between the time points at which two images used for image processing for detecting synchronization deviation are respectively captured. The time interval in frames between the images to be selected as a pair for three-dimensional measurement as described above can be predetermined, for example, before three-dimensional measurement or when three-dimensional measurement is not performed. That is, as an image captured by the monocular camera 102 to be paired with an image of a target frame captured by the monocular camera 101, it is predetermined which of the image of the target frame captured by the monocular camera 102, the image of the previous frame captured by the monocular camera 102, and the image of the next frame captured by the monocular camera 102 is to be used. Alternatively, synchronization can be performed for each three-dimensional measurement in order to determine a frame to be used as an image captured by the monocular camera 102 to be paired with an image captured by the monocular camera 101.

[0160] It is to be noted that, although in the present embodiment, a frame to be used as an image captured by the monocular camera 102 to be paired with an image captured by the monocular camera 101 is determined, conversely, a frame to be used as an image captured by the monocular camera 101 to be paired with an image captured by the monocular camera 102 may be determined. In addition, control of switching the camera used as a standard according to various conditions related to three-dimensional measurement may be performed.

[0161] In addition, the synchronization according to the present embodiment is generally applicable to the synchronization of multiple cameras. However, in the case where the synchronization deviation detection light is turned on during the three-dimensional measurement of the present embodiment, each monocular camera of the stereo camera 1 is preferably composed of a global shutter camera. In the case where the synchronization according to the present embodiment is applied to a stereo camera composed of a rolling shutter camera, a line corresponding to the timing of capturing an image of illumination light appears as a bright band that serves as noise in the image to be used for measurement. Therefore, the three-dimensional measurement may be hindered. By using a global shutter camera as described above, the influence of the synchronization light on the three-dimensional measurement can be reduced.

[0162] Alternatively, in the case of using a stereo camera composed of a rolling shutter camera and lighting a synchronization deviation detection lamp during three-dimensional measurement, the following configuration may be adopted. For example, it may be considered to deploy a synchronization deviation detection lamp in an area not used for three-dimensional measurement outside the common field of view of the monocular cameras 101 and 102 as in the first embodiment. According to this configuration, synchronization of image capture timing can be performed without interfering with three-dimensional measurement of the synchronization deviation detection lamp.

[0163] According to the configuration of selecting the frame image to be used for three-dimensional measurement based on the size of the synchronization deviation amount as in the present embodiment, the time required to complete the synchronization is not a probabilistic event. The time required to complete the synchronization can be shortened and kept constant. In addition, synchronization can also be performed for each three-dimensional measurement, so the effect of not causing synchronization deviation even when the measurement is continued for a long period of time can be expected.

[0164] Sixth embodiment

[0165] Although the stereo camera 1 is composed of two monocular cameras 101 and 102 in the above-mentioned first to fifth embodiments, it is also conceivable to use three or more cameras. If a monocular camera is additionally provided in addition to the two cameras for three-dimensional measurement, for example, the amount of synchronization deviation between the two cameras during three-dimensional measurement can be monitored, and the pair or combination of monocular cameras for three-dimensional measurement can be switched. As the timing of switching the combination of monocular cameras for three-dimensional measurement, the timing at which the amount of synchronization deviation of the two cameras being used for three-dimensional measurement becomes large can be considered. In addition, it is also conceivable to periodically switch the control of the pair or combination of monocular cameras for three-dimensional measurement. In this case, for example, two pairs of monocular cameras 101 and 102 are prepared as will be described later, and control can be performed in which one pair performs synchronization deviation detection and synchronization based on synchronization deviation detection while the other pair performs three-dimensional measurement.

[0166] In the first to fifth embodiments described above, in the case where the difference in exposure period between the monocular cameras 101 and 102 is large, there is a possibility that synchronization deviation occurs again when a long time has passed after synchronization is performed once, and the amount of synchronization deviation increases as time passes. In addition, in some layouts of synchronization deviation detection lights, in the case where the cameras are not synchronized, it is necessary to stop three-dimensional measurement to perform synchronization again. According to the present embodiment, the above problem can be solved by using three or more monocular cameras.

[0167] In the following description, parts of the configuration of the hardware and control system that are different from those of the first embodiment will be illustrated and described, and detailed descriptions of parts similar to those of the first embodiment will be omitted, assuming that the parts can be configured in a similar manner to that described above and can have similar effects.

[0168] As an example in Fig.13 As shown in FIG. 1 , in the present embodiment, the stereo camera 1 includes three monocular cameras 101 , 102 and 110 . Fig.13 With Figure 1BThe configuration of the stereo camera 1 is illustrated in a similar manner. Similar to the first embodiment, the monocular cameras 101 and 102 are arranged to be separated from each other by a predetermined baseline length. The monocular cameras 101, 102, and 110 are arranged so that the baseline length between the imaging optical systems of the monocular cameras 101 and 110 is equal to the baseline length between the monocular cameras 101 and 102.

[0169] In addition, if Fig.13 As shown in FIG. 1 , synchronization deviation detection lights 104 to 107 and 109 are disposed corresponding to the monocular cameras 101, 102, and 110. The synchronization deviation detection light 107 is disposed between the monocular cameras 102 and 110 so that the synchronization deviation detection light 107 can be used for both the monocular cameras 102 and 110.

[0170] According to this configuration, for example, the amount of synchronization deviation between the monocular cameras 101 and 102 performing three-dimensional measurement can be monitored, and when a synchronization deviation occurs, the combination of cameras used for three-dimensional measurement can be switched. For example, the combination of cameras used for three-dimensional measurement is switched from a first combination including the monocular cameras 101 and 102 to a second combination including the monocular cameras 101 and 110. According to this configuration, synchronization can be performed in parallel with three-dimensional measurement without delaying the three-dimensional measurement.

[0171] Conversely, the amount of synchronization deviation between the monocular cameras 101 and 110 can be monitored while the three-dimensional measurement is performed by the monocular cameras 101 and 110. Then, when the synchronization deviation occurs, the camera used for the three-dimensional measurement is switched to the monocular cameras 101 and 102. As a result, the measurement can be continued using the monocular cameras 101 and 102. In this way, the roles of the two cameras in the stereoscopic imaging system can be alternately switched between three-dimensional measurement and synchronization, so the three-dimensional measurement can be continued by the stereoscopic imaging system that is always synchronized without stopping or delaying the three-dimensional measurement.

[0172] Method for switching measurement cameras

[0173] Fig.14 A specific example of control of switching the monocular cameras 101 , 102 , and 110 constituting two stereoscopic imaging systems is illustrated. Fig.14 An example of a camera switching control process is illustrated.

[0174] exist Fig.14 In step S40, three-dimensional measurement is started by the monocular cameras 101 and 102. In step S41, the synchronization deviation amount between the monocular cameras 101 and 102 is calculated. As a method of calculating the synchronization deviation amount, a method substantially the same as the method of the first embodiment using the synchronization deviation detection lights 104 to 107 can be used.

[0175] In step S42, it is determined whether the amount of synchronization deviation between the monocular cameras 101 and 102 is equal to or greater than a predetermined threshold value. In the case where the amount of synchronization deviation between the monocular cameras 101 and 102 is less than the threshold value (i.e., in the case where the result of step S42 is "No"), the process returns to step S41, and the amount of synchronization deviation is checked again. In addition, in the case where the amount of synchronization deviation between the monocular cameras 101 and 102 is equal to or greater than the threshold value (i.e., in the case where the result of step S42 is "Yes"), the process proceeds to step S43.

[0176] In step S43, synchronization of the monocular cameras 101 and 110 is started. This synchronization process can be performed by, for example, repeatedly switching the state of the monocular camera 110 from the power-off state to the initialization state, and then to ... until the synchronization deviation amount becomes equal to or less than a predetermined value as in the first embodiment. Alternatively, the synchronization of the monocular cameras 101 and 110 can be performed by a method different from the above.

[0177] In step S44, the imaging system for three-dimensional measurement is switched to the monocular cameras 101 and 110, and the three-dimensional measurement is started. That is, the camera for three-dimensional measurement is switched from the monocular cameras 101 and 102 to the monocular cameras 101 and 110. Then, while switching the camera whose synchronization deviation amount is monitored and the camera subjected to synchronization, steps S41 to S44 are repeated until the desired three-dimensional measurement is completed.

[0178] Although an example in which the stereo camera 1 is composed of three monocular cameras has been described above, the stereo camera 1 may be composed of more monocular cameras. For example, four or more monocular cameras may be used, and the imaging system whose synchronization deviation amount is calculated and the imaging system subjected to synchronization may be switched. Such a configuration in which the imaging system whose synchronization deviation amount is calculated and the imaging system subjected to synchronization operate in parallel has a great advantage. For example, while the imaging systems of some cameras are performing three-dimensional measurement, synchronization may be performed on the imaging systems of other monocular cameras in parallel. According to such a configuration, for example, in the case where it is required to switch the imaging system for three-dimensional measurement, it is not required Fig.14 The processing time of the synchronization process shown in , and switching to the already synchronized imaging system can be performed at a high speed.

[0179] It should be noted that Fig.13 The horizontal positional relationship between the monocular cameras 102 and 110 and the monocular camera 101 shown in FIG. 1 may be different from Fig.13The positional relationship shown in . For example, a layout in which the monocular cameras 101, 102, and 110 are arranged in a straight line (with the monocular camera 101 in the middle) can be considered, or the three monocular cameras can be arranged at the vertices of an equilateral triangle so that the intervals therebetween are constant. In addition, in the above description, a pair of monocular cameras 101 and 102 or a pair of monocular cameras 101 and 110 are used for three-dimensional measurement. However, depending on the layout of the monocular cameras, a pair of monocular cameras 102 and 110 can be used for measurement.

[0180] The configuration of the above-described embodiment is merely an example, and those skilled in the art may modify its design in various ways within the concept of the present embodiment. For example, in the above description, a plurality of monocular cameras that perform synchronized image capture constitute a stereo camera for three-dimensional measurement. However, it is undoubted that the hardware configuration and image capture control of these embodiments may also be implemented in an imaging system that is composed of a plurality of monocular cameras and that needs to perform synchronized image capture for some purpose. For example, in the case of creating a three-dimensional moving image such as a free viewpoint image, since synchronized image capture is performed by a plurality of cameras, applying the above-described embodiment to detect synchronized deviation is effective for improving the quality of the moving image created by synchronized image capture. In addition, also in the case of performing synchronized image capture by a plurality of cameras incorporated in an image capture device such as a smart phone, applying the above-described embodiment to detect synchronized deviation is effective for improving the quality of the moving image created by synchronized image capture.

[0181] It should be noted that any light source can be used for the light image of the object in the image for synchronous deviation detection as long as the image of its illumination light can be captured by the monocular cameras 101 and 102. That is, the illumination light is not limited to the illumination light of the illumination device included in the imaging system, and may be the illumination light of the illumination device outside the imaging system. It is also possible to radiate the illumination light at a predetermined timing by blocking natural light at a predetermined shading timing. In addition, the configuration of providing the stereo camera 1 of the above-mentioned embodiment on the robot hand 4 has been described, but the configuration is not limited thereto. The above-mentioned embodiment can be applied to a machine that can automatically perform operations such as extension, contraction, bending, vertical movement, horizontal movement, rotation, or a combination of these based on information stored in a storage device provided in a controller.

[0182] Other embodiments

[0183] Embodiments of the present invention may also be implemented by reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a 'non-transitory computer-readable storage medium') to perform one or more of the functions of the above-mentioned embodiments and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more of the functions of the above-mentioned embodiments, and by a computer of the system or device, for example, reading out and executing computer executable instructions from a storage medium to perform one or more of the functions of the above-mentioned embodiments and / or controlling one or more circuits to perform one or more of the functions of the above-mentioned embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)) TM ), one or more of flash memory devices, memory cards, etc.

[0184] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0185] Although the present disclosure includes exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An imaging system, comprising: a first camera and a second camera, the first camera and the second camera configured to capture an image of an area illuminated by the lighting device; as well as Controller, wherein the controller is configured to cause each of the first camera and the second camera to capture, at predetermined time intervals, an image including a brightness pattern of an area illuminated by the illumination device with illumination light in an on / off mode at the predetermined time intervals, wherein the controller is configured to detect a synchronization deviation of image capture timings of the first camera and the second camera by using a brightness pattern of an image captured by the first camera and a brightness pattern of an image captured by the second camera, and The predetermined time interval of the lighting device is shorter than the frame rates of the first camera and the second camera.

2. The imaging system according to claim 1, wherein the images obtained by the first camera and the second camera are still images obtained by cutting out from moving image data, the moving image data being obtained from the first camera and the second camera at the predetermined time intervals.

3. An imaging system according to claim 1, wherein the controller is configured to detect whether the synchronization deviation has occurred by calculating a quantity based on a difference between a first time point and a second time point and determining whether the quantity is equal to or less than a predetermined value, the quantity being calculated by using a first image captured by the first camera at the first time point and a second image captured by the second camera at the second time point. 4 . The imaging system according to claim 1 , wherein the first camera and the second camera constitute a stereo camera configured to obtain three-dimensional information of an object. 5 . The imaging system according to claim 1 , wherein a light emitting member constituting the illumination device is attached to a housing that positions the first camera and the second camera relative to each other. 6 . The imaging system according to claim 1 , wherein a radiation direction of a light emitting member constituting the illumination device includes a radiation direction directed toward imaging optical systems of the first camera and the second camera.

7. The imaging system according to any one of claims 1 to 3, wherein the lighting device comprises a first light emitting member disposed in an imaging area of ​​the first camera and a second light emitting member disposed in an imaging area of ​​the second camera, and The first light emitting member and the second light emitting member are connected to each other through a driving power supply line and radiate illumination light at the same light emission timing.

8. The imaging system according to any one of claims 1 to 3, wherein the illumination device is disposed within a common field of view of the first camera and the second camera.

9. The imaging system of any one of claims 1 to 3, wherein the controller is configured to detect the synchronization deviation based on the position of a brightness pattern in an image.

10. The imaging system of any one of claims 1 to 3, wherein the controller is configured to detect the synchronization deviation based on a ratio of brightness patterns in an image.

11. The imaging system of any one of claims 1 to 3, wherein the brightness pattern in the image is formed by a portion captured with the illumination light on and a portion captured with the illumination light off.

12. A manufacturing system comprising: The imaging system according to claim 1; a robotic device configured to manipulate a workpiece; as well as A robot controller is configured to control the robot device based on the three-dimensional information of the workpiece obtained by the first camera and the second camera of the imaging system.

13. The manufacturing system according to claim 12, wherein a retro-reflective material is attached to a gripping device of the robot apparatus, the retro-reflective material reflects illumination light radiated from an illumination device, an image of the illumination light reflected by the retro-reflective material is captured by the first camera and the second camera, and the controller performs image processing on the image of the reflected illumination light to detect a synchronization deviation of the image capture timing of the first camera and the second camera.

14. A control method for controlling a controller in an imaging system, the imaging system comprising the controller, and a first camera and a second camera for capturing an image of an area illuminated by an illumination device, the control method comprising: controlling the lighting device to emit lighting light in an on / off mode at predetermined time intervals; obtaining an image captured by each of the first camera and the second camera at the image capturing timing at the predetermined time interval when illumination light is applied, the image including a brightness pattern corresponding to the image capturing timing; as well as detecting a synchronization deviation of image capturing timings of the first camera and the second camera by using a brightness pattern of an image captured by the first camera and a brightness pattern of an image captured by the second camera, The predetermined time interval of the lighting device is shorter than the frame rates of the first camera and the second camera.

15. A method for manufacturing a product, the method comprising: A product is manufactured from a workpiece by manipulating the workpiece by a robot device of the manufacturing system according to claim 12 or 13. 16 . A non-transitory computer-readable recording medium storing a control program causing a computer to execute the control method according to claim 14 .

Images