Image processing method and image processing system

Abstract

The system generates images suitable for machine vision processing from the images captured by the imaging device. [Solution] The image processing method determines, based on a predetermined instruction, whether or not to use an image captured by a predetermined imaging device for predetermined machine vision processing. If it is determined that the image should be used for machine vision processing, it performs machine vision image processing on the captured image to generate a machine vision image and provides the machine vision image to the machine vision processing. If it is determined that the image should not be used for machine vision processing, it performs display image processing on the captured image to generate a display image and outputs it to a predetermined display device.

Description

Technical Field

[0001] The present disclosure relates to an image processing method and an image processing system.

Background Art

[0002] For a photographing device mounted on a moving body such as a vehicle, a drone, or a robot, calibration of the position and orientation of the photographing device is performed so that the image projected by the photographing device meets the specified requirements. For example, Patent Document 1 discloses a method for calibrating an in-vehicle stereo camera that acquires stereo images for distance image generation, including a determination step of determining whether the vehicle is stopped or running, and a calibration step of performing calibration of the stereo camera while the vehicle is stopped.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When machine vision processing is performed using an image captured by a photographing device, the image output by the photographing device may be appropriate for visual recognition by a user but not appropriate for machine vision processing.

[0005] An object of the present disclosure is to provide a technique for generating an image suitable for machine vision processing from an image captured by a photographing device.

Means for Solving the Problems

[0006] One aspect of the present disclosure provides an image processing method that, based on a predetermined instruction, determines whether or not to use an image captured by a predetermined imaging device for predetermined machine vision processing; if it is determined to use the image for machine vision processing, performs machine vision image processing on the image to generate a machine vision image and provides the machine vision image to the machine vision processing; and if it is determined not to use the image for machine vision processing, performs display image processing on the image to generate a display image and outputs it to a predetermined display device.

[0007] One aspect of the present disclosure provides an image processing system comprising a shooting device and a control device, wherein the control device determines, based on a predetermined instruction, whether or not to use an image captured by the shooting device for a predetermined machine vision processing, transmits a first instruction to the shooting device if it determines to use the image for the machine vision processing, transmits a second instruction to the shooting device if it determines not to use the image for the machine vision processing, the shooting device, upon receiving the first instruction, performs machine vision image processing on the image to generate a machine vision image and transmits the machine vision image to the control device, the shooting device, upon receiving the second instruction, performs display image processing on the image to generate a display image and transmits the display image to the control device, the control device, upon determining to use the image for the machine vision processing, performs the machine vision processing based on the machine vision image received from the shooting device, and outputs the display image received from the shooting device to a predetermined display device.

[0008] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0009] According to this disclosure, an image suitable for machine vision processing can be generated from an image captured by the imaging device. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram showing example configuration 1 of the image processing system according to Embodiment 1. [Figure 2] A diagram showing an example of setting the marker area according to Embodiment 1. [Figure 3] This figure shows an example of a second calibration image of the marker portion according to Embodiment 1, where the background is bright and the marker is illuminated from an oblique angle. [Figure 4] This figure shows an example of a second calibration image of the marker portion according to Embodiment 1, where the background is dark and the marker is illuminated from an oblique angle. [Figure 5] This figure shows an example of a second calibration image of the marker portion according to Embodiment 1, where the background is colored and the marker is illuminated from an oblique angle. [Figure 6] Block diagram showing example configuration 2 of the image processing system according to Embodiment 1. [Figure 7] Block diagram showing example configuration 3 of the image processing system according to Embodiment 1. [Figure 8] Block diagram showing example configuration 4 of the image processing system according to Embodiment 1. [Figure 9] Block diagram showing example configuration 5 of the image processing system according to Embodiment 1. [Figure 10] Block diagram showing an example of the hardware configuration of the imaging device and control device according to Embodiment 1. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, detailed descriptions that are not necessary may be omitted. For example, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims. Also, even if the functions of one configuration shown in this embodiment are realized by two or more physical configurations, or the functions of two or more configurations are realized by, for example, one physical configuration, it is acceptable.

[0012] (Embodiment 1) <Configuration Example 1> FIG. 1 is a block diagram showing Configuration Example 1 of an image processing system 2 according to Embodiment 1.

[0013] The image processing system 2 includes a photographing device 10, a control device 20, and a display device 30.

[0014] For example, the photographing device 10 may be an external camera mounted on a vehicle, the control device 20 may be an Electronic Control Unit (ECU) mounted on the vehicle, and the display device 30 may be a display mounted inside the vehicle. Alternatively, the display device 30 may be a display arranged outside the vehicle and connected to the ECU.

[0015] In this embodiment, an example in which the photographing device 10 is mounted on a vehicle will be described. However, the photographing device 10 may be mounted on a moving body other than a vehicle. For example, the photographing device 10 may be a camera mounted on a moving body such as a drone or a robot, the control device 20 may be a controller mounted on the moving body such as the drone or the robot, and the display device 30 may be a display connected to the controller.

[0016] The imaging device 10, the control device 20, and the display device 30 may be connected through an in-vehicle network (e.g., Controller Area Network (CAN)). Alternatively, the imaging device 10, the control device 20, and the display device 30 may be connected through a predetermined communication network (e.g., Ethernet (registered trademark), Wi-Fi (registered trademark), or Bluetooth (registered trademark)). Without being limited to this, a one-to-one communication path may be used between the devices mounted in the vehicle.

[0017] The imaging device 10 includes an imaging unit 11, a switching unit 12, a display image processing unit 41, a calibration image processing unit 42, and a transmission unit 13. In the present embodiment, as an example of machine vision processing, a process of calculating calibration information for calibrating the deviation between the position and orientation of the imaging device 10 mounted on the vehicle and the specified display of the image captured by the imaging device 10 will be described. However, the content of the present embodiment is not limited to the calibration process, and for example, it is applicable to various machine vision processes such as detecting pedestrians, other vehicles, and objects from the image captured by the imaging device 10. Therefore, the calibration image processing unit may be read as a machine vision image processing unit.

[0018] The imaging unit 11 includes a lens and an image sensor 105 (see FIG. 10), and generates a RAW image by the image sensor 105. Examples of the image sensor 105 include a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge Coupled Device (CCD) sensor.

[0019] Based on an instruction from the control device 20, the switching unit 12 determines whether to process the RAW image provided from the imaging unit 11 by the display image processing unit 41 or the calibration image processing unit 42. For example, when the switching unit 12 receives a display instruction from the control device 20, it determines to process the RAW image by the display image processing unit 41, and when it receives a calibration instruction from the control device 20, it determines to process the RAW image by the calibration image processing unit 42.

[0020] In this embodiment, for the sake of clarity, the switching unit 12, the display image processing unit 41, and the calibration image processing unit 42 are separated. However, the switching unit 12, the display image processing unit 41, and the calibration image processing unit 42 may be a single image processing unit. In this case, the switching unit 12 deciding to process with the display image processing unit 41 may mean that the image processing unit switches the parameters related to image processing to the parameters related to display image processing. The switching unit 12 deciding to process with the calibration image processing unit 42 may mean that the image processing unit switches the parameters related to image processing to the parameters related to calibration image processing.

[0021] The display image processing unit 41 performs display image processing on the RAW image provided by the imaging unit 11 to generate a display image. The display image processing unit 41 may also perform display image processing after converting the RAW image to an RGB image or a YUB image. Display image processing is image processing that improves the visibility for the end user, and includes, for example, brightness enhancement processing, edge enhancement processing, and / or noise reduction processing. However, display image processing is not limited to these processes.

[0022] The calibration image processing unit 42 performs calibration image processing on the RAW image provided by the imaging unit 11 to generate a first calibration image. The calibration image processing unit 42 may also perform calibration image processing after converting the RAW image to an RGB image or a YUV image. In this embodiment, the calibration image processing is image processing that improves the accuracy or performance of the computer (machine) in detecting calibration markers. Details of the calibration image processing will be described later.

[0023] When the imaging device 10 receives a display instruction, the transmission unit 13 transmits the display image generated by the display image processing unit 41 to the control device 20, and when the imaging device 10 receives a calibration instruction, it transmits the first calibration image generated by the calibration image processing unit 42 to the control device 20.

[0024] The control device 20 includes a receiving unit 21, a calibration determination unit 43, a calibration execution unit 44, and an output unit 22. The calibration determination unit 43 may be read as a machine vision determination unit, and the calibration execution unit 44 may be read as a machine vision processing unit.

[0025] The receiving unit 21 receives the display image or first calibration image transmitted from the transmitting unit 13 of the imaging device 10.

[0026] The calibration determination unit 43 determines, for example, whether or not to perform calibration on the display of an image for display. If the calibration determination unit 43 receives an instruction to perform calibration from the user, another processing unit (not shown) in the control device 20, or another device, it determines to perform calibration, and if it receives an instruction not to perform calibration from the user, another processing unit in the control device 20, or another device (or receives no instruction at all), it determines not to perform calibration.

[0027] If the calibration determination unit 43 determines that calibration should be performed, it sends a calibration instruction to the switching unit 12 of the imaging device 10. If the calibration determination unit 43 determines that calibration should not be performed, it sends a display instruction to the switching unit 12 of the imaging device 10.

[0028] Furthermore, if the calibration determination unit 43 determines that calibration should be performed, it provides the first calibration image received by the receiving unit 21 to the calibration execution unit 44. If the calibration determination unit 43 determines that calibration should not be performed, it provides the display image received by the receiving unit 21 to the output unit 22.

[0029] The calibration unit 44 generates a second calibration image from the first calibration image, detects markers from the second calibration image, and estimates the deviation of the current position and orientation of the imaging device 10 from the specified position and orientation based on the position and shape of the detected markers. The calibration unit 44 then generates calibration information to correct this deviation and sets it in the output unit 22. Details of the calibration unit 44 will be described later.

[0030] The output unit 22 sets the calibration information provided by the calibration execution unit 44. The output unit 22 corrects the misalignment of the display image provided by the calibration judgment unit 43 using the calibration information and outputs it to the display device 30.

[0031] As a result, the control device 20 can display a display image on the display device 30 that is highly visible to the user, with deviations from the specified position and orientation corrected.

[0032] <Details of the image processing unit used for calibration> The calibration image processing unit 42 performs at least one of the calibration image processing steps (A1) and (A2) below to generate a first calibration image. (A1) The calibration image processing unit 42 performs contrast enhancement processing on the RAW image (or RGB image, YUV image) (for example, by gradation correction or gamma correction) to generate a first calibration image. (A2) The calibration image processing unit 42 performs noise reduction processing on the RAW image (or RGB image, YUV image) to generate a first calibration image.

[0033] <Details of the Calibration Department> Figure 2 shows an example of setting the marker area according to Embodiment 1.

[0034] As shown in Figure 2, the calibration unit 44 sets marker regions 50 (50A, 50B) in the area where markers 51 (51A, 51B) can be placed on the first calibration image. The calibration unit 44 performs grayscale processing or binarization processing on the first calibration image to generate a second calibration image (for example, a luminance image or a binarized image). At this time, the calibration unit 44 may generate the second calibration image by performing image processing such as grayscale processing or binarization processing based on the luminance distribution in the marker regions 50 of the first calibration image.

[0035] Figure 3 shows an example of a second calibration image of the marker portion according to Embodiment 1, where the background is bright and the marker 51 is illuminated from an oblique angle. Figure 3(a) shows an example of a second calibration image when the switching unit 12 does not switch to the calibration image processing unit 42, and Figure 3(b) shows an example of a second calibration image when the switching unit 12 switches to the calibration image processing unit 42. In other words, Figure 3(a) shows an example of generating a second calibration image from a display image, and Figure 3(b) shows an example of generating a second calibration image from a first calibration image.

[0036] When the background 61 (see Figure 2) is bright and the marker 51 is illuminated from an oblique angle, if the switching unit 12 does not switch to the calibration image processing unit 42, as shown in Figure 3(a), a portion of the marker 51 (51A, 51B) is missing in the second calibration image, and the black and white pattern of the marker 51 is distorted. In contrast, when the switching unit 12 switches to the calibration image processing unit 42, as shown in Figure 3(b), a portion of the marker 51 (51A, 51B) is almost completely intact in the second calibration image, and the black and white pattern of the marker 51 is clearly recognizable. Therefore, the detection accuracy or performance of the marker 51 in the calibration unit 44 is improved.

[0037] Figure 4 shows an example of a second calibration image of the marker portion according to Embodiment 1, where the background is dark and the marker 51 is illuminated from an oblique angle. Figure 4(a) shows an example of a second calibration image when the switching unit 12 does not switch to the calibration image processing unit 42, and Figure 4(b) shows an example of a second calibration image when the switching unit 12 switches to the calibration image processing unit 42. In other words, Figure 4(a) shows an example of generating a second calibration image from a display image, and Figure 4(b) shows an example of generating a second calibration image from a first calibration image.

[0038] When the background 61 (see Figure 2) is dark and the marker 51 is illuminated from an oblique angle, if the switching unit 12 does not switch to the calibration image processing unit 42, as shown in Figure 4(a), the black and white pattern of the marker 51 (51A, 51B) is obscured in the second calibration image. However, when the switching unit 12 switches to the calibration image processing unit 42, as shown in Figure 4(b), the black and white pattern of the marker can be clearly recognized in the second calibration image. Therefore, the detection accuracy or performance of the marker 51 in the calibration unit 44 is improved.

[0039] Figure 5 shows an example of a second calibration image of the marker portion according to Embodiment 1, where the background is colored and the marker 51 is illuminated from an oblique angle. Figure 5(a) shows an example of a second calibration image when the switching unit 12 does not switch to the calibration image processing unit 42, and Figure 5(b) shows an example of a second calibration image when the switching unit 12 switches to the calibration image processing unit 42. In other words, Figure 5(a) shows an example of generating a second calibration image from a display image, and Figure 5(b) shows an example of generating a second calibration image from a first calibration image.

[0040] When the background 62 (see Figure 2) is colored and the marker 51 is illuminated from an oblique angle, if the switching unit 12 does not switch to the calibration image processing unit 42, as shown in Figure 5(a), the black and white pattern of the marker 51 (51A, 51B) is obscured in the second calibration image. However, when the switching unit 12 switches to the calibration image processing unit 42, as shown in Figure 5(b), the black and white pattern of the marker 51 (51A, 51B) is clearly recognizable in the second calibration image. Therefore, the detection accuracy or performance of the marker 51 in the calibration unit 44 is improved.

[0041] The calibration unit 44 detects two markers 51 (51A, 51B) from the second calibration image. However, the number of markers 51 detected is not limited to two; there may be any number. For example, the calibration unit 44 detects predetermined black and white patterned areas from the second calibration image as markers 51A and 51B. The calibration unit 44 then detects the deviation of the current position and orientation of the imaging device 10 from a predetermined position and orientation based on the deviation of the positions and shapes of the two detected markers 51A and 51B from a predetermined position and orientation. The calibration unit 44 then generates calibration information to correct this deviation and sets it in the output unit 22. The output unit 22 corrects the display image based on the set calibration information and displays it on the display device 30. As a result, the display image with the deviation corrected is displayed on the display device 30.

[0042] <Configuration Example 2> Figure 6 is a block diagram showing an example configuration of the image processing system 2 according to Embodiment 1. Configuration Example 2 shows an example in which the imaging device 10 performs calibration. In Configuration Example 2, components having functions common to Configuration Example 1 are denoted by the same reference numerals, and their descriptions are omitted.

[0043] The control device 20 has a calibration determination unit 43.

[0044] The calibration determination unit 43 transmits a display instruction or calibration instruction to the switching unit 12 of the imaging device 10, similar to the configuration example 1.

[0045] The imaging device 10 includes an imaging unit 11, a switching unit 12, a display image processing unit 41, a calibration image processing unit 42, a calibration unit 44, and an output unit 14. As described above, the switching unit 12, the display image processing unit 41, and the calibration image processing unit 42 are a single image processing unit, and the switching of the switching unit 12 may be by switching parameters related to image processing.

[0046] The calibration image processing unit 42 performs calibration image processing to generate a first calibration image, similar to the configuration example 1, and provides it to the calibration execution unit 44.

[0047] The calibration unit 44 generates a second calibration image (e.g., a luminance image or a binarized image) from the first calibration image, detects markers 51 from the second calibration image, and generates calibration information based on the detected markers 51, similar to the configuration example 1. The calibration unit 44 sets the generated calibration information in the output unit 14.

[0048] The output unit 14 corrects the misalignment of the display image provided by the display image processing unit 41 using the set calibration information and outputs it to the display device 30.

[0049] As a result, the imaging device 10 can display a display image on the display device 30 that has been corrected for misalignment and is highly visible to the user.

[0050] <Configuration Example 3> Figure 7 is a block diagram showing a configuration example 3 of the image processing system according to Embodiment 1. Configuration example 3 shows an example in which the control device 20 performs calibration image processing. In configuration example 3, components having functions common to configuration example 1 are denoted by the same reference numerals, and their descriptions are omitted.

[0051] The imaging device comprises an imaging unit 11, an image processing unit 15, and a transmission unit 13.

[0052] The image processing unit 15 performs predetermined image processing on the RAW image provided by the imaging unit 11 to generate a captured image. These predetermined image processing steps include, for example, automatic gain adjustment, exposure adjustment, white balance adjustment, image format conversion (e.g., JPEG conversion), and edge enhancement. However, automatic gain adjustment, exposure adjustment, white balance adjustment, image format conversion, and edge enhancement may not be performed by the image processing unit 15 but by the calibration image processing unit 42.

[0053] The transmission unit 13 transmits the captured image provided by the image processing unit 15 to the control device 20.

[0054] The control device 20 includes a receiving unit 21, a calibration determination unit 43, a display image processing unit 41, a calibration image processing unit 42, a calibration execution unit 44, and an output unit 22. As described above, the calibration determination unit 43, the display image processing unit 41, and the calibration image processing unit 42 are a single image processing unit, and switching the calibration determination unit 43 may be done by switching parameters related to image processing.

[0055] The receiving unit 21 receives the captured image transmitted from the imaging device 10 and provides it to the calibration determination unit 43.

[0056] If the calibration determination unit 43 determines that calibration should be performed, it sends an instruction regarding image processing to the image processing unit 15 of the imaging device 10. For example, if the image processing unit 15 performs automatic gain adjustment (for example, adjustment to brighten the image) by default, the calibration determination unit 43 sends an instruction to turn off automatic gain adjustment to the image processing unit 15. When the image processing unit 15 receives the instruction to turn off automatic gain adjustment, it generates the captured image without performing automatic gain adjustment on the RAW image. Note that the instruction to turn off automatic gain adjustment is just one example; for example, it could be an instruction to turn off exposure adjustment, an instruction to turn off white balance adjustment, an instruction to turn off automatic adjustment, etc.

[0057] Furthermore, if the calibration determination unit 43 determines that calibration should be performed, it provides the captured image received by the receiving unit 21 to the calibration image processing unit 42. If the calibration determination unit 43 determines that calibration should not be performed, it provides the captured image received by the receiving unit 21 to the display image processing unit 41.

[0058] The display image processing unit 41 performs display image processing on the captured image in the same manner as in Configuration Example 1 to generate a display image.

[0059] The calibration image processing unit 42 performs calibration image processing on the captured image in the same manner as in Configuration Example 1 to generate a first calibration image. Note that the automatic gain adjustment for this captured image is turned OFF by instruction from the calibration determination unit 43.

[0060] The calibration unit 44 generates a second calibration image (e.g., a luminance image or a binarized image) from the first calibration image, detects markers 51 from the second calibration image, and generates calibration information based on the detected markers 51, similar to the configuration example 1.

[0061] If a first calibration image and a second calibration image are generated from a captured image that has been adjusted to be relatively bright by automatic gain adjustment, as shown in Figures 3(a), 4(a), and 5(a), a portion of the marker 51 may be missing or the black and white pattern of the marker 51 may be distorted. In contrast, in this configuration example 3, a first calibration image and a second calibration image are generated from a captured image with automatic gain adjustment turned OFF, so as shown in Figures 3(b), 4(b), and 5(b), a portion of the marker 51 is almost completely intact, and the black and white pattern of the marker 51 can be clearly recognized.

[0062] The output unit 22 corrects the misalignment of the display image provided by the display image processing unit 41 using the set calibration information and outputs it to the display device 30.

[0063] As a result, the control device 20 can display a display image on the display device 30 that has been corrected for misalignment and is highly visible to the user.

[0064] <Configuration Example 4> Figure 8 is a block diagram showing Configuration Example 4 of the image processing system 2 according to Embodiment 1. Configuration Example 4 shows an example in which the control device 20 receives images from the imaging device 10 in two systems and generates a display image, a first calibration image, and a second calibration image in parallel processing. In Configuration Example 4, components having functions common to Configuration Example 1 are denoted by the same reference numerals and their descriptions are omitted.

[0065] The imaging device 10 includes an imaging unit 11, a first image processing unit 16, a first transmission unit 17, a second image processing unit 18, and a second transmission unit 19.

[0066] The first image processing unit 16 performs predetermined image processing on the RAW image provided by the imaging unit 11, similar to the image processing unit 15 in Configuration Example 3, to generate a captured image.

[0067] The first transmission unit 17 transmits the captured image generated by the first image processing unit 16 to the control device 20.

[0068] The second image processing unit 18 performs adjustments such as exposure adjustment or white balance adjustment on the RAW image provided by the imaging unit 11 to generate a YUV image (or RGB image). The second image processing unit 18 provides the RAW image and the YUV image to the second transmission unit 19. Alternatively, the second image processing unit 18 may provide the RAW image provided by the imaging unit 11 directly to the second transmission unit 19.

[0069] The second transmission unit 19 transmits the RAW image and / or YUV image provided by the second image processing unit 18 to the control device 20. In this embodiment, the first transmission unit 17 and the second transmission unit 19 are separated for the sake of clarity, but the first transmission unit 17 and the second transmission unit 19 may be a single transmission unit.

[0070] The control device 20 includes a first receiving unit 23, a second receiving unit 24, a display image processing unit 41, a calibration image processing unit 42, a calibration execution unit 44, and an output unit 22.

[0071] The first receiving unit 23 receives the captured image transmitted from the first transmitting unit 17 of the imaging device 10. The first receiving unit 23 provides the received captured image to the display image processing unit 41.

[0072] The second receiving unit 24 receives the RAW image and / or YUV image transmitted from the second transmitting unit 19 of the imaging device 10. The second receiving unit 24 provides the received RAW image and / or YUV image to the calibration image processing unit 42. In this embodiment, the first receiving unit 23 and the second receiving unit 24 are separated for the sake of clarity, but the first receiving unit 23 and the second receiving unit 24 may be a single receiving unit.

[0073] The display image processing unit 41 performs display image processing on the captured image provided by the first receiving unit 23 in the same manner as in Configuration Example 1, and generates a display image. Alternatively, the display image processing unit 41 may provide the captured image provided by the first receiving unit 23 directly to the output unit 22 as a display image.

[0074] The calibration image processing unit 42 performs calibration image processing on the RAW image and / or YUV image provided by the second receiving unit 24 in the same manner as in Configuration Example 1, and generates a first calibration image. The calibration image processing unit 42 provides the generated first calibration image to the calibration execution unit 44.

[0075] The calibration unit 44, similar to Configuration Example 1, generates a second calibration image (e.g., a luminance image or a binarized image) from the first calibration image provided by the calibration image processing unit 42, detects markers 51 from the second calibration image, and generates calibration information based on the detected markers 51. The calibration unit 44 sets the generated calibration information in the output unit 22.

[0076] The output unit 22 corrects the misalignment of the display image provided by the display image processing unit 41 using the set calibration information and displays it on the display device 30.

[0077] As a result, the imaging device 10 can display a display image on the display device 30 that has been corrected for misalignment and is highly visible to the user.

[0078] <Configuration Example 5> Figure 9 is a block diagram showing Configuration Example 5 of the image processing system 2 according to Embodiment 1. Configuration Example 5 shows an example in which calibration information generated by the calibration unit 44 of the control device 20 is also used by a control device 60 separate from the control device 20. In Configuration Example 5, components having functions common to Configuration Example 4 are given the same reference numerals and their descriptions are omitted.

[0079] The imaging unit 11, first image processing unit 16, first transmission unit 17, and second image processing unit 18 of the imaging device 10 perform the processing described in Configuration Example 4.

[0080] The second transmission unit 19 of the imaging device 10 transmits the RAW image and / or YUV image provided by the second image processing unit 18 to the control device 60.

[0081] The control device 20, comprising a first receiving unit 23, a display image processing unit 41, a calibration image processing unit 42, a calibration execution unit 44, and an output unit 22, performs the processing described in Configuration Example 4.

[0082] The output unit 22 of the control device 20 transmits the calibration information set by the calibration execution unit 44 to the control device 60.

[0083] The control device 60 may perform various processes after correcting the RAW image and / or YUV image received from the second transmission unit 19 using calibration information received from the output unit 22. In other words, the control device 60 may perform various processes after correcting the RAW image and / or YUV image received from the second transmission unit 19 using calibration information received from the control device 20. For example, the control device 60 may perform person, vehicle, and object recognition processing for autonomous driving using the corrected RAW image and / or YUV image. This allows the calibration information generated by the control device 20 to be used by other control devices 60.

[0084] Note that control unit 20 and control unit 60 may be different ECUs. Alternatively, control unit 20 and control unit 60 may be different signal processing circuits provided in a single ECU. Alternatively, control unit 20 and control unit 60 may be implemented as two functions operating on a single ECU.

[0085] <Hardware Configuration> Figure 10 is a block diagram showing an example of the hardware configuration of the imaging device 10 and control device 20 according to Embodiment 1.

[0086] The imaging device 10 and the control device 20 are connected via a predetermined communication network (e.g., Ethernet®, Wi-Fi®, or Bluetooth®) or an in-vehicle network (e.g., CAN).

[0087] The imaging device 10 includes a processor 101, a memory 102, a communication interface 103, an output interface 104, and an image sensor 105.

[0088] The processor 101 reads programs and data from the memory 102 and processes them to realize the functions of the imaging device 10 described above. In the case of Configuration Example 1 shown in Figure 1, the processor 101 realizes the functions of the switching unit 12, the display image processing unit 41, and the calibration image processing unit 42. In the case of Configuration Example 2 shown in Figure 6, the processor 101 realizes the functions of the switching unit 12, the display image processing unit 41, the calibration image processing unit 42, and the calibration execution unit 44. In the case of Configuration Example 3 shown in Figure 7, the processor 101 realizes the function of the image processing unit 15. In the case of Configuration Example 4 shown in Figure 8 and Configuration Example 5 shown in Figure 9, the processor 101 realizes the functions of the first image processing unit 16 and the second image processing unit 18.

[0089] Memory 102 is composed of a volatile storage medium and / or a non-volatile storage medium and stores programs and data, etc. Memory 102 may include Read-Only Memory (ROM) and Random Access Memory (RAM). Memory 102 may also include flash memory or a Solid State Drive (SSD).

[0090] The communication interface 103 controls the transmission and reception of data with the control device 20 via a communication network or in-vehicle network. In the case of Configuration Example 1 shown in Figure 1 and Configuration Example 3 shown in Figure 7, the communication interface 103 implements the functions of the transmitter 13. In the case of Configuration Example 4 shown in Figure 8 and Configuration Example 5 shown in Figure 9, the communication interface 103 implements the functions of the first transmitter 17 and the second transmitter 19. In addition, in Configuration Example 1 shown in Figure 1, Configuration Example 2 shown in Figure 6, and Configuration Example 3 shown in Figure 7, the communication interface 103 receives display instructions or calibration instructions, etc., transmitted from the control device 20.

[0091] The output I / F 104 is connected to the display device 30. In the configuration example 2 shown in Figure 6, the output I / F 104 implements the functions of the output unit 14. Note that the imaging device 10 does not necessarily have to have an output I / F 104.

[0092] As described above, the image sensor 105 is, for example, a CMOS sensor or a CCD sensor, and generates a RAW image. In the case of Configuration Example 1 shown in Figure 1, Configuration Example 2 shown in Figure 6, Configuration Example 3 shown in Figure 7, Configuration Example 4 shown in Figure 8, and Configuration Example 5 shown in Figure 9, the image sensor 105 realizes the functions of the imaging unit 11.

[0093] The control device 20 includes a processor 201, a memory 202, a communication interface 203, an input interface 204, and an output interface 105.

[0094] The processor 201 reads programs and data from the memory 202 and processes them to realize the functions of the control device 20 described above. In the case of Configuration Example 1 shown in Figure 1, the processor 201 realizes the functions of the calibration determination unit 43 and the calibration execution unit 44. In the case of Configuration Example 2 shown in Figure 6, the processor 201 realizes the functions of the calibration determination unit 43. In the case of Configuration Example 3 shown in Figure 7, the processor 201 realizes the functions of the calibration determination unit 43, the display image processing unit 41, the calibration image processing unit 42, and the calibration execution unit 44. In the case of Configuration Example 4 shown in Figure 8 and Configuration Example 5 shown in Figure 9, the processor 201 realizes the functions of the display image processing unit 41, the calibration image processing unit 42, and the calibration execution unit 44.

[0095] Memory 202 is composed of volatile and / or non-volatile storage media and stores programs and data, etc. Memory 202 may include ROM and RAM. Memory 202 may also include flash memory or a Solid State Drive (SSD).

[0096] The communication interface 203 controls the transmission and reception of data with the imaging device 10 via a communication network or in-vehicle network. In the case of Configuration Example 1 shown in Figure 1 and Configuration Example 3 shown in Figure 7, the communication interface 203 implements the functions of the receiving unit 21. In the case of Configuration Example 4 shown in Figure 8, the communication interface 203 implements the functions of the first receiving unit 23 and the second receiving unit 24. In the case of Configuration Example 5 shown in Figure 9, the communication interface 203 implements the functions of the first receiving unit 23. In addition, in Configuration Example 1 shown in Figure 1, Configuration Example 2 shown in Figure 6, and Configuration Example 3 shown in Figure 7, the communication interface 203 transmits display instructions or calibration instructions, etc., to the imaging device 10.

[0097] The input interface 204 is an interface for receiving input from the user. Examples of input interfaces 204 include touch panels, switches, buttons, microphones, keyboards, and mice. For example, the user can instruct the calibration determination unit 43 via the input interface 204 whether or not to perform calibration.

[0098] The output I / F 205 is connected to the display device 30. In the case of Configuration Example 1 shown in Figure 1, Configuration Example 3 shown in Figure 7, and Configuration Example 4 shown in Figure 8, the output I / F 104 realizes the function of the output unit 22.

[0099] (Summary of Embodiment 1) Based on the description of Embodiment 1 above, the following technology is disclosed.

[0100] <Technology 1> The image processing method according to Embodiment 1 determines, based on a predetermined instruction, whether or not to use an image captured by a predetermined imaging device for predetermined machine vision processing. If it is determined that the image should be used for machine vision processing, it performs machine vision image processing on the captured image to generate a machine vision image and provides the machine vision image to the machine vision processing. If it is determined that the image should not be used for machine vision processing, it performs display image processing on the captured image to generate a display image and outputs it to a predetermined display device. As a result, if the captured image is not used for machine vision processing, it is output to a display device as a display image; if it is used for machine vision processing, it is adjusted to a machine vision image and provided to the machine vision processing unit. Therefore, the accuracy or performance of machine vision processing is improved.

[0101] <Technology 2> In the image processing method described in Technical 1, the image processing for machine vision includes at least one of contrast enhancement processing and noise reduction processing. By generating machine vision images in this way, the accuracy or performance of machine vision processing is improved.

[0102] <Technology 3> In the image processing method described in Technology 1 or 2, if it is determined that the image will not be used for machine vision processing, an automatic gain adjustment process is performed on the captured image; if it is determined that the image will be used for machine vision processing, the automatic gain adjustment process is not performed on the captured image. By generating machine vision images without performing automatic gain adjustment processing in this way, the accuracy or performance of machine vision processing is improved.

[0103] <Technology 4> In the image processing method described in any one of the technologies 1 to 3, the machine vision processing is performed by generating a luminance image or a binarized image from the provided machine vision image, detecting predetermined markers from the luminance image or the binarized image, and generating calibration information to correct the misalignment of the display image with respect to the position and orientation of the imaging device based on the detected markers. This allows for the generation of calibration information that can accurately and effectively correct misalignments in display images as part of machine vision processing.

[0104] <Technology 5> In the image processing method described in Technical 4, a marker region is set in the area of ​​the machine vision image where the marker may exist, and image processing is performed based on the brightness distribution within the marker region. This allows for the generation of calibration information that can accurately and effectively correct misalignments in display images as part of machine vision processing.

[0105] <Technology 6> The image processing system (2) comprises a shooting device (10) and a control device (20). The control device determines, based on a predetermined instruction, whether or not to use the captured image taken by the shooting device for a predetermined machine vision process. If it determines that the image should be used for the machine vision process, it transmits a first instruction to the shooting device. If it determines that the image should not be used for the machine vision process, it transmits a second instruction to the shooting device. When the shooting device receives the first instruction, it performs machine vision image processing on the captured image to generate a machine vision image and transmits the machine vision image to the control device. When the shooting device receives the second instruction, it performs display image processing on the captured image to generate a display image and transmits the display image to the control device. When the control device determines that the image should be used for the machine vision process, it performs the machine vision process based on the machine vision image received from the shooting device. If it determines that the image should not be used for the machine vision process, it outputs the display image received from the shooting device to a predetermined display device. As a result, if the captured image is not used for machine vision processing, it is output to a display device as a display image; if it is used for machine vision processing, it is adjusted to a machine vision image and provided to the machine vision processing unit. Therefore, the accuracy or performance of machine vision processing is improved.

[0106] <Technology 7> In the image processing system described in Technical 6, the image processing for machine vision includes at least one of contrast enhancement processing and noise reduction processing. Image processing system. By generating machine vision images in this way, the accuracy or performance of machine vision processing is improved.

[0107] <Technology 8> In the image processing system described in Technical 6, if the imaging device determines that it will not be used for machine vision processing, it performs automatic gain adjustment processing on the captured image; if it determines that it will be used for machine vision processing, it does not perform the automatic gain adjustment processing on the captured image. By generating machine vision images without performing automatic gain adjustment processing in this way, the accuracy or performance of machine vision processing is improved.

[0108] <Technology 9> In the image processing system described in any one of the technical items 6 to 8, the control device generates a luminance image or a binarized image from the machine vision image provided by the imaging device as machine vision processing, detects predetermined markers from the binarized image or the luminance image, and generates calibration information to correct the misalignment of the display image with respect to the position and orientation of the imaging device based on the detected markers. This allows for the generation of calibration information that can accurately correct misalignments in display images as part of machine vision processing.

[0109] <Technology 10> In the image processing system described in Technical 9, the control device sets a marker region in the area of ​​the machine vision image where the marker may exist, and performs image processing based on the brightness distribution within the marker region. This allows for the generation of calibration information that can accurately correct misalignments in display images as part of machine vision processing.

[0110] While embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the embodiments described above may be combined in any way without departing from the spirit of the invention. [Industrial applicability]

[0111] The technology disclosed herein is useful for various machine vision processing applications. [Explanation of Symbols]

[0112] 2 Image Processing System 10. Imaging device 11 Imaging Unit 12 Switching section 13 Transmitter 14 Output section 15 Image Processing Unit 16. First Image Processing Unit 17. First Transmitter 18. Second Image Processing Unit 19. Second Transmitter 20 Control device 21 Receiving unit 22 Output section 23 First Receiving Unit 24 Second Receiving Unit 30 Display device 41 Display Image Processing Unit 42 Image processing unit for calibration 43 Calibration judgment section 44 Calibration Department 50 Marker Areas 51 Marker 60 Control device

Claims

1. An image processing method, Based on predetermined instructions, it is determined whether or not to use the captured image taken by a predetermined imaging device for predetermined machine vision processing. If it is determined that the image will be used for the machine vision processing, the captured image is subjected to machine vision image processing to generate a machine vision image, and the machine vision image is provided to the machine vision processing. If it is determined that the image will not be used for the aforementioned machine vision processing, the captured image is subjected to image processing for display purposes to generate a display image, which is then output to a predetermined display device. Image processing methods.

2. The image processing method according to claim 1, The image processing for machine vision includes at least one of contrast enhancement processing and noise reduction processing. Image processing methods.

3. The image processing method according to claim 1, If it is determined that the image will not be used for the machine vision processing, an automatic gain adjustment process is performed on the captured image. If it is determined that the image will be used for the machine vision processing, the automatic gain adjustment process will not be performed on the captured image. Image processing methods.

4. An image processing method according to any one of claims 1 to 3, A brightness image or a binarized image is generated from the provided machine vision image. A predetermined marker is detected from the brightness image or the binarized image. Based on the detected marker, the machine vision process generates calibration information to correct the misalignment of the display image relative to the position and orientation of the imaging device. Image processing methods.

5. The image processing method according to claim 4, A marker region is set in the area where the marker may exist in the machine vision image. Image processing is performed based on the luminance distribution within the marker region. Image processing methods.

6. An image processing system comprising a shooting device and a control device, The control device is Based on predetermined instructions, it is determined whether or not to use the captured image taken by the imaging device for predetermined machine vision processing. If it is determined that the machine vision processing will be used, a first instruction is transmitted to the imaging device. If it is determined that the image will not be used for the machine vision processing, a second instruction is sent to the imaging device. The aforementioned imaging device is When the first instruction is received, the captured image is subjected to image processing for machine vision to generate a machine vision image, and the machine vision image is transmitted to the control device. When the second instruction is received, image processing for display is performed on the captured image to generate a display image, and the display image is transmitted to the control device. The control device is If it is determined that the image will be used for the machine vision processing, the machine vision processing will be performed based on the machine vision image received from the imaging device. If it is determined that the image will not be used for the machine vision processing, the display image received from the imaging device is output to a predetermined display device. Image processing system.

7. The image processing system according to claim 6, The image processing for machine vision includes at least one of contrast enhancement processing and noise reduction processing. Image processing system.

8. The image processing system according to claim 6, The aforementioned imaging device is If it is determined that the image will not be used for the machine vision processing, an automatic gain adjustment process is performed on the captured image. If it is determined that the image will be used for the machine vision processing, the automatic gain adjustment process will not be performed on the captured image. Image processing system.

9. An image processing system according to any one of claims 6 to 8, The control device, as the machine vision processing, A brightness image or a binarized image is generated from the machine vision image provided by the aforementioned imaging device. A predetermined marker is detected from the binarized image or the brightness image. Based on the detected marker, calibration information is generated to correct the misalignment of the display image relative to the position and orientation of the imaging device. Image processing system.

10. The image processing system according to claim 9, The control device is A marker region is set in the area where the marker may exist in the machine vision image. Image processing is performed based on the brightness distribution within the marker region. Image processing system.

Images