Image processing system and image processing method

Abstract

To adequately prevent a dead angle from arising, without complicating an optical device.SOLUTION: An image processing system 1 comprises: an optical system 3 for object person detection, which includes a stereo camera for imaging the face of an object person SP, to acquire a face image required for detecting the pupils of the object person SP; an optical system 5 for object detection which includes a camera for imaging an observation object SB to be observed of the object person SP, to acquire an object image being the image of the observation object SB; a display device 7 for displaying a processed image obtained by processing the object image, on a display surface 7a; and a computer 9 for executing image processing to generate a processed image, on the basis of the face image and the object image. The computer 9 identifies relative positions of the pupils relative to the display device 7 on the basis of the face image, and generates a processed image on the basis of the relative positions, so that the image of the observation object SB of a range blocked by the display device 7 out of the object image is displayed on the display surface 7a.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing system and an image processing method for processing images.

Background Art

[0002] In recent years, in applications such as automobiles, technologies for preventing blind spots caused by pillars and the like are becoming widespread. For example, Patent Document 1 below discloses a technique for pseudo-transparentizing a pillar of an automobile by an optical method using a prism. Further, Non-Patent Document 1 below discloses photographing an image of the background of a pillar that is in a blind spot using an external camera and projecting the background video onto a pillar cover unit by a projector.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique described in Patent Document 1, the configuration of the optical device becomes complicated. Further, in the technique described in Non-Patent Document 1, when the head of a target person such as a driver moves, a deviation occurs between the background on the projected image and the background when there is no blind spot, and the occurrence of blind spots cannot be sufficiently prevented.

[0006] The present invention has been made in view of the above problems, and an object thereof is to provide an image processing system and an image processing method capable of sufficiently preventing the occurrence of blind spots without complicating the optical device.

Means for Solving the Problems

[0007] To solve the above problems, an image processing system according to an aspect of the present invention includes a subject detection camera that images the face of a subject, and a subject detection optical system for acquiring a face image necessary for detecting the eyes of the subject, and includes an object detection camera that images an observation object of the subject to be observed, and an object detection optical system for acquiring an object image that is an image of the observation object, a display device that displays a processed image obtained by processing the object image on a display surface, and a calculation unit that executes image processing for generating the processed image based on the face image and the object image. The calculation unit specifies the relative position of the eyes with respect to the display device based on the face image, and generates the processed image so that an image of the observation object in a range blocked by the display device in the object image is displayed on the display surface based on the position.

[0008] Alternatively, an image processing method according to another aspect of the present invention is an image processing method executed by an image processing system, including a step of imaging the face of a subject to acquire a face image necessary for detecting the eyes of the subject, a step of imaging an observation object of the subject to be observed to acquire an object image that is an image of the observation object, a step of displaying a processed image obtained by processing the object image on a display surface, and a step of executing image processing for generating the processed image based on the face image and the object image. In the step of executing the image processing, the relative position of the eyes with respect to the display surface is specified based on the face image, and the processed image is generated so that an image of the observation object in a range blocked by the display surface in the object image is displayed on the display surface based on the position.

[0009] In the image processing system or image processing method of the above-described form, based on the face image acquired by the optical system for subject detection, the relative position of the eyes of the subject with respect to the display device is specified, and based on that position, an image of the range blocked by the display device in the object image is displayed on the display surface. As a result, for example, by arranging the display device in front of the shielding object, the occurrence of a blind spot due to the shielding object can be sufficiently prevented. In addition, since it is only necessary to arrange the display device on the shielding object, the configuration of the optical device is not complicated.

[0010] Here, the optical system for subject detection acquires a face image capable of specifying the three-dimensional position of the eyes, the optical system for object detection acquires an object image capable of specifying the three-dimensional position of the observation object, and the calculation unit, based on the three-dimensional position of the eyes and the three-dimensional position of the observation object, may define an image of the range blocked by the display device in the object image. In this case, an image of the range of the observation object blocked by the display device in the object image can be accurately defined. As a result, the occurrence of a blind spot due to the shielding object can be more sufficiently prevented.

[0011] Further, the calculation unit may generate a processed image by repeating an assignment process of assigning pixels of the object image to pixels of the display surface on the line connecting the three-dimensional position corresponding to the pixel and the three-dimensional position of the eyes. In such a configuration, the definition of the image of the range of the observation object blocked by the display device in the object image can be efficiently performed. As a result, the occurrence of a blind spot due to the shielding object can be prevented by efficient processing.

[0012] Further, in the assignment process for a plurality of pixels of the object image, when assigned to the same pixel on the display surface, the calculation unit may determine the pixel value of the same pixel in consideration of the pixel values of the plurality of pixels. In this case, a processed image can be generated with the number of pixels corresponding to the number of pixels of the display device. As a result, the observation object can be displayed on the display surface with good reproducibility.

[0013] Also, when there are pixels on the display surface to which no pixels of the object image are assigned in the assignment process, the calculation unit may determine the pixel value of such pixels based on the pixel values of surrounding pixels. By doing so, in this case, the processed image can be generated with the number of pixels corresponding to the number of pixels of the display device. As a result, the observation object can be displayed on the display surface with good reproducibility.

[0014] Also, the optical system for subject detection acquires a face image capable of specifying the positions of the left and right eyes of the subject, the display device is configured to be able to display a processed left-eye image and a processed right-eye image obtained by processing the object image on the display surface, and the calculation unit is based on the relative position of the left eye part specified based on the face image. A left-eye processed image is generated so that an image of the observation object in the range blocked by the display device in the object image is displayed on the display surface, and based on the relative position of the right eye part specified based on the face image, an image of the observation object in the range blocked by the display device in the object image is displayed on the display surface. It may be to generate a right-eye processed image. According to such a configuration, an image in the range blocked by the display device as viewed from the left eye of the subject in the object image is displayed on the display surface as a processed left-eye image, and an image in the range blocked by the display device as viewed from the right eye of the subject in the object image is displayed on the display surface as a processed right-eye image. Thereby, for example, by arranging the display device in front of the shielding object, it is possible to sufficiently prevent the occurrence of a blind spot due to the shielding object when the subject views from both eyes.

[0015] In addition, the optical system for subject detection is arranged to be covered by the display device on the back surface opposite to the display surface of the display device. The display device is configured to transmit the light incident from the display surface to the back surface. The calculation unit controls the display device to repeat a first period in which a processed image is displayed and a second period in which the processed image is not displayed, and controls the optical system for subject detection to acquire a face image in the second period. This may be the case. In this case, the optical system for subject detection can be hidden on the back side of the display device, and even in that case, the optical system for subject detection can periodically acquire a face image. Thereby, in addition to the occurrence of dead angles due to the shielding object, the occurrence of dead angles due to the optical system for subject detection can also be prevented.

[0016] Further, the display device may be arranged on a pillar or a sun visor of an automobile.

[0017] The image processing system of the embodiment includes [1] "a subject detection camera for imaging the face of a subject, an optical system for subject detection for acquiring a face image necessary for detecting the eyes of the subject, an object detection camera for imaging an observation object of an observation target of the subject, an optical system for object detection for acquiring an object image that is an image of the observation object, a display device for displaying a processed image obtained by processing the object image on a display surface, and a calculation unit that executes image processing for generating the processed image based on the face image and the object image", The calculation unit is identifies the relative position of the eyes with respect to the display device based on the face image, and generates the processed image based on the position so that an image of the observation object in the range blocked by the display device in the object image is displayed on the display surface. "an image processing system".

[0018] The image processing system of the embodiment is [2] "the optical system for subject detection acquires the face image capable of specifying the three-dimensional position of the eyes, The optical system for object detection acquires the object image capable of specifying the three-dimensional position of the observation object, Based on the three-dimensional position of the eye part and the three-dimensional position of the observation object, the calculation unit defines an image of a range blocked by the display device in the object image. It may be the "image processing system" described in [1] above.

[0019] The image processing system of the embodiment is [3] "The calculation unit repeats the assignment process of assigning the pixels of the object image to the pixels of the display surface on the line connecting the three-dimensional position corresponding to the pixel and the three-dimensional position of the eye part, thereby generating the processed image. It may be the "image processing system" described in [2] above.

[0020] The image processing system of the embodiment is [4] "In the assignment process for a plurality of pixels of the object image, when the same pixel of the display surface is assigned, the pixel value of the same pixel is determined in consideration of the pixel values of the plurality of pixels. It may be the "image processing system" described in [3] above.

[0021] The image processing system of the embodiment is [5] "When a pixel of the display surface to which no pixel of the object image is assigned occurs in the assignment process, the pixel value of the pixel is determined based on the pixel values of surrounding pixels. It may be the "image processing system" described in [3] or [4] above.

[0022] The image processing system of the embodiment is [6] "The optical system for subject detection acquires the face image capable of specifying the positions of the left and right eye parts of the subject. The display device is configured to be able to display the processed left-eye image and the processed right-eye image obtained by processing the object image on the display surface. Based on the relative position of the left eye part identified based on the face image, the calculation unit generates the processed image for the left eye such that the image of the observation object in the range blocked by the display device in the object image is displayed on the display surface. Based on the relative position of the right eye part identified based on the face image, the calculation unit generates the processed image for the right eye such that the image of the observation object in the range blocked by the display device in the object image is displayed on the display surface. It may also be the "image processing system according to any one of [1] to [5] above".

[0023] The image processing system of the embodiment is [7] "The optical system for detecting the subject is arranged to be covered by the display device on the back surface on the opposite side of the display surface of the display device. The display device is configured to transmit the light incident from the display surface to the back surface. The calculation unit controls the display device to repeat a first period in which the processed image is displayed and a second period in which the processed image is not displayed, and controls the optical system for detecting the subject to acquire the face image in the second period. It may also be the "image processing system according to any one of [1] to [6] above".

[0024] The image processing system of the embodiment is [8] "The display device is arranged on a pillar or a sun visor of an automobile, and may be the image processing system according to any one of [1] to [7] above".

Effect of the Invention

[0025] According to the present invention, it is possible to sufficiently prevent the occurrence of blind spots without complicating the optical device.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0027] Hereinafter, a preferred embodiment of the image processing system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0028] First, referring to FIG. 1, the overall configuration of the image processing system 1 according to the embodiment will be described. FIG. 1 is a side view of the image processing system 1. The image processing system 1 is a processing system that processes and displays an image in order to prevent the occurrence of a blind spot when a person observes an observation object through an obstacle that blocks the field of view. As the obstacle, in FIG. 1, those that block the line of sight of the subject with respect to the observation object such as a wall or a pillar are exemplified, but in addition to that, a pillar on the driver's seat side of an automobile, a pillar on the passenger seat side of an automobile, a pillar on the rear seat side of an automobile, a sun visor of an automobile, or an object arranged outside the automobile may also be used. This image processing system 1 is the subject S P、 and the subject S P which is the observation object, and is the observation object S located in the background of the obstacle MAB An apparatus group that acquires a face image and an object image by imaging, and performs image processing on the face image and the object image. The image processing system 1 includes an optical system 3 for subject detection, an optical system 5 for object detection, a display device 7, and a computer (calculation unit) 9. Hereinafter, each component of the image processing system 1 will be described.

[0029] The optical system 3 for subject detection is arranged so as to be fixed in position with respect to the display device 7 at the lower part of the shielding object MA, and faces the subject S located with the display device 7 sandwiched between it and the shielding object MA. P It includes two cameras (stereo camera, camera for subject detection) whose optical axes are adjusted to face the face of the subject S. Each camera has been pre-calibrated after the optical system 3 for subject detection is fixed to the display device 7, and images the face of the subject S P and acquires and outputs a face image. In the present embodiment, the two cameras are NTSC-type CMOS (Complementary Metal Oxide Semiconductor) video cameras, which are one of the interlace scanning methods. Near-infrared light sources (for example, two types of light sources with emission wavelength bands of 850 nm and 940 nm) having the configuration described in Japanese Patent No. 4500992 are attached to each camera to detect feature points such as the pupil, corneal reflex, or nostrils of the subject S P . The optical system 3 for subject detection lights the near-infrared light source in response to a command from the computer 9 and images the face of the subject S P and outputs the face image data to the computer 9. These face images are used in the computer 9 to specify the three-dimensional position of the eye part (pupil) of the subject S P .

[0030] The optical system 5 for object detection includes a camera (camera for object detection) that is fixed near the opposite surface of the shielding object MA with respect to the display device 7 and whose optical axis is adjusted to face the direction away from the opposite surface (the right side in FIG. 1). This camera has been pre-calibrated after the optical system 5 for object detection is attached to the display device 7, and the subject SP An observation target, the subject S P An observation target object S located in the background of the shielding object MA as seen from B the subject S of the shielding object MA P is imaged from the opposite side of the surface facing the subject S of the shielding object MA, and a target object image is acquired and output. In the present embodiment, this camera is an RGBD camera, which is a video camera capable of combining color data (RGB data) and depth information for each pixel. For example, as the camera, ZED2 manufactured by STEREOLABS is used. By providing such a camera, the optical system 5 for target detection can obtain a subject image capable of specifying the three-dimensional position of the observation target object S B The optical system 5 for target detection can image the observation target object S according to a command from the computer 9 and output the data of the target object image to the computer 9. These target object images are used in the computer 9 to specify the three-dimensional position of the observation target object S B B

[0031] The display device 7 is a device that displays a processed image processed based on the target object image by the computer 9. This display device 7 is fixed so as to be in contact with the surface on the subject S side of the shielding object MA and has a display surface 7a facing the subject S side (the left side in FIG. 1). Further, the display device 7 is arranged so as to cover and hide the entire shielding object MA and the entire or a part (at least the camera body of the optical system 3 for target detection) of the optical system 3 for target detection on the surface 7b on the opposite side of the display surface 7a P P

[0032] ​​​​Here, the display device 7 can adopt the configuration of the display of the terminal disclosed in Japanese Patent Application Laid-Open No. 2018-124457. That is, the display device 7 has a rectangular or sheet shape and is, for example, an organic EL (electro-luminescence) display. When an image is not displayed on the display surface 7a (when the built-in light-emitting element does not emit light), it has light transmissibility that allows light incident from the display surface 7a to pass through to the surface (back surface) 7b facing the display surface 7a. Further, the display device 7 has a sheet-like filter 7c attached to a range (which may be the entire surface 7b) facing the openings of the two cameras of the optical system 3 for detecting a person on the surface 7b. The filter 7c is a member that cuts the visible light component of the light transmitted through the surface 7b from the display surface 7a and allows the near-infrared component to pass through. With such a configuration, the two cameras constituting the optical system 3 for detecting a person can image the face of the person S P through the display device 7 and output a face image. Note that a cover member (for example, a dark curtain) that covers the entire surface 7b together with the optical system 3 for detecting a person may be provided.

[0033] The computer 9 is a data processing device that executes control of the optical system 3 for detecting a person and the optical system 5 for detecting an object, image processing on the face image and the object image acquired by the optical system 3 for detecting a person and the optical system 5 for detecting an object, and display of the processed image generated by the image processing on the display device 7. When the image processing system 1 is provided in a vehicle or the like, the computer 9 may be used in combination with a computer for various controls of the vehicle. The computer 9 may be constructed by a stationary or portable personal computer (PC), may be constructed by a workstation, or may be constructed by other types of computers. Alternatively, the computer 9 may be constructed by combining a plurality of arbitrary types of computers. When a plurality of computers are used, these computers can be connected via a communication network such as the Internet or an intranet.

[0034] Note that the display device 7 may be provided with a mechanism that can change its position or angle integrally with the shielding object MA. This mechanism may be a mechanical mechanism or a mechanism including a drive unit. In that case, the computer 9 can detect the position and angle of the display device 7 by using signals from the mechanism, or position sensors, angle sensors, etc. attached to the mechanism.

[0035] FIG. 2 is a block diagram showing a general hardware configuration of the computer 9. The computer 9 includes a CPU (processor) 101 that executes an operating system, application programs, etc., a main memory unit 102 composed of a ROM and a RAM, an auxiliary storage unit 103 composed of a hard disk, a flash memory, etc., a communication control unit 104 composed of a network card or a wireless communication module, an input device 105 such as a keyboard and a mouse, and an output device 106 such as a display and a printer.

[0036] Each functional element of the computer 9 described later is realized by causing the CPU 101 or the main memory unit 102 to load a predetermined software, operating the communication control unit 104, the input device 105, the output device 106, the display device 7, etc. under the control of the CPU 101, and reading and writing data in the main memory unit 102 or the auxiliary storage unit 103. Data and databases necessary for processing are stored in the main memory unit 102 or the auxiliary storage unit 103.

[0037] As shown in FIG. 3, the computer 9 includes, as functional components, an imaging control unit 21, an image acquisition unit 23, a position calculation unit 25, a camera calibration unit 27, an image processing unit 29, and an image output unit 31. Hereinafter, the functions of each component of the computer 9 will be described.

[0038] When the image processing by the computer 9 starts, the imaging control unit 21 controls the imaging by the subject detection optical system 3 and the object detection optical system 5. At this time, as described in Japanese Patent No. 4500992, the imaging control unit 21 controls the lighting timing of the near-infrared light sources of two types of emission wavelengths attached to the stereo camera according to the imaging timing of the stereo camera of the subject detection optical system 3. By the control of the imaging control unit 21 as described above, in each camera constituting the stereo camera, as a face image of one frame obtained 30 times per second, an image of an odd field composed of odd-numbered horizontal pixel lines and an image of an even field composed of even-numbered horizontal pixel lines are obtained, and the image of the odd field and the image of the even field are alternately photographed at intervals of 1 / 60 second as a face image (bright pupil image) in which the pupil is relatively brightly imaged and a face image (dark pupil image) in which the pupil is relatively darkly imaged. However, the frame rate of shooting by the stereo camera may be set as appropriate. In addition, the imaging control unit 21 controls the imaging timing of the stereo camera of the subject detection optical system 3 as described above so as to fall within a second period (details will be described later) in which the display of the image on the display device 7 is stopped. Thereby, a face image is acquired by the stereo camera of the subject detection optical system 3 in the second period.

[0039] In addition, the imaging control unit 21 controls the imaging of the object detection optical system 5 simultaneously with the control of the imaging of the subject detection optical system 3. Here, although the frame rate of shooting by the camera of the object detection optical system 5 can be set as appropriate, it is preferably set to be equal to the frame rate of shooting of the bright pupil image and the dark pupil image obtained by the subject detection optical system 3.

[0040] The image acquisition unit 23 acquires face images and object images in real time from the stereo camera of the subject detection optical system 3 and the camera of the object detection optical system 5 via a wired or wireless data transfer interface. Then, the image acquisition unit 23 delivers the acquired face images and object images to the position calculation unit 25 each time.

[0041] The position calculation unit 25 calculates the three-dimensional position of the observation object S for each pixel of the object image based on the object image obtained by the camera of the object detection optical system 5. That is, the position calculation unit 25 refers to the depth information for each pixel of the object image based on the result of pre-camera calibration, and for each pixel of the observation object S B calculates the three-dimensional position as the three-dimensional coordinates in the camera coordinate system C of the object detection optical system 5. The position calculation unit 25 continuously executes the calculation of the three-dimensional position of the observation object S for each pixel for each frame of the object image. B In addition, the position calculation unit 25 performs the detection process of the three-dimensional position of the pupil of the target person S in the same manner as the method described in Japanese Patent No. 4500992 for each face image obtained by the stereo camera of the target person detection optical system 3. At this time, the position calculation unit 25 executes the detection process of the three-dimensional position of the pupil for each frame of the face image continuously obtained by the target person detection optical system 3. BC Specifically, the position calculation unit 25 performs stereo matching based on the result of pre-camera calibration and the position of the pupil detected on the face image, and thus calculates the three-dimensional position of the pupil of the target person S as the three-dimensional coordinates in the camera coordinate system C of the target person detection optical system 3. At this time, the position calculation unit 25 can calculate the three-dimensional coordinates of the pupil position by obtaining the vectors from the pinhole position to the pupil based on the face images of the two cameras constituting the stereo camera and obtaining the intersection point of these two vectors. In addition, the position calculation unit 25 can also detect the nearest point of the two vectors as the pupil position using the method described in International Publication WO2015 / 190204. In this case, the pupil position can be obtained even when there is no intersection point of the two vectors due to errors. B

[0042] P

[0043] P PC

[0044] ​​​​​​Before the display device 7 displays the processed image, the camera calibration unit 27 performs camera calibration between the subject detection optical system 3 and the object detection optical system 5 using Zhang's calibration method. That is, at the time of camera calibration, a first calibration plate (such as a chessboard) that defines the world coordinate system C PW (FIG. 1) of the stereo camera of the subject detection optical system 3 is arranged on the front surface of the stereo camera of the subject detection optical system 3, and the distance and inclination with respect to the first calibration plate are known on the front surface of the camera of the object detection optical system 5. A second calibration plate (such as a chessboard) that defines the world coordinate system C BW (FIG. 1) of the object detection optical system 5 is arranged. Then, the camera calibration unit 27 uses the three-dimensional position of the first calibration plate calculated based on the image obtained in the state where the first calibration plate is arranged, and the three-dimensional coordinates in the camera coordinate system C PC to obtain a rotation matrix R PW for converting the three-dimensional coordinates in the world coordinate system C P and a translation vector T P . In addition, the camera calibration unit 27 uses the three-dimensional position of the second calibration plate calculated based on the image obtained in the state where the second calibration plate is arranged, and the three-dimensional coordinates in the camera coordinate system C BC to obtain a rotation matrix R BW for converting the three-dimensional coordinates in the world coordinate system C B and a translation vector T B .

[0045] When the image processing unit 29 displays the processed image on the display device 7, it processes the object image acquired in real time from the camera of the object detection optical system 5 to generate a processed image. That is, based on the three-dimensional position of the pupil of the subject S P calculated based on the frame of the face image by the position calculation unit 25, the relative position of the pupil with respect to the display surface 7a of the display device 7 is specified, and based on that position, from the object image obtained simultaneously with the frame of the face image, the observation object S B in the range blocked by the display surface 7a of the display device 7 is cut out and processed into a processed image.

[0046] Referring to FIG. 4, the generation function of the processed image by the image processing unit 29 will be described in detail. FIG. 4 is a conceptual diagram for explaining the generation function of the processed image by the image processing unit 29.

[0047] First, the image processing unit 29 uses the calibration result by the camera calibration unit 27 to calculate the three-dimensional coordinate vector V of the pupil of the subject S based on the frame of the face image. P PC The vector V is converted into the three-dimensional coordinate vector V in the world coordinate system C by the following formula (1). PW PW V PW = R P -1 · (V PC - T P ) …(1) Similarly, the three-dimensional coordinate vector V of the point cloud on the observation object S composed of all pixels of the object image obtained simultaneously with the frame of the face image is converted into the three-dimensional coordinate vector V in the world coordinate system C by the following formula (2). B BC BW BW V BW = R B -1 · (V BC - T B ) …(2) Furthermore, since the positional and inclination relationships of the first calibration plate and the second calibration plate arranged during camera calibration are known, the image processing unit 29 uses the fact that the rotation matrix and translation vector for converting the coordinates of the world coordinate system C to the coordinates of the world coordinate system C are known, and converts the three-dimensional coordinate vector V in the world coordinate system C into the three-dimensional coordinate vector V' in the world coordinate system C. The image processing unit 29 performs subsequent processing related to the frame of the face image to be processed by performing operations on the world coordinate system C. BW BW BW BW PW BW PW

[0048] Next, the image processing unit 29 observes the object S.B Select one target point from the above point group, and the vector V of the target point BW ’s indicated position and the subject S P The vector V of the three-dimensional coordinates of the pupil of PW Find the three-dimensional coordinates of the intersection point PI between the straight line connecting the indicated position of and the display surface 7a. Since the shape, position, and inclination of the display surface 7a are known in advance, the intersection point PI with the display surface 7a can be calculated using a function representing the three-dimensional position of the display surface 7a. When the display device 7 is provided with a mechanism capable of changing its position or angle, the three-dimensional coordinates of the intersection point PI can be obtained using the position and angle detected from the signal of the mechanism or the signal of the sensor attached to the mechanism.

[0049] Furthermore, the image processing unit 29 determines whether the three-dimensional coordinates of the intersection point PI fall within the range of any pixel on the display surface 7a. When the intersection point PI falls within the range of a specific pixel, the color data (pixel value) on the object image corresponding to the pixel of the target point that is the basis of the intersection point PI is assigned to the pixel value of the specific pixel of the processed image. The image processing unit 29 repeats the assignment process of color data to the pixels of the processed image for all the point groups specified by the object image of one frame, thereby defining the image of the range blocked by the display surface 7a in the object image.

[0050] Here, in the process of assigning color data to the pixels of the processed image, if the intersection point PI with the display surface 7a is not found for a certain target point, the assignment process for that target point is aborted, and the assignment process is executed for the next target point. Also, in the assignment process for a plurality of target points, when repeatedly assigned to the same pixel on the display surface 7a, the image processing unit 29 determines the color data of the same pixel in consideration of the color data corresponding to the plurality of target points. For example, the color data assigned last may be determined as the color data of the pixel, or the color data corresponding to the target point having the intersection point PI closest to the center of the pixel on the display surface 7a may be determined, or the average value of the plurality of assigned color data may be determined as the color data of the pixel. Also, the image processing unit 29 is the observed object S BAfter performing the assignment process for all the above point clouds, if there are pixels on the display surface 7a for which color data cannot be assigned among all the pixels, the color data for such pixels is determined based on the color data determined for the surrounding pixels. For example, as a method for determining color data based on the color data of surrounding pixels, bilinear interpolation is adopted.

[0051] The image output unit 31 controls to repeatedly display on the display surface 7a of the display device 7 a processed image processed from the object image obtained simultaneously with the frame of the face image. At this time, the image output unit 31 controls to repeat a period (first period) in which the processed image is displayed on the display surface 7a and a period (second period) in which the processed image is not displayed on the display surface 7a based on a predetermined cycle. The lengths of this first period and second period are set from several microseconds to several tens of milliseconds according to the frame rate of the display device 7, the frame rate and sensitivity of the stereo camera of the object detection optical system 3, the speed at which the head of the object person S P moves, etc.

[0052] Next, while explaining the image processing procedure in the above-described image processing system 1, the image processing method according to the present embodiment will be described in detail. FIG. 5 is a flowchart showing the image processing procedure by the image processing system 1.

[0053] First, triggered by the startup of the computer 9 or the reception of an instruction input from the outside, etc., image processing is started, and camera calibration is performed by the camera calibration unit 27 of the computer 9 with the first calibration plate and the second calibration plate arranged (step S1). As a result, calibration information such as the rotation matrix and translation vector of the object detection optical system 3 and the object detection optical system 5 is acquired.

[0054] After that, after the first calibration plate and the second calibration plate are removed, the imaging control unit 21 of the computer 9 uses the stereo camera of the object detection optical system 3 to image the object person S P and the camera of the object detection optical system 5 to image the observation object S BImaging is started (step S2). At the same time, the image acquisition unit 23 of the computer 9 acquires face images and object images in consecutive frames from the stereo camera of the subject detection optical system 3 and the camera of the object detection optical system 5 (step S3).

[0055] Next, the position calculation unit 25 of the computer 9 calculates the three-dimensional position of the pupil of the subject S for one frame of the face image P (step S4). Then, based on the object image acquired simultaneously with the frame of the face image, the position calculation unit 25 calculates the three-dimensional position of the observed object S B for each pixel (step S5). Further, the image processing unit 29 of the computer 9 executes color data assignment processing for each point group on the observed object S B so that an image of the range blocked by the display surface 7a is cut out from the object image, and the object image is processed into a processed image (step S6). The processed image is displayed on the display surface 7a in a first period that is periodically repeated by the image output unit 31 of the computer 9 (step S7).

[0056] The processes up to steps S3 to S7 are repeated for each frame of the face image and the object image captured and acquired at the same timing in the subject detection optical system 3 and the object detection optical system 5 until an instruction input or the like is input from the outside and the image processing is completed (step S8).

[0057] According to the image processing system 1 according to the present embodiment described above, based on the face image acquired by the subject detection optical system 3, the subject S PThe relative position of the pupil of the driver with respect to the display device 7 is identified, and based on that position, an image of the range of the object image that is blocked by the display surface 7a of the display device 7 is displayed on the display surface 7a of the display device 7. As a result, for example, by disposing the display device 7 on the surface of the obstruction MA, it is possible to sufficiently prevent the occurrence of blind spots due to the obstruction MA. In addition, since it is sufficient to dispose the display device 7 on the obstruction MA, the configuration of the optical device does not become complicated. FIG. 6 shows an image of an example of mounting the image processing system 1 according to this embodiment on a pillar of an automobile. According to such an example of mounting, it is possible to sufficiently prevent the occurrence of blind spots for the driver due to the pillar.

[0058] Here, the subject detection optical system 3 detects the subject S P The object detection optical system 5 detects the object S of the subject S from a face image that allows the three-dimensional position of the pupil of the subject S to be identified. B The computer 9 acquires an object image capable of identifying the three-dimensional position of the pupil and the observation object S. B Based on the three-dimensional position of the object S, the image of the range of the object image that is blocked by the display device 7 is defined. In this way, the image of the object image of the observation object S that is blocked by the display device 7 is B Specifically, the image of the subject S can be precisely defined. P On the other hand, the observed object S behind the obstruction MA B Therefore, the image displayed on the obstacle MA can be seen in a continuous, connected state, which makes it possible to more sufficiently prevent blind spots caused by the obstacle MA.

[0059] In this embodiment, the computer 9 generates a processed image by repeating an allocation process of allocating color data of a pixel of the object image to a pixel of the display surface 7a on a line connecting the three-dimensional position of the pixel corresponding to the pixel and the three-dimensional position of the pupil. BThe definition of the image within the range can be efficiently performed. As a result, efficient processing can prevent the occurrence of blind spots due to the shielding object MA.

[0060] Also, in the present embodiment, in the allocation process for a plurality of pixels of the target object image, when assigned to the same pixel on the display surface 7a, the computer 9 determines the color data of the same pixel in consideration of the color data of the plurality of pixels. In this case, the processed image can be generated with the number of pixels corresponding to the number of pixels of the display device 7. As a result, the observation target object S B can be displayed on the display surface 7a with good reproducibility.

[0061] Also, in the present embodiment, when a pixel on the display surface 7a to which no pixel of the target object image is assigned occurs in the allocation process, the computer 9 determines the color data of the pixel based on the color data of the surrounding pixels. In this way, in this case, the processed image can be generated with the number of pixels corresponding to the number of pixels of the display device 7. As a result, the observation target object S B can be displayed on the display surface 7a with good reproducibility.

[0062] Also, in the present embodiment, the optical system 3 for detecting a target person is arranged so as to be covered by the display device 7 on the surface 7b opposite to the display surface 7a of the display device 7. The display device 7 is configured to transmit the light incident from the display surface 7a to the surface 7b. The computer 9 controls the display device 7 to repeat a first period in which the processed image is displayed and a second period in which the processed image is not displayed, and controls the optical system 3 for detecting a target person to acquire a face image in the second period. In this case, the optical system 3 for detecting a target person can be hidden on the back side of the display device 7, and even in that case, the face image can be periodically acquired by the optical system 3 for detecting a target person. Thereby, in addition to the occurrence of blind spots due to the shielding object MA, the occurrence of blind spots due to the optical system 3 for detecting a target person can also be prevented.

[0063] Here, an application example of the image processing system 1 according to the present embodiment will be described.

[0064] The first application example is the detection of the driver S when the sun visor is lowered. P In one example, the display device 7 is disposed in front of the sun visor facing the driver (FIG. 7). With this configuration, the forward field image of the part hidden by the sun visor can be displayed on the display device 7. For example, the driver can be sure to see traffic signals, which are conventionally hidden by the sun visor. Even if the forward field image includes an extremely bright image such as the sun, the brightness of the image can be automatically reduced by saturating the brightness on the processed image. On the other hand, the dark part of the forward field image can be displayed on the processed image with the same brightness. In other words, the part hidden by the sun visor can be displayed superimposed on the forward field of view with only the extremely bright part reduced to a moderate brightness without being particularly darkened. In addition, the sensitivity or gamma value of the camera of the object detection optical system 5, or the brightness or contrast of the display device 7 can be freely adjusted to provide the driver with a comfortable forward field of view. Here, in the first application example, the display device 7 may be configured to include an image projection device and a reflector (mirror), and the reflector may be disposed on the front of the sun visor. Even with this configuration, the image of the forward visual field of the part hidden by the sun visor in front of the reflector is projected from the image projection device onto the reflector, so that the forward visual field image of the subject S is projected on the reflector. P In the first application example, the subject detection optical system 3 may be disposed in a location suitable for capturing a face image of a driver, such as a dashboard on the driver's side of a vehicle, or may be disposed on a steering wheel of a vehicle, as disclosed in JP2020-081756A.

[0065] In the first application example described above, there may be a mechanism for changing the position or angle of the display device 7 together with the sun visor. In this case, as in the above embodiment, the color data assignment process is executed using the signals from the mechanism or the position and angle detected from the signals of the sensors attached to the mechanism. Further, near-infrared light sources or markers with retroreflective characteristics are attached to the four corners of the display surface 7a of the display device 7, and the configuration described in Japanese Patent No. 6430813 is adopted, and another optical system is used to estimate the position and angle of the display surface 7a by the computer 9. Further, the position and angle of the display surface 7a can be estimated by the computer 9 without attaching a light source, a marker, or the like to the display device 7.

[0066] The first application example described above may be arranged on something other than the sun visor. For example, a configuration in which the display device 7 is arranged on the mirror portion of a conventional rearview mirror may be adopted. In this configuration, usually, an image of the rear of the vehicle is displayed on the display device 7, a camera for photographing the rear is installed at the center of the display device 7, and the image can be displayed on the display device 7 in accordance with the position of the driver's eyes. When it is desired to display an image of the front of the vehicle, an image of the front is acquired by a camera installed on the back side of the rearview mirror as seen from the driver, and the image acquired by the camera is displayed on the display device 7. The switching between the rear image and the front image can be made possible, for example, by pressing a switch at the driver's hand. Instead of such a switching function, a function may be provided to automatically recognize whether the driver is trying to look at the rear image or the front image from the change in the face image or the line-of-sight direction (particularly, the convergence angle of both eyes), and automatically switch based on the recognition result. Further, a configuration may be adopted in which an image projected by an image projection device installed at a position different from the rearview mirror is displayed to the driver through the mirror on the rearview mirror.

[0067] The first application example described above may be arranged on the side mirror instead of the rearview mirror.

[0068] As a second application example, a configuration is provided in which the display device 7 is attached to a wall of a building as a shielding object MA, and a processed image in which an external landscape acquired by the object detection optical system 5 is shown is displayed on the display surface 7a of the display device 7. According to such a second application example, a pseudo window can be created on the wall of the building, and the subject S P can feel a sense of presence as if actually observing the landscape through a window. When the viewpoint of the subject S P changes significantly, a plurality of object detection optical systems 5 are provided outside the wall, and the computer 9 selects one object image from a plurality of object images acquired by the plurality of object detection optical systems 5 according to the change in the viewpoint of the subject S P and generates a processed image based on the selected object image.

[0069] The above second application example can also be applied to structures surrounded by shielding objects such as submarines, and structures such as amusement facilities. Although it is conceivable to configure a part of the structure with a translucent material such as high-strength glass, such a configuration has drawbacks such as the inside being visible from the outside and being inferior in strength compared to the shielding object.

[0070] As a third application example, a configuration is provided in which the display device 7 is arranged at an arbitrary position inside a vehicle such as an automobile. For example, if a plurality of display devices 7 are arranged side by side in front of the inside of the automobile, the road surface ahead can be displayed on the display device 7. Also, if a plurality of display devices 7 are arranged side by side on the ceiling inside the automobile, the landscape above can be displayed on the display device 7. According to such an application example, the subject S P can not only enjoy the scenery with a refreshing feeling, but also view an image of an actual visual object instead of an image rendered on a visual object configured based on a map with a three-dimensional structure.

[0071] As a fourth application example, as shown in FIG. 8, the vehicle AM has components of the image processing system 1 other than the object detection optical system 5, and a plurality of object detection optical systems 5 capable of photographing the external scenery are fixed to the upper part of a traveling road RO such as a tunnel for the vehicle AM. In this application example, the computer 9 is configured to obtain the relationship between the world coordinate system C PW and the world coordinate system C BW by referring to the position information of the vehicle AM. The computer 9 can select one object image from a plurality of object images acquired by the plurality of object detection optical systems 5 according to the movement of the vehicle AM, generate a processed image based on the selected object image, and project the processed image onto the ceiling or the like. As a result, the subject S P can realistically feel an external video such as empty clouds.

[0072] The present invention is not limited to the above-described embodiments.

[0073] For example, the position calculation unit 25 of the computer 9 calculates the three-dimensional position of the pupil of the subject S P by stereo matching for the face image. As a modification, the position calculation unit 25 may obtain the three-dimensional position of the pupil of the subject S P using a machine learning model such as a neural network. Further, in addition to the pupil of the subject S P , the three-dimensional position of the eyelid, the entire eye, or other parts of the eye of the subject S P may be calculated. As another modification, when the left and right pupils can be detected from the face image, the three-dimensional position of the pupil of the dominant eye may be detected, or the three-dimensional position of the center point of the left and right pupils may be detected.

[0074] Also, the image processing system 1 according to the above embodiment includes only one object detection optical system 5. As a modification, it includes a plurality of object detection optical systems 5, and the computer 9 is configured to perform the following operations on the subject S PAccording to the three-dimensional position of the pupil, one object image may be selected from a plurality of object images acquired by the plurality of object detection optical systems 5 to generate a processed image. According to such a modification, an observation object S that cannot be photographed by a certain object detection optical system 5 B The part can also be photographed by other object detection optical systems 5, and the occurrence of dead angles can be more completely prevented.

[0075] Also, in the above embodiment, the display device 7 may have a curved display surface 7a. For example, as the display device 7, a flexible organic EL display may be adopted, and the display device 7 may be arranged in a curved state. The computer 9 can store in advance a function representing the three-dimensional position of the display surface 7a and execute the assignment process of color data using the function.

[0076] Also, in the above embodiment, the observation object S that is the imaging object of the object detection optical system 5 B is not limited to a planar shape, and may be a curved object having unevenness.

[0077] Also, the generation of the processed image by the computer 9 according to the above embodiment may be performed as follows. FIG. 9 is a conceptual diagram for explaining the generation function of the processed image in the modification. In this modification, the image processing unit 29 of the computer 9 calculates the vector PX of the three-dimensional coordinates of one pixel on the display surface 7a using the function representing the three-dimensional position of the display surface 7a. Then, the image processing unit 29 uses the vector V PW of the three-dimensional coordinates of the pupil, the vector PX, and the observation object S B On the vector V BW ' of the point group on the basis of the subject S P Identify the point closest to the straight line passing through the pupil of and one pixel on the display surface 7a from the point group. Further, the image processing unit 29 assigns the color data on the object image corresponding to the pixel of the identified point to the pixel value of one pixel on the display surface 7a. The image processing unit 29 generates a processed image by repeating the assignment process of color data for all pixels on the display surface 7a.

[0078] Even with such a modification example, it is possible to define an image of a range blocked by the display surface 7a in the object image. However, in this modification example, in the process of assigning color data for one pixel on the display surface 7a, it is necessary to calculate the distances between a plurality of point groups on the observation object S B and a straight line. In contrast, the above-described embodiment performs the assignment process of color data for each target point selected from the point group, and thus is more preferable from the viewpoint of shortening the calculation time.

[0079] Further, the image processing system 1 according to the above-described embodiment may be used in combination for the purpose of detecting a driver's dozing or inattentive driving in a vehicle and for automatic driving control of the vehicle. For example, the image processing system 1 may function to select and process images of a plurality of external cameras installed for these purposes and display them on the display device 7, or based on an image of a camera for photographing a driver inside the vehicle installed for automatic driving or prevention of dozing driving, detect the position of the eyes of the target person S P by image processing.

[0080] Further, in the above-described embodiment, an RGBD camera is used as the object detection optical system 5, but other types of cameras may be used. For example, a TOF (Time-of-Flight) camera, a stereo vision camera, a structured light camera, a phase shift camera, etc. can be used. By using these cameras, an image capable of specifying the three-dimensional position of the observation object S B can be obtained.

[0081] Further, as a modification example, a stereo display device capable of displaying a processed image for the left eye and a processed image for the right eye on the display surface 7a may be adopted for the display device 7. In this modification example, the object detection optical system 3 is for the left pupil (eye part) of the target person S P and the target person S PAn acquired face image enables specification of the three-dimensional positions of both the left pupil (eye part) and the right pupil. Then, based on the three-dimensional position of the left pupil specified based on the face image, the computer 9 processes the object image in the same manner as in the above-described embodiment to generate a processed image for the left eye, and based on the three-dimensional position of the right pupil specified based on the face image of the same frame, the computer 9 processes the object image in the same manner as in the above-described embodiment to generate a processed image for the right eye. Further, the computer 9 controls the display device 7 to display the generated processed image for the left eye and the processed image for the right eye. According to such a modification, processed images can be displayed separately for the left eye and the right eye in consideration of the binocular parallax of the subject S P As a result, the occurrence of blind spots can be effectively prevented. In this modification, as the stereoscopic display device, for example, a 3D stereoscopic display, a naked-eye 3D display, or the like is adopted.

[0082] Note that the image processing system 1 according to the above-described embodiment may perform face region detection, eye position detection, etc. by image processing (such as AI and pattern matching) of the face image.

Explanation of Reference Numerals

[0083] 1... Image processing system, 3... Optical system for subject detection, 5... Optical system for object detection, 7... Display device, 7a... Display surface, 7b... Surface (rear surface), 9... Computer (calculation unit), S B ... Observation object, S P ... Subject, MA... Shielding object.

Claims

1. including a subject detection camera for imaging the face of a subject, a subject detection optical system for acquiring a face image necessary for detecting the eyes of the subject, including an object detection camera for imaging an object to be observed of the subject, an object detection optical system for acquiring an object image that is an image of the object to be observed, a display device for displaying a processed image obtained by processing the object image on a display surface, a calculation unit that executes image processing for generating the processed image based on the face image and the object image, and the calculation unit identifies the relative position of the eyes with respect to the display device based on the face image, and generates the processed image so that an image of the object to be observed in a range blocked by the display device in the object image is displayed on the display surface based on the position. An image processing system.

2. The subject detection optical system acquires the face image capable of specifying the three-dimensional position of the eyes, the object detection optical system acquires the object image capable of specifying the three-dimensional position of the object to be observed, the calculation unit defines an image of a range blocked by the display device in the object image based on the three-dimensional position of the eyes and the three-dimensional position of the object to be observed. The image processing system according to Claim 1.

3. The calculation unit generates the processed image by repeating an assignment process of assigning a pixel of the object image to a pixel of the display surface on a line connecting the three-dimensional position corresponding to the pixel and the three-dimensional position of the eyes. The image processing system according to Claim 2.

4. In the assignment process for a plurality of pixels of the object image, when the calculation unit assigns to the same pixel of the display surface, the calculation unit determines the pixel value of the same pixel in consideration of the pixel values of the plurality of pixels. The image processing system according to Claim 3.

5. When a pixel of the display surface to which no pixel of the object image is assigned occurs in the assignment process, the calculation unit determines the pixel value of the pixel based on the pixel values of surrounding pixels. The image processing system according to Claim 3.

6. The subject detection optical system acquires the face image capable of specifying the positions of the left and right eyes of the subject, the display device is configured to be able to display a processed image for the left eye and a processed image for the right eye obtained by processing the object image on the display surface. Based on the relative position of the left eye part identified based on the face image, the calculation unit generates the processed image for the left eye such that the image of the observation object in the range of the object image blocked by the display device is displayed on the display surface. Based on the relative position of the right eye part identified based on the face image, the calculation unit generates the processed image for the right eye such that the image of the observation object in the range of the object image blocked by the display device is displayed on the display surface. The image processing system according to any one of claims 1 to 5.

7. The optical system for detecting the subject is arranged to be covered by the display device on the back surface opposite to the display surface of the display device. The display device is configured to transmit the light incident from the display surface to the back surface. The calculation unit controls the display device to repeat a first period in which the processed image is displayed and a second period in which the processed image is not displayed, and controls the optical system for detecting the subject to acquire the face image in the second period. The image processing system according to any one of claims 1 to 5.

8. The display device is arranged on an automobile pillar or a sun visor. The image processing system according to any one of claims 1 to 5.

9. An image processing method executed by an image processing system, comprising the steps of: imaging the face of the subject to obtain a face image necessary for detecting the eye part of the subject; imaging the observation object of the subject to obtain an object image which is an image of the observation object; displaying a processed image obtained by processing the object image on a display surface; executing image processing for generating the processed image based on the face image and the object image, wherein in the step of executing the image processing, the relative position of the eye part with respect to the display surface is specified based on the face image, and based on this position, the processed image is generated such that the image of the observation object in the range of the object image blocked by the display surface is displayed on the display surface. Image processing method.

Images