Deviation amount detection device, driver monitoring system, and deviation amount detection method

Abstract

This deviation amount detection device (2) comprises: an evaluation image acquisition unit (11) that acquires, as an evaluation image, an image which includes feature portions inside a vehicle and is captured by an imaging device (1); a search position setting unit (12) that sets a plurality of search positions distributed within a predetermined range from the positions of the feature portions when it is assumed that there is no optical axis deviation of the imaging device in the evaluation image; a search pattern calculation unit (13) that, when it is assumed that the feature portions are disposed at each search position, calculates a search pattern which indicates the shapes of the feature portions in the evaluation image and is based on the optical axis deviation amount corresponding to each search position; an image comparison unit (14) that compares the feature portions in the evaluation image and each of the plurality of search patterns for each search position; and a deviation amount detection unit (15) that detects the optical axis deviation amount of the imaging device on the basis of the comparison result obtained by the image comparison unit.

Description

Deviation amount detection device, driver monitoring system, and deviation amount detection method

[0001] The present disclosure relates to a deviation amount detection device, a driver monitoring system, and a deviation amount detection method.

[0002] A vehicle equipped with a driver monitoring system is equipped with a camera that captures images of occupants. The installation position of the camera may be shifted from the correct installation position due to a work error by a camera installer, etc. The correct installation position of the camera is, for example, the installation position indicated by the design value. If the installation position of the camera is shifted from the correct installation position, the occupants appearing in the image captured by the camera may become unclear, which may result in a deterioration in the monitoring accuracy of the driver monitoring system.

[0003] There is a deviation amount detection device that detects the deviation amount between the installation position of an on-board camera attached to a vehicle and the correct installation position of the on-board camera (see Patent Document 1). In this deviation amount detection device, an electronic control unit extracts planar symmetric feature points from a planar symmetric object captured in an image captured by the on-board camera, and calculates the deviation amount from the planar symmetric feature points.

[0004] JP 2010-181209 A

[0005] The calculation of the amount of deviation using the deviation detection device disclosed in Patent Document 1 is premised on the assumption that the electronic control unit extracts plane-symmetric feature points. If the correct installation position of the onboard camera is a position directly facing the plane-symmetric object, and the onboard camera is installed in this directly facing position, the electronic control unit can extract the plane-symmetric feature points. However, if the distance between the installation position of the onboard camera and the directly facing position is greater than the allowable range, the electronic control unit may not be able to extract the plane-symmetric feature points. Therefore, the deviation detection device has a problem in that the electronic control unit may not be able to calculate the amount of deviation depending on the installation position of the onboard camera.

[0006] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a deviation detection device that can detect the amount of deviation between the installation position of an imaging device and the correct installation position, regardless of the installation position of the imaging device.

[0007] The deviation amount detection device according to the present disclosure is characterized by comprising an evaluation image acquisition unit that acquires an image including a characteristic part inside a vehicle captured by an imaging device as an evaluation image; a search position setting unit that sets a plurality of search positions distributed within a predetermined range from the position of the characteristic part in the evaluation image assuming that there is no optical axis deviation of the imaging device; a search pattern calculation unit that calculates a search pattern that indicates the shape of the characteristic part in the evaluation image assuming that the characteristic part is located at each search position, the search pattern being based on the optical axis deviation amount corresponding to each search position; an image comparison unit that compares the characteristic part in the evaluation image with each of the plurality of search patterns for each search position; and a deviation amount detection unit that detects the optical axis deviation amount of the imaging device based on the comparison result by the image comparison unit.

[0008] According to the present disclosure, with the above-described configuration, it is possible to detect the amount of deviation between the installation position of the imaging device and the correct installation position, regardless of the installation position of the imaging device.

[0009] 1A and 1B are diagrams illustrating exemplary configurations of a deviation amount detection device 2 and a driver monitoring system 3 according to a first embodiment. FIG. 2A is an explanatory diagram illustrating a captured image when the installation position of the imaging device 1 is correct, and FIG. 2B is an explanatory diagram illustrating a captured image when the imaging device 1 is installed with an angle in the vehicle height direction. FIG. 3A is an explanatory diagram illustrating a captured image when the installation position of the imaging device 1 is correct, and FIG. 3B is an explanatory diagram illustrating a captured image when the imaging device 1 is installed with an angle in the vehicle width direction. FIG. 4A is an explanatory diagram illustrating a captured image when the installation position of the imaging device 1 is correct, and FIG. 4B is an explanatory diagram illustrating a captured image when the imaging device 1 is installed with a rotation in the X-Y plane. FIG. 5A is an explanatory diagram illustrating a captured image when the installation position of the imaging device 1 is correct, and FIG. 5B is an explanatory diagram illustrating a captured image when the imaging device 1 is installed with a deviation in the fore-and-aft direction. A flowchart illustrating an example of the operation of the deviation amount detection device 2 according to the first embodiment. FIG. 7A is an explanatory diagram showing an example of a captured image GP, and FIG. 7B is a diagram showing an example of an optical axis deviation amount ΔΦ detected by the sensing unit 31. Ｘ , ΔΦ Ｙ , ΔΦ ｒｏｌｌ FIG. 10 is an explanatory diagram showing an example of a captured image GP'' after being shifted by a certain amount. FIG. 11 is an explanatory diagram showing an example of an evaluation image (edge ​​image) GE. FIG. 12 is an explanatory diagram showing M captured pixels P that show the outlines represented by the evaluation image GE. Ｅ，１ ~P Ｅ，Ｍ10A is an explanatory diagram showing the relationship between a three-dimensional image GS and the position of an outline R of a characteristic portion when there is no optical axis misalignment of the imaging device 1, and FIG. 10B is an explanatory diagram showing the relationship between the three-dimensional image GS and the position of an outline R of a characteristic portion when there is an optical axis misalignment of the imaging device 1. FIG. 12A is an explanatory diagram showing an image of a pixel search method in a prior example, FIG. 12B is an explanatory diagram showing a case that cannot be handled in the prior example, and FIG. 12C is an explanatory diagram showing a state in which search accuracy decreases in the prior example. FIG. 14A and FIG. 14B are diagrams showing an example of a hardware configuration of a deviation amount detection device 2 according to Embodiment 1. FIG. 14B is a diagram showing an example of the configuration of a deviation amount detection device 2b and a driver monitoring system 3 according to Embodiment 2. FIG. 14B is a flowchart showing an example of the operation of the deviation amount detection device 2b according to Embodiment 2. FIG. 14C is an explanatory diagram showing an example of the operation of the deviation amount detection device 2b according to Embodiment 2.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the embodiments will be described in detail with reference to the drawings. First Embodiment Fig. 1 is a diagram showing an example of the configuration of a deviation amount detecting device 2 and a driver monitoring system 3 according to a first embodiment.

[0011] 1, the imaging device 1 is installed, for example, on the center console, steering column, A-pillar, rearview mirror, dashboard, instrument panel, or ceiling of a vehicle. The imaging device 1 is a camera that captures images of characteristic parts of structures inside the vehicle (hereinafter also referred to as "characteristic parts") or the driver of the vehicle. The camera may be, for example, a visible light camera or an infrared camera. The characteristic parts are, for example, the window frame, assist grip, headrest, B-pillar, door frame, shoulder anchor, or sun visor of the vehicle.

[0012] The imaging device 1 outputs a captured image, which is an image of a characteristic portion, to the deviation amount detection device 2, and outputs a captured image, which is an image of the driver of the vehicle, etc., to the driver monitoring system 3. However, the captured image output from the imaging device 1 to the deviation amount detection device 2 is an image that the deviation amount detection device 2 uses when detecting the amount of deviation between the installation position of the imaging device 1 and the correct installation position. Therefore, the captured image is, for example, an image captured by the imaging device 1 when the imaging device 1 is attached to a vehicle. Furthermore, the captured image output from the imaging device 1 to the driver monitoring system 3 is an image that the driver monitoring system 3 uses when detecting inattentive driving, etc., by the driver. Therefore, the captured image is, for example, an image captured by the imaging device 1 while the driver is driving the vehicle.

[0013] In the first embodiment, the captured image of the imaging device 1 is used in the driver monitoring system 3, and therefore the imaging device 1 needs to be installed so that, for example, the face of the driver of the vehicle is captured when the driver is seated in the driver's seat, and a distinctive part of the vehicle is captured when the driver is not seated in the driver's seat. If the captured image of the imaging device 1 is not used in the driver monitoring system 3, the imaging device 1 does not need to be installed so that the face of the driver is captured.

[0014] <Misalignment detection device 2> The misalignment detection device 2 detects the amount of misalignment between the installation position of the imaging device 1 and the correct installation position of the imaging device 1. The correct installation position of the imaging device 1 is, for example, the installation position indicated by the design value. Furthermore, the misalignment detection device 2 uses the amount of positional misalignment to calculate the amount of optical axis misalignment, which is the amount of misalignment between the optical axis of the imaging device 1 and the optical axis of the imaging device 1 when the installation angle of the imaging device 1 is the correct installation angle. The correct optical axis of the imaging device 1 is, for example, the optical axis indicated by the design value. The misalignment detection device 2 outputs the amount of optical axis misalignment to the driver monitoring system 3.

[0015] It should be noted that the term "optical axis misalignment" here not only refers to the concept of misalignment between the optical axis of the imaging device 1 and the optical axis of the imaging device 1 when the installation angle of the imaging device 1 is correct, but may also include the concept of misalignment of the optical axes between multiple lenses installed inside the imaging device 1.

[0016] 1 , the deviation amount detection device 2 includes an evaluation image acquisition unit 11, a search position setting unit 12, a search pattern calculation unit 13, an image comparison unit 14, and a deviation amount detection unit 15. Of these, the search position setting unit 12, the search pattern calculation unit 13, and the image comparison unit 14 constitute a target search unit 20.

[0017] The evaluation image acquisition unit 11 acquires an image (hereinafter also referred to as a "captured image") including characteristic parts inside the vehicle captured by the imaging device 1 as an evaluation image, and outputs the acquired evaluation image to the search position setting unit 12 and the image comparison unit 14.

[0018] For example, the evaluation image acquisition unit 11 generates an edge image that represents the outline of a characteristic part that appears in the captured image, and outputs the generated edge image as an evaluation image to the search position setting unit 12 and the image comparison unit 14. Note that the evaluation image acquisition unit 11 may not generate the edge image itself, but may acquire it from an external source, and output the acquired edge image as an evaluation image to the search position setting unit 12 and the image comparison unit 14.

[0019] The search position setting unit 12 sets multiple search positions distributed within a predetermined range for the evaluation image based on the evaluation image output from the evaluation image acquisition unit 11, target information input from outside the deviation amount detection device 2, and a predetermined expected range of the optical axis deviation amount of the imaging device 1.

[0020] For example, the search position setting unit 12 sets multiple search positions distributed within a predetermined range for the evaluation image based on the positions of characteristic parts in the evaluation image output from the evaluation image acquisition unit 11 assuming that there is no installation misalignment of the imaging device 1, target information, and a predetermined expected range of optical axis misalignment of the imaging device 1.

[0021] Here, the target information includes information about the characteristic part when it is assumed that there is no installation deviation of the imaging device 1. For example, when a three-dimensional coordinate system is defined based on the position of the imaging device 1 when it is assumed that there is no installation deviation of the imaging device 1, the target information is composed of coordinate values ​​of a point group that constitutes the outline of the characteristic part.

[0022] The position of the characteristic part on the image is expressed, for example, by the coordinates of the center of gravity of the outline of the characteristic part. Therefore, assuming that there is no misalignment of the imaging device 1 in the evaluation image, the position of the characteristic part is expressed as the center of gravity of the outline when the point cloud coordinate values ​​of the characteristic part included in the target information are projected onto the evaluation image.

[0023] The search position set by the search position setting unit 12 is represented by the coordinates of the center of gravity of the outline of the characteristic part on the evaluation image. The search position setting unit 12 outputs information indicating the set search position as search position information to the search pattern calculation unit 13 and the image comparison unit 14.

[0024] The search pattern calculation unit 13 calculates a search pattern that indicates the shape of the characteristic part in the evaluation image, assuming that the characteristic part is located at each of the multiple search positions set by the search position setting unit 12.

[0025] For example, the search pattern calculation unit 13 calculates multiple search patterns (so-called correct patterns) corresponding to each of the multiple search positions based on the above target information and the search position information output from the search position setting unit 12.

[0026] The search pattern is composed of, for example, information associating the center of gravity (center of gravity coordinates) of the outline of a characteristic portion of the evaluation image with a set of coordinates of each point constituting the outline. The set of coordinates of each point constituting the outline is composed of, for example, calculated relative coordinate values ​​from the center of gravity of the outline to each point constituting the outline at each search position. The search pattern calculation unit 13 outputs information indicating the calculated search pattern to the image comparison unit 14 as search pattern information.

[0027] The image comparison unit 14 compares the positions and shapes of characteristic portions between the evaluation image and the search pattern for each search position based on the evaluation image output from the evaluation image acquisition unit 11, the search position information output from the search position setting unit 12, and the search pattern information output from the search pattern calculation unit 13. Then, the image comparison unit 14 outputs the comparison result to the deviation amount detection unit 15. For example, the image comparison unit 14 identifies a search pattern among multiple search patterns whose positions and shapes of characteristic portions correspond to the positions and shapes of characteristic portions in the evaluation image. Then, the image comparison unit 14 outputs information indicating the search position corresponding to the identified search pattern to the deviation amount detection unit 15 as the comparison result.

[0028] The deviation amount detection unit 15 detects the amount of optical axis deviation of the imaging device 1 based on the comparison result output from the image comparison unit 14. The deviation amount detection unit 15 outputs information indicating the detected amount of optical axis deviation to the driver monitoring system 3.

[0029] <Driver Monitoring System 3> The driver monitoring system 3 includes a sensing unit 31 and a warning output unit 32, as shown in FIG. 1, for example.

[0030] The sensing unit 31 acquires the captured image from the imaging device 1. The sensing unit 31 also acquires the optical axis deviation amount from the deviation amount detection unit 15 of the deviation amount detection device 2. The sensing unit 31 then monitors the state of the vehicle occupant based on the optical axis deviation amount and the captured image. By monitoring the state of the occupant, the sensing unit 31 detects, for example, whether the driver is looking away from the road or whether the driver is drowsy at the wheel.

[0031] When the sensing unit 31 detects that the driver is looking away from the road or falling asleep at the wheel, the warning output unit 32 outputs warning information indicating a warning to the driver to the vehicle's electronic control unit (not shown) or the like.

[0032] In the first embodiment, an example will be described in which the driver monitoring system 3 detects the driver's inattentive driving, drowsy driving, etc. However, the driver monitoring system 3 is not limited to detecting the driver's inattentive driving, etc., and may also detect the behavior of vehicle occupants other than the driver, for example. In this case, the vehicle occupants are photographed by the imaging device 1.

[0033] FIG. 2A is an explanatory diagram showing a captured image when the imaging device 1 is installed in the correct installation position, and FIG. 2B is an explanatory diagram showing a captured image when the imaging device 1 is installed with an angle in the vehicle height direction. In FIGS. 2A and 2B , the X axis is parallel to the vehicle width direction, the Y axis is parallel to the vehicle height direction, and the Z axis is parallel to the front-to-rear direction of the vehicle. The driver's face is captured in each of the captured images in FIGS. 2A and 2B . If the imaging device 1 is installed in the correct installation position, the driver's face will be captured in the correct position in the captured image. In the example of FIG. 2A , the correct position is approximately the center of the captured image, and the driver's face is captured in approximately the center of the captured image. If the imaging device 1 is installed with an angle in the vehicle height direction, the driver's face will be captured in a position shifted in the vehicle height direction from the correct position in the captured image, as shown in FIG. 2B . If the angle in the vehicle height direction is large, part of the driver's face will not be visible in the captured image, which may make it difficult for the driver monitoring system 3 to detect the driver's distracted driving, etc.

[0034] FIG. 3A is an explanatory diagram showing a captured image when the imaging device 1 is installed in the correct installation position, and FIG. 3B is an explanatory diagram showing a captured image when the imaging device 1 is installed with an angle in the vehicle width direction. In FIGS. 3A and 3B, the X-axis, Y-axis, and Z-axis are the same as those in FIGS. 2A and 2B. The driver's face is captured in each captured image in FIGS. 3A and 3B. If the imaging device 1 is installed in the correct installation position, the driver's face will be captured in the correct position in the captured image. In the example of FIG. 3A, the correct position is approximately the center of the captured image, and the driver's face is captured in the correct center of the captured image. If the imaging device 1 is installed with an angle in the vehicle width direction, as shown in FIG. 3B, the driver's face will be captured in a position shifted in the vehicle width direction from the correct position in the captured image. If the angle in the vehicle width direction is large, part of the driver's face will not be captured in the captured image, which may make it difficult for the driver monitoring system 3 to detect the driver's distracted driving, etc.

[0035] FIG. 4A is an explanatory diagram showing a captured image when the imaging device 1 is installed in the correct installation position, and FIG. 4B is an explanatory diagram showing a captured image when the imaging device 1 is installed in a rotated state within the X-Y plane. In FIGS. 4A and 4B, the X-axis, Y-axis, and Z-axis are the same as those in FIGS. 2A and 2B. The driver's face is captured in each captured image in FIGS. 4A and 4B. If the imaging device 1 is installed in the correct installation position, the driver's face will be captured in the correct position without rotation in the captured image. In the example of FIG. 4A, the correct position is approximately the center of the captured image, and the driver's face is captured in the correct position without rotation in the captured image. The no-rotation state refers to a state in which the driver's face is captured so that the line connecting the driver's left eye and right eye is approximately parallel to the X-axis. If the imaging device 1 is installed in a rotated state within the X-Y plane, the driver's face will be captured in a rotated state in the captured image, as shown in FIG. 4B. If the angle of rotation is large, it becomes difficult to distinguish the driver's face, and it may become difficult for the driver monitoring system 3 to detect the driver's inattentive driving, etc.

[0036] FIG. 5A is an explanatory diagram showing a captured image when the imaging device 1 is installed in the correct installation position, and FIG. 5B is an explanatory diagram showing a captured image when the imaging device 1 is installed in a state where it is misaligned in the fore-and-aft direction. In FIGS. 5A and 5B, the X-axis, Y-axis, and Z-axis are the same as those in FIGS. 2A and 2B. The driver's face is captured in each captured image in FIGS. 5A and 5B. If the imaging device 1 is installed in the correct installation position, the driver's face is captured in the correct position in the captured image. In the example of FIG. 5A, the correct position is approximately the center of the captured image, and the driver's face is captured in the correct center of the captured image. If the imaging device 1 is installed in a position where it is misaligned in the fore-and-aft direction, the size of the driver's face in the captured image is enlarged or reduced from its original size. In the example of FIG. 5B, the imaging device 1 is installed farther from the driver than the correct installation position, so the size of the driver's face is reduced from its original size. If the magnification is too large, part of the driver's face may not be captured in the captured image. Furthermore, if the reduction ratio is too large, it becomes difficult to distinguish the driver's face. Therefore, if the image capturing device 1 is installed at a position offset in the front-to-rear direction, it may be difficult for the driver monitoring system 3 to detect the driver's inattentive driving, etc.

[0037] Next, an example of the operation of the deviation amount detection device 2 and the driver monitoring system 3 shown in Fig. 1 will be described. Fig. 6 is a flowchart showing an example of the operation of the deviation amount detection device 2.

[0038] Prior to the start of operation of the deviation amount detection device 2, the imaging device 1 captures an image of a characteristic part inside the vehicle and outputs a captured image GP, which is an image of the characteristic part, to the deviation amount detection device 2. Fig. 7A is an explanatory diagram showing an example of a captured image GP. The captured image GP shown in Fig. 7A is an image captured by the imaging device 1 installed on the dashboard, for example, and captures characteristic parts such as a window frame a, an assist grip b, a headrest c, a sun visor d, a B-pillar e, a door frame f, and a shoulder anchor g of the vehicle.

[0039] The captured image GP output from the imaging device 1 to the deviation amount detection device 2 is basically an image captured when the driver is not seated in the driver's seat. However, even if the driver is seated in the driver's seat, the captured image GP may be an image captured when the driver is seated in the driver's seat, as long as the driver is seated in a manner that does not block structures. The deviation amount detection device 2 starts operating when it acquires the captured image GP from the imaging device 1.

[0040] First, the evaluation image acquisition unit 11 acquires a captured image GP including a characteristic part inside the vehicle captured by the imaging device 1, and generates an edge image representing the outline of the characteristic part shown in the captured image GP. The outline of the characteristic part is a line indicating the boundary between the characteristic part and the background, etc., represented in three dimensions, as shown in Fig. 8, for example. The evaluation image acquisition unit 11 outputs the generated edge image as an evaluation image to the search position setting unit 12 and the image comparison unit 14 (step ST1).

[0041] 8 is an explanatory diagram showing an example of the evaluation image (edge ​​image) GE. The evaluation image acquisition unit 11 can obtain the evaluation image (edge ​​image) GE by, for example, passing the captured image GP through a Sobel filter or a Prewitt filter.

[0042] The outline of the evaluation image GE is shown in FIG. Ｅ，１ ~P Ｅ，Ｍ M is an integer equal to or greater than 2, and in the example of FIG. 9, M=60. FIG. 9 shows M captured pixels P Ｅ，１ ~P Ｅ，Ｍ 9 is an explanatory diagram of the evaluation image GE, which is a two-dimensional image, and M captured pixels P Ｅ，１ ~P Ｅ，Ｍ Therefore, the horizontal axis of FIG. 9 is the X axis, and the vertical axis is the Y axis, and the Z axis is not shown. In FIG. 9, a pixel having a luminance value of 0 is an image pixel P Ｅ，１ ~P Ｅ，Ｍ and pixels having a luminance value of 1 are pixels that do not represent outlines.

[0043] For simplicity of explanation, the outlines shown in Fig. 9 only show a portion of the window frame, which is one of the multiple characteristic portions shown in the captured image GP. In the example of Fig. 9, the portion of the window frame that is the characteristic portion includes the upper edge portion of the vehicle window frame and a side edge portion of the vehicle window frame. The upper edge portion of the window frame and the side edge portion of the window frame are in contact with each other. However, this is merely an example, and the portion of the window frame that is the characteristic portion may also include only the upper edge portion of the vehicle window frame.

[0044] Next, the search position setting unit 12 sets multiple search positions on the evaluation image GE based on the evaluation image GE output from the evaluation image acquisition unit 11, target information input from outside the deviation amount detection device 2, and a predetermined expected range of optical axis deviation amount of the imaging device (step ST2).

[0045] For example, the search position setting unit 12 sets the search position as follows: First, the search position setting unit 12 sets the center of gravity position of the characteristic part on the evaluation image GE as (xc, yc) assuming that there is no optical axis misalignment of the imaging device 1.

[0046] Then, using (xc, yc) as a reference, the search position setting unit 12 sets four coordinate points defined by (xc-XW1, yc-YW1), (xc-XW1, yc+YW2), (xc+XW2, yc-YW1), and (xc+XW2, yc+YW2) on the evaluation image GE.The search position setting unit 12 then sets, as search positions, a plurality of coordinate points arranged at intervals XI in the x direction (horizontal direction of the image) and at intervals YI in the y direction (vertical direction of the image) within a rectangular area surrounded by these four coordinate points.

[0047] The intervals XI and YI are set to arbitrary values ​​(e.g., XI=1, YI=1) by, for example, an administrator of the deviation amount detection device 2. Furthermore, XW1, XW2, YW1, and YW2 for determining the four coordinate points are set based on the assumed range of the optical axis deviation of the imaging device 1. For simplicity of explanation, the area in which the search position setting unit 12 sets the search position has been described as a rectangular area, but this area is not limited to a rectangular area. For example, a trapezoidal area may be used depending on the assumed range of the optical axis deviation of the imaging device 1.

[0048] For example, as shown in FIG. 10 , the position of the outline (center of gravity) of the characteristic portion on the evaluation image GE corresponds to the amount of optical axis deviation of the imaging device 1. As shown in FIG. 10 , when an image GS (hereinafter simply referred to as a "three-dimensional image") defined by information indicating the three-dimensional coordinates of the characteristic portion captured by the imaging device 1 is projected onto the evaluation image GE, the position of the outline (center of gravity) of the characteristic portion on the evaluation image GE is determined. Note that, for simplicity's sake, the image GS is described as a three-dimensional image. However, there are cases where not all of the three-dimensional coordinates defining the image GS are located on the evaluation image GE. However, even in such cases, the search position setting unit 12 can operate in the same manner as when all of the three-dimensional coordinates are located on the evaluation image GE.

[0049] Fig. 10A shows the relationship between the three-dimensional image GS and the positions of the outlines of the characteristic parts when there is no optical axis misalignment of the imaging device 1. In this case, information indicating the three-dimensional coordinates of the characteristic parts that define the three-dimensional image GS is included in the target information. Fig. 10B shows the relationship between the three-dimensional image GS and the positions of the outlines of the characteristic parts when there is an optical axis misalignment of the imaging device 1.

[0050] 10B, if there is an amount of optical axis deviation of the imaging device 1, a deviation occurs in the three-dimensional coordinates of the characteristic part captured by the imaging device 1 according to the amount of deviation, which in turn causes a deviation in the position of the outline of the characteristic part (position of center of gravity) obtained by projecting the three-dimensional image GS onto the evaluation image GE. In this case, it is possible to roughly predict how much the position of the outline of the characteristic part (position of center of gravity) will deviate depending on the amount of optical axis deviation of the imaging device 1.

[0051] For example, assume that when the optical axis deviation of the imaging device 1 corresponding to the horizontal direction (x direction) on the evaluation image GE is greatest, the center of gravity position of the outline R of the characteristic portion on the evaluation image GE is (xc-XW3,yc) or (xc+XW4,yc) with respect to (xc,yc). In this case, the search position setting unit 12 sets XW1 and XW2 such that xc-XW3>xc-XW1 and xc+XW4<xc+XW2 are satisfied, so that the positions (xc-XW3,yc) and (xc+XW4,yc) are included in the search positions. Note that the specific amount of optical axis deviation when the optical axis deviation of the imaging device 1 is greatest can be obtained in advance from, for example, design information of the imaging device 1.

[0052] Similarly, the search position setting unit 12 sets YW1 and YW2 in the vertical direction (y direction). In this way, the search position setting unit 12 determines a rectangular area surrounded by the four coordinate points described above, and sets, as search positions, a plurality of coordinate points spaced apart by an interval XI in the x direction (horizontal direction of the image) and an interval YI in the y direction (vertical direction of the image) in this rectangular area.

[0053] Note that when calculating the position of the characteristic part on the evaluation image GE corresponding to the amount of optical axis deviation of the imaging device 1, the search position setting unit 12 performs the calculation, for example, as follows: For example, the search position setting unit 12 first calculates a coordinate group T3g' by multiplying a three-dimensional coordinate group defining the three-dimensional image GS when it is assumed that there is no optical axis deviation shown in Fig. 10A, that is, a three-dimensional coordinate group T3g of the outline of the characteristic part included in the target information, by a rotation matrix corresponding to the amount of optical axis deviation.

[0054] Next, the search position setting unit 12 projects the three-dimensional image GS defined by the coordinate set T3g' onto the evaluation image GE, thereby obtaining the outline of the characteristic part on the evaluation image GE corresponding to the amount of optical axis deviation.The search position setting unit 12 then calculates the coordinates of the center of gravity of the outline of the characteristic part on the evaluation image GE, and calculates the obtained coordinate position as the position of the outline of the characteristic part on the evaluation image GE.The search position setting unit 12 repeats the above process while changing the amount of optical axis deviation.In this way, the search position setting unit 12 calculates the position of the outline of the characteristic part on the evaluation image GE corresponding to the amount of optical axis deviation of the imaging device 1.

[0055] The calculated position of the outline of the characteristic portion (the center of gravity of the outline) corresponds to one search position and one approximate value of the amount of optical axis deviation, for example, as follows: In the following, for ease of understanding, the approximate value of the amount of optical axis deviation is represented by U. Search position Center of gravity position of outline Optical axis deviation amount (approximate value) (x1, y1) (x1', y1') U1 (x2, y1) (x2', y1') U2 (x3, y1) (x3', y1') U3 (x4, y1) (x4', y1') U4 (x1, y2) (x1', y2') U5 (x2, y2) (x2', y2') U6 (x3, y2) (x3', y2') U7 (x4, y2) (x4', y2') U8, ... etc.

[0056] The search position setting unit 12 outputs information indicating the search position set as described above to the search pattern calculation unit 13 and the image comparison unit 14 as search position information.

[0057] Next, the search pattern calculation unit 13 calculates search patterns that indicate the shapes of the characteristic portions in the evaluation image GE, assuming that the characteristic portions are located at each of the plurality of search positions (step ST3). For example, the search pattern calculation unit 13 calculates a plurality of search patterns based on the target information and the search position information output from the search position setting unit 12. Then, the search pattern calculation unit 13 outputs information indicating the calculated search patterns to the image comparison unit 14 as search pattern information.

[0058] 11 shows an example of a method for calculating a search pattern by the search pattern calculation unit 13. The search pattern calculation unit 13 calculates a search pattern according to (a) to (d) shown in FIG.

[0059] First, the search pattern calculation unit 13 identifies one search position on the evaluation image GE indicated by the search position information (a). Next, the search pattern calculation unit 13 calculates the amount of optical axis deviation corresponding to the identified search position (b).

[0060] For example, the search pattern calculation unit 13 calculates, as the amount of positional deviation, the difference between the center of gravity of the outline of the characteristic part included in the target information when it is assumed that there is no optical axis deviation of the imaging device 1 and the center of gravity of the outline of the characteristic part corresponding to the coordinate values ​​of the specified search position. Then, based on the calculated amount of positional deviation, the search pattern calculation unit 13 calculates the amount of optical axis deviation corresponding to the specified search position.

[0061] For example, the search pattern calculation unit 13 calculates the positional deviation amount dx from the centroid position (xm, ym, zm) of the outline of the characteristic part when it is assumed that there is no optical axis deviation of the imaging device 1 and the centroid position (xn, yn, zn) of the outline of the characteristic part corresponding to the coordinate values ​​of the specified search position, as shown in the following equations (1) to (3). ｎ－ｍ , dy ｎ－ｍ , dz ｎ－ｍ Calculate dx ｎ－ｍ =xn-xm (1) dy ｎ－ｍ =yn-ym (2) dz ｎ－ｍ =zn-zm (3)

[0062] In the first embodiment, the image including the outline of the characteristic part when it is assumed that there is no optical axis deviation of the imaging device 1 and the image including the outline of the characteristic part corresponding to the coordinate value of the specified search position are both two-dimensional images. ｎ－ｍ Even if both of the above images are three-dimensional images, if zm = zn = constant, then dz ｎ－ｍ =0.

[0063] The search pattern calculation unit 13 calculates the positional deviation amount dx in the X-axis direction as shown in the following equation (4). ｎ－ｍ , the optical axis Φ in the X-axis direction of the imaging device 1 is Ｘ and the optical axis Φ in the X-axis direction of the imaging device 1 when there is no optical axis misalignment in the X-axis direction. ＸＲＥＦ The amount of optical axis deviation ΔΦ Ｘ Calculate.

[0064] In equation (4), FL is a known focal length of the imaging device 1. The focal length is, for example, the distance between an image sensor (not shown) included in the imaging device 1 and a lens (not shown) included in the imaging device 1.

[0065] Furthermore, the search pattern calculation unit 13 calculates the positional deviation amount dy in the Y-axis direction as shown in the following equation (5). ｎ－ｍ , the optical axis Φ in the Y-axis direction of the imaging device 1 is Ｙ and the optical axis Φ in the Y-axis direction of the imaging device 1 when there is no optical axis misalignment in the Y-axis direction. ＹＲＥＦ The amount of optical axis deviation ΔΦ Ｙ Calculate.

[0066] In addition, the search pattern calculation unit 13 determines whether the outlines represented by both images match by rotating either an image containing the outline of the characteristic part when it is assumed that there is no optical axis misalignment of the imaging device 1 within the X-Y plane, or an image containing the outline of the characteristic part corresponding to the coordinate values ​​of the identified search position.

[0067] The search pattern calculation unit 13 rotates one of the two images, and if the outlines of the two images match, specifies the amount of rotation of one of the images when the two images match. Then, the search pattern calculation unit 13 converts the specified amount of rotation into the amount of optical axis deviation ΔΦ in the rotation direction in the X-Y plane. ｒｏｌｌ If the outlines of both images match without rotating them in the XY plane, the optical axis deviation amount ΔΦ ｒｏｌｌ is 0.

[0068] In this way, the search pattern calculation unit 13 calculates the optical axis deviation amount ΔΦ corresponding to the specified search position. Ｘ , ΔΦ Ｙ , ΔΦ ｒｏｌｌ The amount of optical axis deviation calculated here is different from the approximate value U of the amount of optical axis deviation used by the search position setting unit 12 when setting the search position, and indicates an accurate amount of optical axis deviation calculated based on the amount of positional deviation at the specified search position.

[0069] Next, based on the calculated amount of optical axis shift, the search pattern calculation unit 13 rotates the group of three-dimensional coordinates that define the three-dimensional image GS assuming that there is no optical axis shift of the imaging device 1, i.e., the group of three-dimensional coordinates of the outline of the characteristic parts included in the target information (c).

[0070] For example, the search pattern calculation unit 13 calculates a coordinate set T3g' by multiplying the three-dimensional coordinate set T3g of the outline of the characteristic part included in the target information by a rotation matrix calculated from the optical axis deviation amount corresponding to the specified search position.The search pattern calculation unit 13 then projects the three-dimensional image GS defined by the calculated coordinate set T3g' onto the evaluation image GE (d).In this way, the search pattern calculation unit 13 obtains the outline of the characteristic part on the evaluation image GE corresponding to the optical axis deviation amount corresponding to the specified search position.

[0071] Then, the search pattern calculation unit 13 associates the obtained center of gravity (center of gravity position) of the outline with a group of coordinate values ​​of the outline consisting of coordinate values ​​indicating the relative distance from the center of gravity of the outline to each point that makes up the outline, and calculates this as a search pattern corresponding to the above-specified search position.

[0072] Thereafter, the search pattern calculation unit 13 calculates a plurality of search patterns corresponding to each search position by repeatedly performing the above process for the plurality of search positions. For example, if there are 100 search positions, the search pattern calculation unit 13 calculates 100 search patterns.

[0073] The calculated search pattern is shown below. For ease of understanding, the optical axis deviation amount ΔΦ is used below. Ｘ , ΔΦ Ｙ , ΔΦ ｒｏｌｌis represented by V. Search position Center of gravity position of outline line Coordinate value group of outline line Optical axis deviation amount (x1, y1) (x1', y1') Point a11, a12, a13, a14,... V1 (x2, y1) (x2', y1') Point a21, a22, a23, a24,... V2 (x3, y1) (x3', y1') Point a31,a32,a33,a34,... V3 (x4, y1) (x4', y1') Point a41,a42,a43,a44,... V4 (x1, y2) (x1', y2') Point b11,b12,b13,b14,... V5 (x2, y2) (x2', y2') Points b21, b22, b23, b24, ... V6 (x3, y2) (x3', y2') Points b31, b32, b33, b34, ... V7 (x4, y2) (x4', y2') Points b41, b42, b43, b44, ... V8 ... etc.

[0074] Next, the image comparison unit 14 compares the positions and shapes of the characteristic parts between the evaluation image GE and the search pattern for each search position based on the evaluation image GE output from the evaluation image acquisition unit 11, the search position information output from the search position setting unit 12, and the search pattern output from the search pattern calculation unit 13 (step ST4).The image comparison unit 14 then outputs the comparison result to the deviation amount detection unit 15.

[0075] For example, the image comparison unit 14 identifies a search pattern among the multiple search patterns PT whose position and shape correspond to the position and shape of the characteristic part in the evaluation image GE, and outputs information indicating the search position corresponding to the identified search pattern as the comparison result to the deviation amount detection unit 15. Here, the "corresponding search pattern" includes not only a search pattern whose position and shape completely match the position and shape of the characteristic part, but also a search pattern whose position and shape match the position and shape of the characteristic part with the highest degree of match.

[0076] The image comparison unit 14 performs the comparison using a common technique such as pattern matching. Since pattern matching itself is a well-known technique, detailed description thereof will be omitted. However, the pattern matching process can compare the line shapes of two outlines to determine whether the two outlines have the same line shape. The same line shape includes similar line shapes. Furthermore, when two outlines have the same line shape, the pattern matching process can search for pixels among the pixels representing one outline that correspond to each of the pixels representing the other outline.

[0077] Next, the deviation amount detection unit 15 detects the amount of optical axis deviation of the imaging device 1 based on the comparison result output from the image comparison unit 14 (step ST5).

[0078] For example, if the comparison result is information indicating a search position where the matching rate of the position and shape of the characteristic part between the evaluation image GE and the search pattern is the highest, the deviation amount detection unit 15 detects the optical axis deviation amount (the above-mentioned V) corresponding to the search position as the optical axis deviation amount of the imaging device 1. The deviation amount detection unit 15 outputs information indicating the detected optical axis deviation amount to the driver monitoring system 3.

[0079] The sensing unit 31 of the driver monitoring system 3 acquires the captured image GP' from the imaging device 1. The captured image GP' is an image captured while the driver is driving the vehicle. The sensing unit 31 also receives the optical axis deviation amount ΔΦ from the deviation amount detection unit 15. Ｘ , ΔΦ Ｙ , ΔΦ ｒｏｌｌ Get.

[0080] The sensing unit 31 performs occupant sensing to identify the position of the driver's head shown in the captured image GP', for example. When performing occupant sensing, the sensing unit 31 takes into consideration the possibility that the installation position of the imaging device 1 may be shifted from the correct installation position. For this reason, when sensing the position of the driver's head, for example, the sensing unit 31 calculates the optical axis deviation amount ΔΦ from the position of the driver's head in the captured image GP'. Ｘ , ΔΦ Ｙ , Δφ ｒｏｌｌThe position that is shifted by only this amount is sensed.

[0081] Specifically, if the coordinates of the driver's head position in the captured image GP' are (x, y, z), the coordinates (x', y', z') of the position sensed by the sensing unit 31 are expressed as follows:

[0082] FIG. 7B shows the amount of optical axis deviation ΔΦ measured by the sensing unit 31. Ｘ , ΔΦ Ｙ , Δφ ｒｏｌｌ FIG. 7B is an explanatory diagram showing an example of a captured image GP'' after being shifted by a certain distance. For the sake of simplicity, FIG. 7B shows a captured image GP'' in which the driver is not shown.

[0083] The sensing unit 31 monitors the state of the vehicle occupants based on pixel values ​​at coordinates (x', y', z') in the captured image GP'. By monitoring the state of the occupants, the sensing unit 31 detects, for example, whether the driver is looking away from the wheel or whether the driver is falling asleep at the wheel. The process of detecting whether the driver is looking away from the wheel or whether the driver is falling asleep at the wheel is a known technique, and therefore a detailed description thereof will be omitted.

[0084] When the sensing unit 31 detects, for example, that the driver is distracted while driving or that the driver is drowsy while driving, the warning output unit 32 outputs warning information indicating a warning to the driver to the vehicle's electronic control unit or the like.

[0085] Next, a description will be given of the effects of the deviation amount detection device 2 according to embodiment 1. As described above, the deviation amount detection device 2 includes the evaluation image acquisition unit 11, the search position setting unit 12, the search pattern calculation unit 13, the image comparison unit 14, and the deviation amount detection unit 15, and is thereby able to detect the amount of deviation between the installation position of the imaging device 1 and the correct installation position, regardless of the installation position of the imaging device 1.

[0086] Furthermore, by being configured as described above, the deviation amount detection device 2 can calculate the amount of optical axis deviation by taking into account changes in the shape of characteristic parts in the evaluation image GE when optical axis deviation occurs in the imaging device 1. Therefore, the deviation amount detection device 2 can further improve the detection accuracy of the amount of deviation.

[0087] Regarding this point, referring to Figures 12 and 13, for example, a deviation amount detection device disclosed in Japanese Patent No. 7134384 (hereinafter referred to as the "prior example") is known as a technology that can detect the amount of deviation between the installation position of the imaging device 1 and the correct installation position, regardless of the installation position of the imaging device 1, similar to the deviation amount detection device 2 of embodiment 1.

[0088] This prior art includes an edge image generation unit, a pixel search unit, and a deviation amount calculation unit. The edge image generation unit acquires an image of the interior of a vehicle captured by an imaging device and generates an edge image that represents the outline of a structure shown in the image.

[0089] The pixel search unit acquires a reference image representing the outline of the structure when the installation position of the imaging device is correct, searches for reference pixels, which are multiple pixels that represent the outline represented by the reference image, that correspond to each of the imaging pixels, which are multiple pixels that represent the outline represented by the edge image generated by the edge image generation unit, among the reference pixels, which are multiple pixels that represent the outline represented by the reference image, and outputs the pixel position of each imaging pixel and the pixel position of the reference pixel corresponding to each imaging pixel.

[0090] The deviation amount calculation unit calculates a positional deviation amount, which is the amount of deviation between the installation position of the imaging device and the correct installation position, from the pixel position of each imaging pixel output from the pixel search unit and the pixel position of a reference pixel corresponding to each imaging pixel output from the pixel search unit. The deviation amount calculation unit also uses the positional deviation amount to calculate an optical axis deviation amount, which is the amount of deviation between the optical axis of the imaging device and the optical axis of the imaging device when the installation angle of the imaging device is the correct installation angle.

[0091] An image of the pixel search method in this prior example is shown in FIG. 12A. In the prior example, as shown in FIG. 12A, a search is performed by translating a so-called correct pattern (reference image) relative to an edge image (evaluation image). In this case, the shape of the structure reflected in the reference image does not change and always remains the same. In other words, in the prior example, the shape of the outline of the characteristic part in the evaluation image, assuming that there is no optical axis misalignment of the imaging device 1, is used as the search pattern (correct pattern) at all search positions. Specifically, the relative coordinate values ​​in the search pattern (representing the shape of the outline) are the same at all search positions, with only the center of gravity position using different values ​​depending on the search position.

[0092] Here, when the optical axis of the imaging device 1 is misaligned, not only will the structure (feature portion) be displaced in the evaluation image, but the shape of the structure (feature portion) will also change, more specifically, the shape of the outline of the feature portion will change. For example, if the optical axis of the imaging device 1 is rotated 10 degrees in both the yaw direction and the pitch direction with respect to the center of the evaluation image, not only will the feature portion be displaced in the evaluation image, but the shape (appearance) of the outline of the feature portion will also change, as shown in FIG. 12B .

[0093] However, in the prior art, although the positional shift of the characteristic portion is taken into consideration when searching for pixels, the change in the shape (appearance) of the outline of the characteristic portion is not taken into consideration. Therefore, in the prior art, if the shape (appearance) of the characteristic portion changes, the shape of the characteristic portion and the shape of the search pattern will not match, as shown in Fig. 12C , which may result in an increase in calculation error of the amount of optical axis deviation or a decrease in the calculation accuracy of the amount of optical axis deviation.

[0094] In contrast, the deviation amount detection device 2 according to the first embodiment calculates a search pattern (correct pattern) according to the search position (amount of optical axis deviation), as shown in FIG. 13 , and performs a search (image comparison) using this search pattern. For example, FIG. 13A shows an example of a search pattern corresponding to a certain search position A, and FIG. 13B shows an example of a search pattern corresponding to another search position B. In this case, search position A corresponds to the position of the characteristic part when the optical axis is deviated in both the yaw direction and the pitch direction, and search position B corresponds to the position of the characteristic part when the optical axis is rotated 10 degrees in each of the yaw direction and the pitch direction. As a result, the deviation amount detection device 2 according to the first embodiment can perform a search that takes into account not only the positional deviation of the characteristic part but also changes in the shape (appearance) of the outline of the characteristic part, thereby improving the accuracy of calculating the amount of optical axis deviation compared to the prior art.

[0095] Next, an example of the hardware configuration of the deviation amount detection device 2 according to the first embodiment will be described with reference to Fig. 14. The functions of the evaluation image acquisition unit 11, search position setting unit 12, search pattern calculation unit 13, image comparison unit 14, and deviation amount detection unit 15 in the deviation amount detection device 2 are realized by processing circuits. The processing circuit may be dedicated hardware as shown in Fig. 14A, or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in a memory 53 as shown in Fig. 14B.

[0096] When the processing circuit is dedicated hardware, the processing circuit 51 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each of the evaluation image acquisition unit 11, the search position setting unit 12, the search pattern calculation unit 13, the image comparison unit 14, and the deviation amount detection unit 15 may be realized individually by the processing circuit 51, or the functions of each unit may be realized collectively by the processing circuit 51.

[0097] When the processing circuit is a CPU 52, the functions of the evaluation image acquisition unit 11, search position setting unit 12, search pattern calculation unit 13, image comparison unit 14, and deviation amount detection unit 15 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuit realizes the functions of each unit by reading and executing the programs stored in the memory 53. In other words, the deviation amount detection device 2 includes a memory for storing a program that, when executed by the processing circuit, results in the execution of each step shown in FIG. 6 . Furthermore, these programs can also be said to cause a computer to execute the procedures and methods of the evaluation image acquisition unit 11, search position setting unit 12, search pattern calculation unit 13, image comparison unit 14, and deviation amount detection unit 15. Here, examples of the memory 53 include non-volatile or volatile semiconductor memories such as RAM (Radio Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0098] It is also possible to realize some of the functions of the evaluation image acquisition unit 11, the search position setting unit 12, the search pattern calculation unit 13, the image comparison unit 14, and the deviation amount detection unit 15 with dedicated hardware and some with software or firmware. For example, the function of the evaluation image acquisition unit 11 can be realized by a processing circuit as dedicated hardware, and the functions of the search position setting unit 12, the search pattern calculation unit 13, the image comparison unit 14, and the deviation amount detection unit 15 can be realized by the processing circuit reading and executing programs stored in the memory 53.

[0099] Thus, the processing circuitry can implement each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0100] As described above, according to the first embodiment, the deviation amount detection device 2 includes an evaluation image acquisition unit 11 that acquires, as an evaluation image GE, an image including a characteristic part inside the vehicle captured by the imaging device 1, a search position setting unit 12 that sets a plurality of search positions distributed within a predetermined range from the position of the characteristic part in the evaluation image GE assuming that there is no optical axis deviation of the imaging device 1, a search pattern calculation unit 13 that calculates a search pattern that indicates the shape of the characteristic part in the evaluation image GE assuming that the characteristic part is located at each search position, the search pattern being based on the optical axis deviation corresponding to each search position, an image comparison unit 14 that compares the characteristic part in the evaluation image GE with each of the plurality of search patterns for each search position, and a deviation amount detection unit 15 that detects the amount of optical axis deviation of the imaging device 1 based on the comparison result by the image comparison unit 14. As a result, the deviation amount detection device 2 according to the first embodiment can detect the amount of deviation between the installation position of the imaging device 1 and the correct installation position, regardless of the installation position of the imaging device 1. Furthermore, the deviation amount detection device 2 according to embodiment 1 can calculate the amount of optical axis deviation taking into account changes in the shape of characteristic parts on the evaluation image GE when an optical axis deviation occurs in the imaging device 1, thereby improving the detection accuracy of the deviation amount.

[0101] Furthermore, the image comparison unit 14 identifies a search pattern among the multiple search patterns whose position and shape correspond to the position and shape of a characteristic part in the evaluation image GE, and sets information indicating the search position corresponding to the identified search pattern as the comparison result, and the deviation amount detection unit 15 detects the amount of optical axis deviation corresponding to the search position indicated by the comparison result as the amount of optical axis deviation of the imaging device 1. In this way, the deviation amount detection device 2 according to the first embodiment can accurately detect the amount of deviation of the imaging device 1.

[0102] Furthermore, the search pattern calculation unit 13 calculates a search pattern based on information indicating the three-dimensional position of the characteristic part, with reference to the position of the imaging device 1 when it is assumed that there is no optical axis deviation of the imaging device 1, and the amount of optical axis deviation corresponding to each search position. This allows the deviation amount detection device 2 according to the first embodiment to accurately calculate a search pattern corresponding to each search position.

[0103] Furthermore, the search pattern calculation unit 13 calculates a search pattern by multiplying coordinate values ​​indicating the three-dimensional position of the characteristic part, based on the position of the imaging device 1 when it is assumed that there is no optical axis deviation of the imaging device 1, by a rotation matrix obtained based on the amount of optical axis deviation corresponding to each search position, and projecting the coordinate values ​​obtained by this multiplication onto the evaluation image GE. This allows the deviation amount detection device 2 according to the first embodiment to accurately calculate a search pattern corresponding to each search position.

[0104] The search pattern calculation unit 13 also acquires the outline of a characteristic part on the evaluation image GE corresponding to the amount of optical axis deviation corresponding to each search position by projecting the coordinate values ​​obtained by multiplication onto the evaluation image GE, and sets a search pattern that associates the coordinate value indicating the center of gravity of the acquired outline with a group of coordinate values ​​indicating the relative distance from the center of gravity of the outline to each point constituting the outline. This allows the deviation amount detection device 2 according to the first embodiment to define a search pattern corresponding to each search position using a group of coordinate values.

[0105] Furthermore, the search pattern calculation unit 13 calculates, as the amount of positional deviation, the difference between the center of gravity of the outline of the characteristic part in the evaluation image GE when it is assumed that there is no optical axis deviation of the imaging device 1 and the center of gravity of the outline of the characteristic part in the evaluation image GE when it is assumed that the characteristic part is located at each search position, and calculates the amount of optical axis deviation corresponding to each search position based on the calculated amount of positional deviation. This allows the deviation amount detection device 2 according to the first embodiment to accurately calculate the amount of optical axis deviation corresponding to each search position.

[0106] Furthermore, the search position setting unit 12 sets a plurality of search positions so that the search positions include at least the center of gravity of the outline of the characteristic part on the evaluation image GE when the amount of optical axis deviation of the imaging device 1 is maximum within the assumed range of the amount of optical axis deviation of the imaging device 1. This allows the deviation amount detection device 2 according to the first embodiment to appropriately set a plurality of search positions for the evaluation image GE.

[0107] The characteristic part is a window frame, an assist grip, a headrest, a B-pillar, a door frame, a shoulder anchor, or a sun visor of the vehicle. As a result, the deviation amount detection device 2 according to the first embodiment can easily detect the amount of deviation between the installation position of the imaging device 1 and the correct installation position, regardless of the installation position of the imaging device 1.

[0108] The characteristic portion is a part of the window frame of the vehicle, and the part of the window frame includes only the top edge portion of the window frame, or the top edge portion of the window frame and a side edge portion of the window frame that is in contact with the top edge portion of the window frame. This allows the deviation amount detection device 2 according to the first embodiment to easily detect the amount of deviation between the installation position of the imaging device 1 and the correct installation position, regardless of the installation position of the imaging device 1.

[0109] The imaging device 1 is installed on a center console, a steering column, an A-pillar, a rearview mirror, a dashboard, an instrument panel, or a ceiling, so that the deviation amount detection device 2 according to the first embodiment can easily detect the amount of deviation between the installation position of the imaging device 1 and the correct installation position, regardless of the installation position of the imaging device 1.

[0110] Furthermore, according to this embodiment 1, the driver monitoring system 3 includes a sensing unit 31 that monitors the state of the vehicle occupant using the amount of optical axis deviation detected by the deviation amount detection unit 15 of the deviation amount detection device 2 and an image of the inside of the vehicle captured by the imaging device 1. This enables the driver monitoring system 3 according to embodiment 1 to improve the accuracy of monitoring the state of the occupant.

[0111] Embodiment 2. In the first embodiment, a deviation amount detection device has been described that can detect the amount of deviation between the installation position of the imaging device 1 and the correct installation position, regardless of the installation position of the imaging device 1. In the second embodiment, a deviation amount detection device will be described that can reduce the amount of calculation for detecting the amount of deviation compared to the first embodiment.

[0112] FIG. 15 is a diagram showing an example of the configuration of a deviation amount detection device 2b and a driver monitoring system 3 according to the second embodiment. In the deviation amount detection device 2b according to the second embodiment, the target search unit 20 of the deviation amount detection device 2 according to the first embodiment shown in FIG. 1 is changed to a target search unit 20b. In the target search unit 20b, the search position setting unit 12, the search pattern calculation unit 13, and the image comparison unit 14 are each divided into two functional units. In other words, the deviation amount detection device 2b according to the second embodiment performs search (image comparison) in two stages. The other configurations of the deviation amount detection device 2b and the driver monitoring system 3 according to the second embodiment are the same as those of the deviation amount detection device 2 and the driver monitoring system 3 according to the first embodiment shown in FIG. 1, and therefore the same reference numerals are used and description thereof will be omitted.

[0113] The search position setting unit 12 includes a first search position setting unit 12a and a second search position setting unit 12b. The search pattern calculation unit 13 includes a first search pattern calculation unit 13a and a second search pattern calculation unit 13b. The image comparison unit 14 includes a first image comparison unit 14a and a second image comparison unit 14b. Of these, the functional units marked with the symbol a are functional units for performing the first search, and the functional units marked with the symbol b are functional units for performing the second search.

[0114] <Regarding the first search> The first search position setting unit 12a sets a plurality of search positions for the first search based on the evaluation image output from the evaluation image acquisition unit 11 and target information input from outside the deviation amount detection device 2.

[0115] For example, the first search position setting unit 12a sets multiple search positions distributed within a predetermined range for the evaluation image based on the positions of characteristic parts in the evaluation image output from the evaluation image acquisition unit 11, assuming that there is no installation misalignment of the imaging device 1, and on target information.

[0116] At this time, the first search position setting unit 12a, unlike the search position setting unit 12 in Embodiment 1, sets the search position without considering the amount of optical axis deviation of the imaging device 1. Furthermore, the first search position setting unit 12a sets the search position so that the search range in the first search (hereinafter also referred to as the "first range") is wider than the search range in the second search (hereinafter also referred to as the "second range") described later.

[0117] For example, the first search position setting unit 12a sets search positions within a first range, which is a predetermined range centered on the position of a characteristic part in the evaluation image when it is assumed that there is no misalignment of the imaging device 1. In addition, at this time, the first search position setting unit 12a sets the intervals of the search positions set within the first range narrower than the intervals of the search positions set within the second range.

[0118] The first search pattern calculation unit 13 a calculates a first search pattern based on the target information. At this time, unlike the search pattern calculation unit 13 in the first embodiment, the first search pattern calculation unit 13 a calculates the search pattern without taking into account the amount of optical axis deviation of the imaging device 1.

[0119] For example, based on the target information, the first search pattern calculation unit 13a associates the center of gravity (center of gravity position) of the outline of the characteristic part, assuming that there is no installation misalignment of the imaging device 1, with a group of coordinates of the outline consisting of relative coordinate values ​​from the center of gravity of the outline to each point that makes up the outline, and calculates this as a search pattern.

[0120] The first image comparison unit 14a compares the positions and shapes of characteristic parts between the evaluation image and the search pattern for each search position based on the evaluation image output from the evaluation image acquisition unit 11, the search position information set by the first search position setting unit 12a, and the search pattern calculated by the first search pattern calculation unit 13a.The first image comparison unit 14a then outputs the comparison result to the second search position setting unit 12b.

[0121] For example, the first image comparison unit 14 a outputs, as a comparison result, information indicating a search position when the position and shape of a characteristic part in the evaluation image correspond to the position and shape of the first search pattern among a plurality of search positions distributed in the first range to the second search position setting unit 12 b. Note that, as in the prior example described above, the search here (first search) is a search that takes into account only the positional shift of the characteristic part, and does not take into account the change in the shape (appearance) of the outline of the characteristic part due to the shift in the optical axis of the imaging device 1.

[0122] <Regarding the second search> The second search position setting unit 12b sets multiple search positions for the second search based on the evaluation image output from the evaluation image acquisition unit 11 and the comparison result output from the first image comparison unit 14a.

[0123] For example, the second search position setting unit 12b sets a predetermined range from the search position indicated by the comparison result by the first image comparison unit 14a as a second range, and sets multiple search positions distributed in the second range. At this time, the second search position setting unit 12b sets the search positions so that the second range is narrower than the above-mentioned first range and so that the intervals between the search positions are wider than the intervals between the search positions in the above-mentioned first range.

[0124] The second search pattern calculation unit 13 b calculates a plurality of second search patterns based on the target information and the search position information set by the first search position setting unit 12 a. For example, the second search pattern calculation unit 13 b calculates the shape of the outline of the characteristic part, assuming that the characteristic part is located at each search position set by the second search position setting unit 12 b, assuming that there is no installation deviation of the imaging device 1.

[0125] At this time, the second search pattern calculation unit 13b knows the approximate value of the optical axis deviation amount corresponding to each search position set by the second search position setting unit 12b, as described in Embodiment 1, and calculates the positional deviation amount of the characteristic part for each search position from this approximate value of the optical axis deviation amount. Thereafter, similar to the search pattern calculation unit 13 in Embodiment 1, the second search pattern calculation unit 13b calculates the optical axis deviation amount for each search position from the calculated positional deviation amount, and calculates the shape of the outline of the characteristic part if it were arranged at each search position from the calculated optical axis deviation amount, and sets these as the search pattern for each search position.

[0126] The second image comparison unit 14b compares the positions and shapes of characteristic parts between the evaluation image and the search pattern for each search position based on the evaluation image output from the evaluation image acquisition unit 11, the search position information set by the second search position setting unit 12b, and the search pattern calculated by the second search pattern calculation unit 13b.The second image comparison unit 14b then outputs the comparison result to the deviation amount detection unit 15.

[0127] For example, the second image comparison unit 14b identifies a second search pattern among the plurality of second search patterns, the second search pattern having a position and shape of a characteristic portion corresponding to the position and shape of the characteristic portion in the evaluation image. Then, the image comparison unit 14 outputs information indicating the search position corresponding to the identified second search pattern as the comparison result to the deviation amount detection unit 15. Note that, as in the first embodiment described above, the search here (second search) is a search that takes into account not only the positional deviation of the characteristic portion but also the change in the shape (appearance) of the outline of the characteristic portion due to the deviation of the optical axis of the imaging device 1.

[0128] Next, an example of the operation of the misalignment detection device 2b according to the second embodiment will be described with reference to the flowchart shown in Fig. 16. Unlike the misalignment detection device 2 according to the first embodiment, the misalignment detection device 2b according to the second embodiment performs a search in two stages (first search and second search).

[0129] First, the evaluation image acquisition unit 11 acquires an image GP including characteristic parts inside the vehicle captured by the imaging device 1, generates an edge image representing the outline of the characteristic parts shown in the image GP, and outputs the generated edge image as an evaluation image to the first search position setting unit 12a, the second search position setting unit 12b, the first image comparison unit 14a, and the second image comparison unit 14b (step ST11).

[0130] Next, the first search pattern calculation unit 13a calculates a first search pattern based on the target information (step ST12). At this time, unlike the search pattern calculation unit 13 in the first embodiment, the first search pattern calculation unit 13a calculates the search pattern without taking into account the amount of optical axis deviation of the imaging device 1.

[0131] Next, the first search position setting unit 12a sets a plurality of search positions distributed within a predetermined range for the evaluation image GE based on the positions of the characteristic parts in the evaluation image output from the evaluation image acquisition unit 11, assuming that there is no installation deviation of the imaging device 1, and on the target information (step ST13). At this time, unlike the search position setting unit 12 in the first embodiment, the first search position setting unit 12a sets the search positions without taking into account the amount of optical axis deviation of the imaging device 1.

[0132] Next, the first image comparison unit 14a compares the positions and shapes of the characteristic parts between the evaluation image GE output from the evaluation image acquisition unit 11 and the first search pattern based on the search position information set by the first search position setting unit 12a and the first search pattern calculated by the first search pattern calculation unit 13a (step ST14).Then, the first image comparison unit 14a outputs the comparison result to the second search position setting unit 12b.

[0133] Next, the second search position setting unit 12b sets multiple search positions for the second search based on the evaluation image GE output from the evaluation image acquisition unit 11 and the comparison result output from the first image comparison unit 14a (step ST15).

[0134] For example, the second search position setting unit 12b sets a predetermined range from the search position indicated by the comparison result by the first image comparison unit 14a as a second range, and sets multiple search positions distributed in the second range. At this time, the second search position setting unit 12b sets the search positions so that the second range is narrower than the above-mentioned first range and so that the intervals between the search positions are wider than the intervals between the search positions in the above-mentioned first range.

[0135] Next, the second search pattern calculation unit 13b calculates a plurality of second search patterns based on the target information and the search position information set by the first search position setting unit 12a (step ST16). At this time, the second search pattern calculation unit 13b calculates the search pattern taking into account the amount of optical axis deviation for each search position, as described above.

[0136] Next, the second image comparison unit 14b compares the positions and shapes of the characteristic parts between the evaluation image GE output from the evaluation image acquisition unit 11 and the search pattern for each search position based on the search position information set by the second search position setting unit 12b and the search pattern calculated by the second search pattern calculation unit 13b (step ST17).The second image comparison unit 14b then outputs the comparison result to the deviation amount detection unit 15.

[0137] Next, the deviation amount detection unit 15 detects the amount of optical axis deviation of the imaging device 1 based on the comparison result output from the second image comparison unit 14b (step ST18).

[0138] Next, a description will be given of the effects of the deviation amount detection device 2 b according to embodiment 2. As described above, the deviation amount detection device 2 b includes the first search position setting unit 12 a and the second search position setting unit 12 b, the first search pattern calculation unit 13 a and the second search pattern calculation unit 13 b, and the first image comparison unit 14 a and the second image comparison unit 14 b, and thus, like the deviation amount detection device 2 according to embodiment 1, it can detect the amount of deviation between the installation position of the imaging device 1 and the correct installation position, regardless of the installation position of the imaging device 1.

[0139] Furthermore, by having the above-described configuration, the deviation amount detection device 2b can not only improve the detection accuracy of the deviation amount, as with the deviation amount detection device 2 according to embodiment 1, but also reduce the amount of calculation required to detect the deviation amount compared to the deviation amount detection device 2 according to embodiment 1.

[0140] For example, the deviation amount detection device 2 according to the first embodiment may require a large amount of calculation because the search pattern calculation unit 13 calculates a search pattern (relative position coordinates of the outline of the characteristic portion) for each search position set by the search position setting unit 12. In contrast, in the deviation amount detection device 2b according to the second embodiment, as described above, only the second search pattern calculation unit 13b calculates a search pattern for each search position, and the first search pattern calculation unit 13a calculates a search pattern similar to that in the preceding example.

[0141] That is, as shown in FIG. 17 , the misalignment detection device 2b according to the second embodiment first performs a first search (search using one search pattern) similar to that of the prior art over a relatively wide range, and then narrows down the approximate position and range to be searched intensively in the evaluation image. Then, in the second search, the misalignment detection device 2b sets a search position within the narrowed-down range and calculates a search pattern corresponding to the set search position. The misalignment detection device 2b then performs a search using the calculated search pattern. This allows the misalignment detection device 2b to improve the accuracy of misalignment detection, similar to the misalignment detection device 2 according to the first embodiment. Furthermore, because the misalignment detection device 2b calculates a search pattern for each search position only over a relatively limited range, the amount of calculation required to detect the misalignment can be reduced compared to the misalignment detection device 2 according to the first embodiment.

[0142] In the above description, the second search pattern calculation unit 13b calculates a search pattern for each of the multiple search positions set by the second search position setting unit 12b, and the second image comparison unit 14b compares the position and shape of the characteristic part between the evaluation image and the search pattern for each search position. However, the second search pattern calculation unit 13b may calculate a search pattern corresponding to any one of the multiple search positions set by the second search position setting unit 12b, and the second image comparison unit 14b may use this one search pattern to compare the position and shape of the characteristic part between the evaluation image and the search pattern for each search position.

[0143] In other words, the second search range determined by the search position setting unit 12b is narrower than the first search range. Therefore, even if the second search pattern calculation unit 13b calculates a search pattern for each search position, there is a possibility that there will not be much difference between the search patterns. Therefore, the second search pattern calculation unit 13b may calculate a search pattern corresponding to one of the multiple search positions set by the second search position setting unit 12b, and the second image comparison unit 14b may perform the second search using this one search pattern. In this case, the deviation amount detection device 2b can further reduce the amount of calculation required to detect the deviation amount.

[0144] As described above, according to the second embodiment, the deviation amount detection device 2b includes an evaluation image acquisition unit 11 that acquires an image including a characteristic part inside the vehicle captured by the imaging device 1 as an evaluation image GE; a first search position setting unit 12a that sets a plurality of search positions distributed in a predetermined first range from the position of the characteristic part in the evaluation image GE when it is assumed that there is no optical axis deviation of the imaging device 1; a first search pattern calculation unit 13a that calculates a first search pattern that indicates the shape of the characteristic part in the evaluation image GE when it is assumed that there is no optical axis deviation of the imaging device 1; a first image comparison unit 14a that compares the characteristic part in the evaluation image GE with the first search pattern for each search position distributed in the first range; and a first image comparison unit 14a that compares the characteristic part in the evaluation image GE with the first search pattern for each search position distributed in the first range. The deviation amount detection device according to the second embodiment includes a second search position setting unit (12b) that sets a plurality of search positions distributed in a predetermined second range from the position of the characteristic portion when it is assumed that there is no optical axis deviation of the imaging device (1), a second search pattern calculation unit (13b) that calculates a second search pattern that indicates the shape of the characteristic portion in the evaluation image (GE) when it is assumed that the characteristic portion is located at each of the search positions distributed in the second range, the second search pattern being based on the amount of optical axis deviation corresponding to each of the search positions distributed in the second range, a second image comparison unit (14b) that compares the characteristic portion in the evaluation image (GE) with each of the plurality of second search patterns for each of the search positions distributed in the second range, and a deviation amount detection unit (15) that detects the amount of optical axis deviation of the imaging device (1) based on the comparison result by the second image comparison unit (14b). As a result, the deviation amount detection device (2b) according to the second embodiment can reduce the amount of calculation for detecting the amount of deviation compared to the deviation amount detection device (2) according to the first embodiment in addition to the effects of the first embodiment.

[0145] Furthermore, the first image comparison unit 14a sets, as a comparison result, information indicating a search position when the position and shape of a characteristic part in the evaluation image GE corresponds to the position and shape of the first search pattern, among the plurality of search positions distributed in the first range, and the second search position setting unit 12b sets, as a second range, a predetermined range from the search position indicated by the comparison result by the first image comparison unit 14a, and sets a plurality of search positions distributed in the second range. This allows the deviation amount detection device 2b according to the second embodiment to appropriately set the second range.

[0146] Furthermore, the first range is wider than the second range, and the distance between the search positions distributed in the first range is narrower than the distance between the search positions distributed in the second range, thereby enabling the deviation amount detection device 2b according to the second embodiment to improve the calculation efficiency for detecting the deviation amount.

[0147] Furthermore, the second search pattern calculation unit 13b calculates the second search pattern based on information indicating the three-dimensional position of the characteristic part, with reference to the position of the imaging device 1 when it is assumed that there is no optical axis misalignment of the imaging device 1, and the amount of optical axis misalignment corresponding to each search position distributed in the second range. This allows the deviation amount detection device 2b according to the second embodiment to accurately calculate the second search pattern corresponding to each search position distributed in the second range.

[0148] In addition, the present disclosure allows for free combination of the embodiments, modification of any of the components of the embodiments, or omission of any of the components of the embodiments.

[0149] The present disclosure makes it possible to detect the amount of deviation between the installation position of an imaging device and the correct installation position regardless of the installation position of the imaging device, and is suitable for use in a deviation detection device, a driver monitoring system, and a deviation detection method.

[0150] REFERENCE SIGNS LIST 1 Imaging device, 2, 2b Deviation amount detection device, 3 Driver monitoring system, 11 Evaluation image acquisition unit, 12 Search position setting unit, 12a First search position setting unit, 12b Second search position setting unit, 13 Search pattern calculation unit, 13a First search pattern calculation unit, 13b Second search pattern calculation unit, 14 Image comparison unit, 14a First image comparison unit, 14b Second image comparison unit, 15 Deviation amount calculation unit, 20, 20b Target search unit, 31 Sensing unit, 32 Warning output unit, 51 Processing circuit, 52 CPU, 53 Memory.

Claims

1. A deviation amount detection device comprising: an evaluation image acquisition unit that acquires an image including a characteristic part inside a vehicle captured by an imaging device as an evaluation image; a search position setting unit that sets a plurality of search positions distributed within a predetermined range from the position of the characteristic part in the evaluation image when it is assumed that there is no optical axis misalignment of the imaging device; a search pattern calculation unit that calculates a search pattern indicating the shape of the characteristic part in the evaluation image when it is assumed that the characteristic part is located at each of the search positions, the search pattern being based on an optical axis misalignment amount corresponding to each of the search positions; an image comparison unit that compares the characteristic part in the evaluation image with each of the plurality of search patterns for each search position; and a deviation amount detection unit that detects the amount of optical axis misalignment of the imaging device based on a comparison result by the image comparison unit.

2. The shift amount detection device according to claim 1, characterized in that the image comparison unit identifies a search pattern from among the multiple search patterns whose position and shape correspond to the position and shape of the characteristic part in the evaluation image, and sets information indicating the search position corresponding to the identified search pattern as a comparison result, and the shift amount detection unit detects the optical axis shift amount corresponding to the search position indicated by the comparison result as the optical axis shift amount of the imaging device.

3. The deviation amount detection device according to claim 1 or 2, characterized in that the search pattern calculation unit calculates the search pattern based on information indicating the three-dimensional position of the characteristic part, based on the position of the imaging device when it is assumed that there is no optical axis deviation of the imaging device, and on the amount of optical axis deviation corresponding to each search position.

4. The deviation amount detection device according to claim 3, characterized in that the search pattern calculation unit calculates the search pattern by multiplying coordinate values ​​indicating the three-dimensional position of the characteristic part, based on the position of the imaging device when it is assumed that there is no optical axis deviation of the imaging device, by a rotation matrix obtained based on the amount of optical axis deviation corresponding to each search position, and projecting the coordinate values ​​obtained by this multiplication onto the evaluation image.

5. The deviation detection device described in claim 4, characterized in that the search pattern calculation unit acquires an outline of a characteristic part on the evaluation image corresponding to the amount of optical axis deviation corresponding to each search position by projecting the coordinate values ​​obtained by the multiplication onto the evaluation image, and defines as the search pattern information that associates a coordinate value indicating the center of gravity of the acquired outline with a group of coordinate values ​​indicating the relative distance from the center of gravity of the outline to each point that constitutes the outline.

6. A deviation detection device as described in any one of claims 1 to 5, characterized in that the search pattern calculation unit calculates, as a positional deviation amount, the difference between the center of gravity of the outline of the characteristic part in the evaluation image when it is assumed that there is no optical axis deviation of the imaging device, and the center of gravity of the outline of the characteristic part in the evaluation image when it is assumed that the characteristic part is located at each of the search positions, and calculates the optical axis deviation amount corresponding to each of the search positions based on the calculated positional deviation amount.

7. A deviation detection device as described in any one of claims 1 to 6, characterized in that the search position setting unit sets a plurality of search positions so that at least the center of gravity position of the outline of the characteristic part on the evaluation image when the optical axis deviation of the imaging device is maximum within an assumed range of the optical axis deviation of the imaging device is included in the search positions.

8. An evaluation image acquisition unit that acquires an image including a characteristic part inside a vehicle captured by an imaging device as an evaluation image; a first search position setting unit that sets a plurality of search positions distributed in a predetermined first range from the position of the characteristic part in the evaluation image when it is assumed that there is no optical axis misalignment of the imaging device; a first search pattern calculation unit that calculates a first search pattern that indicates the shape of the characteristic part in the evaluation image when it is assumed that there is no optical axis misalignment of the imaging device; a first image comparison unit that compares the characteristic part in the evaluation image with the first search pattern for each search position distributed in the first range; and a second search position setting unit that sets a plurality of search positions distributed in a predetermined second range from the position of the characteristic part in the evaluation image when it is assumed that there is no optical axis misalignment of the imaging device based on a comparison result by the first image comparison unit. a second search pattern calculation unit that calculates a second search pattern indicating a shape of the characteristic part in the evaluation image when it is assumed that the characteristic part is located at each of the search positions distributed in the second range, the second search pattern being based on an optical axis shift amount corresponding to each of the search positions distributed in the second range; a second image comparison unit that compares the characteristic part in the evaluation image with each of the multiple second search patterns for each search position distributed in the second range; and a shift amount detection unit that detects an optical axis shift amount of the imaging device based on a comparison result by the second image comparison unit.

9. The deviation detection device described in claim 8, characterized in that the first image comparison unit sets as a comparison result information indicating a search position when the position and shape of the characteristic part in the evaluation image correspond to the position and shape of the first search pattern, among the multiple search positions distributed in the first range, and the second search position setting unit sets as the second range a predetermined range from the search position indicated by the comparison result by the first image comparison unit, and sets the multiple search positions distributed in the second range.

10. A deviation detection device as described in claim 8 or claim 9, characterized in that the first range is wider than the second range, and the spacing between the multiple search positions distributed in the first range is narrower than the spacing between the multiple search positions distributed in the second range.

11. A deviation amount detection device as described in any one of claims 8 to 10, characterized in that the second search pattern calculation unit calculates the second search pattern based on information indicating the three-dimensional position of the characteristic part based on the position of the imaging device when it is assumed that there is no optical axis deviation of the imaging device, and on the amount of optical axis deviation corresponding to each of the search positions distributed in the second range.

12. The deviation detection device according to any one of claims 1 to 11, characterized in that the characteristic part is a window frame, an assist grip, a headrest, a B-pillar, a door frame, a shoulder anchor, or a sun visor of the vehicle.

13. A deviation detection device as claimed in any one of claims 1 to 11, characterized in that the characteristic portion is a part of a window frame of the vehicle, and the part of the window frame includes only a top edge portion of the window frame, or the top edge portion of the window frame and a side edge portion of the window frame that contacts the top edge portion of the window frame.

14. The deviation amount detection device according to any one of claims 1 to 13, characterized in that the imaging device is installed in a center console, a steering column, an A-pillar, a rearview mirror, a dashboard, an instrument panel, or a ceiling.

15. A driver monitoring system comprising a sensing unit that monitors the state of occupants of a vehicle using the amount of optical axis deviation calculated by the deviation amount detection unit of the deviation amount detection device described in any one of claims 1 to 14 and an image of the interior of the vehicle captured by an imaging device.

16. A method for detecting an amount of deviation using a deviation detection device, comprising: an evaluation image acquisition unit acquiring an image including a characteristic part inside a vehicle captured by an imaging device as an evaluation image; a search position setting unit setting a plurality of search positions distributed within a predetermined range from the position of the characteristic part in the evaluation image when it is assumed that there is no optical axis deviation of the imaging device; a search pattern calculation unit calculating a search pattern indicating the shape of the characteristic part in the evaluation image when it is assumed that the characteristic part is located at each of the search positions, the search pattern being based on the amount of optical axis deviation corresponding to each of the search positions; an image comparison unit comparing the characteristic part in the evaluation image with each of the plurality of search patterns for each search position; and a deviation detection unit detecting the amount of optical axis deviation of the imaging device based on the comparison result by the image comparison unit.

17. A method for detecting an amount of deviation using a deviation detection device, comprising: an evaluation image acquisition unit acquiring an image including a characteristic part inside a vehicle captured by an imaging device as an evaluation image; a first search position setting unit setting a plurality of search positions distributed in a predetermined first range from the position of the characteristic part in the evaluation image when it is assumed that there is no optical axis deviation of the imaging device; a first search pattern calculation unit calculating a first search pattern indicating the shape of the characteristic part in the evaluation image when it is assumed that there is no optical axis deviation of the imaging device; a first image comparison unit comparing the characteristic part in the evaluation image with the first search pattern for each search position distributed in the first range; and a second search position setting unit setting a plurality of search positions distributed in a predetermined second range from the position of the characteristic part in the evaluation image when it is assumed that there is no optical axis deviation of the imaging device based on a comparison result by the first image comparison unit. A deviation amount detection method comprising: a step of a second search pattern calculation unit calculating a second search pattern indicating a shape of the characteristic part in the evaluation image when it is assumed that the characteristic part is located at each of the search positions distributed in the second range, the second search pattern being based on an optical axis deviation amount corresponding to each of the search positions distributed in the second range; a step of a second image comparison unit comparing the characteristic part in the evaluation image with each of the multiple second search patterns for each search position distributed in the second range; and a step of a deviation amount detection unit detecting an optical axis deviation amount of the imaging device based on a comparison result by the second image comparison unit.

Images