Image processing device and image processing method

Abstract

To more accurately measure the distance to a subject.SOLUTION: An image processing apparatus comprises: an acquisition unit that acquires data of a standard image created by performing imaging at a standard position and data of a reference image created by performing imaging at a position different from the standard position; and a control unit. The control unit projects the reference image on a first virtual plane and converts the reference image to acquire a first virtual image, and projects the reference image on a second virtual plane and converts the reference image to acquire a second virtual image. The control unit calculates the distance from the standard position to a subject based on a result of comparison of the position of the subject in the standard image with the position of the subject in the first virtual image or the second virtual image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]

[0002] Patent Document 1 discloses an imaging system including a first camera having a first field of view and a first optical axis, and a second camera having a second field of view and a second optical axis. Patent Document 1 describes that the first field of view and the second field of view at least partially overlap to form a combined field of view. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2020-522906 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a demand for more accurate measurement of the distance to the subject.

[0005] In view of the above, an object of the present disclosure is to provide an image processing device and an image processing method that are capable of measuring the distance to a subject with higher accuracy. [Means for solving the problem]

[0006] An image processing device according to an embodiment of the present disclosure includes: an acquisition unit that acquires data of a standard image generated by imaging at a standard position and data of a reference image generated by imaging at a position different from the standard position; a control unit that performs projective transformation of the reference image onto a first virtual plane to obtain a first virtual image, and performs projective transformation of the reference image onto a second virtual plane to obtain a second virtual image, and calculates a distance from the reference position to the subject based on a result of comparing a position of the subject in the reference image with a position of the subject in the first virtual image or the second virtual image, a normal direction of the first virtual plane coincides with a first direction that is an optical axis direction of a camera that generated the reference image at the reference position, and the first virtual plane is located a first distance away from the reference position along the first direction; A normal direction of the second virtual plane coincides with a second direction different from the first direction, and the second virtual plane is located a second distance away from the reference position along the second direction.

[0007] An image processing method according to an embodiment of the present disclosure includes: acquiring data of a standard image generated by imaging at a standard position and data of a reference image generated by imaging at a position different from the standard position; obtaining a first virtual image by projecting the reference image onto a first virtual plane, obtaining a second virtual image by projecting the reference image onto a second virtual plane, and calculating a distance from the reference position to the subject based on a result of comparing a position of the subject in the reference image with a position of the subject in the first virtual image or the second virtual image, a normal direction of the first virtual plane coincides with a first direction that is an optical axis direction of a camera that generated the reference image at the reference position, and the first virtual plane is located a first distance away from the reference position along the first direction; A normal direction of the second virtual plane coincides with a second direction different from the first direction, and the second virtual plane is located a second distance away from the reference position along the second direction. Effect of the Invention

[0008] According to an embodiment of the present disclosure, the distance to a subject can be measured with higher accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of a stereo camera system according to an embodiment of the present disclosure. [Diagram 2] 2 is a side view showing a schematic diagram of a moving object on which the stereo camera system shown in FIG. 1 is mounted. [Diagram 3] 2 is a front view showing a schematic diagram of a moving object on which the stereo camera system shown in FIG. 1 is mounted. [Figure 4] FIG. 2 is a diagram for explaining a first virtual image and a first virtual plane. [Diagram 5] 10 is a diagram for explaining the position of a first subject in a first virtual image. FIG. [Figure 6] 13 is a diagram for explaining a second virtual image and a second virtual plane. FIG. [Figure 7] 13 is a diagram for explaining the position of a second subject in a second virtual image. FIG. [Figure 8] 1 is a flowchart showing a procedure of an image processing method according to an embodiment of the present disclosure. [Figure 9] 1A and 1B are diagrams for explaining a reference image captured by a fisheye lens. [Figure 10] FIG. 11 is a block diagram showing a schematic configuration of a stereo camera system according to another embodiment of the present disclosure. [Figure 11] 10 is a diagram for explaining the relationship between the normal direction of a road surface and a second imaginary plane. FIG. [Figure 12] 10 is a diagram for explaining the relationship between the normal direction of a road surface and a second imaginary plane. FIG. [Figure 13] 10 is a flowchart showing the procedure of an image processing method according to another embodiment of the present disclosure. [Figure 14] FIG. 11 is a diagram for explaining a group of feature points with high linearity. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In this disclosure, the term "world coordinate system" refers to a three-dimensional coordinate system set based on the real space outside the camera. In this disclosure, the term "camera coordinate system" refers to a three-dimensional coordinate system set based on the position of the camera. The conversion between the camera coordinate system and the world coordinate system is defined by the external parameters of the camera. In this disclosure, the term "external parameters of the camera" refers to the position and orientation of the camera. In this disclosure, the term "position and orientation of the camera" refers to the position of the camera relative to a reference position and the tilt (orientation) of the camera relative to a reference direction in the world coordinate system.

[0011] In this disclosure, the term "image coordinate system" refers to a two-dimensional coordinate system that is set in an image generated by a camera. In this disclosure, the term "camera internal parameters" refers to the focal length of the camera and information on the image center of the image coordinate system.

[0012] In this disclosure, a "normalized image coordinate system" is a two-dimensional coordinate system obtained by normalizing the image coordinate system of a camera. The normalized image coordinate system is normalized so that a point on the optical axis of the camera is the origin, and the focal length of the camera is 1. The two axes that make up the normalized image coordinate system are perpendicular to the optical axis of the camera.

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0014] (System Configuration) 1, the stereo camera system 1 includes a stereo camera 10 and an image processing device 20. As shown in FIGS.

[0015] The moving body 30 travels on a road surface. The road surface is, for example, the surface of a roadway including a road or a runway. The moving body 30 is, for example, a vehicle such as an automobile, an industrial vehicle, a railroad vehicle, or a vehicle for daily life. However, the moving body 30 is not limited to a vehicle. As another example, the moving body 30 may be an aircraft, a ship, a drone, or the like. The aircraft is, for example, a fixed-wing aircraft or a rotary-wing aircraft.

[0016] The stereo camera 10 and the image processing device 20 can communicate with each other by wired or wireless communication. The stereo camera 10 and the image processing device 20 may communicate with each other via a network. The network is, for example, a wired LAN (Local Area Network), a wireless LAN, or a CAN (Controller Area Network). The stereo camera 10 and the image processing device 20 may be configured to be able to communicate with an electronic control unit (ECU: Electronic Control Unit) in the moving object 30.

[0017] The stereo camera 10 may be mounted at various locations on the moving body 30. In one of a plurality of embodiments, the stereo camera 10 may be mounted inside the moving body 30, which is a vehicle, and may capture an image of the outside of the moving body 30 through a windshield. For example, the stereo camera 10 may be disposed in front of a rearview mirror or on a dashboard. In one of a plurality of embodiments, the stereo camera 10 may be fixed to any one of a front bumper, a fender grill, a side fender, a light module, and a bonnet of the vehicle.

[0018] The image processing device 20 may be mounted in various locations of the vehicle 30. For example, the image processing device 20 may be mounted in the dashboard of the vehicle 30.

[0019] 2 and 3, the stereo camera 10 and the image processing device 20 are positioned apart from each other. However, the stereo camera 10 and the image processing device 20 may be housed in the same housing and configured as an integrated device. A device in which the stereo camera 10 and the image processing device 20 are housed in the same housing is also called a stereo camera device.

[0020] The stereo camera 10 includes a plurality of cameras that have parallax and cooperate with each other. The stereo camera 10 includes at least two or more cameras. In the present embodiment, the stereo camera 10 includes a first camera 11 and a second camera 12. The stereo camera 10 can image an object from a plurality of directions by making the plurality of cameras cooperate with each other. The stereo camera 10 may be a device including a plurality of cameras in the same housing. The stereo camera 10 may be a device including two or more cameras that are independent of each other and located apart from each other. However, the stereo camera 10 is not limited to a plurality of cameras that are independent of each other. For example, the stereo camera 10 of the present disclosure may employ a camera having an optical mechanism that guides light incident at two separate points to one light receiving element. In the present disclosure, a plurality of images generated by imaging the same object from different viewpoints is also described as a "stereo image".

[0021] The first camera 11 includes an optical system that defines an optical axis OX1 and an imaging element. The second camera 12 includes an optical system that defines an optical axis OX2 and an imaging element. The optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera are different. The optical system includes a lens or a mirror. The optical system may form an image of a subject on a light receiving surface of the imaging element. The optical system of each of the first camera 11 and the second camera 12 may include a wide-angle lens such as a fisheye lens. However, the optical system of each of the first camera 11 and the second camera 12 is not limited to a wide-angle lens and may include any lens. The imaging element includes a CCD image sensor (Charge-Coupled Device Image Sensor) and a CMOS image sensor (Complementary MOS Image Sensor). The imaging elements of each of the first camera 11 and the second camera 12 may be present in the same plane perpendicular to the optical axis OX1 and the optical axis OX2, respectively. The first camera 11 and the second camera 12 generate image data. The image data is data that represents an image formed by an image sensor.

[0022] For example, the first camera 11 and the second camera 12 are fixed to each other. For example, the first camera 11 and the second camera 12 are fixed to each other, so that the relative positions and orientations of the first camera 11 and the second camera 12 are determined.

[0023] The optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are oriented in a direction that allows the same object to be captured. The position and optical axis OX1 of the first camera 11 and the position and optical axis OX2 of the second camera 12 are determined so that the images captured by the first camera 11 and the second camera 12 respectively include at least the same object. The optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are oriented so as to be parallel to each other. This parallelism is not limited to strict parallelism, and allows for misalignment in assembly, misalignment in installation, and misalignment over time. The optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are not limited to being parallel, and may be oriented in different directions. Even if the optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are not parallel to each other, a stereo image can be generated by converting the images in the stereo camera 10 or the image processing device 20. The base length is the distance between the optical center of the first camera 11 and the optical center of the second camera 12. The base line length corresponds to the distance between the centers of the lenses of first camera 11 and second camera 12. The base line length direction is the direction connecting the optical center of first camera 11 and the optical center of second camera 12.

[0024] 2, the first camera 11 and the second camera 12 are disposed so as to be able to capture an image of the area in front of the moving body 30. For example, the first camera 11 and the second camera 12 are disposed so that the optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are parallel to the area in front of the moving body 30.

[0025] The first camera 11 and the second camera 12 are positioned apart in a direction intersecting the optical axis OX1 and the optical axis OX2. In one of a plurality of embodiments, the first camera 11 and the second camera 12 are positioned along the left-right direction of the moving body 30. The first camera 11 is positioned to the left of the second camera 12 when facing forward. The second camera 12 is positioned to the right of the first camera 11 when facing forward.

[0026] The first camera 11 captures an image of a subject to generate a reference image. The position where the reference image is generated by imaging, i.e., the position of the first camera 11, is also referred to as the "reference position". The second camera 12 captures an image of a subject to generate a reference image. The position where the reference image is generated by imaging, i.e., the position of the second camera 12, is different from the position of the first camera 11. Since the positions of the first camera 11 and the second camera 12 are different, the position of the subject in the reference image is different from the position of the subject in the reference image, even if the subject is the same. The reference image and the reference image are stereo images generated by imaging the subject from different viewpoints. The first camera 11 and the second camera 12 may capture the subject at a predetermined frame rate (e.g., 30 fps).

[0027] As shown in FIG. 1, the image processing device 20 includes an acquisition unit 21, an output unit 22, a storage unit 23, and a control unit 24.

[0028] The acquisition unit 21 is an input interface for the image processing device 20. The acquisition unit 21 can receive input of information from the stereo camera 10 and other devices. The acquisition unit 21 can employ a physical connector and a wireless communication module. The physical connector includes an electrical connector compatible with transmission by electrical signals, an optical connector compatible with transmission by optical signals, and an electromagnetic connector compatible with transmission by electromagnetic waves. The electrical connector includes a connector conforming to IEC60603, a connector conforming to the USB standard, a connector compatible with an RCA terminal, a connector compatible with an S terminal defined in EIAJ CP-1211A, a connector compatible with a D terminal defined in EIAJ RC-5237, a connector conforming to the HDMI (registered trademark) standard, and a connector compatible with a coaxial cable including BNC. The optical connector includes various connectors conforming to IEC 61754. The wireless communication module includes a wireless communication module conforming to each standard including Bluetooth (registered trademark) and IEEE802.11.

[0029] The acquisition unit 21 acquires data of the base image generated by the first camera 11 and data of the reference image generated by the second camera 12. The acquisition unit 21 outputs the acquired data of the base image and data of the reference image to the control unit 24. The acquisition unit 21 may be compatible with a data transmission method of the stereo camera 10. The acquisition unit 21 may be connected to an output interface of the stereo camera 10 via a network. The acquisition unit 21 may be connected to an electronic control unit in the mobile object 30 via a network.

[0030] The output unit 22 is an interface for output of the image processing device 20. The output unit 22 can output the processing result of the image processing device 20 to other devices in the moving body 30 or other devices outside the moving body 30. The other devices in the moving body 30 may include a driving support device such as an auto cruise control, and a safety device such as an automatic braking device. The other devices outside the moving body 30 may include other vehicles and road measuring instruments. The other devices in the moving body 30 or other devices outside the moving body 30 can use the information received from the image processing device 20 as appropriate. The output unit 22 may include various interfaces compatible with wired and wireless communication, the same as or similar to the acquisition unit 21.

[0031] The storage unit 23 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The RAM is, for example, a SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The ROM is, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 23 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 23 stores data used in the operation of the image processing device 20 and data obtained by the operation of the image processing device 20.

[0032] The storage unit 23 stores internal parameters of the first camera 11 and the second camera 12. The storage unit 23 stores external parameters of the first camera 11 and the second camera 12. The storage unit 23 stores a rotation matrix R and a translation vector t, which will be described later.

[0033] The control unit 24 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 24 executes processes related to the operation of the image processing device 20 while controlling each part of the image processing device 20.

[0034] The control unit 24 acquires data of the standard image and the reference image from the acquisition unit 21. The control unit 24 executes a distortion correction process on the standard image and the reference image. The distortion correction process is a process for correcting lens distortion aberration. In addition to the distortion correction process, the control unit 24 may execute any image processing such as a brightness adjustment process, a contrast adjustment process, and a gamma correction process. Hereinafter, the standard image after the distortion correction process and the like is also referred to as a "standard image 40". The reference image after the distortion correction process and the like is also referred to as a "reference image 41".

[0035] The control unit 24 converts the reference image 41 into a first virtual image 42 and a second virtual image 43, as described below. The control unit 24 calculates the distance from the reference position to the subject based on a result of comparing the position of the subject in the base image 40 with the position of the subject in the first virtual image 42 or the second virtual image 43, as described below.

[0036] <Get distance in the first direction> The control unit 24 converts the reference image 41 into a first virtual image 42. The first virtual image 42 is a virtual point X1 in real space as shown in FIG. q This is an image that is assumed to be captured at the reference position (q=1, ..., m1) (m1 is an integer equal to or greater than 1). q is a first distance d1 from a reference position along a first direction D1 in real space. qThe first direction D1 is a direction along the optical axis OX1 of the first camera 11, that is, the optical axis direction of the first camera 11. Information on the first direction D1 may be stored in advance in the storage unit 23 in association with the camera coordinate system. The control unit 24 may specify the first direction D1 by acquiring information on the first direction D1 associated with the camera coordinate system from the storage unit 23. Hereinafter, the first distance d1 will be referred to as q The first virtual image 42 corresponding to q " is also written.

[0037] The control unit 24 maps the reference image 41 onto a first virtual plane V1 q The reference image 41 after the projective transformation is converted into a first virtual image 42. q The first virtual plane V1 q is the virtual point X1 q The first virtual plane V1 is a virtual plane on which the q is a first distance d1 from the reference position along the first direction D1 q That is, the first distance d1 q A first virtual image 42 corresponding to q means a first distance d1 from a reference position along a first direction D1 q A first imaginary plane V1 located at a distance q The first virtual plane V1 can be said to be an image obtained by projectively transforming the reference image 41. q The normal direction of the reference image 41 coincides with the first direction D1. q Then, the first virtual image 42 is obtained by projecting the q The process of acquiring the normalized image coordinate x1 can be said to be a process by a plane sweep method. Here, in FIG. 4, the normalized image coordinate x1 is a virtual point X1 in the normalized image coordinate system C1 of the reference image 40. q The normalized image coordinate x2 is the coordinate of the virtual point X1 in the normalized image coordinate system C2 of the reference image 41. q The relationship between the normalized image coordinates x1 and x2 is expressed by the planar projection transformation H q By "x2 = H q x1". In other words, the control unit 24 performs the planar projective transformation H qThe reference image 41 can be transformed into the first virtual image 42 by the planar projective transformation H q is expressed by equation (1).

number

[0038] The control unit 24 calculates the first distance d1 in the formula (1). q While changing the value of q Here, when the subject is the first subject, the control unit 24 acquires the first virtual image 42 q The distance to the first object in real space can be obtained by the above. The first object is located on a first virtual plane V1. q In other words, the normal direction of at least a part of the surface of the first object is parallel to the first virtual plane V1. q Since the normal direction of the surface included in the first object coincides with the first direction D1, the first distance 1 q A plurality of first virtual images 42 are displayed while changing q When the first virtual image 42 is acquired, q The position of the first object in is a first distance d1 q Furthermore, the first distance d1 q corresponds to the distance from the reference position to the first object in the first direction D1, the first virtual image 42 qThe position of the first subject in the image 10 coincides with the position of the first subject in the reference image 40.

[0039] Therefore, the control unit 24 calculates the position of the first subject in the reference image 40 and the position of the first virtual image 42. q For example, the control unit 24 calculates the distance to the first subject based on a result of comparing the position of the first subject in the reference image 40 with the position of the first subject in the plurality of first virtual images 42. q The control unit 24 compares the position of the first subject in each of the first virtual images 42. q From among these, the first virtual image 42 q The position of the first object in the first virtual image 42 is closest to the position of the first object in the reference image 40. q The control unit 24 identifies the first virtual image 42. q The first distance d1 corresponding to q is acquired as the distance from the reference position to the first subject in the first direction D1 in real space.

[0040] 5 shows the base image 40a and the reference image 41a. The base image 40a includes a partial image 40оb1. The partial image 40оb1 is an image generated by the first camera 11 capturing an image of the first object оb1. The reference image 41a includes a partial image 41оb1. The partial image 41оb1 is an image generated by the second camera 12 capturing an image of the first object оb1. The normal direction of the front of the first object оb1 coincides with the first direction D1.

[0041] In FIG. 5, q=1, 2, 3. The first virtual image 42a1, the first virtual image 42a2, and the first virtual image 42a3 are shown on the right side of the reference image 41a. The control unit 24 acquires the first virtual image 42a1 by projecting the reference image 41a onto the first virtual plane V11 with the first distance d11=10 cm. The control unit 24 acquires the first virtual image 42a2 by projecting the reference image 41a onto the first virtual plane V12 with the first distance d12=20 cm. The control unit 24 acquires the first virtual image 42a3 by projecting the reference image 41a onto the first virtual plane V13 with the first distance d13=30 cm.

[0042] In FIG. 5, the control unit 24 compares the position of the first subject оb1 in the reference image 40a, for example, the position of the partial image 40оb1, with the position of the first subject оb1 in each of the first virtual images 42a1 to 42a3, for example, the position of the partial image 41оb1. In FIG. 5, for the sake of explanation, the partial image 40оb1 of the reference image 40a is shown by a broken line in each of the first virtual images 42a1 to 42a3. Of the positions of the first subject оb1 in each of the first virtual images 42a1 to 42a3, the position of the first subject оb1 in the first virtual image 42a2 is closest to the position of the first subject оb1 in the reference image 40a. The control unit 24 specifies the first virtual image 42a2, of the first virtual images 42a1 to 42a3, in which the position of the first subject оb1 in the reference image 40a is closest to the position of the first subject оb1. The control unit 24 acquires the first distance d12=20 cm corresponding to the identified first virtual image 42a2 as the distance from the reference position in the first direction D1 in real space to the first subject OB1.

[0043] The control unit 24 determines the position of the first subject in the reference image 40 and the plurality of first virtual images 42. q The position of the first object in each of the reference image 40 and the plurality of first virtual images 42 is compared. q In this case, the control unit 24 may perform feature matching using the reference image 40 and the plurality of first virtual images 42. q The control unit 24 extracts a feature amount from each pixel of the reference image 40 and the plurality of first virtual images 42. The feature amount includes, for example, at least one of the brightness and color of the pixel. The control unit 24 may extract the feature amount by any algorithm such as AKAZE (Accelerated-KAZE), ORB (Oriented FAST and Rotated BRIEF), or SIFT (Scale-Invariant Feature Transform). The feature amount may be expressed in a format such as AKAZE, ORB, or SIFT, or in various other formats. The control unit 24 extracts a feature amount from each pixel of the reference image 40 and the plurality of first virtual images 42. q The control unit 24 compares the feature amount of the pixel at the same coordinate in each of the first virtual images 42. qAmong them, the first virtual image 42 has the highest similarity to the feature amount of the pixel at the same coordinate of the reference image 40. q In order to evaluate the similarity of the feature amount, the control unit 24 compares the reference image 40 and the plurality of first virtual images 42. q The control unit 24 may calculate the evaluation value of the feature amount of the pixel having the same coordinate in each of the first virtual images 42 by ZNCC (Zero Means Normalized Cross Correlation). q Among them, the first virtual image 42 with the smallest evaluation value is q The first virtual image 42 has the highest similarity to the feature amount of the pixel at the same coordinate in the reference image 40. q The control unit 24 may specify the specified first virtual image 42 as q The first distance d1 corresponding to q The first distance d1 associated with the pixel is q is the distance from the reference position in the first direction D1 to the subject appearing in that pixel. That is, in the subsequent processing, the control unit 24 determines the first distance d1 q By acquiring the distance from the reference position in the first direction D1 to the subject, the distance can be acquired. q The feature amount of a block at the same position may be compared in each of the above. The block includes at least one pixel. The feature amount of a block may be an average value of the feature amounts of the pixels included in the block.

[0044] The control unit 24 determines the acquired first distance d1 q That is, the distance from the reference position to the first subject is calculated based on the distance from the reference position to the first subject in the first direction D1. The control unit 24 may calculate the linear distance Z from the reference position to the first subject using equation (2).

number

[0045] <Get distance in the second direction> The control unit 24 converts the reference image 41 into a second virtual image 43. The second virtual image 43 is a virtual point X2 in real space as shown in FIG. q This is an image that is assumed to be captured at the reference position (q=1, ..., m2) (m2 is an integer equal to or greater than 1). q is a second distance d2 from the reference position along the second direction D2 in the real space. q The second direction D2 is a direction different from the first direction D1. For example, the inner product of a vector in the second direction D2 and a vector in the first direction D1 is other than 0. The second direction D2 may be a normal direction to a horizontal plane. Hereinafter, the second direction D2 is assumed to be a normal direction to a road surface. However, the second direction D2 may be any direction different from the first direction D1, and is not limited to the normal direction to the road surface. Information on the second direction D2 may be stored in advance in the memory unit 23 in association with the camera coordinate system. The control unit 24 may identify the second direction D2 by acquiring information on the second direction D2 associated with the camera coordinate system from the memory unit 23. Hereinafter, the second distance d2 q The second virtual image 43 corresponding to the second virtual image 43 is q " is also written.

[0046] The control unit 24 maps the reference image 41 onto the second virtual plane V2 q The reference image 41 after the projective transformation is converted into a second virtual image 43. q The second virtual plane V2 q is the virtual point X2 q The second virtual plane V2 is a virtual plane on which the q is the second distance d2 from the reference position q That is, the second distance d2 q A second virtual image 43 corresponding to q means a second distance d2 from the reference position along the second direction D2 qA second imaginary plane V2 located at a distance q The second virtual plane V2 can be said to be an image obtained by projectively transforming the reference image 41. q The normal direction of the plane projection transformation H q The reference image 41 is converted into a second virtual image 43 q The second virtual image 43 q In the transformation, the first distance d1 in Eq. q Instead of the second distance d2 q In addition, the normal vector n q is the second virtual plane V2 q becomes the normal vector of

[0047] The control unit 24 calculates the second distance d2 in the formula (1). q While changing the second virtual image 43 q Here, when the subject is the second subject, the control unit 24 acquires the second virtual image 43 q The distance to the second object in real space can be obtained by the above. The second object is located on the second virtual plane V2. q In other words, the normal direction of at least a part of the surface of the second object is parallel to the second virtual plane V2. q The first virtual plane V1 corresponds to the normal direction of the first virtual plane V2, i.e., the second direction D2. Examples of the second object include a car stop and an uneven road surface. The second object may be the same object as the first object, or may be a different object from the first object. When the second object is the same object as the first object, the second object is located within the first virtual plane V1. q and a second imaginary plane V2 q The normal direction of the surface included in the second object coincides with the second direction D2, so that the second distance 2 q While changing the second virtual image 43 q If the second virtual image 43 is acquired, q The position of the second object in is a second distance d2 q The second distance d2 qcorresponds to the distance from the reference position to the second object in the second direction D2, the second virtual image 43 q The position of the second subject in the reference image 40 coincides with the position of the second subject in the reference image 40.

[0048] Therefore, the control unit 24 calculates the position of the second subject in the reference image 40 and the position of the second virtual image 43. q For example, the control unit 24 calculates the distance to the second subject based on a result of comparing the position of the second subject in the reference image 40 with the position of the second subject in the plurality of second virtual images 43. q The control unit 24 compares the position of the second subject in each of the second virtual images 43. q From among them, the second virtual image 43 q The position of the second subject in the second virtual image 43 is closest to the position of the second subject in the reference image 40. q The control unit 24 identifies the identified second virtual image 43. q The second distance d2 corresponding to q is acquired as the distance from the reference position to the second subject in the second direction D2 in real space. This process will be described with reference to FIG.

[0049] 7 shows the base image 40b and the reference image 41b. The base image 40b includes a partial image 40оb2. The partial image 40оb2 is an image generated by the first camera 11 capturing an image of the second object оb2. The reference image 41b includes a partial image 41оb2. The partial image 41оb2 is an image generated by the second camera 12 capturing an image of the second object оb2. The normal direction of the top surface of the second object оb2 coincides with the second direction D2.

[0050] In FIG. 7, q=1, 2, 3. The second virtual image 43b1, the second virtual image 43b2, and the second virtual image 43b3 are shown on the right side of the reference image 41b. The control unit 24 acquires the second virtual image 43b1 by projecting the reference image 41b onto the second virtual plane V21 with the second distance d21=10 cm. The control unit 24 acquires the second virtual image 43b2 by projecting the reference image 41b onto the second virtual plane V22 with the second distance d22=20 cm. The control unit 24 acquires the second virtual image 43b3 by projecting the reference image 41b onto the second virtual plane V23 with the second distance d23=30 cm.

[0051] In FIG. 7, the control unit 24 compares the position of the second subject оb2 in the reference image 40b, for example, the position of the partial image 40оb2, with the position of the second subject оb2 in each of the second virtual images 43b1 to 43b3, for example, the position of the partial image 41оb2. In FIG. 7, for the sake of explanation, the partial image 40оb2 of the reference image 40b is shown by a broken line in each of the second virtual images 43b1 to 43b3. Of the positions of the second subject оb2 in each of the second virtual images 43b1 to 43b3, the position of the second subject оb2 in the second virtual image 43b2 is closest to the position of the second subject оb2 in the reference image 40b. The control unit 24 specifies the second virtual image 43b2 in which the position of the second subject оb2 in the reference image 40b is closest to the position of the second subject оb2 from among the second virtual images 43b1 to 43b3. The control unit 24 acquires the second distance d22=20 cm corresponding to the identified second virtual image 43b2 as the distance from the reference position in the second direction D2 in real space to the second subject оb2.

[0052] The control unit 24 determines the position of the second subject in the reference image 40 and the plurality of second virtual images 43. q The position of the second object in each of the reference image 40 and the plurality of second virtual images 43 is compared. q In this case, the control unit 24 may perform feature matching using the reference image 40 and the plurality of second virtual images 43. qAs described above, the control unit 24 may extract the feature amounts by any algorithm such as AKAZE, ORB, or SIFT. The control unit 24 extracts the feature amounts from the reference image 40 and the plurality of second virtual images 43. q The control unit 24 compares the feature amount of the pixel at the same coordinate in each of the second virtual images 43. q Among them, the second virtual image 43 having the highest similarity to the feature amount of the pixel at the same coordinate of the reference image 40 is q In order to evaluate the similarity of the feature amount, the control unit 24 compares the reference image 40 and a plurality of second virtual images 43 q The control unit 24 may calculate the evaluation value of the feature amount of the pixel having the same coordinate in each of the second virtual images 43 by using the ZNCC. q Among them, the second virtual image 43 with the smallest evaluation value is q The second virtual image 43 has the highest similarity to the feature amount of the pixel at the same coordinate in the reference image 40. q The control unit 24 may specify the specified second virtual image 43 as q The second distance d2 corresponding to q The second distance d2 associated with the pixel is q is the distance from the reference position in the second direction D2 to the subject appearing in that pixel. That is, in the subsequent processing, the control unit 24 determines the second distance d2 q By acquiring the distance from the reference position in the second direction D2 to the subject, the distance can be acquired. q In the same manner or similarly, the control unit 24 may also be configured to generate a reference image 40 and a plurality of second virtual images 43. q The feature amounts of blocks at the same position may be compared in each of the blocks.

[0053] The control unit 24 determines the acquired second distance d2 qThat is, the distance from the reference position to the second object is calculated based on the distance from the reference position to the second object in the second direction D2. The control unit 24 may calculate the linear distance Z from the reference position to the second object using the formula (2). In the second direction D2, the normal vector n q is the second virtual plane V2 q Moreover, the unit vector b in equation (2) is a unit vector in the direction of the camera coordinate system corresponding to the direction from the origin of the normalized image coordinate system C1 toward the pixel in which the second object is captured.

[0054] (System Operation) Fig. 8 is a flowchart showing the procedure of an image processing method according to an embodiment of the present disclosure. In this embodiment, the image processing device 20 executes the image processing method as shown in Fig. 8. When the stereo camera 10 starts capturing images, the control unit 24 starts the process of step S1.

[0055] The control unit 24 acquires data of the base image and the reference image from the acquisition unit 21 (step S1). The control unit 24 executes a distortion correction process on the base image and the reference image (step S2). As described above, the base image after the distortion correction process and the like is also referred to as the "base image 40". The reference image after the distortion correction process and the like is also referred to as the "reference image 41".

[0056] The control unit 24 determines the first distance d1 q While changing the reference image 41, the first virtual plane V1 q By projecting the image onto the target object, a plurality of first virtual images 42 are obtained. q (Step S3). The control unit 24 acquires the reference image 40 and the multiple first virtual images 42. q The control unit 24 executes feature matching using the above (step S4). The control unit 24 acquires the distance from the reference position to the first object in the first direction D1 in real space (step S5). The control unit 24 calculates the distance from the reference position to the first object based on the distance in the first direction D1 acquired in the process of step S5 (step S6).

[0057] The control unit 24 determines the second distance d2 q While changing the reference image 41, the second virtual plane V2 q , and a plurality of second virtual images 43 are obtained by projecting the q (Step S7). The control unit 24 acquires the reference image 40 and the plurality of second virtual images 43. q The control unit 24 executes feature matching using the above (step S8). The control unit 24 acquires the distance from the reference position to the second object in the second direction D2 in real space (step S9). The control unit 24 calculates the distance from the reference position to the second object based on the distance in the second direction D2 acquired in the process of step S9 (step S10).

[0058] In this manner, in the image processing device 20 according to this embodiment, the control unit 24 aligns the reference image 41 with the first virtual plane V1. q Then, the first virtual image 42 is obtained by projecting the q and the reference image 41 is placed on the second virtual plane V2 q Then, projective transformation is performed to obtain the second virtual image 43 q Furthermore, the control unit 24 acquires the position of the subject in the reference image 40 and the first virtual image 42. q or second virtual image 43 q Based on the result of comparing the position of the subject in the first virtual plane V1 with the reference position, the distance from the reference position to the subject is calculated. q Then, the first virtual image 42 is obtained by projecting the q The process of acquiring the first virtual image 42 can be said to be a process by the plane sweep method. In the present embodiment, in addition to the process by the plane sweep method, the control unit 24 acquires the first virtual image 42 q In addition to the process of acquiring the reference image 41, the second virtual plane V2 q Then, projective transformation is performed to obtain the second virtual image 43 q With this configuration, the distance to the subject can be measured with higher accuracy, as will be described below.

[0059] As a first comparative example, the control unit 24 performs processing using the plane sweep method, i.e., the first virtual image 42 qIn the first comparative example, when the control unit 24 calculates the distance to the second object оb2 as shown in FIG. 7, the control unit 24 calculates the distance to the second object оb2 by locating the reference image 41b on the first virtual plane V1. q Then, the first virtual image 42 is obtained by projecting the q Here, the second object оb2 is located on the first virtual plane V1 q In other words, the normal direction of any surface of the second object оb2 does not include a surface parallel to the first virtual plane V1 q Therefore, the first distance d1 in the first direction D1 does not match the normal direction of the q A plurality of first virtual images 42 are displayed while changing q Even if multiple first virtual images 42 are acquired, q The position of the second object оb2 in each of the above is the first distance d1 q The first virtual images 42 q The position of the second object оb2 in each of the above is the first distance d1 q The first virtual image 42 does not change depending on q Therefore, it is not possible to obtain the distance from the reference position to the second object OB2.

[0060] In contrast to the first comparative example, in this embodiment, the control unit 24 controls the second virtual plane V2 whose normal direction coincides with the second direction D2. q The reference image 41 is projectively transformed to a second virtual image 43. q With this configuration, the control unit 24 acquires the first virtual plane V1 q For the second object that does not include a surface parallel to the second virtual image 43 q Thus, the distance from the reference position to the second object can be obtained. Therefore, according to this embodiment, the distance to the object can be measured with higher accuracy. As described above, an example of the second object is unevenness of the road surface. In this embodiment, the distance to the unevenness of the road surface can be calculated, so that the road surface shape can be detected with higher accuracy.

[0061] As a second comparative example, consider measuring the distance to the subject using a standard image and a reference image captured by a fisheye lens. FIG. 9 shows a diagram for explaining an image 40X captured by a fisheye lens. The solid lines in the image 40X are lines connecting points at the same height from the road surface. The image 40X includes regions R1 and R2. In the region R1, the intervals between the solid lines are wider than in the region R2. In the region R1, the intervals between the solid lines are wider, so that when distortion correction is performed on the image 40X, points at different heights from the road surface correspond to different pixels. However, in the region R2, the intervals between the solid lines are narrower, so that when distortion correction is performed on the image 40X, points at different heights from the road surface may correspond to the same pixel. In the region R2, points at different heights from the road surface correspond to the same pixel, so that even if the plane sweep method is applied to a standard image and a reference image such as the image 40X, the distance to the subject may not be obtained.

[0062] In this embodiment, in addition to the processing by the plane sweep method, the control unit 24 also performs the processing by the first virtual image 42 q In addition to the process of acquiring the reference image 41, the second virtual plane V2 q Then, projective transformation is performed to obtain the second virtual image 43 q The control unit 24 converts the reference image 41 into a second virtual image 43. q By converting the distance to the subject captured in region R2 into the distance to the subject captured in region R3, it is possible to obtain the distance to the subject captured in region R4. That is, in this embodiment, even if fisheye lenses are used in the optical systems of first camera 11 and second camera 12, the distance to the subject can be obtained. In this embodiment, by using fisheye lenses in the optical systems of first camera 11 and second camera 12, it is possible to obtain the distance to the subject present in a wider region.

[0063] (System configuration according to other embodiments) 10, a stereo camera system 101 according to another embodiment includes a stereo camera 10 and an image processing device 120. The stereo camera system 101 may be mounted at various locations on a moving object 30 in the same manner as or similar to the stereo camera 10 shown in FIG. 2. The image processing device 120 includes an acquisition unit 21, an output unit 22, a storage unit 23, a control unit 24, and a sensor unit 25.

[0064] The sensor unit 25 includes an inertial measurement unit (IMU) or an inertial sensor. The sensor unit 25 detects acceleration and angular velocity in three mutually orthogonal axial directions. The sensor unit 25 can detect attitude information of the stereo camera 10 by mounting the stereo camera 10 and the image processing device 20 on the same moving body 30. The attitude information of the stereo camera 10 is the inclination (attitude) of the stereo camera 10 with respect to a reference direction in a world coordinate system.

[0065] Here, as shown in FIG. 11, the attitude of the stereo camera 10 may tilt when the moving body 30 carrying the stereo camera 10 runs over an object 2 on the road surface. In FIG. 11, only the tires of the moving body 30 are shown among the components of the moving body 30. The object 2 is, for example, a stone on the road surface. When the attitude of the stereo camera 10 tilts, the relative positional relationship between the world coordinate system and the camera coordinate system changes. When the relative positional relationship between the world coordinate system and the camera coordinate system changes, for example, when a direction previously associated with the camera coordinate system is specified as the second direction D2, the specified second direction D2 deviates from the normal direction to the road surface. When the specified second direction D2 deviates from the normal direction to the road surface, the second virtual plane V2 in formula (1) q Normal vector n q is deviated from the normal direction of the road surface. q When the direction of the second virtual image 43 deviates from the normal direction of the road surface, q This may result in a failure to accurately calculate the distance to the second object.

[0066] Therefore, the control unit 24 acquires the attitude information of the stereo camera 10 by the sensor unit 25. The control unit 24 detects the normal direction of the road surface from the attitude information of the stereo camera 10. The control unit 24 converts the detected normal direction of the road surface into a second direction D2, that is, a second virtual plane 2V q , and the reference image 41 is the second virtual plane V2 q The reference image 41 after the projective transformation is converted into a second virtual image 43. q For example, the control unit 24 obtains the normal vector n qC and the second distance d2 is obtained by equation (4). qC The control unit 24 acquires the acquired normal vector n qC Let n be the normal vector of equation (1). q Substituting in and obtaining the second distance d2 qC The first distance d1 in equation (1) q and the reference image 41 is converted into the second virtual image 43 q Convert to.

number

number

[0067] By detecting the normal direction of the road surface in this manner, the detected normal direction of the road surface is converted into a second direction D2, i.e., a second virtual plane 2V q , and the reference image 41 is the second virtual image 43 qWith this configuration, even if the relative positional relationship between the world coordinate system and the camera coordinate system changes, the control unit 24 can convert the reference image 40 and the second virtual image 43 into q In this way, the distance to the second subject can be detected with high accuracy. In another embodiment, when the second direction D2 is the normal direction to the horizontal plane, the control unit 24 may detect the normal direction to the horizontal plane based on the posture information of the first camera 11.

[0068] (System Operation According to Other Embodiments) Fig. 13 is a flowchart showing the procedure of an image processing method according to another embodiment of the present disclosure. The image processing device 120 executes the image processing method as shown in Fig. 13. When the stereo camera 10 starts capturing images, the control unit 24 starts the process of step S21.

[0069] The control unit 24 executes the processes of steps S21 and S22 in the same manner or similarly to the processes of steps S1 and S2 shown in FIG.

[0070] The control unit 24 acquires the attitude information of the stereo camera 10 by the sensor unit 25, and detects the normal direction of the road surface based on the acquired attitude information of the stereo camera 10 (step S23).

[0071] The control unit 24 executes the processes of steps S24 to S27 in a manner that is the same as or similar to the processes of steps S3 to S6 shown in FIG.

[0072] In the process of step S28, the control unit 24 aligns the normal direction of the road surface detected in the process of step S23 with the second virtual plane 2V q , and the reference image 41 is the second virtual plane V2 q The reference image 41 after the projective transformation is converted into a second virtual image 43. q Obtain as.

[0073] The control unit 24 executes the processes of steps S29 to S31 in the same manner or similarly to the processes of steps S8 to S10 shown in FIG.

[0074] The image processing device 120 according to another embodiment may not include the sensor unit 25. In this case, the control unit 24 may acquire posture information of the moving object 30 from an electronic control unit in the moving object 30 by the acquisition unit 21. The control unit 24 may use the posture information of the moving object 30 as posture information of the stereo camera 10 to detect the normal direction of the road surface.

[0075] Other effects and configurations of the image processing device 120 according to the other embodiment are the same as or similar to those of the image processing device 20 shown in FIG.

[0076] (System configuration according to still another embodiment) A stereo camera system according to still another embodiment can employ the same configuration as the stereo camera system 1 shown in Fig. 1. Hereinafter, a stereo camera system according to still another embodiment will be described with reference to Fig. 1.

[0077] The control unit 24 acquires the posture information of the stereo camera 10 by bundle adjustment using the base image 40 and the reference image 41. The bundle adjustment is a process for estimating the posture of the camera such that the deviation between the coordinates of each of the feature points extracted from the image and the coordinates of each of the reprojected points obtained by reprojecting the restored points restored from the feature points onto the image is minimized. For example, the control unit 24 extracts feature points from each of the base image 40 and the reference image 41. The feature points are parts of the subject captured in the image that stand out from other adjacent parts. The feature points are parts that can be extracted by, for example, comparing pixel values. Examples of feature points include a point on the contour of the subject captured in the image and a point in the image where the pixel value is maximum or minimum. The control unit 24 may extract the feature points by any algorithm such as AKAZE, ORB, or SIFT. In particular, with ORB, highly reliable feature points can be extracted from the edge part of the subject captured in the image. When the control unit 24 extracts the feature points, it identifies the corresponding feature points in the base image 40 and the reference image 41. The corresponding feature points are feature points that indicate the same target point in the base image 40 and the reference image 41. The target point is a part of the subject in the three-dimensional real space. The control unit 24 regards the points that correspond in the three-dimensional real space to the corresponding feature points included in the base image 40 and the reference image 41 as restoration points and restores them. The control unit 24 may restore the corresponding feature points by utilizing the principle of triangulation using the attitude of the stereo camera 10, i.e., external parameters, and the internal parameters of the camera. The control unit 24 acquires the coordinates of the restoration points in the world coordinate system. The control unit 24 reprojects the restoration points onto the image coordinate systems of the base image 40 and the reference image 41, respectively, to acquire the two-dimensional coordinates of the reprojected points. The control unit 24 estimates the orientation of the stereo camera 10 such that the deviation between the two-dimensional coordinates of the reprojected points acquired by reprojecting the restoration points and the two-dimensional coordinates of the corresponding feature points used for restoring the restoration points is minimized. The control unit 24 estimates the orientation of the stereo camera 10 to obtain orientation information of the stereo camera 10.

[0078] As described above, the control unit 24 detects the normal direction of the road surface based on the posture information of the stereo camera 10. As described above, the control unit 24 converts the detected normal direction of the road surface into the second direction D2, that is, the second virtual plane 2V q , and the reference image 41 is the second virtual plane V2 q The reference image 41 after the projective transformation is converted into a second virtual image 43. q Obtain as.

[0079] By using such bundle adjustment, the control unit 24 can obtain the attitude information of the stereo camera 10 even without an inertial measurement unit. Furthermore, the control unit 24 can detect the normal direction of the road surface using the attitude information of the stereo camera 10. With this configuration, as described above with reference to FIG. 12, even if the relative positional relationship between the world coordinate system and the camera coordinate system changes, the control unit 24 can easily detect the normal direction of the reference image 40 and the second virtual image 43. q In this way, the distance to the second subject can be detected with high accuracy. In still another embodiment, when the second direction D2 is the normal direction to the horizontal plane, the control unit 24 may detect the normal direction to the horizontal plane based on the attitude information of the first camera 11.

[0080] (System Operation According to Further Embodiments) An image processing method according to still another embodiment can be executed by the procedure shown in Fig. 13. However, in the process of step S23, the control unit 24 acquires the attitude information of the stereo camera 10 by bundle adjustment using the base image 40 and the reference image 41, and detects the normal direction of the road surface based on the acquired attitude information of the stereo camera 10.

[0081] Other effects and configurations of the image processing device 20 according to the further embodiment are the same as or similar to those of the image processing device 20 according to the embodiment described above.

[0082] (System configuration according to still another embodiment) A stereo camera system according to still another embodiment can employ the same configuration as the stereo camera system 1 shown in Fig. 1. Hereinafter, a stereo camera system according to still another embodiment will be described with reference to Fig. 1.

[0083] The control unit 24 extracts a plurality of feature points from the base image 40 or the reference image 41. The control unit 24 may extract the feature points by any algorithm such as AKAZE, ORB, or SIFT. In particular, by using ORB, highly reliable feature points can be extracted from the edge portion of the object captured in the image. The control unit 24 detects a group of feature points with high linearity from the extracted plurality of feature points. A group of feature points with high linearity is a group of feature points in which the deviation between an approximate line derived from a plurality of feature points included in the group of feature points and each of the plurality of feature points is smaller than a threshold value. The threshold value may be set in advance based on a desired accuracy of measuring the distance to the object. The control unit 24 acquires the coordinates of a world coordinate system of a corresponding point that corresponds to the plurality of feature points included in the detected group of feature points in a three-dimensional real space. The control unit 24 acquires the coordinates of the world coordinate system of the corresponding point by utilizing the principle of triangulation using the attitude of the stereo camera 10, i.e., external parameters, and the internal parameters of the camera. The control unit 24 detects the orientation information of the stereo camera 10 based on the coordinates of the corresponding points in the world coordinate system.

[0084] For example, the control unit 24 extracts a plurality of feature points 44 from a reference image 40c as shown in Fig. 14. The control unit 24 detects a feature point group 45 having high linearity from the plurality of feature points 44. The control unit 24 detects posture information of the stereo camera 10 based on the detected feature point group 45.

[0085] The control unit 24 detects the normal direction of the road surface, as described above, based on the posture information of the stereo camera 10. In still another embodiment, when the second direction D2 is the normal direction to the horizontal plane, the control unit 24 may detect the normal direction to the horizontal plane based on the posture information of the stereo camera 10. As described above, the control unit 24 may detect the normal direction of the detected road surface as the second direction D2, that is, the second virtual plane 2Vq , and the reference image 41 is the second virtual plane V2 q The reference image 41 after the projective transformation is converted into a second virtual image 43. q Obtain as.

[0086] In this way, by detecting a group of feature points with high linearity from the base image 40 or the reference image 41, it is possible to detect the normal direction of the road surface without an inertial measurement unit. With this configuration, as described above with reference to FIG. 12, even if the relative positional relationship between the world coordinate system and the camera coordinate system changes, the control unit 24 can detect the normal direction of the base image 40 and the second virtual image 43. q This makes it possible to detect the distance to the second subject with high accuracy.

[0087] (System Operation According to Further Embodiments) An image processing method according to still another embodiment can be performed by the procedure shown in FIG. However, in the process of step S23, the control unit 24 obtains the attitude information of the stereo camera 10 by detecting a group of feature points with high linearity, and detects the normal direction of the road surface from the obtained attitude information of the stereo camera 10.

[0088] Other effects and configurations of the image processing device 20 according to the further embodiment are the same as or similar to those of the image processing device 20 according to the embodiment described above.

[0089] In one embodiment, (1) an image processing device includes: an acquisition unit that acquires data of a standard image generated by imaging at a standard position and data of a reference image generated by imaging at a position different from the standard position; a control unit that performs projective transformation of the reference image onto a first virtual plane to obtain a first virtual image, and performs projective transformation of the reference image onto a second virtual plane to obtain a second virtual image, and calculates a distance from the reference position to the subject based on a result of comparing a position of the subject in the reference image with a position of the subject in the first virtual image or the second virtual image, a normal direction of the first virtual plane coincides with a first direction that is an optical axis direction of a camera that generated the reference image at the reference position, and the first virtual plane is located a first distance away from the reference position along the first direction; A normal direction of the second virtual plane coincides with a second direction different from the first direction, and the second virtual plane is located a second distance away from the reference position along the second direction.

[0090] (2) In the image processing device of (1) above, The control unit is A plurality of the first virtual images are acquired while varying the first distance, and a plurality of the second virtual images are acquired while varying the second distance.

[0091] (3) In the image processing device of (1) or (2) above, the subject is a first subject including a surface parallel to the first imaginary plane, The control unit calculates the distance from the reference position to the first subject based on a result of comparing the position of the first subject in the reference image with the position of the first subject in each of the multiple first virtual images.

[0092] (4) In any one of the image processing devices (1) to (3) above, the subject is a second subject including a surface parallel to the second imaginary plane, The control unit calculates the distance from the reference position to the second subject based on a result of comparing the position of the second subject in the reference image with the position of the second subject in each of the multiple second virtual images.

[0093] (5) In any one of the image processing devices according to (1) to (4), The control unit executes a distortion correction process on the standard image and the reference image.

[0094] (6) In any one of the image processing devices according to (1) to (5), The second direction is a normal direction to a horizontal plane.

[0095] (7) In any one of the image processing devices (1) to (6) above, The control unit detects a normal direction to a horizontal plane based on attitude information of a camera that generated the reference image at the reference position.

[0096] (8) In any one of the image processing devices according to (1) to (7), The control unit obtains attitude information of the camera by a sensor unit.

[0097] (9) In any one of the image processing devices according to (1) to (8), The control unit obtains posture information of the camera by bundle adjustment using the base image and the reference image.

[0098] (10) In any one of the image processing devices according to (1) to (9), The control unit obtains the posture information of the camera by detecting a group of feature points having high linearity from the base image or the reference image.

[0099] (11) The image processing method is acquiring data of a standard image generated by imaging at a standard position and data of a reference image generated by imaging at a position different from the standard position; obtaining a first virtual image by projecting the reference image onto a first virtual plane, obtaining a second virtual image by projecting the reference image onto a second virtual plane, and calculating a distance from the reference position to the subject based on a result of comparing a position of the subject in the reference image with a position of the subject in the first virtual image or the second virtual image, a normal direction of the first virtual plane coincides with a first direction that is an optical axis direction of a camera that generated the reference image at the reference position, and the first virtual plane is located a first distance away from the reference position along the first direction; A normal direction of the second virtual plane coincides with a second direction different from the first direction, and the second virtual plane is located a second distance away from the reference position along the second direction.

[0100] Although the present disclosure has been described based on the drawings and examples, it should be noted that a person skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions, etc. included in each functional unit can be rearranged so as not to be logically inconsistent. Multiple functional units, etc. may be combined into one or divided. Each embodiment according to the present disclosure described above is not limited to being implemented faithfully to each of the embodiments described respectively, and may be implemented by combining each feature as appropriate or omitting a part. In other words, the contents of the present disclosure can be modified and corrected in various ways by a person skilled in the art based on the present disclosure. Therefore, these modifications and corrections are included in the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. can be added to other embodiments so as not to be logically inconsistent, or replaced with each functional unit, each means, each step, etc. of other embodiments. In addition, in each embodiment, multiple functional units, each means, each step, etc. can be combined into one or divided. Furthermore, each of the above-described embodiments of the present disclosure is not limited to being implemented faithfully according to each of the described embodiments, but may be implemented by combining each feature or omitting some features as appropriate.

[0101] For example, in the above-described embodiment, the control unit 24 of the image processing device 20 has been described as performing distortion correction processing on the standard image and the reference image. However, the first camera 11 may perform distortion correction processing on the generated standard image. The second camera 12 may perform distortion correction processing on the generated reference image. The first camera 11 and the second camera 12 may perform any image processing, such as brightness adjustment processing, contrast adjustment processing, gamma correction processing, etc., on the standard image and the reference image, in addition to the distortion correction processing.

[0102] For example, in the above-described embodiment, the first camera 11 and the second camera 12 are described as being positioned along the left-right direction of the moving body 30 as shown in Fig. 3. However, the positions of the first camera 11 and the second camera 12 are not limited to this. As another example, the first camera 11 and the second camera 12 may be positioned along the up-down direction of the moving body 30.

[0103] For example, in the above-described embodiment, it has been described that the first camera 11 generates a base image, and the second camera 12 generates a reference image. However, the base image and the reference image may be generated by one camera. In this case, the camera moves to generate a base image by capturing an image at a base position, and generate a reference image by capturing an image at a position different from the base position. [Explanation of symbols]

[0104] 1,101 Stereo Camera System 2 objects V1q First virtual plane V2q Second virtual plane d1q 1st distance d2q 2nd distance 10 Stereo Camera 11 Camera 1 12 Second Camera 20,120 Image processing device 21 Acquisition Department 22 Output section 23 Memory section 24 Control Unit 25 Sensor section 30 Mobile 40,40a,40b,40c Reference images 40оb1,40оb2,41оb1,41оb2 Partial images 41,41a,41b Reference images 42,42q First virtual image 43,43q 2nd virtual image 44 feature points 45 Feature points D1 1st direction D2 2nd direction OX1 Optical Axis OX2 Optical Axis R1, R2 fields

Claims

1. an acquisition unit that acquires data of a standard image generated by imaging at a standard position and data of a reference image generated by imaging at a position different from the standard position; a control unit that performs projective transformation of the reference image onto a first virtual plane to obtain a first virtual image, and performs projective transformation of the reference image onto a second virtual plane to obtain a second virtual image, and calculates a distance from the reference position to the subject based on a result of comparing a position of the subject in the reference image with a position of the subject in the first virtual image or the second virtual image, a normal direction of the first virtual plane coincides with a first direction which is an optical axis direction of a camera that generated the reference image at the reference position, and the first virtual plane is located a first distance away from the reference position along the first direction; An image processing device, wherein a normal direction of the second virtual plane coincides with a second direction different from the first direction, and the second virtual plane is located a second distance away from the reference position along the second direction.

2. The control unit is The image processing device according to claim 1 , further comprising: acquiring a plurality of the first virtual images while changing the first distance; and acquiring a plurality of the second virtual images while changing the second distance.

3. the subject is a first subject including a surface parallel to the first virtual plane, The image processing device of claim 2 , wherein the control unit calculates the distance from the reference position to the first subject based on a result of comparing the position of the first subject in the reference image with the position of the first subject in each of the multiple first virtual images.

4. the subject is a second subject including a surface parallel to the second imaginary plane, The image processing device of claim 2, wherein the control unit calculates the distance from the reference position to the second subject based on a result of comparing the position of the second subject in the reference image with the position of the second subject in each of the multiple second virtual images.

5. The image processing device according to claim 1 , wherein the control unit performs distortion correction processing on the standard image and the reference image.

6. The image processing device according to claim 1 , wherein the second direction is a normal direction to a horizontal plane.

7. The image processing device according to claim 6 , wherein the control unit detects a normal direction to a horizontal plane based on attitude information of a camera that generated the reference image at the reference position.

8. The image processing device according to claim 7 , wherein the control unit obtains the attitude information of the camera by a sensor unit.

9. The image processing device according to claim 7 , wherein the control unit acquires the attitude information of the camera by bundle adjustment using the base image and the reference image.

10. The image processing device according to claim 7 , wherein the control unit acquires the attitude information of the camera by detecting a group of feature points having high linearity from the base image or the reference image.

11. acquiring data of a standard image generated by imaging at a standard position and data of a reference image generated by imaging at a position different from the standard position; obtaining a first virtual image by projecting the reference image onto a first virtual plane, obtaining a second virtual image by projecting the reference image onto a second virtual plane, and calculating a distance from the reference position to the subject based on a result of comparing a position of the subject in the reference image with a position of the subject in the first virtual image or the second virtual image, a normal direction of the first virtual plane coincides with a first direction which is an optical axis direction of a camera that generated the reference image at the reference position, and the first virtual plane is located a first distance away from the reference position along the first direction; An image processing method, wherein a normal direction of the second virtual plane coincides with a second direction different from the first direction, and the second virtual plane is located a second distance away from the reference position along the second direction.

Images