Image processing device, stereo camera device, and image processing method

Abstract

To provide a technique for accurately measuring a distance to a subject.SOLUTION: An image processing apparatus includes an acquisition unit and a control unit. The control unit transforms coordinates of a first reference pixel of a reference image by a plurality of homographies into coordinates of a plurality of referred pixels included in a referred image, specifies a referred pixel including the same subject with the first reference pixel out of the referred pixels with transformed coordinates, and executes association processing to associate the first reference pixel with the specified referred pixel. The control unit transforms the coordinates of the referred pixel associated with the first reference pixel by inverse transformation of the homographies into coordinates of multiple reference pixels included in the reference image, specifies a second reference pixel including the same subject with the referred pixel associated with the first reference pixel out of the reference pixels with transformed coordinates, and executes determination processing to determine, based on the first reference pixel and the second reference pixel, whether the association between the first reference pixel and the referred pixel is wrong or not.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to an image processing device, a stereo camera device, and an image processing method. [Background technology]

[0002] Patent Document 1 discloses an imaging system comprising a first camera having a first field of view and a first optical axis, and a second camera having a second field of view and a second optical axis. Patent Document 1 describes that the first field of view and the second field of view overlap at least partially to form a combined field of view. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Special Publication No. 2020-522906 [Overview of the project] [Problems that the invention aims to solve]

[0004] It is necessary to measure the distance to the subject with greater accuracy.

[0005] In view of these points, the purpose of this disclosure is to provide a technology that can measure the distance to an object with greater accuracy. [Means for solving the problem]

[0006] An image processing apparatus according to one embodiment of the present disclosure is An acquisition unit that acquires data of a reference image generated by imaging at a reference position and data of a reference image generated by imaging at a position different from the reference image, The system includes a control unit that performs a mapping process to associate the first reference pixel of the reference image with the specified reference pixel by performing multiple planar projection transformations, the first reference pixel with the specified reference pixel, the first reference pixel with the specified reference pixel, and the first reference pixel with the specified reference pixel. The control unit, The coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations are transformed into the coordinates of the plurality of reference pixels included in the reference image, a second reference pixel is identified among the plurality of reference pixels whose coordinates have been transformed and which contains the same subject as the reference pixel associated with the first reference pixel, and a determination process is performed to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations transform rays directed to the camera that generated the reference image, which pass through a plurality of virtual points located in each of the plurality of virtual planes, into rays directed to the camera that generated the reference image, which pass through the plurality of virtual points.

[0007] A stereo camera device according to one embodiment of the present disclosure, A stereo camera including a first camera that generates a reference image by capturing an image at a reference position, and a second camera that generates a reference image by capturing an image at a position different from the reference image, The image processing device includes: an image processing device that performs a mapping process to associate the first reference pixel of the reference image with the coordinates of a plurality of reference pixels included in the reference image by performing a plurality of planar projection transformations, to identify a reference pixel that contains the same subject as the first reference pixel among the plurality of reference pixels whose coordinates have been transformed, and to associate the first reference pixel with the identified reference pixel; The aforementioned image processing device is The coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations are transformed into the coordinates of the plurality of reference pixels included in the reference image, a second reference pixel is identified among the plurality of reference pixels whose coordinates have been transformed and which contains the same subject as the reference pixel associated with the first reference pixel, and a determination process is performed to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations convert rays from a plurality of virtual points located in each of the plurality of virtual planes to rays from a plurality of virtual points to a second camera.

[0008] An image processing method according to one embodiment of this disclosure is: The process involves acquiring data for a reference image generated by imaging at a reference position, and data for a reference image generated by imaging at a position different from the reference image. The process involves transforming the coordinates of the first reference pixel of the reference image into the coordinates of multiple reference pixels included in the reference image using multiple planar projection transformations, identifying a reference pixel among the multiple reference pixels whose coordinates have been transformed that contains the same subject as the first reference pixel, and performing a mapping process to associate the first reference pixel with the identified reference pixel. The process includes: converting the coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations into the coordinates of a plurality of reference pixels included in the reference image; identifying a second reference pixel among the plurality of reference pixels whose coordinates have been converted that contains the same subject as the reference pixel associated with the first reference pixel; and performing a determination process to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations transform rays directed to the camera that generated the reference image, which pass through a plurality of virtual points located in each of the plurality of virtual planes, into rays directed to the camera that generated the reference image, which pass through the plurality of virtual points. [Effects of the Invention]

[0009] According to one embodiment of this disclosure, a technology can be provided that can measure the distance to an object with greater accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This block diagram shows a schematic configuration of a stereo camera device according to one embodiment of the present disclosure. [Figure 2] Figure 1 is a schematic side view showing a mobile body on which the stereo camera device is mounted. [Figure 3] Figure 1 is a schematic front view showing a mobile body equipped with the stereo camera device shown. [Figure 4] This is a diagram illustrating the first virtual image and the first virtual plane. [Figure 5] This is a diagram illustrating the position of the first subject in the first virtual image. [Figure 6] This is a diagram illustrating the second virtual image and the second virtual plane. [Figure 7] This is a diagram illustrating the position of the second subject in the second virtual image. [Figure 8] This figure shows an example of a distance image. [Figure 9] This is a flowchart showing the steps for the mapping process. [Figure 10] This is a flowchart showing the steps for the mapping process. [Figure 11] This flowchart shows the procedure for determining incorrect responses. [Figure 12] This flowchart shows the procedure for determining incorrect responses. [Figure 13] This is a diagram illustrating an image captured with a fisheye lens. [Modes for carrying out the invention]

[0011] In this disclosure, “world coordinate system” refers to a three-dimensional coordinate system defined based on real space outside the camera. In this disclosure, “camera coordinate system” refers to a three-dimensional coordinate system defined based on the camera’s position. The transformation between the camera coordinate system and the world coordinate system is defined by the camera’s external parameters. In this disclosure, “camera’s external parameters” refers to the camera’s position and orientation. In this disclosure, “camera position and orientation” refers to the camera’s position relative to a reference position and the camera’s tilt (orientation) relative to a reference direction in the world coordinate system.

[0012] In this disclosure, “image coordinate system” refers to the two-dimensional coordinate system set in the image generated by the camera. In this disclosure, “camera intrinsic parameters” include the camera’s focal length and information about the image center of the image coordinate system.

[0013] In this disclosure, "normalized image coordinate system" refers to a two-dimensional coordinate system obtained by normalizing the image coordinate system set in an image generated by a camera. The normalized image coordinate system is normalized such that the origin is a point on the camera's optical axis and the camera's focal length is 1. The two axes constituting the normalized image coordinate system are orthogonal to the camera's optical axis.

[0014] In this disclosure, an image containing a subject means an image in which a subject is depicted. A pixel containing a subject means a pixel in which a subject is depicted. In an image, multiple pixels are arranged along vertical and horizontal directions. In this disclosure, one of the vertical and horizontal directions in which pixels are arranged in an image is referred to as a "row," and the other as a "column."

[0015] The embodiments relating to this disclosure will be described below with reference to the drawings.

[0016] (Configuration of the stereo camera system) As shown in Figure 1, the stereo camera device 1 comprises a stereo camera 10 and an image processing device 20. As shown in Figures 2 and 3, the stereo camera device 1 is mounted on, for example, a mobile body 30.

[0017] The mobile body 30 travels on a road surface. The road surface is, for example, the surface of a road or runway. The mobile body 30 is, for example, a vehicle such as an automobile, industrial vehicle, railway vehicle, or residential vehicle. However, the mobile body 30 is not limited to a vehicle. As another example, the mobile body 30 may be an aircraft, ship, or drone. The aircraft may be, for example, a fixed-wing aircraft or a rotary-wing aircraft.

[0018] The stereo camera 10 and the image processing device 20 can communicate by wired or wireless connection. The stereo camera 10 and the image processing device 20 may communicate via a network. The network may be, for example, a wired LAN (Local Area Network), a wireless LAN, or a CAN (Controller Area Network). The stereo camera 10 and the image processing device 20 may be configured to communicate with an electronic control unit (ECU) in the mobile body 30.

[0019] The stereo camera 10 may be mounted at various locations on the mobile body 30. In one of the multiple embodiments, the stereo camera 10 may be mounted inside the mobile body 30, which is a vehicle, and may image the outside of the mobile body 30 through the windshield. For example, the stereo camera 10 may be positioned in front of the rearview mirror or on the dashboard. In one of the multiple embodiments, the stereo camera 10 may be fixed to the vehicle's front bumper, fender grille, side fender, light module, or hood.

[0020] The image processing device 20 may be mounted in various locations on the mobile device 30. For example, the image processing device 20 may be mounted inside the dashboard of the mobile device 30.

[0021] In Figures 2 and 3, the stereo camera 10 and the image processing device 20 are located separately. However, the stereo camera 10 and the image processing device 20 may be housed in the same enclosure and configured as a single unit.

[0022] The stereo camera 10 includes a plurality of cameras that have parallax with each other and cooperate with each other. The stereo camera 10 includes at least two or more cameras. In this embodiment, the stereo camera 10 includes a first camera 11 and a second camera 12. The stereo camera 10 is capable of imaging an object from multiple directions by having the plurality of cameras cooperate. The stereo camera 10 may be a device that includes a plurality of cameras in the same housing. The stereo camera 10 may be a device that includes two or more cameras that are independent of each other and located far apart from each other. However, the stereo camera 10 is not limited to a plurality of independent cameras. The stereo camera 10 of this disclosure may employ, for example, a camera having an optical mechanism that guides light incident at two distant locations to a single photodetector. In this disclosure, a plurality of images generated by imaging the same subject from different viewpoints are also referred to as a “stereo image”.

[0023] The first camera 11 comprises an optical system defining the optical axis OX1 and an image sensor. The second camera 12 comprises an optical system defining the optical axis OX2 and an image sensor. The optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera are different. The optical systems of the first camera 11 and the second camera 12 each include lenses or mirrors. The optical systems of the first camera 11 and the second camera 12 may form an image of the subject on the light-receiving surface of the image sensor. The optical systems of the first camera 11 and the second camera 12 each may include wide-angle lenses such as fisheye lenses. However, the optical systems of the first camera 11 and the second camera 12 are not limited to wide-angle lenses and may include any lens. The image sensors of the first camera 11 and the second camera 12 each include CCD image sensors (Charge-Coupled Device Image Sensors) and CMOS image sensors (Complementary MOS Image Sensors). The image sensors of the first camera 11 and the second camera 12 may be located in the same plane perpendicular to the optical axes OX1 and OX2, respectively. The first camera 11 and the second camera 12 generate image data by capturing images. The image data is data representing the image formed by the image sensor.

[0024] The first camera 11 and the second camera 12 are fixed to each other, for example. Between the first camera 11 and the second camera 12, their relative positions and orientations are determined by, for example, the fixing of the first camera 11 and the second camera 12 to each other.

[0025] The optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are oriented in directions that allow them to image the same subject. The position and optical axis OX1 of the first camera 11 and the position and optical axis OX2 of the second camera 12 are determined so that the images captured by the first camera 11 and the second camera 12 each contain at least the same subject. The optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are oriented so that they are parallel to each other. This parallelism is not limited to strict parallelism, and allows for misalignment due to assembly, mounting, and these misalignments over time. The optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are not limited to parallel, and may be oriented in different directions. Even if the optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are not parallel to each other, a stereo image can be generated by converting the image within the stereo camera 10 or the image processing device 20. The baseline length is the distance between the optical center of the first camera 11 and the optical center of the second camera 12. For example, the baseline length corresponds to the distance between the intersection of the optical axis OX1 with the light-receiving surface of the first camera 11 and the intersection of the optical axis OX2 with the light-receiving surface of the second camera 12. The baseline length direction is the direction connecting the optical center of the first camera 11 and the optical center of the second camera 12.

[0026] As shown in Figure 2, the first camera 11 and the second camera 12 are arranged to capture images in front of the moving body 30. For example, the first camera 11 and the second camera 12 are arranged so that the optical axis OX1 of the first camera 11 and the optical axis OX2 of the second camera 12 are parallel to the direction of straight-line movement of the moving body 30.

[0027] The first camera 11 and the second camera 12 are positioned apart in directions intersecting the optical axes OX1 and OX2. In one of the multiple embodiments, the first camera 11 and the second camera 12 are positioned apart along the left-right direction of the moving body 30. The first camera 11 is positioned to the left of the second camera 12 when facing forward. The second camera 12 is positioned to the right of the first camera 11 when facing forward.

[0028] The first camera 11 captures an image of the subject and generates a reference image. The position from which the reference image is generated, i.e., the position of the first camera 11, is also referred to as the "reference position." The second camera 12 captures an image of the subject and generates a reference image. The position from which the reference image is generated, i.e., the position of the second camera 12, is different from the position of the first camera 11. Because the positions of the first camera 11 and the second camera 12 are different, even for the same subject, the position of the subject in the reference image and the position of the subject in the reference image will be different. The reference image and the reference image are stereo images generated by capturing the subject from different viewpoints. The first camera 11 and the second camera 12 may capture the subject at a predetermined frame rate (e.g., 30fps).

[0029] As shown in Figure 1, the image processing device 20 includes an acquisition unit 21, an output unit 22, a storage unit 23, and a control unit 24.

[0030] The acquisition unit 21 is an input interface for the image processing device 20. The acquisition unit 21 can accept information input from the stereo camera 10 and other devices. The acquisition unit 21 can employ physical connectors and wireless communication modules. Physical connectors include electrical connectors that support transmission by electrical signals, optical connectors that support transmission by optical signals, and electromagnetic connectors that support transmission by electromagnetic waves. Electrical connectors include connectors compliant with IEC60603, connectors compliant with the USB standard, connectors compatible with RCA terminals, connectors compatible with S terminals as defined in EIAJ CP-1211A, connectors compatible with D terminals as defined in EIAJ RC-5237, connectors compliant with the HDMI® standard, and connectors compatible with coaxial cables including BNC. Optical connectors include various connectors compliant with IEC 61754. Wireless communication modules include wireless communication modules compliant with various standards, including Bluetooth® and IEEE802.11.

[0031] The acquisition unit 21 acquires data of the reference image 40 generated by the first camera 11 and data of the reference image 41 generated by the second camera 12. The acquisition unit 21 outputs the acquired data of the reference image 40 and the reference image 41 to the control unit 24. The acquisition unit 21 may be compatible with the data transmission method of the stereo camera 10. The acquisition unit 21 may be connected to the output interface of the stereo camera 10 via a network. The acquisition unit 21 may be connected to the electronic control unit in the mobile unit 30 via a network.

[0032] The output unit 22 is an output interface for the image processing device 20. The output unit 22 can output the processing results of the image processing device 20 to an external device. The external device may be, for example, another device within the mobile body 30 or another device outside the mobile body 30. The other device within the mobile body 30 may include driving assistance devices such as automatic cruise control and safety devices such as automatic braking devices. The other device outside the mobile body 30 may include other vehicles and road surveying equipment. The other device within the mobile body 30 or the other device outside the mobile body 30 can use the information received from the image processing device 20 as appropriate. The output unit 22 may include various interfaces that support wired and wireless communication, similar to or in the same way as the acquisition unit 21.

[0033] The storage unit 23 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The RAM is, for example, SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The ROM is, for example, EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 23 may function as a main memory, auxiliary memory, or cache memory. The storage unit 23 stores data used for the operation of the image processing device 20 and data obtained by the operation of the image processing device 20.

[0034] The memory unit 23 stores the internal parameters of the first camera 11 and the second camera 12. The memory unit 23 stores the external parameters of the first camera 11 and the second camera 12. The memory unit 23 stores the rotation matrix R, translation vector t, and parameters K1, K2, ρ1, ρ2, which will be described later.

[0035] The control unit 24 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 24 controls each part of the image processing device 20 and executes processes related to the operation of the image processing device 20.

[0036] The control unit 24 acquires data from the reference image 40 and the reference image 41 using the acquisition unit 21.

[0037] Hereafter, the normalized image coordinate system of the first camera 11 will also be referred to as "normalized image coordinate system C1". In other words, normalized image coordinate system C1 is a two-dimensional coordinate system that normalizes the image coordinate system set in the image generated by the first camera 11. The normalized image coordinate system of the second camera 12 will also be referred to as "normalized image coordinate system C2". In other words, normalized image coordinate system C2 is a two-dimensional coordinate system that normalizes the image coordinate system set in the image generated by the second camera 12.

[0038] Hereafter, the camera coordinate system of the first camera 11 will also be referred to as "camera coordinate system C11". The camera coordinate system of the second camera 12 will also be referred to as "camera coordinate system C12".

[0039] The following describes the overview of the acquisition process for obtaining the distance to the subject, followed by the correspondence process according to this embodiment. This acquisition process consists of a first acquisition process that uses the first virtual image 42, which will be described later, and a second acquisition process that uses the second virtual image 43, which will be described later.

[0040] <First acquisition process> The control unit 24 converts the reference image 41 into a first virtual image 42. The first virtual image 42 represents a virtual point X1 in real space as shown in Figure 4.q (q = 1, …, m1) (where m1 is an integer greater than or equal to 1) is an image assumed to be captured at the reference position. Virtual point X1 q is located at a first distance d1 from the reference position along a first direction D1 in the real space q away. The first direction D1 is the direction along the optical axis OX1 of the first camera 11, that is, the optical axis direction of the first camera 11. Information on the first direction D1 may be stored in advance in the storage unit 23 in association with the camera coordinate system C11 of the first camera 11. The control unit 24 may identify the first direction D1 by acquiring information on the first direction D1 associated with the camera coordinate system C11 from the storage unit 23. Hereinafter, the first virtual image 42 corresponding to the first distance d1 q is also described as "the first virtual image 42 q ".

[0041] The control unit 24 performs projective transformation on the reference image 41 to the first virtual plane V1 q and acquires the reference image 41 after projective transformation as the first virtual image 42[[ID=1 VI]] q . The first virtual plane V1 q is a virtual plane where the virtual point X1 q is located. The first virtual plane V1 q is located at a first distance d1 from the reference position along the first direction D1 q away. That is, the first virtual image 42 q corresponding to the first distance d1 q is also an image obtained by performing projective transformation on the reference image 41 to the first virtual plane V1 q located at a first distance d1 from the reference position along the first direction D1 q . The normal direction of the first virtual plane V1 q agrees with the first direction D1. The process of performing projective transformation on the reference image 41 to the first virtual plane V1 q to obtain the first virtual image 42 q can also be said to be a process by the Plane Sweep method. Here, in FIG. 4, the coordinate x1 is the virtual point X1 q when projected onto the reference image 40, and the virtual point X1 qThe coordinates of the pixels drawn on the reference image 40 are given by x1. Coordinate x1 is given as the coordinate in the normalized image coordinate system C1. Coordinate x2 is the virtual point X1 q When projected onto reference image 41, the virtual point X1 q x² is the coordinate of the pixel to be drawn on the reference image 41. The coordinate x² is given as the coordinate in the normalized image coordinate system C2. The control unit 24 performs a plane projection transformation H q Therefore, virtual point X1 q The light ray from the first camera 11 passing through is directed to virtual point X1 q It can be converted into the light rays of the second camera 12 passing through. Planar projection transformation H q This is expressed by equation (1).

number

[0042] The control unit 24 controls the first virtual image 42 q This allows us to obtain the distance to the first object in real space. The first object is located on the first virtual plane V1. q The subject includes a plane parallel to the first virtual plane V1. In other words, the normal direction of at least some of the planes of the first subject is the first virtual plane V1. q The normal direction of the surface included in the first subject coincides with the first direction D1, thus the first distance d1 along the first direction D1q Multiple first virtual images 42 while changing q If obtained, the first virtual image 42 q The position of the first subject in this context is given by the first distance d1 q It changes according to the first distance d1 q If this matches the distance from the reference position in the first direction D1 to the first subject, then the first virtual image 42 q The position of the first subject in the first virtual image 42 coincides with the position of the first subject in the reference image 40. q If the position of the first subject in the first virtual image 42 matches the position of the first subject in the reference image 40, then the first virtual image 42 q The first distance d1 corresponding to q This represents the distance from the reference position in the first direction D1 to the first subject.

[0043] Therefore, the control unit 24 determines the position of the first subject in the reference image 40 and the multiple first virtual images 42 q The control unit 24 compares the position of the first subject in each of the multiple first virtual images 42 q From among them, the first virtual image 42 q The first virtual image 42 is the closest to the position of the first subject in the reference image 40, where the position of the first subject in the reference image 40 is the closest to the position of the first subject in the reference image 40. q The control unit 24 identifies the first virtual image 42. q The first distance d1 corresponding to q This is obtained as the distance from the reference position in the first direction D1 in real space to the first subject. This process can be performed by the mapping process described later. The following explanation will be given with reference to Figure 5.

[0044] Figure 5 shows the reference image 40a and the reference image 41a. The reference image 40a includes a partial image 40оb1. Partial image 40оb1 is the portion of the reference image 40a in which the first subject оb1, captured by the first camera 11, is drawn. The reference image 41a includes a partial image 41оb1. Partial image 41оb1 is the portion of the reference image 41a in which the first subject оb1, captured by the second camera 12, is drawn. The normal direction of the front of the first subject оb1 coincides with the first direction D1.

[0045] In Figure 5, let q = 1, 2, 3. To the right of reference image 41a are the first virtual images 42a1, 42a2, and 42a3. The first virtual image 42a1 corresponds to the first distance d11. The first virtual image 42a2 corresponds to the first distance d12. The first virtual image 42a3 corresponds to the first distance d13.

[0046] In Figure 5, for the sake of explanation, in each of the first virtual images 42a1 to 42a3, the partial image 40оb1 of the reference image 40a is shown by a dashed line. In each of the first virtual images 42a1 to 42a3, the position of the partial image 41оb1, i.e., the position of the first subject оb1, is the first distance d1 q It changes accordingly. Here, the position of the first subject оb1 in the first virtual image 42a2 and the position of the first subject оb1 in the reference image 40a coincide. In other words, the distance from the reference position in the first direction D1 to the first subject оb1 is the first distance d12. The control unit 24 identifies the first virtual image 42a2 from among the first virtual images 42a1 to 42a3 in which the position of the first subject оb1 in the reference image 40a is closest to the position of the first subject оb1. The control unit 24 acquires the first distance d12 corresponding to the identified first virtual image 42a2 as the distance from the reference position in the first direction D1 in real space to the first subject оb1.

[0047] The control unit 24 receives the acquired first distance d1 qIn other words, the straight-line distance from the reference position to the first subject may be calculated based on the distance from the reference position to the first subject in the first direction D1. For example, the control unit 24 may calculate the straight-line distance Z from the reference position to the first subject using equation (2).

number

[0048] <Second acquisition process> The control unit 24 converts the reference image 41 into a second virtual image 43. The second virtual image 43 represents a virtual point X2 in real space, as shown in Figure 6. q This is a hypothetical image taken at a reference position (q=1,…,m2) (where m2 is an integer greater than or equal to 1). (Virtual point X2) q This is the second distance d2 from the reference position along the second direction D2 in real space. q They are located at a distance. The second direction D2 is a different direction from the first direction D1. For example, the dot product of the vector in the second direction D2 and the vector in the first direction D1 is a non-zero value. The second direction D2 may be the normal direction of the horizontal plane. Hereafter, the second direction D2 will be assumed to be the normal direction of the road surface. However, the second direction D2 may be a different direction from the first direction D1 and is not limited to the normal direction of the road surface. Information on the second direction D2 may be stored in the storage unit 23 in association with the camera coordinate system C11. The control unit 24 may identify the second direction D2 by obtaining the information on the second direction D2 associated with the camera coordinate system C11 from the storage unit 23. Hereafter, the second distance d2 q The corresponding second virtual image 43 is, "Second virtual image 43 q It is also written as ".

[0049] The control unit 24 controls the reference image 41 in the second virtual plane V2 q The reference image 41 after the projection transformation is transformed into the second virtual image 43. q It is acquired as the second virtual plane V2. q This is virtual point X2q This is a virtual plane on which it is located. Second virtual plane V2 q This is the second distance d2 from the reference position. q They are located at a distance. That is, the second distance d2 q The corresponding second virtual image 43 q This is along the second direction D2 from the reference position at a second distance d2 q Second virtual plane V2 located at a distance q This can also be described as the image obtained after projective transformation of reference image 41. Second virtual plane V2 q The normal direction of the first direction D1 coincides with the second direction D2. In the same or similar manner as the first direction D1, the control unit 24 performs the planar projection transformation H of equation (1). q Therefore, virtual point X2 q The light ray passing through to the first camera 11 is directed to virtual point X2 q This can be converted into a ray to the second camera 12 passing through. In this conversion, the first distance d1 in equation (1) q Instead, the second distance d2 q The normal vector n in equation (1) is used. q This is the second virtual plane V2 q This becomes the normal vector.

[0050] The control unit 24 controls the second virtual image 43 q This allows us to obtain the distance to the second subject in real space. The second subject is located on the second virtual plane V2. q The subject includes a plane parallel to the second virtual plane V2. In other words, the normal direction of at least some of the planes of the second subject is the second virtual plane V2. q It coincides with the normal direction of the first virtual plane V1, i.e., the second direction D2. Examples of the second subject include bollards and unevenness in the road surface. The second subject may be the same object as the first subject, or it may be a different object. If the second subject is the same object as the first subject, the second subject is the first virtual plane V1 q A plane parallel to and the second virtual plane V2 q This includes a plane parallel to the second subject. Here, the normal direction of the plane included in the second subject coincides with the second direction D2, so the second distance 2 along the second direction D2 q Multiple second virtual images 43 while changing q If obtained, the second virtual image 43q The position of the second subject in q changes according to the second distance d2. q When the second distance d2 coincides with the distance from the reference position in the second direction D2 to the second subject, the position of the second subject in the second virtual image 43 q coincides with the position of the second subject in the reference image 40. That is, when the position of the second subject in the second virtual image 43 q coincides with the position of the second subject in the reference image 40, the second distance d2 q corresponding to the second virtual image 43 becomes the distance from the reference position in the second direction D2 to the second subject. q [[ID=ll]]Therefore, the control unit 24 compares the position of the second subject in the reference image 40 with the position of the second subject in each of the plurality of second virtual images 43. q The control unit 24 identifies the second virtual image 43 among the plurality of second virtual images 43

[0051] in which the position of the second subject is closest to the position of the second subject in the reference image 40. The control unit 24 obtains the second distance d2 q corresponding to the identified second virtual image 43 as the distance from the reference position in the second direction D2 in the real space to the second subject. This process can be executed by the association process described later. Hereinafter, it will be described with reference to FIG. 7. q FIG. 7 shows a reference image 40b and a reference image 41b. The reference image 40b includes a partial image 40ob2. The partial image 40ob2 is a part of the reference image 40b in which the second subject ob2 imaged by the first camera 11 is drawn. The reference image 41b includes a partial image 41ob2. The partial image 41ob2 is a part of the reference image 41b in which the second subject ob2 imaged by the second camera 12 is drawn. The normal direction of the upper surface of the second subject ob2 coincides with the second direction D2. q q q q

[0052]

[0053] ​​​​​In FIG. 7, let q = 1, 2, 3. On the right side of the reference image 41b, the second virtual images 43b1, 43b2, and 43b3 are shown. The second virtual image 43b1 corresponds to the second distance d21. The second virtual image 43b2 corresponds to the second distance d22. The second virtual image 43b3 corresponds to the second distance d23.

[0054] In FIG. 7, for convenience of explanation, in each of the second virtual images 43b1 to 43b3, the partial image 40оb2 of the reference image 40b is indicated by a dashed line. In each of the second virtual images 43b1 to 43b3, the position of the partial image 41оb2, that is, the position of the second subject оb2, is the second distance d2 q which changes according to. Here, the position of the second subject оb2 in the second virtual image 43b2 and the position of the second subject оb2 in the reference image 40b coincide. That is, the distance from the reference position in the second direction D2 to the second subject оb2 is the second distance d22. The control unit 24 specifies the second virtual image 43b2 in which the position of the second subject оb2 in the reference image 40b and the position of the second subject оb2 are the closest among the second virtual images 43b1 to 43b3. The control unit 24 acquires the second distance d22 corresponding to the specified second virtual image 43b2 as the distance from the reference position in the second direction D2 in the real space to the second subject оb2.

[0055] The control unit 24 q That is, the straight-line distance from the reference position to the second subject may be calculated based on the distance from the reference position in the second direction D2 to the second subject. The control unit 24 may calculate the straight-line distance Z from the reference position to the second subject according to Equation (2). In the second direction D2, the normal vector n q of Equation (2) is the normal vector of the second virtual plane V2 q The unit vector b of Equation (2) is the unit vector in the direction of the camera coordinate system C11 corresponding to the direction from the origin of the normalized image coordinate system C1 to the pixel including the second subject.

[0056] <Association process> Hereafter, pixels included in reference image 40 will also be referred to as "reference pixels P40". Pixels included in reference image 41 will also be referred to as "reference pixels P41".

[0057] In the following, virtual plane V3 is any virtual plane whose normal direction is set based on the direction along the optical axis OX1 of the first camera 11. Virtual plane V3 may be the first virtual plane V1 or the second virtual plane V2. However, virtual plane V3 is not limited to the first virtual plane V1 and the second virtual plane V2. Virtual plane V3 may be any virtual plane as long as its normal direction is set based on the direction along the optical axis OX1 of the first camera 11. In other words, the angle between the normal direction of virtual plane V3 and the direction along the optical axis OX1 of the first camera 11 may be set arbitrarily.

[0058] In the following, the plane projection transformation H k (where k is an integer satisfying "1 ≤ k ≤ N") is the virtual plane V3 k Virtual point X3 located at k The light ray passing through to the first camera 11 is directed to the virtual point X3 k This is a conversion formula for converting the light to a ray passing through the virtual plane V3 to the second camera 12. k The set of integers (where k satisfies "1 ≤ k ≤ N"), i.e., the virtual plane V31~V3. N This may include both the first virtual plane V1 and the second virtual plane V2, or it may include only one of the first virtual plane V1 and the second virtual plane V2. Or, virtual plane V3 k The set of (where k is an integer satisfying "1 ≤ k ≤ N") may include virtual planes other than the first virtual plane V1 and the second virtual plane V2, in addition to or in place of either the first virtual plane V1 or the second virtual plane V2. Plane projection transformation H k This is expressed by equation (3).

number

[0059] The mapping process according to this embodiment will be described below.

[0060] The control unit 24 controls the first reference pixel P40 i,j Coordinates u i,j Obtain the data for coordinate u. i,j This is the coordinate of the image coordinate system set in the reference image 40. Row number i corresponds to the row of reference pixels P40 arranged in the reference image 40. Column number j corresponds to the column of reference pixels P40 arranged in the reference image 40.

[0061] The control unit 24 controls the first reference pixel P40 i,j Coordinates u i,j The light ray ρ1 to the first camera 11 -1 (K1 -1 u i,jProject back onto ). Parameter K1 is an intrinsic parameter of the first camera 11. That is, coordinate u i,j and parameter K1 -1 The coordinate K1 calculated by -1 u i,j This represents the coordinates in the normalized image coordinate system C1. Furthermore, parameter ρ1 is a parameter for projecting the light rays to the first camera 11 onto the coordinates in the normalized image coordinate system C1. That is, parameter ρ1 -1 This parameter is used to back-project the coordinates of the normalized image coordinate system C1 onto the rays directed to the first camera. Therefore, coordinate K1 -1 u i,j and parameter ρ1 -1 As a result, the light ray ρ1 to the first camera 11 -1 (K1 -1 u i,j ) is calculated. Here, the angle θ1 between the optical axis OX1 of the first camera 11 and the light ray to the first camera 11, and the height r=(x) of the image at coordinate (x,y) in the normalized image coordinate system C1 are calculated. 2 +y 2 ) 1 / 2 The relationship between them is expressed by equation (3). r = ρ(θ1) ≡ θ1 + k1θ1 + k2θ1 3 +k3θ1 5 +k4θ1 7 Formula (4) In equation (4), the coefficient k j (j=1~4) are coefficients that represent the distortion characteristics in the lens of the first camera 11.

[0062] The control unit 24 directs the light ray ρ1 to the first camera 11. -1 (K1 -1 u i,j ) to the plane projection transformation H k By means of the light ray H to the second camera 12 k ρ1 -1 (K1 -1 u i,j The control unit 24 converts the light ray H to the second camera 12 after conversion. k ρ1 -1 (K1 -1 u i,j Project ) onto reference image 41, and coordinate K2ρ2{H kρ1 -1 (K1 -1 u i,j )} is obtained. Coordinates K2ρ2{H k ρ1 -1 (K1 -1 u i,j )} represents the coordinates of the image coordinate system set in reference image 41. Parameter K2 is an intrinsic parameter of the second camera 12. Parameter ρ2 is a parameter for projecting the rays to the second camera 12 onto the coordinates of the normalized image coordinate system C2. Here, θ2 is the angle between the optical axis OX2 of the second camera 12 and the rays to the second camera 12, and r=(x) is the height of the image at coordinates (x,y) in the normalized image coordinate system C2. 2 +y 2 ) 1 / 2 The relationship between them is expressed by equation (5). r = ρ(θ2) ≡ θ2 + k1θ2 + k2θ2 3 +k3θ2 5 +k4θ2 7 Formula (5) In equation (5), the coefficient k j (j=1~4) are coefficients that represent the distortion characteristics in the lens of the second camera 12.

[0063] The control unit 24 controls the first reference pixel P40 i,j Coordinates u i,j In the coordinates K2ρ2{H k ρ1 -1 (K1 -1 u i,j The control unit 24 repeatedly performs the above process while changing the row number i and column number j, thereby associating all first reference pixels P40 i,j Coordinates u i,j In the coordinates K2ρ2{H k ρ1 -1 (K1 -1 u i,j )} are associated with each other.

[0064] The control unit 24 controls multiple reference pixels P41 k,i,j Among them, the first reference pixel P40 i,j Reference pixel P41 containing the same subject k,i,j Identify the reference pixel P41. k,i,jThe first reference pixel P40 i,j Coordinates u i,j Coordinates K2ρ2{H k ρ1 -1 (K1 -1 u i,j This is the reference pixel P41 located at ). In this embodiment, the control unit 24 performs a matching process to find the reference pixel P41 that contains the same subject as the first reference pixel. k,i,j Identify.

[0065] In the matching process, the control unit 24 controls the first reference pixel P40 i,j Feature quantities and reference pixel P41 k,i,j The control unit 24 extracts feature quantities. These feature quantities include, for example, at least one of the pixel's brightness and color. The control unit 24 may extract the feature quantities using any algorithm such as AKAZE (Accelerated-KAZE), ORB (Oriented FAST and Rotated BRIEF), or SIFT (Scale-Invariant Feature Transform). The feature quantities may be expressed in the form of AKAZE, ORB, or SIFT, or in various other forms.

[0066] In the matching process, the control unit 24 changes k and then adjusts the reference pixel P41 k,i,j While extracting the features of the first reference pixel P40 i,j Features and reference pixel P41 k,i,j Cost value C indicates the similarity between the feature quantities. k,i,j The cost value C is calculated. In this embodiment, the cost value C is calculated. k,i,j The smaller the value, the smaller the first reference pixel P40 i,j Features and reference pixel P41 k,i,jThe similarity between the feature quantities is assumed to be high. The control unit 24 calculates the cost value C for reference block B40 and reference block B41 by determining SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), ZSSD (Zero-mean Sum of Squared Difference), or ZNCC (Zero-means Normalized Cross Correlation), etc. k,i,j You may calculate the following. The reference block B40 has a cost value of C k,i,j The first reference pixel P40 is the one used for calculation. i,j This is a block of multiple reference pixels P40 centered around [the specified value]. Reference block B41 has a cost value of C k,i,j Reference pixel P41 that is subject to calculation k,i,j This is a block of multiple reference pixels P41 centered around [a specific point]. The reference block B40 and reference block B41 are, for example, 3x3 pixel blocks.

[0067] The control unit 24 calculates the cost value C k,i,j The data is stored in the storage unit 23. The cost value C stored in the storage unit 23 k,i,j This is used in the process of determining incorrect responses, which will be described later.

[0068] The control unit 24 controls the first reference pixel P40 i,j In each of these, the cost value C k,i,j We identify the k at which this is minimized. Below, when k=α, the first reference pixel P40 i,j Cost value C k,i,j This is to be minimized. In other words, the control unit 24 controls multiple reference pixels P41 k,i,j Among them, reference pixel P41 α,i,j The first reference pixel P40 i,j Reference pixel P41 containing the same subject k,i,j It is identified as such. The control unit 24 identifies the first reference pixel P40 i,j Distance d α The data is associated with the first reference pixel P40 and stored in the memory unit 23. The control unit 24 then processes the data. i,j and reference pixel P41 α,i,j To associate with it.

[0069] <Incorrect handling detection process> The control unit 24 performs a mismatch detection process. The mismatch detection process is performed on the first reference pixel P40 i,j and reference pixel P41 α,i,j This process determines whether the correspondence is incorrect or not.

[0070] The control unit 24 directs the light ray H to the second camera 12. α ρ1 -1 (K1 -1 u i,j ) Data is obtained. Light ray H to the second camera 12. α ρ1 -1 (K1 -1 u i,j ) is the first reference pixel P40 i,j Reference pixel P41, which is associated with this pixel. α,i,j It is a ray of light passing through [the object].

[0071] The control unit 24 directs the light ray H to the second camera 12. α ρ1 -1 (K1 -1 u i,j ) to the plane projection transformation H k The inverse transformation, i.e., the plane projection transformation H k -1 (where k is an integer satisfying "1 ≤ k ≤ N") Light rays H to the first camera 11 k -1 H α ρ1 -1 (K1 -1 u i,j The control unit 24 converts the light ray H to the first camera 11. k -1 H α ρ1 -1 (K1 -1 u i,j ) is projected onto the reference image 40, and the reference pixel P40 k,I,J Coordinates u k,I,J Obtain the coordinate u. k,I,Jare the coordinates in the image coordinate system set in the reference image 40. The row number I corresponds to the rows of the reference pixels P40 arranged in the reference image 40. The column number J corresponds to the columns of the reference pixels P40 arranged in the reference image 40. The control unit 24 obtains the coordinate u k,I,J by Equation (6). However, when the calculation result on the right side of Equation (6) is not an integer, the control unit 24 obtains the coordinate u k,I,J after rounding the calculation result on the right side of Equation (6) to an integer by rounding or the like u k,I,J = K1ρ1{H k -1 H α ρ1 -1 (K1 -1 u i,j )} Equation (6)

[0072] The control unit 24 identifies a second reference pixel P40 k,I,J that includes the same subject as the reference pixel P41 α,i,j among the plurality of reference pixels P40 k,I,J . In the present embodiment, the control unit 24 identifies a second reference pixel P40 α,i,j that includes the same subject as the reference pixel P41 k,I,J by matching processing

[0073] In the matching process, the control unit 24 changes k and identifies the k when the cost value C α,i,j indicating the similarity between the feature amount of the reference pixel P41 and the feature amount of the reference pixel P40 k,I,J becomes the minimum. The control unit 24 identifies the second reference pixel P40 k,I,J by identifying the k when the cost value C k,I,J becomes the minimum. Here, in the false correspondence determination process, the control unit 24 identifies the k when the cost value C k,I,J becomes the minimum using the data of the cost value C k,i,j stored in the storage unit 23. With such a configuration, in the false correspondence determination process, it is not necessary to calculate the cost value C k,I,J . Hereinafter, when k = β, the feature amount of the reference pixel P41 k,I,J and the reference pixel P40 α,i,j and the reference pixel P40 k,I,JThe cost value C indicating the similarity with the feature amount k,I,J shall be minimized. That is, the control unit 24 selects the plurality of reference pixels P40 k,I,J among the reference pixels P40 β,I,J in which the reference pixel P40 α,i,j contains the same subject as the reference pixel P41 β,I,J as the second reference pixel P40

[0074] The control unit 24 determines whether the association between the first reference pixel P40 i,j and the second reference pixel P40 β,I,J is a mis-association based on the first reference pixel P40 i,j and the reference pixel P41 α,i,j or not.

[0075] In the present embodiment, the control unit 24 determines whether the association between the first reference pixel P40 i,j and the second reference pixel P40 β,I,J is a mis-association based on the distance between the first reference pixel P40 i,j and the reference pixel P41 α,i,j The control unit 24 may obtain the distance between the first reference pixel P40 i,j and the reference pixel P40 β,I,J by the difference from the coordinates u i,j of the first reference pixel P40 i,j and the coordinates K1ρ1{H β,I,J of the reference pixel P40 β -1 H α ρ1 -1 (K1 -1 u i,j )}. Here, when the association between the first reference pixel P40 i,j and the reference pixel P41 α,i,j is not a mis-association, ideally, the first reference pixel P40 i,j and the second reference pixel P40 β,I,J become the same reference pixel P40. Therefore, when the association is not a mis-association, the distance between the first reference pixel P40[[ID=6 )] i,j and the second reference pixel P40 β,I,J becomes zero. However, in reality, the virtual plane V3 kBecause the number of pixels is finite, or because there is distortion in the lenses of the first camera 11 and the second camera 12, the first reference pixel P40 i,j and the second reference pixel P40 β,I,J Therefore, it will not be the same reference pixel P40. In other words, virtual plane V3 k The number of elements is N, or the parameters ρ1, ρ2, K1, K2 are included in equation (6), etc., which results in the first reference pixel P40 i,j and the second reference pixel P40 β,I,J The distance between them is not zero.

[0076] Therefore, the control unit 24 controls the first reference pixel P40 i,j and the second reference pixel P40 β,I,J If it is determined that the distance between the first reference pixel P40 falls below the threshold, i,j and reference pixel P41 α,i,j The control unit 24 determines that the correspondence is not incorrect. Meanwhile, the control unit 24 determines that the first reference pixel P40 i,j and the second reference pixel P40 β,I,J If it is determined that the distance between the first reference pixel P40 is greater than or equal to the threshold, i,j and reference pixel P41 α,i,j The correspondence is determined to be incorrect. In this embodiment, the first reference pixel P40 i,j and the second reference pixel P40 β,I,J The distance between the two points is given by the number of pixels. The threshold may be set based on the degree of lens distortion of the first camera 11 and the second camera 12. The threshold is, for example, 1 pixel.

[0077] The control unit 24 repeatedly executes the above-described process while changing the row number i and column number j. By repeatedly executing the above-described process, the control unit 24 determines the first reference pixel P40 i,j and reference pixel P41 α,i,j From among the matching pairs, we detect the ones that are incorrectly matched.

[0078] <Processing to obtain distance> The control unit 24 controls the reference pixel P40 i,jThe distance to the subject included in the image is obtained. For example, the control unit 24 obtains the first reference pixel P40 from the storage unit 23. i,j The distance d associated with α The control unit 24 obtains the obtained distance d. α The linear distance Z from the reference position to the subject is calculated using equation (2). The control unit 24 then displays the calculated linear distance Z to the first reference pixel P40 i,j The distance to the subject included in the first reference pixel P40 is acquired. Alternatively, the control unit 24 acquires the distance to the subject included in the first reference pixel P40 i,j The distance d associated with α The first reference pixel P40 i,j The distance to the subject included in the image may also be obtained.

[0079] <Generation process for depth images> The control unit 24 controls the first reference pixel P40 i,j When the distance to the subject included in the image is obtained, a distance image is generated based on the obtained distance information to the subject. The distance image contains multiple distance pixels. The pixel value of each distance pixel in the distance image corresponds to the distance to the subject. The control unit 24 uses the pixel value of the distance pixel to determine the first reference pixel P40 corresponding to that distance pixel. i,j A distance image is generated by creating it based on the distance information to the subject contained within the image.

[0080] For example, the control unit 24 generates a distance image 44 as shown in Figure 8. The distance image 44 includes a plurality of distance pixels P44. Each of the plurality of distance pixels P44 corresponds to each of the plurality of first reference pixels P40. In the plurality of distance pixels P44 arranged in the distance image 44, the distance pixel P44 located at row number i and column number j is "distance pixel P44 i,j It is also written as "Distance Pixel P44". i,j The first reference pixel P40 i,j The control unit 24 corresponds to the distance pixel P44 i,j The pixel value of the first reference pixel P40 i,j It is generated based on the distance to the subject included in the image.

[0081] The control unit 24 controls the first reference pixel P40 i,jand reference pixel P41 α,i,j If it is determined that the correspondence with the first reference pixel P40 is incorrect, i,j P44 distance pixels corresponding to this i,j You may disable it. Disabling a distance pixel means that the distance pixel does not contain distance information. For example, disabling a distance pixel means setting the pixel value of the distance pixel to zero.

[0082] For example, the multiple distance pixels P44 shown in Figure 8 include multiple distance pixels P44a and multiple distance pixels P44b. In Figure 8, hatching is applied to the distance pixels P44a. Hatching is not applied to the distance pixels P44b. The control unit 24 controls the first reference pixel P40 i,j and reference pixel P41 α,i,j This pixel was invalidated because its association with the reference pixel P41 was determined to be incorrect. Distance pixel P44b is the first reference pixel P40, which was determined not to be incorrectly associated with the reference pixel P41. i,j This corresponds to the following. The pixel value of distance pixel P44b is generated based on the distance information to the subject contained in the first reference pixel P40.

[0083] The control unit 24 may interpolate the pixel value of a disabled distance pixel among a plurality of distance pixels using the pixel values ​​of other distance pixels. For example, in Figure 8, the control unit 24 may interpolate the pixel value of distance pixel P44a using the pixel value of distance pixel P44b located around distance pixel P44a.

[0084] The control unit 24 controls the first reference pixel P40 i,j and reference pixel P41 α,i,j Even if it is determined that the correspondence with the first reference pixel P40 is incorrect, i,j It is not necessary to disable the distance pixel corresponding to the first reference pixel P40. i,j and reference pixel P41 α,i,jEven if it is determined that the correspondence is incorrect, the pixel values ​​of all distance pixels in the distance image may be generated based on the distance information to the subject. In this case, the control unit 24 may output the distance image and the confidence score (described later) to an external device via the output unit 22.

[0085] <Reliability Assignment Process> The control unit 24 controls the first reference pixel P40 i,j When the distance to a subject included in the image is obtained, a confidence level may be assigned to the obtained distance to the subject. The confidence level is an indicator of the reliability of the obtained distance to the subject. In this embodiment, a higher confidence level indicates higher reliability of the distance to the subject to which the confidence level has been assigned.

[0086] The control unit 24 controls the first reference pixel P40 i,j and the second reference pixel P40 β,I,J Based on the distance between, the first reference pixel P40 i,j A confidence level may be assigned to the distance to the subject included in the first reference pixel P40. i,j and reference pixel P41 α,i,j If the mapping is not incorrect, ideally, the first reference pixel P40 i,j and the second reference pixel P40 β,I,J The distance between them becomes zero. Therefore, the control unit 24 determines the first reference pixel P40 i,j and the second reference pixel P40 β,I,J The shorter the distance between the two points, the higher the confidence level that can be assigned.

[0087] <Output Processing> The control unit 24 outputs the generated distance image to an external device via the output unit 22. If the control unit 24 has assigned a confidence level to the distance to the subject, it may also output the distance image and the confidence level to an external device via the output unit 22.

[0088] (Operation of the stereo camera device) FIG. 9 and FIG. 10 are flowcharts showing the procedure of the association process. The association process as shown in FIGS. 9 and 10 corresponds to an example of the image processing method according to the present embodiment. When the stereo camera 10 starts imaging, the control unit 24 of the image processing apparatus 20 starts the process of step S1 as shown in FIG. 9.

[0089] The control unit 24 acquires the data of the reference image 40 and the data of the reference image 41 by the acquisition unit 21 (step S1).

[0090] The control unit 24 i,j acquires the data of the coordinate u i,j of the first reference pixel P40 (step S2).

[0091] The control unit 24 i,j projects the coordinate u i,j of the first reference pixel P40 onto the light ray ρ1 -1 (K1 -1 u i,j ) to the first camera 11 (step S3).

[0092] The control unit 24 -1 converts the light ray ρ1 -1 to the first camera 11 (K1 i,j u k ) to the light ray H k ρ1 -1 to the second camera 12 by the planar projective transformation H -1 u i,j ) (step S4).

[0093] The control unit 24 i,j associates the coordinate u i,j of the first reference pixel P40 with the coordinate K2ρ2{H k ρ1 -1 (K1 -1 u i,j )} (step S5).

[0094] The control unit 24 i,j associates the coordinate u i,j of all the first reference pixels P40 with the coordinate K2ρ2{H k ρ1 -1 (K1-1 u i,j The control unit 24 determines whether or not the )} has been associated (step S6). i,j Coordinates u i,j In the coordinates K2ρ2{H k ρ1 -1 (K1 -1 u i,j If it is not determined that the )} has been associated (step S6: NO), the row number i and column number j are changed, and the process returns to step S1. Meanwhile, the control unit 24 checks all first reference pixels P40 i,j Coordinates u i,j In the coordinates K2ρ2{H k ρ1 -1 (K1 -1 u i,j If it is determined that the )} has been matched (Step S6: YES), the process proceeds to Step S7 as shown in Figure 10.

[0095] In the process of step S7 as shown in Figure 10, the control unit 24 controls the first reference pixel P40 i,j Features and multiple reference pixels P41 k,i,j The control unit 24 extracts the features of the cost value C. k,i,j The control unit 24 calculates the cost value C calculated in the process of step S8 (step S8). k,i,j The data is stored in the storage unit 23 (step S9).

[0096] The control unit 24 controls the first reference pixel P40 i,j In each of these, the cost value C k,i,j The control unit 24 determines the k at which the value is minimized (step S10). i,j and reference pixel P41 α,i,j This is associated with (step S11). In the process of step S11, the control unit 24 associates the first reference pixel P40 i,j Distance d α The data is associated with each other and stored in the memory unit 23.

[0097] Figures 11 and 12 are flowcharts showing the procedure for determining incorrect matching. The incorrect matching determination process shown in Figures 11 and 12 corresponds to an example of the image processing method according to this embodiment. After processing in step S11 as shown in Figure 10, the control unit 24 of the image processing apparatus 20 starts processing in step S21 as shown in Figure 11.

[0098] The control unit 24 directs the light ray H to the second camera 12. α ρ1 -1 (K1 -1 u i,j ) obtain the data (step S21).

[0099] The control unit 24 directs the light ray H to the second camera 12. α ρ1 -1 (K1 -1 u i,j ) to the plane projection transformation H k The inverse transformation, i.e., the plane projection transformation H k -1 Therefore, the light ray H to the first camera 11 k -1 H α ρ1 -1 (K1 -1 u i,j Convert to (step S22).

[0100] The control unit 24 sends a light ray H to the first camera 11. k -1 H α ρ1 -1 (K1 -1 u i,j ) is projected onto the reference image 40, and the reference pixel P40 k,I,J Coordinates u k,I,J Obtain (step S23).

[0101] The control unit 24 receives the cost value C from the storage unit 23. k,i,j The data of cost value C k,I,J This data is obtained (step S24).

[0102] The control unit 24 changes the cost value C while changing k. k,I,JThe control unit 24 determines the k at which the cost value C is minimized (step S25). k,I,J By identifying the k at which is minimized, the second reference pixel P40 β,I,J Identify.

[0103] After executing the process in step S25, the control unit 24 proceeds to the process in step S26, as shown in Figure 12.

[0104] The control unit 24 controls the first reference pixel P40 i,j and the second reference pixel P40 β,I,J It is determined whether the distance between the two points falls below a threshold (step S26).

[0105] The control unit 24 controls the first reference pixel P40 i,j and the second reference pixel P40 β,I,J If it is determined that the distance between the first reference pixel P40 is below the threshold (step S26: YES), i,j and reference pixel P41 α,i,j The control unit 24 determines that the correspondence is not a mismatch (step S27). Meanwhile, the control unit 24 determines the first reference pixel P40 i,j and the second reference pixel P40 β,I,J If it is determined that the distance between the first reference pixel P40 is greater than or equal to the threshold (step S26: NO), then the first reference pixel P40 i,j and reference pixel P41 α,i,j The system determines that the correspondence is incorrect (step S28).

[0106] In step S29, all first reference pixels P40 i,j and reference pixel P41 α,i,j The control unit 24 determines whether the processes in steps S1 to S28 have been executed regarding the correspondence with all first reference pixels P40 i,j and reference pixel P41 α,i,j If it is not determined that the processes in steps S1 to S28 have been executed regarding the correspondence with (step S29: NO), the row number i and column number j are changed, and the process returns to step S21. The control unit 24 checks all first reference pixels P40 i,j and reference pixel P41 α,i,jIf it is determined that the processes in steps S1 to S28 have been executed regarding the correspondence (step S29: YES), the process for determining incorrect correspondence is terminated.

[0107] Thus, in the image processing apparatus 20 according to this embodiment, the control unit 24 performs a plurality of planar projection transformations H k by the first reference pixel P40 i,j Coordinates u i,j The coordinates K2ρ2{H k ρ1 -1 (K1 -1 u i,j )} Convert to coordinates. The control unit 24 converts to multiple reference pixels P41 k,i,j Among them, the first reference pixel P40 i,j Reference pixel P40 containing the same subject α,i,j The control unit 25 identifies the first reference pixel P40 i,j and reference pixel P40 α,i,j A mapping process is performed to associate the two. Here, as described above, the plane projection transformation H k (where k is an integer satisfying "1 ≤ k ≤ N") is the virtual plane V3 k Virtual point X3 located at k The light ray passing through to the first camera 11 is directed to the virtual point X3 k This is a conversion formula for converting the light to a ray passing through the virtual plane V3 to the second camera 12. k This is virtual plane V3 k The normal direction of the first virtual plane V1 may be any virtual plane set based on the direction along the optical axis OX1 of the first camera 11. In other words, in this embodiment, the control unit 24 sets the first virtual plane V1 k In addition to processing using the plane sweep method, the first virtual plane V1 k Other virtual plane V3 k Processing using this method can be performed. With this configuration, the distance to the subject can be measured with greater accuracy, as described below.

[0108] As the first comparative example, virtual plane V3 k The set of integers (where k satisfies "1 ≤ k ≤ N"), i.e., the virtual plane V31~V3. NLet's consider the case where it includes only the first virtual plane V1. In other words, in the first comparative example, the control unit 24 performs only processing by the plane sweep method, and the first reference pixel P40 i,j and reference pixel P41 α,i,j A mapping process is performed to associate the two. Here, as shown in Figure 7, the second subject оb2 does not contain any planes parallel to the first virtual plane V1. In other words, the normal direction of any plane of the second subject оb2 is parallel to the first virtual plane V1. k The normal direction, i.e., the first direction D1, does not coincide with the first distance d1 in the first direction D1. k Multiple planar projection transformations H while varying k Multiple reference pixels P41 k,i,j Even if data is acquired, the first reference pixel P40 i,j Reference pixel P41 containing the same subject α,i,j It is not possible to identify it with high accuracy.

[0109] In contrast to the first comparative example, in this embodiment, virtual plane V3 k This is the first virtual plane V1 k Other virtual plane V3 k This configuration may include the first reference pixel P40 that includes the subject, even if the subject does not have a plane parallel to the first virtual plane V1. i,j and reference pixel P41 α,i,j This allows for a highly accurate correspondence between the two. For example, in Figure 7, the virtual plane V3 k If the second virtual plane V2 is included, the control unit 24 includes the first reference pixel P40 which includes the second subject оb2. i,j and reference pixel P41 α,i,j This allows for the association between the two. Here, for example, if the second subject оb2 is the unevenness of the road surface, in this embodiment, the first reference pixel P40 including the second subject оb2 i,j and reference pixel P41 α,i,j By associating these with each other, the distance to the unevenness of the road surface can be calculated. Therefore, in this embodiment, the road surface shape can be detected with higher accuracy.

[0110] As a second comparative example, we consider measuring the distance to a subject using a reference image and a reference image captured with a fisheye lens. Figure 13 shows a diagram illustrating image 40X captured with a fisheye lens. The solid lines in image 40X connect points where the height from the road surface is the same. Image 40X includes regions R1 and R2. In region R1, the spacing of the solid lines is wider than in region R2. In region R1, because the spacing of the solid lines is wide, when distortion correction is applied to image 40X, points with different heights from the road surface correspond to different pixels. However, in region R2, because the spacing of the solid lines is narrow, when distortion correction is applied to image 40X, points with different heights from the road surface may correspond to the same pixel. In region R2, because points with different heights from the road surface correspond to the same pixel, even if the plane sweep method is applied to a reference image and a reference image like image 40X, it may not be possible to obtain the distance to the subject.

[0111] In contrast to this second comparative example, in this embodiment, the control unit 24 controls the virtual plane V3 k This is the first virtual plane V1 obtained by the plane sweep method. k Other virtual plane V3 k It can include virtual plane V3. k If set appropriately, the control unit 24 will determine the first reference pixel P40 which includes the subject that is captured in region R2. i,j and reference pixel P41 α,i,j This allows for a correspondence between the two. In other words, in this embodiment, even if fisheye lenses are used in the optical systems of the first camera 11 and the second camera 12, the distance to the subject can be obtained. In this embodiment, by using fisheye lenses in the optical systems of the first camera 11 and the second camera 12, the distance to a subject located in a wider area can be obtained.

[0112] Furthermore, in this embodiment, the control unit 24 controls the first reference pixel P40 i,j and the second reference pixel P40 β,I,J Based on this, the first reference pixel P40 i,j and reference pixel P41 α,i,jThe control unit 24 determines whether the correspondence is incorrect. Here, occlusion regions may occur in stereo images. An occlusion region is a region that is captured in one of the reference image 40 and the reference image 41, but not in the other. When an occlusion region occurs, the control unit 24 determines whether the first reference pixel P40 i,j There are cases where the first reference pixel P40 is mistakenly associated with the reference pixel P41. Also, if periodic patterns such as grid patterns are included in the reference image 40 and the reference image 41, the control unit 24 will... i,j There are cases where the first reference pixel P40 is mistakenly associated with the reference pixel P41. In such cases, in this embodiment, the first reference pixel P40 i,j and the second reference pixel P41 β,I,J Based on this, the first reference pixel P40 i,j and reference pixel P41 α,i,j It is possible to determine whether the correspondence is incorrect or not. With this configuration, the distance to the subject can be measured with greater accuracy.

[0113] Therefore, according to this embodiment, it is possible to provide a technology that can measure the distance to the subject with greater accuracy.

[0114] Furthermore, in this embodiment, the lenses of the first camera 11 and the second camera 12 may each be wide-angle lenses. In this case, the control unit 24, in the mismatch determination process, determines the first reference pixel P40 i,j Reference pixel P41, which is associated with this pixel. α,i,j Light ray H passing through to the second camera 12 α ρ1 -1 (K1 -1 u i,j The control unit 24 may acquire data of the light ray H to the second camera that it has acquired. α ρ1 -1 (K1 -1 u i,j ) to multiple planar projection transformations H k The inverse transformation, i.e., multiple planar projection transformations H k -1 By means of the light ray H to the first camera 11 k -1 Hα ρ1 -1 (K1 -1 u i,j The light ray H to the first camera after conversion may be converted. The control unit 24 sends the converted light ray H to the first camera. k -1 H α ρ1 -1 (K1 -1 u i,j By projecting ) onto the reference image 40, multiple reference pixels P40 k,I,J Coordinates u k,I,J You may obtain the following. Here, if wide-angle lenses are used for the lenses of the first camera 11 and the second camera 12, the degree of distortion of the lenses of the first camera 11 and the second camera 12 will increase. When the degree of distortion of the lenses of the first camera 11 and the second camera 12 increases, a simple transformation formula from the image coordinate system set in the reference image 40 to the image coordinate system set in the reference image 41 will result in the reference pixel P41 k,I,J The coordinates of the object cannot be obtained with high accuracy. In this embodiment, the light ray H to the second camera 12 α ρ1 -1 (K1 -1 u i,j ) to the plane projection transformation H k The inverse transformation of the light ray H to the first camera 11 k -1 H α ρ1 -1 (K1 -1 u i,j ) can be converted to this. With this configuration, even if the degree of lens distortion of the first camera 11 and the second camera 12 is large, multiple reference pixels P40 k,I,J The coordinates of the first reference pixel P40 can be obtained with high accuracy. Therefore, in this embodiment, the coordinates of the first reference pixel P40 i,j and the second reference pixel P40 β,I,J Based on this, the first reference pixel P40 i,j and reference pixel P41 α,i,j It is possible to accurately determine whether the correspondence is incorrect or not.

[0115] Furthermore, in this embodiment, multiple virtual planes V3 kThis may include both the first virtual plane V1 and the second virtual plane V2. With this configuration, the control unit 24, for a second subject оb2 that does not include a plane parallel to the first virtual plane V1 as described above, includes the first reference pixel P40 which includes the second subject оb2. i,j and reference pixel P41 α,i,j This allows for the correspondence between the two. Here, depending on the subject, it may include both a plane parallel to the first virtual plane V1 and a plane parallel to the second virtual plane V2, thus falling under both the first and second subjects. For such subjects that fall under both the first and second subjects, the control unit 24 determines the first reference pixel P40 i,j There are cases where the correspondence between the first reference pixel P40 and the reference pixel P41 is incorrect. Even in such cases, the control unit 24 will i,j and the second reference pixel P40 β,I,J Based on this, the first reference pixel P40 i,j It is possible to determine whether the correspondence between the virtual plane V3 and the reference pixel P41 is incorrect. k Even if the data includes both the first virtual plane V1 and the second virtual plane V2, it is possible to determine whether or not it is a mismatch, thus enabling accurate measurement of the distance to a greater number of subjects.

[0116] Furthermore, in this embodiment, the control unit 24, in the mismatch determination process, uses the cost value C calculated in the matching process. k,i,j Based on this, multiple reference pixels P40 k,I,J Among them, reference pixel P41 α,i,j Second reference pixel P40 containing the same subject β,I,J The following may be identified. With this configuration, the mismatch detection process does not need to perform the process of extracting features. Also, in the mismatch detection process, the cost value C k,i,j The process of calculating the reference pixel P41 does not need to be performed. As a result, the control unit 24 can perform the calculation of the reference pixel P41 at a faster speed. α,i,j Second reference pixel P40 containing the same subject β,I,J It is possible to identify this.

[0117] Furthermore, in this embodiment, the first reference pixel P40 i,jand the second reference pixel P40 β,I,J A confidence level may be assigned to the distance to the subject based on the distance between the two. The control unit 24 may output the distance information to the subject and the confidence level to an external device via the output unit 22. The control unit 24 may output the distance image as information about the distance to the subject and the confidence level to an external device via the output unit 22. With this configuration, the external device can appropriately process the distance information to the subject by referring to the confidence level.

[0118] In one embodiment, (1) the image processing apparatus is An acquisition unit that acquires data of a reference image generated by imaging at a reference position and data of a reference image generated by imaging at a position different from the reference image, The system includes a control unit that performs a mapping process to associate the first reference pixel of the reference image with the specified reference pixel by performing multiple planar projection transformations, the first reference pixel with the specified reference pixel, the first reference pixel with the specified reference pixel, and the first reference pixel with the specified reference pixel. The control unit, The coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations are transformed into the coordinates of the plurality of reference pixels included in the reference image, a second reference pixel is identified among the plurality of reference pixels whose coordinates have been transformed and which contains the same subject as the reference pixel associated with the first reference pixel, and a determination process is performed to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations transform rays directed to the camera that generated the reference image, which pass through a plurality of virtual points located in each of the plurality of virtual planes, into rays directed to the camera that generated the reference image, which pass through the plurality of virtual points.

[0119] (2) In the image processing apparatus described in (1) above, The aforementioned reference image and the aforementioned reference image may be images captured using a wide-angle lens.

[0120] (3) In the image processing apparatus described in (1) or (2) above, The control unit, in the determination process, The data of the light ray passing through the reference pixel associated with the first reference pixel is acquired from the camera that generated the aforementioned reference image. The acquired light rays may be converted into light rays directed to the camera that generated the reference image by an inverse transformation of the plurality of planar projection transformations, and the coordinates of the plurality of reference pixels may be obtained by projecting the converted light rays onto the reference image.

[0121] (4) In the image processing apparatus described in any one of (1) to (3) above, If the control unit determines in the determination process that the distance between the first reference pixel and the second reference pixel is greater than or equal to a threshold, it may determine that the correspondence between the first reference pixel and the reference pixel is incorrect.

[0122] (5) In the image processing apparatus described in any one of (1) to (4) above, The normal direction of the virtual plane is set based on the optical axis direction of the camera that generated the reference image. The control unit may obtain the distance to the subject included in the first reference pixel based on the distance from the reference position to the virtual plane used to obtain the coordinates of the reference pixel associated with the first reference pixel.

[0123] (6) In the image processing apparatus described in any one of (1) to (5) above, The aforementioned plurality of virtual planes include both the first virtual plane and the second virtual plane, The normal direction of the first virtual plane coincides with the first direction, which is the optical axis direction of the camera that generated the reference image. The first virtual plane is located at a first distance from the reference position along the first direction, The normal direction of the second virtual plane coincides with a second direction that is different from the first direction. The second virtual plane may be located at a second distance from the reference position along the second direction.

[0124] (7) In the image processing apparatus described in any one of (1) to (6) above, The control unit, In the aforementioned mapping process, the feature quantities of the first reference pixel and the feature quantities of the plurality of reference pixels are extracted, a plurality of cost values ​​are calculated that represent the similarity between the extracted feature quantities of the first reference pixel and the feature quantities of each of the plurality of reference pixels, and based on the plurality of cost values, a reference pixel containing the same subject as the first reference pixel is identified. In the determination process, a second reference pixel containing the same subject as the reference pixel associated with the first reference pixel may be identified based on the multiple cost values ​​calculated in the correspondence process.

[0125] (8) In the image processing apparatus described in any one of (1) to (7) above, The control unit may generate a distance image that includes distance pixels having pixel values ​​based on information about the distance to the subject.

[0126] (9) In any one of the image processing devices described in (1) to (8) above, The control unit may disable the distance pixel corresponding to the first reference pixel that it has determined to be an incorrect match.

[0127] (10) In the image processing apparatus described in any one of (1) to (9) above, The control unit may interpolate the pixel value of the disabled distance pixel among the plurality of distance pixels using the pixel values ​​of the other distance pixels.

[0128] (11) In the image processing apparatus described in any one of (1) to (10) above, It also has an output section, The control unit may output the distance image data to an external device using the output unit.

[0129] (12) In the image processing apparatus described in any one of (1) to (11) above, The control unit may assign a confidence level to the acquired distance to the subject based on the distance between the first reference pixel and the second reference pixel.

[0130] (13) In the image processing apparatus described in any one of (1) to (12) above, It also has an output section, The control unit may output information about the distance to the subject and the reliability to an external device via the output unit.

[0131] In one embodiment, (14) the stereo camera device is A stereo camera including a first camera that generates a reference image by capturing an image at a reference position, and a second camera that generates a reference image by capturing an image at a position different from the reference image, The image processing device includes: an image processing device that performs a mapping process to associate the first reference pixel of the reference image with the coordinates of a plurality of reference pixels included in the reference image by performing a plurality of planar projection transformations, to identify a reference pixel that contains the same subject as the first reference pixel among the plurality of reference pixels whose coordinates have been transformed, and to associate the first reference pixel with the identified reference pixel; The aforementioned image processing device is The coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations are transformed into the coordinates of the plurality of reference pixels included in the reference image, a second reference pixel is identified among the plurality of reference pixels whose coordinates have been transformed and which contains the same subject as the reference pixel associated with the first reference pixel, and a determination process is performed to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations convert rays from a plurality of virtual points located in each of the plurality of virtual planes to rays from a plurality of virtual points to a second camera.

[0132] In one embodiment, (15) the image processing method is The process involves acquiring data for a reference image generated by imaging at a reference position, and data for a reference image generated by imaging at a position different from the reference image. The process involves transforming the coordinates of the first reference pixel of the reference image into the coordinates of multiple reference pixels included in the reference image using multiple planar projection transformations, identifying a reference pixel among the multiple reference pixels whose coordinates have been transformed that contains the same subject as the first reference pixel, and performing a mapping process to associate the first reference pixel with the identified reference pixel. The process includes: converting the coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations into the coordinates of a plurality of reference pixels included in the reference image; identifying a second reference pixel among the plurality of reference pixels whose coordinates have been converted that contains the same subject as the reference pixel associated with the first reference pixel; and performing a determination process to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations transform rays directed to the camera that generated the reference image, which pass through a plurality of virtual points located in each of the plurality of virtual planes, into rays directed to the camera that generated the reference image, which pass through the plurality of virtual points.

[0133] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art will find it easy to make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are within the scope of this disclosure. For example, the functions, etc., included in each functional part can be rearranged in a logically consistent manner. Multiple functional parts, etc., may be combined into one or divided. The embodiments relating to this disclosure described above are not limited to being implemented strictly according to the respective embodiments, but can be implemented by combining features or omitting some as appropriate. In other words, the contents of this disclosure can be modified and altered in various ways based on this disclosure by those skilled in the art. Therefore, these modifications and alterations are within the scope of this disclosure. For example, in each embodiment, each functional part, each means, each step, etc., can be added to other embodiments in a logically consistent manner, or replaced with each functional part, each means, each step, etc., from other embodiments. Also, in each embodiment, multiple functional parts, each means, or each step, etc., can be combined into one or divided. Furthermore, the embodiments of this disclosure described above are not limited to being implemented strictly according to the respective embodiments described, but can also be implemented by combining or omitting some of the features as appropriate.

[0134] For example, the control unit 24 of the image processing device 20 may perform any image processing, such as brightness adjustment, contrast adjustment, or gamma correction, on the reference image 40 and the reference image 41. Alternatively, the first camera 11 and the second camera 12 may each perform any image processing, such as brightness adjustment, contrast adjustment, or gamma correction, on the reference image 40 and the reference image 41, respectively.

[0135] For example, in the embodiment described above, the first camera 11 and the second camera 12 were described as being positioned along the left-right direction of the moving body 30, as shown in Figure 3. However, the positions of the first camera 11 and the second camera 12 are not limited to this. As another example, the first camera 11 and the second camera 12 may be positioned along the up-down direction of the moving body 30.

[0136] For example, in the embodiment described above, it was explained that the first camera 11 generates a reference image 40 and the second camera 12 generates a reference image 41. However, the reference image 40 and the reference image 41 may be generated by a single camera. That is, the first camera and the second camera may be the same camera. In this case, the camera generates the reference image 40 by imaging at a reference position and generates the reference image 41 by imaging at a position different from the reference position by moving. [Explanation of symbols]

[0137] 1. Stereo camera device 10 Stereo Cameras 11. Camera 1 12. Second camera 20 Image Processing Devices 21 Acquisition Department 22 Output section 23 Memory section 24 Control Unit 30 Mobile Units Reference images 40, 40a, 40b 40оb1,40оb2,41оb1,41оb2 Partial images 40X image Reference images 41, 41a, 41b 42,42a1,42a2,42a3,42 q First virtual image 43, 43b1, 43b2, 43b3, 43 q Second virtual image 45 Distance image P40 Reference Pixel P41 Reference Pixel P44, P44a, P44b distance pixels V1, V1 k ,V1 q First virtual plane V2, V2 k ,V2 q Second virtual plane V3, V3 k virtual plane

Claims

1. An acquisition unit that acquires data of a reference image generated by imaging at a reference position and data of a reference image generated by imaging at a position different from the reference image, The system includes a control unit that performs a mapping process to associate the first reference pixel of the reference image with the specified reference pixel by performing multiple planar projection transformations, the first reference pixel with the specified reference pixel by performing multiple planar projection transformations, the second reference pixel with the specified reference pixel by performing a mapping process, and the third reference pixel with the specified reference pixel. The control unit, The coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations are transformed into the coordinates of the plurality of reference pixels included in the reference image, a second reference pixel is identified among the plurality of reference pixels whose coordinates have been transformed that contains the same subject as the reference pixel associated with the first reference pixel, and a determination process is performed to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the distance between the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations transform rays to the camera that generated the reference image, which pass through a plurality of virtual points located in each of the plurality of virtual planes, into rays to the camera that generated the reference image, which pass through the plurality of virtual points. Image processing device.

2. The image processing apparatus according to claim 1, wherein the reference image and the reference image are images captured with a wide-angle lens.

3. The control unit, in the determination process, The data of the light ray passing through the first reference pixel, which is associated with the first reference pixel, is obtained from the camera that generated the aforementioned reference image. The image processing apparatus according to claim 2, wherein the acquired light ray is converted into a light ray directed to a camera that generated the reference image by an inverse transformation of the plurality of planar projection transformations, and the coordinates of the plurality of reference pixels are obtained by projecting the converted light ray onto the reference image.

4. The image processing apparatus according to claim 1, wherein the control unit determines in the determination process that the distance between the first reference pixel and the second reference pixel is greater than or equal to a threshold, and determines that the correspondence between the first reference pixel and the reference pixel is incorrect.

5. The normal direction of the virtual plane is set based on the optical axis direction of the camera that generated the reference image. The image processing apparatus according to claim 1, wherein the control unit obtains the distance to the subject included in the first reference pixel based on the distance from the reference position to the virtual plane used to obtain the coordinates of the reference pixel associated with the first reference pixel.

6. The aforementioned plurality of virtual planes include both the first virtual plane and the second virtual plane, The normal direction of the first virtual plane coincides with the first direction, which is the optical axis direction of the camera that generated the reference image. The first virtual plane is located at a first distance from the reference position along the first direction, The normal direction of the second virtual plane coincides with a second direction that is different from the first direction. The image processing apparatus according to claim 1, wherein the second virtual plane is located at a second distance from the reference position along the second direction.

7. The control unit, In the aforementioned mapping process, the feature quantities of the first reference pixel and the feature quantities of the plurality of reference pixels are extracted, a plurality of cost values ​​are calculated that represent the similarity between the extracted feature quantities of the first reference pixel and the feature quantities of each of the plurality of reference pixels, and based on the plurality of cost values, a reference pixel containing the same subject as the first reference pixel is identified. The image processing apparatus according to claim 1, wherein, in the determination process, a second reference pixel containing the same subject as the reference pixel associated with the first reference pixel is identified based on the plurality of cost values ​​calculated in the association process.

8. The image processing apparatus according to claim 5, wherein the control unit generates a distance image including distance pixels having pixel values ​​based on information about the distance to the subject.

9. The image processing apparatus according to claim 8, wherein the control unit invalidates the distance pixel corresponding to the first reference pixel that has been determined to be an incorrect match.

10. The image processing apparatus according to claim 9, wherein the control unit interpolates the pixel value of the invalidated distance pixel among the plurality of distance pixels with the pixel values ​​of the other distance pixels.

11. It also has an output section, The image processing apparatus according to any one of claims 8 to 10, wherein the control unit outputs the distance image data to an external device using the output unit.

12. The image processing apparatus according to claim 5, wherein the control unit assigns a confidence level to the acquired distance to the subject based on the distance between the first reference pixel and the second reference pixel.

13. It also has an output section, The image processing apparatus according to claim 12, wherein the control unit outputs information on the distance to the subject and the reliability to an external device via the output unit.

14. A stereo camera including a first camera that generates a reference image by capturing an image at a reference position, and a second camera that generates a reference image by capturing an image at a position different from the reference image, The image processing device includes: an image processing device that performs a mapping process to associate the first reference pixel of the reference image with the coordinates of a plurality of reference pixels included in the reference image by performing a plurality of planar projection transformations, to identify a reference pixel that contains the same subject as the first reference pixel among the plurality of reference pixels whose coordinates have been transformed, and to associate the first reference pixel with the identified reference pixel; The aforementioned image processing device is The coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations are transformed into the coordinates of the plurality of reference pixels included in the reference image, a second reference pixel is identified among the plurality of reference pixels whose coordinates have been transformed that contains the same subject as the reference pixel associated with the first reference pixel, and a determination process is performed to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the distance between the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations transform rays from rays passing through a plurality of virtual points located in each of the plurality of virtual planes to rays passing through the plurality of virtual points to the second camera. Stereo camera device.

15. The process involves acquiring data for a reference image generated by imaging at a reference position, and data for a reference image generated by imaging at a position different from the reference image. The process involves transforming the coordinates of the first reference pixel of the reference image into the coordinates of multiple reference pixels included in the reference image using multiple planar projection transformations, identifying a reference pixel among the multiple reference pixels whose coordinates have been transformed that contains the same subject as the first reference pixel, and performing a mapping process to associate the first reference pixel with the identified reference pixel. The process includes: converting the coordinates of the reference pixels associated with the first reference pixels by the inverse transformation of the plurality of plane projection transformations into the coordinates of a plurality of reference pixels included in the reference image; identifying a second reference pixel among the plurality of reference pixels whose coordinates have been converted that contains the same subject as the reference pixel associated with the first reference pixel; and performing a determination process to determine whether the association between the first reference pixel and the reference pixel is incorrect based on the distance between the first reference pixel and the second reference pixel. The aforementioned plurality of planar projection transformations transform rays to the camera that generated the reference image, which pass through a plurality of virtual points located in each of the plurality of virtual planes, into rays to the camera that generated the reference image, which pass through the plurality of virtual points. Image processing methods.

Images