Image processing apparatus and image processing method

Abstract

This allows for more precise setting of parking spaces. [Solution] The image processing device 20 includes an image acquisition unit 22 that acquires images taken by cameras 11 to 14 around the vehicle, a detection unit 23 that detects a pair of lane markings based on the images taken around the vehicle, a setting unit 29 that sets a parking area in which the vehicle can park according to the type of each lane marking that makes up the detected pair of lane markings, and a correction unit 28 that corrects the width of the parking area set based on the types of lane markings when the types of the detected pair of lane markings are different from each other.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus and an image processing method.

Background Art

[0002] There is known a technique for detecting a pair of lane lines from images taken of the surroundings of a vehicle by a plurality of cameras, and detecting a parking area where the vehicle can park based on the detected pair of lane lines. The types of lane lines are classified according to the shape of the lane lines, etc., and mainly include single lines composed of one straight line, etc., double lines composed of two parallel straight lines or lines such as U-shapes, etc.

[0003] Patent Document 1 discloses a technique for detecting a parking area based on corrected lane lines, where when the shapes of a pair of lane lines detected from an image are different, the shape of one lane line is corrected to the shape of the other lane line. Patent Document 2 discloses a technique for grouping double lines detected from an image and setting a parking area based on the double line located inside among the double lines included in one group. Patent Document 3 discloses a technique for calculating the median value of the edge strengths of a plurality of detected lane line candidates from an image, and not specifying a lane line candidate whose edge strength is smaller than the median value as a lane line.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a real-world parking lot where a parking area is defined by a pair of double lines, a camera may capture the entire width of one of the double lines, but only partially capture the other double line in the width direction. In this case, the type of line captured in its entirety in the width direction may be recognized as a double line, while the partially captured line may be recognized as a single line. For this reason, in the inventions described in Patent Documents 1 to 3, the parking area may be set to a width that is larger or smaller than the width that should be set.

[0006] This disclosure aims to enable more precise setting of parking areas. [Means for solving the problem]

[0007] To achieve the above objective, the image processing apparatus of the present disclosure includes: an image acquisition unit that acquires an image of the area around a vehicle captured by a camera; a detection unit that detects a pair of lane markings based on the image of the area around the vehicle; a setting unit that sets a parking area in which the vehicle can park according to the type of each lane marking that constitutes the detected pair of lane markings; and a correction unit that corrects the width of the parking area set based on the types of lane markings when the types of the detected pair of lane markings are different from each other. [Effects of the Invention]

[0008] According to this disclosure, when a pair of lane markings are of different types, the width of the parking area is appropriately adjusted. Therefore, this disclosure allows for more precise setting of the parking area. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the functional configuration of an image processing system equipped with an image processing device according to the first embodiment. [Figure 2] This figure shows the camera capturing images of the vehicle's surroundings at time Ta, and an example of an overhead view image generated based on the images of the vehicle's surroundings. [Figure 3]This figure shows the camera capturing images of the vehicle's surroundings at time Tb, and an example of an overhead view image generated based on the images of the vehicle's surroundings. [Figure 4] This figure shows the camera capturing images of the vehicle's surroundings at time Tc, and an example of an overhead view image generated based on the images of the vehicle's surroundings. [Figure 5] This diagram illustrates an example of the detection process in the detection unit. [Figure 6] This diagram illustrates an example of the comparison process in the comparison unit. [Figure 7] This diagram illustrates an example of the correction process in the correction section. [Figure 8] This diagram illustrates an example of the comparison process in the comparison unit and the correction process in the correction unit. [Figure 9] This diagram illustrates other examples of comparison processing in the comparison unit and correction processing in the correction unit. [Figure 10] This is a flowchart illustrating an example of how an image processing device operates. [Figure 11] Figure 10 is a flowchart detailing step S8. [Figure 12] Figure 10 is a flowchart detailing step S9. [Modes for carrying out the invention]

[0010] (First Embodiment) An image processing system 100, comprising an image processing device 20 according to the first embodiment, will be described below with reference to the drawings. Referring to Figure 1, the image processing system 100 is mounted on a vehicle. As shown in Figure 1, the image processing system 100 mainly comprises an imaging device 10 and an image processing device 20. The image processing system 100 may further include a display device for displaying images, an operating unit for user operation, etc.

[0011] The image processing system 100 is connected to an external device 50 via a communication network. The external device 50 is composed of, but not limited to, a display device 51 on which an image is displayed, a parking control device 52 that controls parking of a vehicle, etc. The external device 50 acquires the position, type, etc. of a parking area from the image processing system 100. These are used for display on the display device 51 and parking control by the parking control device 52.

[0012] The imaging device 10 is a device that captures an image of the surroundings of the host vehicle 4. The imaging device 10 includes a front camera 11, a left side camera 12, a right side camera 13, and a rear camera 14, and may further include another camera.

[0013] The front camera 11 is attached to the front bumper or front grill of the host vehicle 4 and captures the front of the host vehicle 4 as shown in FIGS. 2 to 4. The left side camera 12 is attached to the left door mirror or its vicinity and captures the left side of the host vehicle 4. The right side camera 13 is attached to the right door mirror or its vicinity and captures the right side of the host vehicle 4. The rear camera 14 is attached to the rear bumper or rear garnish of the host vehicle 4 and captures the rear of the host vehicle 4.

[0014] In the following description, the front camera 11, the left side camera 12, the right side camera 13, and the rear camera 14 are referred to as cameras 11 to 14 when not distinguishing them. The cameras 11 to 14 have an imaging lens composed of a wide-angle lens or a fish-eye lens capable of observing a wide range, an imaging element such as a CCD sensor, etc. The cameras 11 to 14 capture the surroundings of the host vehicle 4 at a frame rate according to their performance and output the captured images to the image processing device 20 one frame at a time. The time interval of the image output from the cameras 11 to 14 to the image processing device 20 is not limited. For example, the interval is 10 msec or 20 msec.

[0015] The image processing device 20 detects one or more pairs of section lines based on the images input from the cameras 11 to 14, and sets a parking area where the host vehicle 4 can park according to the type of each section line constituting each pair of detected section lines. When the types of a pair of detected section lines are different from each other, the image processing device 20 corrects the width of the parking area set based on the type of the section line.

[0016] The image processing device 20 is composed of an ECU having a processing device such as a CPU and a GPU, and a storage device such as a RAM and a ROM. The image processing device 20 can also be composed of one ECU, or can be composed of a plurality of ECUs to disperse each function of the control unit 21 or to disperse the data to be stored. The image processing device 20 can also be configured with an FPGA, an ASIC, etc.

[0017] As shown in FIG. 1, the image processing device 20 functions as a control unit 21 and a storage unit 40. The control unit 21 is mainly composed of a processing device such as a CPU included in the ECU, and controls the operation of the entire image processing device 20 by expanding and executing a predetermined program stored in the ROM in the RAM. The storage unit 40 is mainly composed of a storage device such as a RAM included in the ECU, but can also be provided with an external server or database.

[0018] The storage unit 40 stores a control program for operating the image processing device 20, an application program including an image processing program, etc. The storage unit 40 temporarily or non-temporarily stores thresholds, parameters, etc. used in the control unit 21. The storage unit 40 temporarily or non-temporarily stores the images from the cameras 11 to 14 and the bird's-eye view images generated by the image acquisition unit 22.

[0019] The storage unit 40 has a first area 41, a second area 42, and a third area. The first area 41 stores the positions, types, etc. of a pair of section lines detected by the detection unit 23 at the first time. The second area 42 stores the positions, types, etc. of a pair of section lines constituting the candidate of the parking area set by the setting unit 28 over time. The third area 43 stores the position, type, etc. of the parking area created by the creation unit 30.

[0020] The control unit 21 functions as an image acquisition unit 22, a detection unit 23, a determination unit 24, a recording unit 25, a judgment unit 26, a comparison unit 27, a setting unit 28, a correction unit 29, and a creation unit 30. When each unit of the control unit 21 performs processing using images, lane markings, types, positions, thresholds, parameters, etc., these are treated and processed as data. The image acquisition unit 22 acquires images of the area around the vehicle taken by the front camera 11, left side camera 12, right side camera 13, and rear camera 14. The images of the area around the vehicle include front images, left side images, right side images, and rear images input from the front camera 11, left side camera 12, right side camera 13, and rear camera 14. The image acquisition unit 22 processes the acquired images using a known method to transform the viewpoint and generate an overhead view image of the vehicle 4 as if viewed from directly above. The detection unit 23 detects a pair of lane markings based on the overhead view image generated by the image acquisition unit 22. The detection unit 23 may detect one pair or more pairs of boundary lines.

[0021] The procedure for generating an overhead image in the image acquisition unit 22 and detecting a pair of parking space lines in the detection unit 23 is described below with reference to Figures 2 and 5. Figure 2 shows the state of cameras 11-14 photographing the area around the vehicle 4 at time Ta, and an example of an overhead image G generated based on the images taken around the vehicle 4. The left side of Figure 2 shows the parking lot 1 in real space. The parking lot 1 in real space is divided by multiple parking space lines 2A, 2B, 2C, 2D, and 2E, and adjacent pairs of parking space lines define parking spaces 3 where the vehicle 4 and other vehicles 5 can be parked. The white arrows in the left side of Figure 2 indicate the direction of travel of the vehicle 4.

[0022] Types of parking lines include single lines, which consist of one straight line, and double lines, which consist of two parallel straight lines. Straight lines can be solid or dashed. Single lines include those consisting of only one straight line, and those consisting of two straight lines perpendicular to each other in a T-shape or L-shape in plan view. Double lines include those consisting of two parallel straight lines, and those in which one or both ends of two parallel straight lines are connected by a straight or curved line, resulting in a U-shape, oval, rectangle, etc. in plan view. Parking lot 1 in the example in Figure 2 has parking lines 2A to 2E consisting of U-shaped double lines.

[0023] The right-hand diagram in Figure 2 shows an example of an overhead image G generated by the image acquisition unit 22. The image acquisition unit 22 generates overhead images g1, g2, g3, and g4 by performing viewpoint transformation processing on images acquired from each camera 11 to 14, and then combines these to generate the overhead image G. The overhead image G displays three lane line images I1, I2, and I3, which are images of lane lines 2A, 2B, and 2C. The image acquisition unit 22 places a vehicle model V, which is modeled after the vehicle 4, in the center of the overhead image G.

[0024] The detection unit 23 detects a section line image from the overhead image G using known methods such as edge detection, and detects a pair of section lines based on the detected section line image. For example, the detection unit 23 scans the overhead image G in a direction intersecting the section lines to detect pixels in the image whose brightness value changes significantly above a threshold, and detects the portion where the sequence of detected pixels is longer than a predetermined length as an edge. Scanning means selecting pixels one by one in a predetermined direction and comparing the brightness between adjacent pixels. The detection unit 23 detects edges where the direction of these changes changes in the positive direction as positive edges, and edges where the direction changes in the negative direction as negative edges.

[0025] In the right-hand diagram of Figure 2, the lower left of the overhead image G is set as the origin 0, the width direction of the lane marking image, which is in the direction along the vehicle's travel direction and perpendicular to the extension direction of the lane marking image, is set as the X-axis, and the length direction of the lane marking image, which is the extension direction of the lane marking image, is set as the Y-axis. The detection unit 23 scans the overhead image G in the positive X-axis direction and detects positive and negative edges, detecting line segments of positive and negative edges that are continuous for a predetermined length. Figure 5(1) is a schematic representation of the edge line segments detected from the lane marking images I1, I2, I3 of the overhead image G in Figure 2, using thick solid lines and thick dashed lines. The thick solid lines are the positive edge line segments Ep1, Ep2, Ep3, Ep4, Ep5, Ep6, and the thick dashed lines are the negative edge line segments Em1, Em2, Em3, Em4, Em5, Em6.

[0026] The detection unit 23 acquires the coordinates of the start and end points of each of the detected positive edge line segments and negative edge line segments. In this embodiment, the coordinates are in the world coordinate system, and the coordinates on the overhead image are converted to world coordinate system coordinates by the program and output to each part of the control unit 21, including the detection unit 23.

[0027] Based on the acquired coordinates, the detection unit 23 extracts adjacent positive edge line segments and negative edge line segments at predetermined intervals and determines that they constitute a pair of edges forming a single boundary line. For example, if the distance between a positive edge line segment and a negative edge line segment is within the range of the boundary line width ± threshold, it is determined that they are a pair of edges.

[0028] In the example of Figure 5 (1), pairs of line segments consisting of a positive edge and a negative edge were detected on both sides of the section line images I1, I2, and I3, respectively: the pair of line segment Ep1 and line segment Em1, the pair of line segment Ep2 and line segment Em2, the pair of line segment Ep3 and line segment Em3, the pair of line segment Ep4 and line segment Em4, the pair of line segment Ep5 and line segment Em5, and the pair of line segment Ep6 and line segment Ep6.

[0029] The detection unit 23 uses known methods such as pattern matching to compare the boundary line images with various preset patterns and determines whether the boundary line corresponding to each boundary line image is a single line or a double line. Alternatively, the detection unit 23 can calculate the distance between the line segments of opposing edges of two adjacent edge pairs, specifically the line segment of the positive edge of one edge pair and the line segment of the negative edge of the other edge pair, and determine the type of boundary line according to this distance. If the calculated distance is within a predetermined interval, for example, within the range of the interval of a double line ± a threshold, the detection unit 23 determines that the line composed of the two edge pairs is a double line; otherwise, it determines that the two edge pairs are each different single lines.

[0030] In the example in Figure 5, the pairs of line segments Ep1 and Em1, and Ep2 and Em2 are determined to be pairs of double line edges that constitute boundary line 2A in real space. The pairs of line segments Ep3 and Em3, and Ep4 and Em4 are determined to be pairs of double line edges that constitute boundary line 2B in real space. The pairs of line segments Ep5 and Em5, and Ep6 and Em6 are determined to be pairs of double line edges that constitute boundary line 2C in real space.

[0031] The detection unit 23 sets a reference line for defining the parking area based on the type of parking space line and the set of edges. If the parking space line is determined to be a single line, the detection unit 23 sets a line segment parallel to the positive edge and the negative edge of the set of edges constituting the single line at the center in the width direction of these line segments, and uses this as the reference line. In other words, the reference line is set at the center in the width direction of the parking space line image.

[0032] If the type of boundary line is determined to be a double line, the detection unit 23 sets two line segments, the first and second, parallel to the positive edge line segment and the negative edge line segment of the pair of edges that make up each line of the double line, in the center in the width direction. The detection unit 23 sets a third line segment parallel to the set first and second line segments in the center in the width direction, and uses this third line segment as the reference line. In Figure 5(1), the reference lines L1, L2, and L3, which are set based on the pair of two positive edge line segments and negative edge line segments that make up each boundary line image I1, L2, and I3, are shown as dashed lines.

[0033] The detection unit 23 detects pairs of parking lines that constitute a parking area based on the detected parking lines. Specifically, the detection unit 23 detects parking lines 2A and 2B corresponding to adjacent parking line images I1 and I2 in the width direction as a pair of parking lines that constitute one parking area. The detection unit 23 also detects parking lines 2B and 2C corresponding to parking line images I2 and I3 as a pair of parking lines that constitute another parking area.

[0034] The determination unit 24 determines the type of parking area based on the pair of parking lines detected by the detection unit 23. The determination unit 24 determines whether the parking area composed of the pair of parking lines is a parallel parking area or a tandem parking area, depending on the distance between each parking line of the pair. If the distance between each parking line of the pair of parking lines corresponds to the vehicle width, the determination unit 24 determines the type of parking area to be a parallel parking area. If the distance between each parking line of the pair of parking lines corresponds to the vehicle length, the determination unit 24 determines the type of parking area to be a tandem parking area. The vehicle width can be 2 to 3 m and the vehicle length can be 3 to 6 m, but is not limited to these values. In the examples in Figures 2 and 5, the type of parking area is determined to be a parallel parking area. The types of parking areas are not limited to parallel parking areas and tandem parking areas; diagonal parking areas, priority parking areas, electric vehicle parking areas, etc., may be added based on the angle of the parking lines, marks drawn within the parking area, etc.

[0035] The recording unit 25 records information including the type and position of each of the pair of parking lines detected by the detection unit 23 at the current time by writing it to the storage unit 40 over time. Recording over time includes accumulating and recording the information at regular intervals, or updating and recording it at regular intervals. The recording unit 25 writes the position and type of each of the pair of parking lines detected by the detection unit 23 and the type of parking area determined by the determination unit 24 to the first area 41. In this embodiment, when a vehicle is parked facing backward in the parking area, the parking line located to the right of the vehicle in a pair of parking lines is designated as the first parking line, and the parking line located to the left is designated as the second parking line. The information for the pair of parking lines includes the coordinates of the start and end points of the reference line, which is the position of the first and second parking lines, and their types. The coordinates of the start and end points are not coordinates on the overhead image or coordinates relative to the vehicle 4, but coordinates in the world coordinate system.

[0036] The recording unit 25 updates the pair of boundary lines stored in the second area 42 based on the comparison result from the comparison unit 27. When the comparison unit 27 determines that the pair of boundary lines overlap with the pair of boundary lines stored in the second area 42, the recording unit 25 updates the pair of boundary lines stored in the second area 42 with the more recent pair of boundary lines from the first area 41 detected at the current time. When the comparison unit 27 determines that the pair of boundary lines do not overlap with the pair of boundary lines in the second area 42, the recording unit 25 adds the pair of boundary lines to the second area 42 as newly detected.

[0037] The recording unit 25 sets a timer for a pair of dividing lines that have been overwritten or added to the second area 42. The recording unit 25 sets a predetermined time, for example 0.5 seconds, to the timer and counts down the time. The recording unit 25 may also count up the timer from 0. The time interval for counting down or counting up can be the same as the time interval at which cameras 11-14 output images, which is 10 msec or 20 msec.

[0038] The recording unit 25 deletes the information of a pair of parking space lines from the second area 42 when the timer reaches 0 or when the timer reaches a predetermined time. The recording unit 25 moves up any valid pairs of parking space lines that are not to be deleted and stores them in the second area 42. As the vehicle 4 moves within the parking lot 1, the parking space lines detected by the detection unit 23 are updated over time. Therefore, parking space lines detected before the predetermined time move away from the vehicle 4. Since there is no need to detect a parking area based on parking space lines that are away from the vehicle 4, these parking space lines are deleted.

[0039] The determination unit 26 determines whether the type of the first and second boundary lines of a pair of boundary lines are the same or different. The determination unit 26 determines that the types are the same if both the first and second boundary lines are single lines or both are double lines. The determination unit 26 determines that the types are different if one of the first and second boundary lines is a single line and the other is a double line.

[0040] If the types of the first and second lane markings are different, please refer to Figures 3 and 5 for further explanation. Figure 3 shows the camera capturing images of the area around the vehicle 4 at time Tb, and an example of an overhead image G generated based on the images of the area around the vehicle 4. Time Tb is later than time Ta, and the vehicle 4 has moved forward compared to time Ta. In the example in Figure 3, lane marking 2A is located between the shooting range of the left side camera 12 and the shooting range of the rear camera 14. The entire width of lane marking 2B is captured by the left side camera 12, but only a portion of lane marking 2A in the width direction is captured by the rear camera 14. In the generated overhead image G, as shown in the right-hand diagram of Figure 3, at the boundary between overhead images g2 and g4, i.e., the image seam, one of the double lines of lane marking 2A is missing, and lane marking image I1 is generated.

[0041] "Capturing the entire width" means that the entire image of the lane markings is discernible and the center in the width direction can be properly detected. For example, this means that at least a portion of one line is captured without being cut off in the width direction, to the extent that it can be determined that it is a single line, or that at least a portion of each of two lines is captured without being cut off in the width direction, to the extent that it can be determined that they are double lines. "Capturing only a portion in the width direction" means that the entire image of the lane markings in the width direction is not discernible. For example, this means that a portion of a single line is cut off in the width direction, or that only one of the two lines of a double line is captured.

[0042] When the detection unit 23 scans the overhead image G shown in Figure 3, as shown in (2) of Figure 5, only the pair of line segments Ep1 and Em1 is detected in the section line image I1, and the pair of line segments Ep2 and Em2 detected at time Ta is not detected. Section line 2A corresponding to section line image I1 is determined to be a single line, and section line 2B which is paired with section line 2A is determined to be a double line. As a result, the determination unit 26 determines that the type of the first section line 2A and the type of the second section line 2B are different.

[0043] Figure 4 shows the camera capturing images of the area around the vehicle 4 at time Tc, and an example of an overhead image G generated based on the images of the area around the vehicle 4. Time Tc is later than time Tb, and the vehicle 4 has moved forward compared to time Tb. At time Tc, the entire width of the first lane marking 2A is captured by the rear camera 14, and the entire width of the second lane marking 2B is captured by the left side camera 12, so the type of the first lane marking 2A and the type of the second lane marking 2B of the pair of lane markings are determined to be the same. Even when the vehicle 4 moves backward and the positional relationship between each lane marking and the vehicle 4 returns to the same positional relationship as in Figure 2 from Figure 3, the type of the first lane marking 2A and the type of the second lane marking 2B of the pair of lane markings are determined to be the same. According to the image processing device 20 of this embodiment, even if the type of the first parking space line 2A is not properly determined at time Tb, the type of the first parking space line 2A is properly determined at a different time Tc, and the parking area is set with high accuracy.

[0044] The comparison unit 27 compares the position of the parking space line detected at the current time (the first time) with the position of the parking space line detected at a second time (a time earlier than the first time). The comparison unit 27 compares the coordinates of the reference line of the first or second parking space line stored in the first region 41 with the coordinates of the reference line of the first or second parking space line stored in the second region 42. The comparison unit 27 determines that the parking space lines are the same if the difference between the position of the parking space line detected at the first time and the position of the parking space line detected at the second time is within a threshold. The comparison unit 27 determines that the parking space lines are different if the difference exceeds the threshold. The setting unit 28 sets the parking area set based on the pair of parking space lines detected at the first time as a different parking area from the parking area set based on the pair of parking space lines detected at the second time, when the difference between the position of the parking space line detected at the first time and the position of the parking space line detected at the second time exceeds a threshold.

[0045] The specific determination procedure of the comparison unit 27 is explained below using Figure 6. In Figure 6, the black line indicates a pair of lane markings detected at the current time, which is the first time, and the white line indicates a pair of lane markings detected at a second time, which is earlier than the first time. The black line in (1) of Figure 6 is a pair of lane markings detected at a certain time, and at this point, that certain time is the first time. The black lines in (2), (3), (4), and (5) of Figure 6 indicate a pair of lane markings detected at a time later than the time in (1), and the white line indicates a pair of lane markings detected at the time in (1). At this point, the times in (2), (3), (4), and (5) become the current time, which is the first time, and the time in (1) becomes the second time, which is earlier than the first time. The arrows shown in Figure 6 indicate the direction of travel of the vehicle 4.

[0046] Figures 6(2) and (3) show the state in which each of a pair of boundary lines detected at the first time is determined to be a double line. Figure 6(2) shows the state in which the difference d between the positions of the pair of boundary lines detected at the first time and the positions of the pair of boundary lines detected at the second time is less than or equal to the threshold t. In this case, the comparison unit 27 determines that the pair of boundary lines are the same because they overlap. Figure 6(3) shows the state in which the difference d between the positions of the pair of boundary lines detected at the first time and the positions of the pair of boundary lines detected at the second time exceeds the threshold t. In this case, the comparison unit 27 determines that the pair of boundary lines are different because they do not overlap. The threshold t is the allowable range of error when detecting boundary lines, i.e., the tolerance. The threshold t can be set to about 10 pixels, i.e., about 30 cm.

[0047] Figures 6(4) and (5) show the state in which the first of a pair of boundary lines detected at the first time is determined to be a single line and the second of a pair of boundary lines is determined to be a double line. Figure 6(4) shows the state in which the difference d between the position of the pair of boundary lines detected at the first time and the position of the pair of boundary lines detected at the second time is less than or equal to the threshold t. In this case, the comparison unit 27 determines that the position of the pair of boundary lines detected at the first time and the pair of boundary lines detected at the second time are the same. Figure 6(5) shows the state in which the difference d between the position of the pair of boundary lines detected at the first time and the position of the pair of boundary lines detected at the second time exceeds the threshold t. In this case, the comparison unit 27 determines that the position of the pair of boundary lines detected at the first time and the pair of boundary lines detected at the second time are different from each other.

[0048] The setting unit 28 sets a parking area in which the vehicle 4 can park, according to the type of each section line that constitutes the detected pair of section lines, and more specifically, based on the reference lines set according to the type of each section line. Setting a parking area means recognizing the area defined by the first and second section lines that constitute the pair of section lines as a parking area and storing the location of the parking area. At this stage, the parking area set by the setting unit 28 is a candidate for a parking area and is not yet confirmed as a parking area. The setting unit 28 defines the area enclosed by the four points of the start and end points of the reference lines of each section line of the pair as a parking area and stores the coordinates of the four points as the location of the parking area. In the example of (1) in Figure 5, parking area R1 is set in the rectangular area shown by the dashed line with the start and end points of reference line L1 and reference line L2 as vertices. Parking area R2 is set in the rectangular area shown by the dashed line with the start and end points of reference line L2 and reference line L3 as vertices. In the example of (2) in Figure 5, parking area R1' is set in the rectangular region indicated by the dashed line with the start and end points of reference line L1' and reference line L2 as vertices. Parking area R2 is set in the rectangular region indicated by the dashed line with the start and end points of reference line L2 and reference line L3 as vertices.

[0049] When the width of the parking area, which is composed of a pair of dividing lines, is changed by the modification unit 29, that is, when the position of the pair of dividing lines is corrected, the setting unit 28 readjusts the position of the parking area based on the changed pair of dividing lines. The setting unit 28 can set the parking area more accurately.

[0050] When the setting unit 28 determines that the types of each of the pair of parking lines in the parking area are different, the correction unit 29 corrects the width of the parking area set based on the types of parking lines. When the setting unit 28 determines that the types of each of the pair of parking lines in the parking area are different, the correction unit 29 corrects the width of the parking area at the first time based on the types of the pair of parking lines detected at the second time recorded by the recording unit 25 in the storage unit 40, which are of the same type. The correction unit 29 can correct the width of the parking area set based on the types of parking lines when the setting unit 28 determines that the types of each of the pair of parking lines in the parking area are different, and the difference between the position of the pair of parking lines and the position of the pair of parking lines detected at the second time in the second area 42 is less than or equal to a threshold.

[0051] The correction unit 29 does not correct the parking space when it determines that each of the lines constituting a pair of parking space lines is of the same type. This is because the parking area is appropriately set when it is determined that each of the lines is of the same type. Figures 6(2) and (3) show examples where it is determined that each of the lines in a pair of parking space lines is of the same type. Figures 6(4) and (5) show examples where it is determined that each of the lines in a pair of parking space lines is of a different type.

[0052] As explained with reference to Figures 3 and 5(2), a situation in which a pair of parking lines are of different types is when, although a pair of parking lines 2A and 2B in real space are double lines, the second parking line 2B is captured in its entirety in the width direction and determined to be a double line, while the first parking line 2A is captured in only a portion and determined to be a single line. In such a case, for the first parking line 2A, which is determined to be a single line, the reference line L1' of the parking line image I1 is set on the outer line of the double line, as shown in Figure 5(2). The width W' of the parking area R1' set between reference line L1' and reference line L2 is wider than the width W of the parking area R1 set between reference line L1 and reference line L2 of each parking line of the pair of parking lines that were determined to be double lines in (1). Therefore, it is necessary to correct the width W' of the parking area R1'.

[0053] The process of correcting the width of the parking area by the correction unit 29 is explained below using Figure 7. The left side of Figure 7 shows the parking area before correction, and the right side shows the parking area after correction. As shown in the right side of Figure 7, the correction unit 29 moves the reference line L1' of the first parking area from the position of the reference line L2 of the second parking area image I2 of the pair of parking area lines represented by black lines to a position of width W, and updates the position of the reference line L1'. As a result, the width of the parking area R1' is appropriately corrected. The coordinates of the start and end points of the moved reference line L1' are overwritten in the second area 42.

[0054] As shown in (5) of Figure 6, if the difference between the position of a pair of parking lines and the position of a pair of parking lines detected at the second time exceeds a threshold, the correction unit 29 does not correct the width of the parking area. The coordinates of the pair of parking lines that constitute this parking area are discarded and not stored in the second area 42. This parking area is set based on parking lines that were not properly detected and were not detected at times prior to the current time. When the vehicle 4 moves forward or backward so that the entire width of each parking line of the pair is captured, the type of each parking line is determined to be the same double line, and the parking area is set appropriately. Therefore, there is no need to store a parking area that is currently set based on different parking lines, and this parking area is discarded.

[0055] Figure 8 illustrates an example of the comparison process in the comparison unit 27 and the correction process in the correction unit 29. The times T2 or T3 shown in Figure 8 are the current time, which is the first time, and time T1 is the second time, which is earlier than the first time. At the second time T1, each pair of parking lines is determined to be a double line. At the first time T2, the first parking line constituting the pair of parking lines is determined to be a double line, and as a result of detecting the outer line of the double line as the second parking line, the second parking line is determined to be a single line. At the first time T3, which shows another case different from the first time T2, as a result of detecting the outer line of the double line as the first parking line constituting the pair of parking lines, the first parking line is determined to be a single line, and the second parking line is determined to be a double line. For this reason, the width of the parking area at the first times T2 and T3 is wider than the width of the parking area at the second time T1.

[0056] If the difference between the positions of a pair of parking space lines detected at the first time T2 and the positions of a pair of parking space lines detected at the second time T1 is less than or equal to a threshold, the correction unit 29 moves the reference line of the second parking space line from the reference line of the first parking space line at the first time T2 to the position of the width of the parking area at the second time T1, thereby correcting the width of the parking area. If the difference between the positions of a pair of parking space lines detected at the first time T3 and the positions of a pair of parking space lines detected at the second time T1 is less than or equal to a threshold, the correction unit 29 moves the reference line of the first parking space line from the reference line of the second parking space line at the first time T3 to the position of the width of the parking area at the second time T1, thereby correcting the width of the parking area. If the above difference exceeds the threshold, the pair of parking space lines detected at the first time T2 or T3 is discarded.

[0057] Figure 9 illustrates other examples of the comparison process in the comparison unit 27 and the correction process in the correction unit 29. The times T4 or T5 shown in Figure 9 are the current time, which is the first time, and time T1 is the second time, which is earlier than the first time. At the first time T4, the inner line of the double line that constitutes the first set of dividing lines is detected, resulting in the first dividing line being determined to be a single line and the second dividing line being determined to be a double line. At the first time T5, the first dividing line that constitutes the pair of dividing lines is determined to be a double line, and the inner line of the double line that constitutes the second dividing line is detected, resulting in the second dividing line being determined to be a single line. Therefore, the width of the parking area at the first times T4 and T5 is narrower than the width of the parking area at the second time T1.

[0058] If the difference between the positions of a pair of parking space lines detected at the first time T4 and the positions of a pair of parking space lines detected at the second time T1 is less than or equal to a threshold, the correction unit 29 moves the reference line of the first parking space lines from the reference line of the first parking space lines at the first time T4 to the position of the width of the parking area at the second time T1, thereby correcting the width of the parking area. If the difference between the positions of a pair of parking space lines detected at the first time T5 and the positions of a pair of parking space lines detected at the second time T1 is less than or equal to a threshold, the correction unit 29 moves the reference line of the first parking space lines from the reference line of the second parking space lines at the first time T5 to the position of the width of the parking area at the second time T1, thereby correcting the width of the parking area. On the other hand, if the difference in the above positions exceeds the threshold, the parking area consisting of a pair of parking space lines detected at the first time T4 or T5 is discarded.

[0059] The creation unit 30 creates parking areas based on the new parking area candidates set in the setting unit 28 and the parking area candidates whose widths have been modified in the modification unit 29. From these parking area candidates, the creation unit 30 determines the parking area based on the width of the parking area and other conditions. The creation unit 30 writes and registers the coordinates of the start and end points of the pair of reference lines of the parking area, which are the positions of the determined parking area, in the third area 43. The positions of the parking areas in the third area 43 are output to the external device 50 and used by the external device 50. For example, an overhead image with a frame indicating the parking area determined by the creation unit 30 superimposed is displayed on the display device 51. The parking control device 52 selects the optimal parking area based on the positions of the parking areas and controls the vehicle 4 to automatically park in the selected parking area.

[0060] An example of the operation of the image processing system 100 according to the first embodiment will be described below with reference to the flowcharts in Figures 10 to 12. The flowcharts in Figures 10 to 12 show an example of the image processing operation of the image processing device 20. The operation of the image processing system 100 and the image processing device 20 is not limited to the operation described below.

[0061] The operation shown in the flowchart of Figure 10 is initiated when the vehicle 4 enters the parking lot 1, or when the driver operates an operation switch (not shown) to input the instruction to start automatic parking. The operation shown in the flowchart of Figure 10 is repeated each time images of the area around the vehicle 4 are input from cameras 11-14.

[0062] In step S1, the image acquisition unit 22 acquires images of the area around the vehicle 4 from cameras 11-14 at the current time (first time), and generates an overhead view image G based on the acquired images. In step S2, the detection unit 23 detects one or more pairs of lane markings based on the overhead view image G. The recording unit 25 records the position and type of each lane marking for all pairs of lane markings detected at the first time by writing them to the first area 41.

[0063] In step S3, the recording unit 25 counts down the timers for all pairs of dividing lines in the second region 42. In step S4, the recording unit 25 deletes the information of the pair of dividing lines to be deleted from the second region 42 whose timer has reached 0, and counts down the number of valid dividing lines. The recording unit 25 updates the second region by advancing all valid pairs of dividing lines that are not to be deleted. Note that during the first execution of image processing, dividing lines have not been detected at times prior to the current time, so steps S3 and S4 may be skipped.

[0064] The processes from step S5 onward are executed until processing is completed for all pairs of dividing lines detected in step S2. Hereinafter, "a pair of dividing lines" refers to a pair of dividing lines detected at the first time and stored in the first region 41. In step S5, the control unit 21 determines whether there is a pair of dividing lines to be processed. If the determination result is YES, the program proceeds to step S6. If the determination result is NO, the program proceeds to END because processing for all pairs of dividing lines has been completed, and the image processing ends.

[0065] In step S6, the determination unit 24 determines whether the type of parking area formed by the pair of parking lines is a parallel parking area or a tandem parking area, based on the distance between the first and second parking lines. In step S7, the judgment unit 26 determines whether the type of the first parking line and the type of the second parking line are the same. If the judgment is YES, the program proceeds to step S8. If the judgment is NO, the program proceeds to step S9.

[0066] The comparison and setting process in step S8 is performed when it is determined that the type of the first boundary line and the second boundary line are the same. Step S8 is explained with reference to the flowchart in Figure 11. In step S801, the control unit 21 turns off the discovery flag. The discovery flag is turned on when a pair of boundary lines in the second region 42 that overlaps with one pair of boundary lines is discovered. The processes in steps S802 to S805 are repeated until the comparison between the pair of boundary lines and all pairs of boundary lines in the second region 42 is completed, or until the discovery flag is turned on.

[0067] In step S803, the comparison unit 27 compares the positions of the pair of boundary lines with the positions of the pair of boundary lines in the second region 42 to determine if they overlap. Whether they overlap is determined by whether the difference in the positions of these boundary lines is less than or equal to a threshold. If the result of the determination is YES, the program proceeds to step S804. If the result of the determination is NO, these boundary lines are considered to be different boundary lines, and the program proceeds to step S807 to compare them with another different pair of boundary lines in the second region 42.

[0068] In step S804, the recording unit 25 overwrites and updates the information of the pair of parking lines in the second area 42 with the information of the pair of parking lines and resets the timer for the pair of parking lines. In step S805, the setting unit 28 sets the parking area based on the updated pair of parking lines. That is, the area enclosed by the start and end points of the reference lines of each updated parking line is reset as the parking area. In step S806, the control unit 21 turns on the discovery flag because it has detected overlapping parking lines. The program then proceeds to step S807.

[0069] In step S807, the control unit 21 determines whether the comparison between one pair of boundary lines and all pairs of boundary lines in the second region 42 is complete, or whether the discovery flag is turned on. If the determination result is YES, the program exits the comparison and setting loop and proceeds to step S808. If the determination result is NO, the program returns to step S802 to compare with another different pair of boundary lines in the second region 42, and steps S802 to S807 are repeated.

[0070] In step S808, the control unit 21 determines whether the detection flag is off. If the determination result is YES, the program proceeds to step S809, assuming that the pair of boundary lines have been newly detected. If the determination result is NO, the program proceeds to END, since the information on the pair of boundary lines in the second region 42 has been updated in the previous step S804.

[0071] In step S809, the recording unit 25 adds information for a pair of parking lines to the second area 42, sets a timer for this pair of parking lines, and counts up the number of valid parking lines. In step S810, the setting unit 28 sets the parking area based on the added pair of parking lines. That is, the area enclosed by the start and end points of the reference lines of each parking line is set as the new parking area. After that, the program proceeds to END. When the program proceeds to END, the comparison and setting process is completed, and then the program proceeds to step S10 in Figure 10.

[0072] The comparison, setting, and modification process in step S9 is performed when it is determined that the type of the first boundary line and the second boundary line are different from each other. Details of step S9 are explained below with reference to the flowchart in Figure 12. In step S901, the control unit 21 turns off the detection flag. Steps S902 to S905 are repeated until the comparison between one pair of boundary lines and all pairs of boundary lines in the second region 42 is completed, or until the detection flag is turned on.

[0073] In step S903, the comparison unit 27 compares the positions of the pair of boundary lines with the positions of the pair of boundary lines in the second region 42 to determine if they overlap. If the result of the determination is YES, the program proceeds to step S904. If the result of the determination is NO, the program proceeds to step S908 to perform a comparison with another different pair of boundary lines in the second region 42.

[0074] In step S904, the correction unit 29 corrects the width of the parking area composed of a pair of lane markings based on the width of the parking area composed of a pair of lane markings in the second area that is deemed to overlap. In step S905, the recording unit 25 overwrites and updates the information of the pair of lane markings in the second area 42 with the information of the pair of lane markings and resets the timer for this pair of lane markings. In step S906, the setting unit 28 sets the parking area based on the updated pair of lane markings. That is, the area enclosed by the start and end points of the reference lines of each lane marking after the update is reset as the parking area. In step S907, the control unit 21 turns on the detection flag. The program proceeds to step S908.

[0075] In step S908, the control unit 21 determines whether the comparison between one pair of boundary lines and all pairs of boundary lines in the second region 42 is complete, or whether the detection flag is turned on. If the determination result is YES, the program exits the comparison, setting, and modification loop and proceeds to END, and the comparison, setting, and modification process is completed. After that, the program proceeds to step S10 in Figure 10. If the determination result is NO, the program returns to step S902 to compare with another different pair of boundary lines in the second region 42, and the process from steps S902 to S908 is repeated.

[0076] In the process of step S9 described above, if the type of the first and second lines of a pair of parking lines are different, and a pair of parking lines overlapping the second area 42 is found, the width of the parking area composed of the pair of parking lines is corrected, and the information of the pair of parking lines in the second area 42 is updated. If the type of the first and second lines of a pair of parking lines are different, and no pair of parking lines overlapping the second area 42 is found, the pair of parking lines is discarded without being stored in the second area. In this way, even if a pair of parking lines detected at the first time is discarded, no problems occur because the pair of parking lines is properly photographed at a time after the first time due to the movement or reversing of the vehicle 4, and the type is properly detected and processed.

[0077] Returning to Figure 10, in step S10, the creation unit 30 determines the parking area to be registered based on the parking area set by the setting unit 28, i.e., the candidate parking area. The creation unit 30 registers the coordinates and type of the start and end points of the reference lines of each of the pair of parking lines that represent the location of the parking area in the third area 43. With this, the operation of the image processing system 100 is completed.

[0078] As described above, the image processing apparatus 20 according to the above embodiment includes an image acquisition unit 22 that acquires images taken by cameras 11 to 14 around the vehicle, a detection unit 23 that detects a pair of parking lines based on the images taken around the vehicle, a setting unit 28 that sets a parking area in which the vehicle can park according to the type of each parking line constituting the detected pair of parking lines, and a correction unit 29 that corrects the width of the parking area set based on the type of parking lines when the types of the detected pair of parking lines are different from each other. With this configuration, the image processing apparatus 20 and the image processing method executed by the control unit 21 of the image processing apparatus 20 according to this embodiment can appropriately correct a parking area that is set to a different size from the parking space 3 defined by the parking lines drawn in the parking lot 1 in real space, and can set the parking area with greater accuracy.

[0079] The image processing device 20 of this embodiment includes a recording unit 25 that records the types of parking space lines over time. When the types of a pair of parking space lines forming the parking area are different at a first time, the correction unit 29 corrects the width of the parking area at the first time based on the types of a pair of parking space lines of the same type detected at a second time, which is a time earlier than the first time recorded by the recording unit 25. With this configuration, the image processing device 20 and image processing method of this embodiment can adjust a parking area that is set to a different size from the actual size to an appropriate size that matches the actual parking space lines.

[0080] The image processing device 20 of this embodiment includes a comparison unit 27 that compares the position of a parking space line detected at a first time with the position of a parking space line detected at a second time, which is earlier than the first time. The setting unit 28 sets the parking area set based on a pair of parking space lines detected at the first time as a different parking area from the parking area set based on a pair of parking space lines detected at the second time, when the difference between the position of the parking space line detected at the first time and the position of the parking space line detected at the second time exceeds a threshold. The modification unit 29 does not modify the width of the parking area set based on a pair of parking space lines detected at the first time, when the difference between the position of the parking space line detected at the first time and the position of the parking space line detected at the second time exceeds a threshold. With this configuration, the image processing device 20 and image processing method of this embodiment do not perform unnecessary processing if the parking space line newly detected at the first time is a different parking space line from the parking space line detected at a second time, which is earlier than the first time, thus enabling improved processing speed and reduced load.

[0081] In the image processing device 20 of this embodiment, the image captured around the vehicle may include the entire width of the first lane markings and a portion of the width of the second lane markings. The correction unit 29 determines that the types of a pair of lane markings forming the parking area are different from each other, based on the type of the first lane markings captured in their entirety and the type of the second lane markings captured in their entirety. In this configuration, the image processing device 20 and image processing method of this embodiment can set a parking area composed of a pair of lane markings at an appropriate position and of an appropriate size, even if the type of lane markings is not appropriately determined by the timing of the cameras 11 to 14.

[0082] While embodiments of this disclosure have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments, and design modifications that do not depart from the gist of this disclosure are included.

[0083] In the above embodiment, the image acquisition unit 22 generates an overhead image G and outputs it to the detection unit 23, but is not limited to this. In another different embodiment, instead of generating an overhead image G, the image acquisition unit 22 can directly output an image that has not undergone viewpoint transformation processing to the detection unit 23. Based on the image that has not undergone viewpoint transformation processing, the detection unit 23 can detect a pair of boundary lines and associate the coordinates of the points on the image coordinate system without viewpoint transformation processing that constitute the detected boundary lines with the coordinates of the overhead image coordinate system. [Explanation of Symbols]

[0084] 4: Own vehicle, 5: Other vehicles, 11: Front camera, 12: Left side camera, 13: Right side camera, 14: Rear camera, 20: Image processing device, 21: Control unit, 22: Image acquisition unit, 23: Detection unit, 25: Recording unit, 27: Comparison unit, 28: Setting unit, 29: Correction unit

Claims

1. An image acquisition unit that acquires images taken by a camera around the vehicle, A detection unit that detects a pair of lane markings based on an image taken of the area around the vehicle, A setting unit sets a parking area in which the vehicle can park, according to the type of each section line that constitutes the detected pair of section lines, An image processing apparatus comprising: a correction unit that corrects the width of the parking area set based on the type of the parking lines when the type of the detected pair of parking lines is different from the type of the parking lines.

2. The system includes a recording unit that records the type of the aforementioned road markings over time. The image processing apparatus according to claim 1, characterized in that the correction unit corrects the width of the parking area at the first time based on the types of a pair of demarcation lines that are the same type, which were detected at a second time that is earlier than the first time recorded by the recording unit, when the types of a pair of demarcation lines that form the parking area at the first time are different from each other.

3. The system includes a comparison unit that compares the position of the lane line detected at a first time with the position of the lane line detected at a second time, which is earlier than the first time. When the difference between the position of the lane markings detected at the first time and the position of the lane markings detected at the second time exceeds a threshold, the setting unit sets the parking area set based on the pair of lane markings detected at the first time as a different parking area from the parking area set based on the pair of lane markings detected at the second time. The image processing apparatus according to claim 1, characterized in that the correction unit does not correct the width of the parking area set based on a pair of boundary lines detected at the first time when the difference between the position of the boundary lines detected at the first time and the position of the boundary lines detected at the second time exceeds a threshold.

4. The image taken around the vehicle includes the entire width of the first lane marking and a portion of the width of the second lane marking. The image processing apparatus according to claim 1, characterized in that the modification unit modifies the width of the parking area set based on the types of the pair of partition lines when it is determined that the types of the pair of partition lines forming the parking area are different from each other, based on the type of the first partition line from which the entire width direction is photographed and the type of the second partition line from which a part of the width direction is photographed.

5. An image processing method performed in the control unit of an image processing device, The image acquisition process involves the camera capturing images of the area around the vehicle, A detection step of detecting a pair of lane markings based on an image taken of the area around the vehicle, A setting step to set a parking area in which the vehicle can park, according to the type of each section line that constitutes the detected pair of section lines, An image processing method characterized by including a correction step of correcting the width of the parking area set based on the type of the parking lines when the type of the detected pair of parking lines is different from the type of parking lines.

6. The recording step includes recording the type of the aforementioned road markings over time. The image processing method according to claim 5, characterized in that the correction step corrects the width of the parking area at the first time based on the types of a pair of demarcation lines that are the same type and were detected at a second time, which is a time earlier than the first time recorded in the recording step, when the types of a pair of demarcation lines that form the parking area at the first time are different from each other.

7. The process includes a comparison step of comparing the position of the lane line detected at a first time with the position of the lane line detected at a second time, which is earlier than the first time. The setting step, when the difference between the position of the lane markings detected at the first time and the position of the lane markings detected at the second time exceeds a threshold, sets the parking area set based on the pair of lane markings detected at the first time as a different parking area from the parking area set based on the pair of lane markings detected at the second time. The image processing method according to claim 5, characterized in that the correction step does not correct the width of the parking area set based on the pair of boundary lines detected at the first time when the difference between the position of the boundary lines detected at the first time and the position of the boundary lines detected at the second time exceeds a threshold.

8. The image taken around the vehicle includes the entire width of the first lane marking and a portion of the width of the second lane marking. The image processing method according to claim 5, characterized in that the correction step corrects the width of the parking area set based on the pair of section lines when it is determined that the types of section lines forming the parking area are different from each other, based on the type of first section line from which the entire width direction is photographed and the type of second section line from which a part of the width direction is photographed.

Images