Image processing device and image processing program

Abstract

To improve white line detection performance.SOLUTION: An image processing device 10 includes a controller 12 for detecting a white line in a detection area which is set in an image photographed by a camera 21 arranged in a vehicle. The controller 12 detects that the vehicle is moving laterally based on information related to the vehicle. The controller 12 enlarges the detection area in the opposite direction of a horizontal movement direction when it is detected that the vehicle is moving laterally.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing program. [Background technology]

[0002] When detecting white lines on a road from an image captured by a camera mounted on a vehicle, a technique is known in which a detection region for the white lines (ROI: Region of Interest) is set within the image and the white lines are detected within the ROI (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 10-208047 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem in that the detection performance of white lines is low.

[0005] For example, white lines are detected by performing image recognition processing on the ROI. In this case, a narrower ROI improves the accuracy of the image recognition processing and reduces the load on the image recognition processing. On the other hand, if the ROI is too narrow, the ROI may not be able to track the white lines when the vehicle moves sideways to change lanes, etc.

[0006] In contrast, the technology described in Patent Document 1 does not solve the problem of the ROI not being able to track the white lines when the vehicle is moving laterally because the ROI is constant regardless of whether the vehicle is moving forward or laterally. For this reason, the technology described in Patent Document 1 does not provide sufficient white line detection performance.

[0007] The present invention has been made in view of the above, and has an object to provide an image processing device and an image processing program that have high performance in detecting white lines. [Means for solving the problem]

[0008] The image processing device according to the present invention includes a controller that detects white lines within a detection area set in an image captured by a camera mounted on a vehicle. The controller detects lateral movement of the vehicle based on information about the vehicle. When the controller detects lateral movement of the vehicle, it expands the detection area in the direction opposite to the direction of lateral movement. [Effects of the Invention]

[0009] According to the present invention, the ROI of the captured image is changed in accordance with the movement of the vehicle, thereby improving the ability of the ROI to follow the white lines, thereby improving the white line detection performance. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a vehicle according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating white lines on a road. [Figure 3] FIG. 3 is a diagram illustrating the lateral movement. [Figure 4] FIG. 4 is a diagram showing the amount of movement of the white line. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of an in-vehicle device according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing a processing procedure of the in-vehicle device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating the expansion of the ROI. [Figure 8] FIG. 8 is a diagram illustrating the expansion of the ROI. [Figure 9] FIG. 9 is a diagram illustrating the expansion of the ROI. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of an image processing device and an image processing program disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments.

[0012] In this embodiment, the image processing device is realized as an in-vehicle device provided in a vehicle, but the image processing device may also be a terminal or a server connected to the vehicle via a network.

[0013] Fig. 1 is a diagram showing an example of the configuration of a vehicle according to an embodiment. As shown in Fig. 1, the vehicle V includes an in-vehicle device 10. The in-vehicle device 10 also includes a camera. The camera captures images in the traveling direction of the vehicle V. In this embodiment, the camera captures moving images.

[0014] The in-vehicle device 10 detects a white line within a ROI set in an image captured by a camera. The ROI may also be referred to as a detection area.

[0015] For example, the in-vehicle device 10 is a drive recorder. The in-vehicle device 10 may also be a device for assisting automatic or manual driving of the vehicle V. Based on the detection result of the white lines, the in-vehicle device 10 can warn the driver of deviation from the white lines and perform steering control of the vehicle V.

[0016] FIG. 2 is a diagram illustrating white lines on a road. The dashed arrow indicates the traveling direction of a vehicle V. As shown in FIG. 2, there may be multiple white lines drawn on a road. A white line 41 is a white line that needs to be detected. On the other hand, a white line 42 is a white line that does not need to be detected. In this case, in order to improve the detection accuracy of the white line 41, it is desirable that the ROI includes the white line 41 but does not include the white line 42.

[0017] Here, the width of each of the white lines 41 and 42 is 0.15 [m]. Also, the distance between the white lines 41 and 42 is 0.15 [m]. In this case, assuming that the ROI is a rectangle with sides parallel to the white line 41, it is desirable that the width of the ROI in the direction perpendicular to the white line 41 be 0.3 [m] or less in order to improve the accuracy of white line detection.

[0018] On the other hand, while the vehicle V is moving laterally to change lanes or the like, the white line in the captured image moves in the direction opposite to the direction of the lateral movement. In this case, the in-vehicle device 10 needs to make the ROI follow the moving white line.

[0019] Figure 3 is a diagram explaining lateral movement. Here, we consider a case where vehicle V moves laterally from the left lane to the right lane over a travel time of 3 [sec]. As a result, vehicle V changes lanes. The lane width is 3.5 [m].

[0020] The speed of lateral movement is not constant. As shown in Figure 3, vehicle V takes 1 / 4 of the travel time to increase its lateral movement speed from 0 to maximum speed, maintains the lateral movement speed at maximum speed for 1 / 2 of the travel time, and then takes 1 / 4 of the travel time to decrease its lateral movement speed from maximum speed back to 0. In this case, vehicle V travels 2 / 3 of the travel distance (e.g., equivalent to the width of the lane) while maintaining the maximum speed.

[0021] The in-vehicle device 10 processes images (for example, frames of a moving image) acquired at regular intervals. Meanwhile, while the vehicle V is moving laterally, the position of the white line in the image moves in the direction opposite to the lateral movement.

[0022] When an image is acquired, if the white line deviates from the ROI set in the previous process, the in-vehicle device 10 cannot detect the white line. In order to prevent the ROI from deviating from the white line, the in-vehicle device 10 needs to set a large ROI in advance.

[0023] The amount of movement of the white line in the image for each process varies depending on the time required for lateral movement, the maximum speed of lateral movement, and the time interval between processes. Figure 4 shows the amount of movement of the white line. Note that the vehicle V moves laterally in the procedure described in Figure 3.

[0024] "100 [ms] processing" means that the in-vehicle device 10 acquires and processes an image every 100 [ms]. For example, Fig. 4 shows that when the lateral movement time is 2 [sec], the maximum speed reaches 2.33 [m / s], and with 100 [ms] processing, the white line moves 0.23 [m] in the lateral direction between processes.

[0025] In particular, if the movement of the white line exceeds 0.3 m, the width of the ROI must be 0.3 m or more to prevent the white line from moving out of the ROI. However, in this case, the ROI will simultaneously include the white line 41 and the white line 42 in Figure 2, which will reduce the accuracy of white line detection by the in-vehicle device 10.

[0026] Therefore, the in-vehicle device 10 prevents both a decrease in detection accuracy and the white line from moving out of the ROI by expanding the ROI at the minimum necessary timing according to the situation of the vehicle V. As a result, according to this embodiment, the detection performance of the white line is improved.

[0027] The configuration of the in-vehicle device 10 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of the in-vehicle device according to the embodiment. As shown in Fig. 5, the in-vehicle device 10 includes an interface 11, a controller 12, and a memory 13.

[0028] The in-vehicle device 10 is connected to devices provided in the vehicle V. The in-vehicle device 10 is connected to a camera 21, a G sensor 22, a vehicle speed sensor 23, a direction indicator 24, and an output device 25.

[0029] The camera 21 captures images in the traveling direction of the vehicle V. The G-sensor 22 measures the acceleration of the vehicle V. The G-sensor 22 is capable of measuring at least the acceleration in a direction perpendicular to the traveling direction of the vehicle V, i.e., the left-right direction. The vehicle speed sensor 23 measures the speed of the vehicle V. The turn signal 24 indicates the direction in which the vehicle V is traveling, either in response to the driver's operation or automatically. The output device 25 is a device that outputs sound or images. For example, the output device 25 is a speaker and a display. The in-vehicle device 10 can acquire information from each device. The in-vehicle device 10 may also be connected to other ECUs (Electronic Control Units) and the like (not shown).

[0030] The interface 11 inputs and outputs data between the in-vehicle device 10 and other devices. For example, the interface 11 is a port compatible with USB (Universal Serial Bus), Ethernet (registered trademark), or the like.

[0031] The controller 12 reads and executes a program stored in the memory 13. The controller 12 is a central processing unit (CPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a graphics processing unit (GPU), a system on a chip (SoC), or the like.

[0032] The controller 12 may be a single processor, a multiprocessor configuration, or a multicore configuration having multiple cores in a single chip connected via a single socket.

[0033] The memory 13 is a storage medium such as an eMMC (embedded multi media card), etc. The memory 13 functions as a ROM (read only memory) or a RAM (random access memory).

[0034] The flow of processing in the in-vehicle device 10 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the processing procedure of the in-vehicle device according to the embodiment. The processing shown in Fig. 6 is executed by the controller 12.

[0035] 6, first, the controller 12 acquires the traveling state of the vehicle (step S101). For example, the controller 12 acquires the speed of the vehicle V from the vehicle speed sensor 23.

[0036] If the vehicle is not moving (step S102, No), the controller 12 returns to step S101. On the other hand, if the vehicle is moving (step S102, Yes), the controller 12 proceeds to step S103. For example, if the vehicle speed is greater than 0, the controller 12 determines that the vehicle V is moving.

[0037] The controller 12 acquires images captured by the camera 21 of the vehicle V (step S103). For example, the controller 12 acquires one or more frames included in a moving image.

[0038] Here, the controller 12 determines whether or not initialization of the ROI is necessary (step S104). The controller 12 determines that initialization of the ROI is necessary when the ROI is not set, when a white line cannot be detected in the process of detecting the white line (step S108), when the ROI for the white line detected in the process of detecting the white line is too wide, etc.

[0039] In addition, the controller 12 may determine that initialization of the ROI is necessary if a certain time has elapsed after lateral movement of the vehicle is detected (step S106) or after the process of enlarging the ROI (step S107) is performed.

[0040] If ROI initialization is not required (step S104, No), the controller 12 proceeds to step S106 without initializing the ROI. On the other hand, if ROI initialization is required (step S104, Yes), the controller 12 initializes the ROI based on the image (step S105).

[0041] For example, the controller 12 detects white lines from the entire image and sets an area including the detected white lines as the ROI.

[0042] Next, if the controller 12 detects a lateral movement of the vehicle V (step S106, Yes), the controller 12 proceeds to step S107 and expands the ROI. On the other hand, if the controller 12 does not detect a lateral movement of the vehicle V (step S106, No), the controller 12 proceeds to step S108 without expanding the ROI.

[0043] Specifically, the controller 12 detects, based on information about the vehicle V, that the vehicle V is about to start moving laterally or that the vehicle V is in the middle of moving laterally.

[0044] The controller 12 can detect that the vehicle V is moving laterally based on the direction indicated by the turn indicator 24 of the vehicle V. For example, when the turn indicator 24 indicates the right direction, the controller 12 detects that the vehicle V is moving laterally to the right.

[0045] The controller 12 can detect that the vehicle V is moving laterally based on the sensor value of the G sensor 22 of the vehicle V. For example, when the rightward component of the acceleration indicated by the sensor value of the G sensor 22 is positive or the leftward component is negative, the controller 12 detects that the vehicle V is moving laterally to the right.

[0046] The controller 12 can detect that the vehicle V is moving laterally based on the amount of movement of the white line captured in the image. For example, when the controller 12 detects a white line from multiple images captured at different times, if the position of the detected white line is moving leftward over time and the amount of movement exceeds a threshold, the controller 12 detects that the vehicle V is moving laterally to the right.

[0047] In this way, the controller 12 can easily detect lateral movement based on information acquired from devices provided in the vehicle V.

[0048] When the controller 12 detects a lateral movement (step S106, Yes), the controller 12 enlarges the ROI according to the direction of the lateral movement (step S107).

[0049] When the controller 12 detects that the vehicle V is moving laterally, it expands the ROI in the direction opposite to the direction of the lateral movement. For example, when the controller 12 detects that the vehicle V is moving leftward, it expands the right side of the ROI, and when the controller 12 detects that the vehicle V is moving rightward, it expands the left side of the ROI.

[0050] By expanding the ROI when lateral movement is detected, the controller 12 prevents both a decrease in detection accuracy and the white line from moving out of the ROI. As a result, according to this embodiment, the detection performance of the white line is improved. Furthermore, when the vehicle V does not move laterally, the size of the ROI is maintained at the minimum necessary size that includes the white line, preventing erroneous detection of the white line.

[0051] The expansion of ROI will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining the expansion of ROI. Fig. 7(a) is an example of the case where ROI is not expanded. Fig. 7(b) is an example of the case where ROI is expanded, that is, an example of this embodiment.

[0052] First, (a) of Fig. 7 will be described. As shown in Fig. 7, an ROI 61a is set in an image 511a. An image 512a is an image taken after the image 511a. In the image 512a, the position of the white line and the size of the ROI 61a have not changed compared to the image 511a.

[0053] Image 513a was captured after image 512a. In image 513a, the position of the white line has moved to the left compared to image 512a. This means that vehicle V has moved laterally to the right. However, because the size of ROI 61a has not changed, part of the white line is outside of ROI 61a.

[0054] Next, (b) of Fig. 7 will be described. As shown in Fig. 7, an ROI 61 is set in an image 511. An image 512 is an image taken after the image 511. In the image 512, the position of the white line has not changed compared to the image 511.

[0055] However, the controller 12 detects that the vehicle V is moving laterally to the right when the image 512 is captured, so the controller 12 expands the ROI 61 to the left.

[0056] The depth of each image shown in FIG. 7 is the traveling direction of the vehicle V. The horizontal axis in FIG. 7 represents the position in the left-right direction. The position of the right end of the ROI 61 before enlargement is x 11 The left edge of ROI61 before enlargement is x 12 The left edge of the enlarged ROI61 is located at x 13 is.

[0057] That is, the controller 12 sets the ROI 61 as x 12 -x 13 The amount of expansion of the ROI by the controller 12 may be a predetermined amount, or may be proportional to the absolute value of the lateral acceleration.

[0058] Image 513 is an image taken after image 512. In image 513, the position of the white line has moved to the left compared to image 512. In addition, since ROI 61 has already been enlarged, the white line does not move out of ROI 61.

[0059] The shape of the ROI is not limited to that shown in FIG. 7. For example, the shape of the ROI may be a trapezoid as shown in FIG. 8. FIG. 8 is a diagram for explaining the enlargement of the ROI. When an image is captured using a camera 21 equipped with a normal lens, the white line appears in the image in a shape similar to the left and right sides of a trapezoid. By making the shape of the ROI more closely match the shape of the white line, the detection accuracy can be further improved.

[0060] 8, an ROI 62 is set in image 521. Image 522 is an image captured after image 521. In image 522, the position of the white line has not changed compared to image 521.

[0061] Here, the controller 12 expands the ROI 62. The position of the right end of the ROI 62 before expansion is x 21 The left edge of ROI62 before enlargement is x 22 The left edge of the enlarged ROI62 is located at x 23 is.

[0062] Image 523 is an image taken after image 522. In image 523, the position of the white line has moved to the left compared to image 522. In addition, since ROI 62 has already been enlarged, the white line does not move out of ROI 62.

[0063] As shown in Fig. 9, an ROI may be set for each of the white lines on the left and right sides of a lane. As shown in Fig. 9, ROI 63 and ROI 64 are set in image 531. Image 532 is an image captured after image 531. In image 532, the positions of the white lines have not changed compared to image 531.

[0064] Here, the controller 12 expands the ROI 63 and the ROI 64. The position of the right end of the ROI 63 before expansion is x 31 The left edge of ROI63 before enlargement is x 32 The left edge of the enlarged ROI63 is located at x 33 The right edge of ROI64 before enlargement is x 41 The left edge of ROI64 before enlargement is x 42 The left edge of the enlarged ROI64 is located at x 43 In this way, the controller 12 can expand multiple ROIs set on an image.

[0065] Image 533 is an image taken after image 532. In image 533, the position of the white line has moved to the left compared to image 532. Furthermore, because ROI 63 and ROI 64 have already been enlarged, neither of the left nor right white lines has moved out of the ROI.

[0066] Furthermore, when the controller 12 detects that the vehicle V is moving laterally, the controller 12 may expand the ROI in both the direction of the lateral movement and the opposite direction. For example, when the controller 12 detects that the vehicle V is moving laterally, the controller 12 expands the left and right sides of the ROI. This allows the controller 12 to prevent the white line from deviating from the ROI even when the direction of the lateral movement cannot be clearly identified.

[0067] 6, the controller 12 detects a white line within the ROI (step S108). Then, the controller 12 outputs information according to the detection result (step S109). For example, if the controller 12 cannot detect a white line, the controller 12 causes the output device 25 to output a warning sound or a warning message.

[0068] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0069] V vehicle 10 Onboard equipment 11 Interface 12 Controllers 13. Memory 21 Camera 22 G sensor 23 Vehicle speed sensor 24 Turn signal 25 Output Devices 41, 42 White lines 61, 61a, 62, 63, 64 ROIs 511, 512, 513, 521, 522, 523, 531, 532, 533 Images

Claims

1. a controller that detects a white line within a detection area set in an image captured by a camera provided on the vehicle; The controller Detecting that the vehicle is moving laterally based on information about the vehicle; When it is detected that the vehicle is moving laterally, the detection area is expanded in a direction opposite to the direction of the lateral movement. Image processing device.

2. The controller When the vehicle is detected to be moving laterally, the left and right sides of the detection area are expanded. The image processing device according to claim 1 .

3. The controller When it is detected that the vehicle is moving leftward, the right side of the detection area is expanded, and when it is detected that the vehicle is moving rightward, the left side of the detection area is expanded. The image processing device according to claim 1 .

4. The controller Detecting lateral movement of the vehicle based on the direction indicated by the turn signal of the vehicle. The image processing device according to claim 1 .

5. The controller Detecting lateral movement of the vehicle based on the sensor value of the G sensor of the vehicle The image processing device according to claim 1 .

6. The controller Detecting lateral movement of the vehicle based on the amount of movement of the white line in the image The image processing device according to claim 1 .

7. A computer that executes a process of detecting a white line within a detection area set in an image captured by a camera provided on a vehicle, Detecting that the vehicle is moving laterally based on information about the vehicle; When it is detected that the vehicle is moving laterally, the detection area is expanded in a direction opposite to the direction of the lateral movement. An image processing program that performs the processing.

Images