Vehicle control device
The vehicle control device uses a side camera with image processing to generate rear and side views, addressing the cost issue of multiple camera setups and enhancing visibility for drivers.
Patent Information
- Application Number
- JP2023078748
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-21
AI Technical Summary
Existing vehicle monitoring systems require separate cameras for rear and side views, increasing costs due to the need for multiple camera setups.
A vehicle control device that utilizes a side camera with a wide-angle lens to generate images for both rear and side monitoring by cutting out a partial area from the side image, correcting distortion, and displaying it on an in-car monitor to mimic a conventional mirror view.
Reduces costs by utilizing a single side camera for both rear and side monitoring, providing a comfortable and effective visual representation of the vehicle's surroundings without the need for additional cameras, thereby reducing driver discomfort and visual blind spots.
Smart Images

Figure 2024162828000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a vehicle control device. [Background technology]
[0002] In recent years, electronic side mirrors capable of displaying camera images have been developed to replace side mirrors for checking the rear side of a vehicle. In addition, there are cases where a vehicle is equipped with multiple cameras for monitoring the surroundings, including the sides of the vehicle.
[0003] For example, there is a technology that shares some functions between a device that controls a camera for checking the rear side of the vehicle to monitor the rear side of the vehicle and a device that controls a side camera for monitoring the surroundings of the vehicle (for example, Patent Document 1). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent Publication No. 2021-069075 Summary of the Invention [Problem to be solved by the invention]
[0005] However, the technique of Patent Document 1 requires the combined use of a camera for checking the rear side and a side camera for monitoring the surroundings, which increases costs.
[0006] An object of the present disclosure is to provide a vehicle control device that enables a side camera for periphery monitoring to also be used for rear-side monitoring. [Means for solving the problem]
[0007] The vehicle control device disclosed herein comprises an input unit that receives a side image of the vehicle from a camera having a wide-angle lens mounted on the side of the vehicle, and a generation unit that cuts out a partial area from the side image and generates first and second images to be displayed on an in-vehicle monitor.When generating the first image, the generation unit cuts out the partial area so that the cut-out center in the horizontal direction of the side image is to the rear side of the vehicle, starting from the mounting position of the camera in the fore-and-aft direction of the vehicle, mirror-inverts the cut-out partial area, and corrects distortion of the side image caused by the wide-angle lens.When generating the second image, the generation unit cuts out the partial area so that the cut-out center in the horizontal direction of the side image is to coincide with a line perpendicular to a line connecting the front and rear ends of the vehicle, starting from the mounting position of the camera in the fore-and-aft direction of the vehicle. Effect of the Invention
[0008] According to the vehicle control device of the present disclosure, the side camera for monitoring the surroundings can also be used for monitoring the rear side. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a vehicle including an ECU according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of the ECU according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating the range of the angle of view cut out from the angle of view of the side camera when the ECU according to the embodiment generates an image for monitoring the rear side. [Figure 4] FIG. 4 is a diagram illustrating a state in which the ECU according to the embodiment generates an image for monitoring the rear side. [Diagram 5] FIG. 5 is a diagram illustrating a state in which the ECU according to the embodiment generates an image for periphery monitoring. [Figure 6] FIG. 6 is a flowchart illustrating an example of a procedure of an image generation process performed by the ECU according to the embodiment. [Figure 7] FIG. 7 is a diagram showing other examples of various images displayed on the CID according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an embodiment of a vehicle control device according to the present disclosure will be described with reference to the drawings.
[0011] (Example of ECU configuration) FIG. 1 is a block diagram showing an example of a configuration of a vehicle 10 including an ECU (Electronic Control Unit) 20 according to an embodiment.
[0012] As shown in FIG. 1, the vehicle 10 of the embodiment is equipped with a front camera 11F, side cameras 11R, 11L, a rear camera 11B, a meter display 12M, a center information display (CID) 12C, a wheel speed sensor 13, a steering angle sensor 14, a shift sensor 15, a turn switch 16, an ignition switch 17, and an ECU 20.
[0013] The ECU 20 is connected to the front camera 11F, the side cameras 11R and 11L, the rear camera 11B, the meter display 12M, and the CID 12C by a display cable such as an HDMI (High-Definition Multimedia Interface) (registered trademark) cable. This allows images captured by the front camera 11F, the side cameras 11R and 11L, and the rear camera 11B to be converted into digital signals and transmitted.
[0014] The ECU 20 is connected to the wheel speed sensor 13, the steering angle sensor 14, the shift sensor 15, the turn switch 16, and the ignition switch 17 via an in-vehicle network such as a Controller Area Network (CAN). This allows the ECU 20 to receive signals from, for example, the wheel speed sensor 13 and the turn switch 16, as described later.
[0015] The front camera 11F is provided at the front of the vehicle 10, for example, at the front bumper, and captures a forward image of the vehicle 10. The side camera 11R as a first camera is provided at the right side of the vehicle 10, for example, at the right door mirror, as viewed from the driver's seat, and captures a right side image of the vehicle 10. The side camera 11L as a second camera is provided at the left side of the vehicle 10, for example, at the left door mirror, as viewed from the driver's seat, and captures a left side image of the vehicle 10. The back camera 11B is provided at the rear of the vehicle 10, for example, at the rear bumper, and captures a rear image of the vehicle 10.
[0016] The surrounding images captured by the front camera 11F, the side cameras 11R and 11L, and the back camera 11B are used to monitor the surroundings of the vehicle 10, for example, when parking or exiting the vehicle. For this reason, in order to minimize blind spots, a wide-angle lens such as a fisheye lens is used for the front camera 11F, the side cameras 11R and 11L, and the back camera 11B.
[0017] The meter display 12M is a meter panel type liquid crystal display (LCD) or the like, and is provided, for example, on an instrument panel in front of the driver's seat and behind a steering wheel. The meter display 12M digitally displays at least one meter, for example, a speedometer, an engine tachometer, a fuel gauge, a water temperature gauge, a distance gauge, and various warning lights and indicator lights, and is configured to be capable of displaying, for example, side images of the vehicle 10 captured by the side cameras 11R and 11L.
[0018] The meters digitally displayed on the meter display 12M may display various warning lights or indicator lights in addition to the above-mentioned instruments. In this manner, the meters on the meter display 12M are configured to be capable of presenting the driver with information necessary for driving the vehicle 10.
[0019] The CID 12C is an LCD or the like, and is provided, for example, near the center of an instrument panel in front of the driver's seat. The CID 12C is configured to display, for example, images of the surroundings of the vehicle 10 captured by the front camera 11F, the side cameras 11R and 11L, and the back camera 11B, and to display an overhead image obtained by synthesizing these images of the surroundings, as well as to present various information to the driver.
[0020] In addition, CID 12C may be configured to be able to display a navigation screen by being used in a navigation system. Also, CID 12C may be configured to be able to display broadcast programs, etc. by being connected to a receiver for broadcast signals, etc. In this way, CID 12C is configured to be able to display information that is not necessarily required for driving vehicle 10.
[0021] The meter display 12M is an example of a second monitor, the CID 12C is an example of a first monitor, and the meter display 12M and the CID 12C are an example of an in-vehicle monitor.
[0022] CID 12C is provided with a function switch 12s used for various operations such as starting the display of CID 12C, changing the display contents, or switching the display destination between meter display 12M and CID 12C. As described above, the display of CID 12C includes the display of a peripheral image and an overhead image, and the display of various information. Thus, function switch 12s is an example of a display switch that turns on the camera functions of front camera 11F, side cameras 11R and 11L, and rear camera 11B to start image display.
[0023] The wheel speed sensor 13 is configured using, for example, a Hall element, and outputs a wheel speed pulse representing the wheel speed as a detection result based on the amount of rotation or the number of rotations per unit time of the wheels of the vehicle 10.
[0024] The steering angle sensor 14 detects the angle of the steering wheel, and outputs a steering angle signal as a detection result.
[0025] The shift sensor 15 includes, for example, a displacement sensor, and detects the position of a lever, arm, button, or the like for performing a gear shift operation of the vehicle 10, and outputs a signal indicating the shift position as the detection result.
[0026] The turn switch 16 is a lever, arm, button, or the like for flashing the turn signal lights that show the direction of the vehicle 10 when the vehicle 10 makes a right or left turn or changes lanes, and when this operation is performed, it outputs a control signal for turning on the turn signal lights.
[0027] The ignition switch 17 is a switch that starts the engine or motor and the electrical system, and is operated by rotating the key cylinder with the engine key, or by pressing a button, etc. When the ignition switch 17 is operated, it outputs a control signal for starting the engine, etc.
[0028] The ECU 20 is configured as a computer equipped with, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), which are not shown, and generates images that have been subjected to various processing from peripheral images acquired from the front camera 11F, the side cameras 11R and 11L, the back camera 11B, etc., and outputs the images appropriately to the meter display 12M and the CID 12C. At this time, the ECU 20 generates images suited to the situation at the time according to the state of the vehicle 10 at that time and the driver's instructions, and also changes the output destination of the images.
[0029] In this manner, the ECU 20 is configured as a vehicle control device that processes various peripheral images and outputs them to the meter display 12M and the CID 12C.
[0030] The ECU 20 may be composed of a plurality of ECUs, such as a camera ECU, a meter ECU, and a display ECU. The camera ECU controls, for example, the front camera 11F, the side cameras 11R and 11L, and the rear camera 11B, and performs various processing on the surrounding images acquired from these cameras. The meter ECU controls the meter display 12M to display the images processed by the camera ECU and various information including the digital display of various meters on the meter display 12M. The display ECU controls the CID 12C to display various information including the images processed by the camera ECU on the CID 12C.
[0031] Furthermore, the ECU 20 may obtain signals from various sensors via other ECUs such as a body ECU and a brake ECU. The body ECU controls the turn switch 16, a headlight switch (not shown), and a door lock motor, which need to be operated even when the ignition switch of the vehicle 1 is off. The brake ECU obtains the vehicle speed of the vehicle 10 by, for example, counting the vehicle speed pulses of the wheel speed sensor 13, and applies braking to the wheels including the drive wheels by the brakes provided on each wheel.
[0032] Fig. 2 is a block diagram showing an example of a functional configuration of the ECU 20 according to the embodiment. As shown in Fig. 2, the ECU 20 includes, as its functional configuration, an input unit 21, a determination unit 22, a generation unit 23, a detection unit 24, an output unit 25, and a storage unit 26. These functional units are realized, for example, by a CPU (not shown) included in the ECU 20 expanding a control program stored in a ROM or the like into a RAM and executing the program.
[0033] The input unit 21 acquires surrounding images captured by the front camera 11F, the side cameras 11R, 11L, and the back camera 11B via the above-mentioned display cable, etc. The input unit 21 also acquires the wheel speed detection result and a control signal for blinking the turn signal lamp from the wheel speed sensor 13 and the turn switch 16 via the above-mentioned in-vehicle network, etc. The input unit 21 also acquires a control signal indicating that the above-mentioned on operation, etc. has been performed from the function switch 12s via the main body of the CID 12C when the image display function is turned on by the function switch 12s.
[0034] The determination unit 22 determines whether or not the speed of the vehicle 10 at that time is less than a predetermined speed based on the detection result of the wheel speed sensor 13. The predetermined speed of the vehicle 10 used by the determination unit 22 for the determination can be a very low speed, for example, less than 10 km / h to 15 km / h.
[0035] Furthermore, the determination unit 22 determines that the vehicle 10 is about to change course based on the generation of a control signal for the turn signal lamp by the operation of the turn switch 16. The determination unit 22 also determines whether or not an operation to turn on the image display function has been performed by the function switch 12s provided in the CID 12C.
[0036] The generation unit 23 generates an image for monitoring the periphery of the vehicle 10, an image for monitoring the rear side, an overhead image, etc., based on images captured by the front camera 11F, the side cameras 11R and 11L, and the back camera 11B.
[0037] As described above, the surroundings monitoring image is an image for showing the driver the state of the surroundings of the vehicle 10, such as obstacles or pedestrians around the vehicle 10, for example, when parking or exiting the vehicle. The surroundings monitoring image is generated based on any of the images captured by the front camera 11F, the side cameras 11R and 11L, and the back camera 11B. Among these surroundings monitoring images, the side monitoring image generated based on the side images of the side cameras 11R and 11L may be displayed to check the state of the parking space, for example, when the vehicle 10 is parked next to the parking space. The side monitoring image is an example of a second image.
[0038] The overhead image is a composite image showing 360° around the vehicle 10 with the vehicle icon at the center, and is used for driving assistance when parking or leaving the vehicle, for example.
[0039] The image for monitoring the rear side as the first image is an image for showing the driver the state of the rear side when the vehicle 10 changes course, for example.
[0040] The generation unit 23 determines an image to be generated as appropriate according to various states of the vehicle 10 determined by the determination unit 22.
[0041] More specifically, when the turn switch 16 is operated and the vehicle 10 is about to change course, the generation unit 23 generates, for example, an image for monitoring the rear side. When the speed of the vehicle 10 is less than a predetermined speed, the generation unit 23 generates, for example, an image for monitoring the surroundings. When parking or exiting assistance control is started by the function switch 12s of the CID 12C, the generation unit 23 may generate an overhead image in addition to the image for monitoring the surroundings.
[0042] The detection unit 24 detects events that may affect the traveling of the vehicle 10, such as the presence of another vehicle or a following vehicle such as a motorbike, a bicycle, or a pedestrian, etc., on the rear side of the vehicle 10 within a predetermined distance of the vehicle 10, from the rear side monitoring image generated when the vehicle 10 changes course. Detection by the detection unit 24 can be performed using image recognition technology, etc.
[0043] When an event that may affect the traveling of the vehicle 10 occurs, the detection unit 24 generates warning information linked to the rear side monitoring image generated by the generation unit 23. The warning information is, for example, a mark, a character, or other symbol indicating the presence of an event that may affect the traveling of the vehicle 10. These are appropriately displayed on the meter display 12M or the CID 12C together with the rear side monitoring image. At this time, these may be displayed superimposed on the rear side monitoring image.
[0044] Moreover, the warning information may include a warning sound such as an alarm, a buzzer sound, or an announcement, instead of or in addition to the above display. The warning sound may be output from a speaker (not shown) provided in the CID 12C, for example.
[0045] The output unit 25 outputs the periphery monitoring image, rear side monitoring image, and overhead image generated by the generation unit 23, as well as warning information linked to the rear side monitoring image, to the meter display 12M or CiD 12C as appropriate via the above-mentioned display cable or the like. At this time, the periphery monitoring image and the overhead image are output to, for example, CiD 12C. In addition, the rear side monitoring image can be output to the meter display 12M or CiD 12C depending on the situation. As a result, these various images and the like are displayed on the meter display 12M or CiD 12C as appropriate.
[0046] The storage unit 26 stores control parameters and control programs used to realize various functions of the ECU 20. The storage unit 26 may store surrounding images acquired from the front camera 11F, the side cameras 11R and 11L, and the rear camera 11B. The storage unit 26 may also store surroundings monitoring images, rear-side monitoring images, overhead images, and the like generated by the generation unit 23. When there is warning information linked to the rear-side monitoring image, the storage unit 26 may store the warning information together with the rear-side monitoring image.
[0047] (Example of ECU functions) Next, a detailed example of the functions of the ECU 20 according to the embodiment will be described with reference to FIGS.
[0048] 3A and 3B are diagrams illustrating the range of the angle of view cut out from the angle of view of the side cameras 11R and 11L when the ECU 20 according to the embodiment generates an image for monitoring the rear side. More specifically, FIG. 3A shows the cut-out angle of view as viewed from above the vehicle 10, FIG. 3B shows the cut-out angle of view as viewed from the left side of the vehicle 10, and FIG. 3C shows the cut-out angle of view as viewed from the right side of the vehicle 10.
[0049] As shown in Fig. 3, the side camera 11R is attached to the door mirror on the right side of the vehicle 10 as seen from the driver's seat, with the angle of view facing slightly downward. The side camera 11L is attached to the door mirror on the left side of the vehicle 10 as seen from the driver's seat, with the angle of view facing slightly downward, similar to the side camera 11R. As described above, the cameras for monitoring the surroundings, including the side cameras 11R and 11L, use, for example, wide-angle lenses. Therefore, the surrounding images, including the side images of the vehicle 10 captured by the side cameras 11R and 11L, include a range wider than, for example, the human field of vision.
[0050] As shown in FIG. 3(a), the horizontal angles of view VArh, VAlh of the side cameras 11R, 11L are both, for example, approximately 190°, and the optical axes VXrt, VXlt of the side cameras 11R, 11L each extend horizontally from the mounting positions of the side cameras 11R, 11L in the fore-and-aft direction of the vehicle 10 toward the side of the vehicle 10 on a line perpendicular to the line connecting the front and rear ends of the vehicle 10.
[0051] That is, the optical axis VXrt of the side camera 11R starts near the door mirror on the right side of the vehicle 10 and extends perpendicular to the right side surface of the vehicle 10 to the right side of the vehicle 10. The optical axis VXlt of the side camera 11L starts near the door mirror on the left side of the vehicle 10 and extends perpendicular to the left side surface of the vehicle 10 to the left side of the vehicle 10.
[0052] As shown in Figures 3(b) and (c), the vertical viewing angles VArv, VAlv of the side cameras 11R, 11L are both, for example, approximately 160°, and the optical axes VXrt, VXlt of the side cameras 11R, 11L are both directed diagonally downward on a line connecting the left and right ends of the vehicle 10 at the mounting heights of the side cameras 11R, 11L in the vertical direction.
[0053] On the other hand, when generating an image for rear-side monitoring, the generating unit 23 of the ECU 20 cuts out a predetermined range from the above-mentioned angle of view, and generates an image that captures an angle of view range similar to that of a normal door mirror, for example. As a result, when the image for rear-side monitoring is displayed on the meter display 12M or the CID 12C, the driver can monitor the rear-side without feeling uncomfortable, as if he or she were looking through a normal door mirror.
[0054] As shown in FIG. 3(a), when cutting out a specified range from the side images of the side cameras 11R and 11L, the generation unit 23, as an example, sets the horizontal viewing angles FArh and FAlh of the cut-out range to be greater than or equal to 60° and less than 110°, and cuts out the range so that the central axes FXrt and FXlt of the cut-out range are both to the rear side of the vehicle 10 in the horizontal direction.
[0055] That is, when generating a right rear side monitoring image based on the right side image captured by the side camera 11R, the generating unit 23 cuts out a partial area so that the cut-out center is the right rear side of the vehicle 10 from the right side image. When generating a left rear side monitoring image based on the left side image captured by the side camera 11L, the generating unit 23 cuts out a partial area so that the cut-out center is the left rear side of the vehicle 10 from the left side image.
[0056] As a result, the centers of cut-outs from the side images of the side cameras 11R and 11L are shifted toward the rear of the vehicle 10 from the optical axes VXrt and VXlt of the side cameras 11R and 11L, respectively.
[0057] As shown in Figures 3(b) and (c), the generation unit 23 can cut out the image so that the vertical angles of view FArv, FAlv of the cut-out range in the lateral image of the side cameras 11R, 11L are both greater than or equal to 45° and less than 70°, and the central axes FXrt, FXlt of the cut-out range are both positioned vertically on a line connecting the front and rear ends of the vehicle 10 at the mounting height of the side cameras 11R, 11L, or slightly below that line.
[0058] As a result, the center of cropping from the side images of the side cameras 11R, 11L is shifted upward from the optical axes VXrt, VXlt of the side cameras 11R, 11L. As an example, the generation unit 23 can arrange the rear position of the vehicle 10 in the side image on a line connecting the front and rear ends of the vehicle 10 at the mounting height of the side cameras 11R, 11L in an area of 30% to 60% from the upper end of the rear side monitoring image.
[0059] When the generating unit 23 generates an image for monitoring the surroundings from the side images of the side cameras 11R and 11L, an area of a predetermined range may be cut out from the angle of view of these side images.
[0060] In this case, the horizontal angle of view in the cut-out range from the lateral image of the side cameras 11R, 11L can be set to 60° or more and 110° or less, as when generating an image for rear-side monitoring, while the central axis of the image for peripheral monitoring can be aligned with the optical axes VXrt, VXlt of the side cameras 11R, 11L, respectively.
[0061] Also, the vertical angle of view in the range cut out from the side images of the side cameras 11R, 11L can be set to 45° or more and 70° or less, for example, as in the case of generating a rear side monitoring image, and the image can be cut out so that the central axis of the periphery monitoring image is located below a line connecting the left and right ends of the vehicle 10 at the mounting height of the side cameras 11R, 11L. In this case, the central axis of the periphery monitoring image may be aligned with, for example, the optical axes VXrt, VXlt of the side cameras 11R, 11L, respectively.
[0062] In the above case, the side position of the vehicle 10 in the side image on the line connecting the left and right ends of the vehicle 10 at the mounting height of the side cameras 11R, 11L will be located above the vertical center position of the cut-out range. As an example, the generation unit 23 can arrange the side position of the vehicle 10 in the side image on the line connecting the left and right ends of the vehicle 10 in an area between 10% and 20% from the upper end of the periphery monitoring image.
[0063] FIG. 4 is a diagram showing how the ECU 20 according to the embodiment generates an image for monitoring the rear side.
[0064] In the example of Fig. 4, for example, the turn switch 16 is operated and the vehicle 10 is about to change lanes to the lane immediately to the left of the own lane. In this case, the generation unit 23 of the ECU 20 generates a left rear side image based on a left side image of the vehicle 10 captured by the side camera 11L. As described below, the left rear side image is generated as if it were a mirror image seen through the left door mirror.
[0065] Fig. 4(a) is an example of a side image of the vehicle 10 captured by the side camera 11L. In the example of Fig. 4(a), the side image shows the vehicle's own lane in front of and behind the vehicle 10 traveling on the road, an adjacent lane to the left of the vehicle's own lane, and a following vehicle traveling in the adjacent lane. The image on the lower side of the side image in Fig. 4(a) is the body of the vehicle 10.
[0066] As shown in FIG. 4(a), the unprocessed side image includes the front, rear, and sides of the vehicle 10 centered on the mounting position of the side camera 11L, and captures an area wider than a human field of vision, for example, and also contains distortion caused by the wide-angle lens.
[0067] The generation unit 23 of the ECU 20 cuts out a predetermined range from such a side image of Fig. 4(a) and generates an image for rear-side monitoring as seen through a door mirror. Therefore, the cut-out range from the side image including the front, rear, and sides of the vehicle 10 can be mainly an area including the adjacent lane on the rear side of the vehicle 10 and the following vehicle traveling in the adjacent lane.
[0068] 4(b) to 4(d) show how an image for monitoring the rear side is generated from the side image in FIG. 4(a).
[0069] As shown in Fig. 4(b), the generation unit 23 mirror-inverts the side image captured by the side camera 11L. This is because the image seen through the door mirror should be a mirror image of the side image captured by the side camera 11L. In addition, the generation unit 23 cuts out, as the cut-out area CAb used for the rear-side monitoring image, a range narrower than the entire mirror-inverted side image in both the horizontal and vertical directions.
[0070] At this time, the generation unit 23 cuts out an area toward the rear of the vehicle 10 from the mirror-inverted side image so that the cut-out center in the horizontal direction is on the rear side of the vehicle 10. The generation unit 23 also cuts out an area from the mirror-inverted side image so that the cut-out center in the vertical direction is located on a line connecting the front and rear ends of the vehicle 10 at the mounting height of the side camera 11L or slightly below that line.
[0071] FIG. 4(c) shows how the generation unit 23 corrects distortion caused by the wide-angle lens in the cutout area CAb used for the rear-side monitoring image.
[0072] As shown in Fig. 4(c), the generating unit 23 corrects the image distortion in the cutout area CAb by using an image processing technique such as warping. For example, when correcting the image distortion by warping, the generating unit 23 divides the image of the cutout area CAb into a mesh shape and moves a plurality of feature points FP present at each intersection of the mesh to predetermined coordinates. At this time, the moving distance of each feature point FP is appropriately changed, thereby correcting the image distortion in the cutout area CAb.
[0073] Fig. 4(d) is an example of a rear-side monitoring image generated from the side image of the side camera 11L by the generating unit 23. As shown in Fig. 4(d), the rear-side monitoring image is generated in a state in which the range of the cutout area CAb in Fig. 4(b) is corrected for distortion.
[0074] The rear side portion of the left side of the vehicle 10 is reflected in the rear side monitoring image generated by the generation unit 23. As described above, the cut-out area CAb in Fig. 4(b) is set so that the cut-out center in the horizontal direction is the rear side of the vehicle 10. Therefore, the proportion of the rear side portion of the vehicle 10 in the rear side monitoring image is, for example, 20% to 30%.
[0075] As described above, the cut-out area CAb in Fig. 4(b) is set so that the cut-out center in the vertical direction is located on the line connecting the front and rear ends of the vehicle 10 at the mounting height of the side camera 11L or slightly below that line. Therefore, the rear position of the vehicle 10 in the side image on a line horizontal to the line connecting the front and rear ends of the vehicle 10 at the mounting height of the side camera 11L is located within an area of 30% to 60% from the upper end of the rear side monitoring image.
[0076] The detection unit 24 of the ECU 20 detects, for example, by using image recognition technology, events that may affect the traveling of the vehicle 10 in the rear-side monitoring image generated by the generation unit 23. That is, when, in the rear-side monitoring image, another vehicle, motorbike, bicycle, pedestrian, or the like is present within a predetermined distance of the vehicle 10 on the road surface or the like in the direction in which the vehicle 10 is about to change course, the detection unit 24 detects these as events that may affect the traveling of the vehicle 10.
[0077] In the example of Fig. 4(d), it is assumed that a following vehicle is detected in the rear-side monitoring image in the adjacent lane on the left side, into which the vehicle 10 is to change course, and is sufficiently close to the vehicle 10. In this case, the detection unit 24 regards this following vehicle as an event that may affect the traveling of the vehicle 10, and generates warning information linked to this rear-side image.
[0078] Fig. 4(e) shows a state in which the rear-side image generated by the generation unit 23 is displayed on the CID 12C of the vehicle 10. As shown in Fig. 4(e), a triangular mark 29c is attached to the following vehicle in the rear-side image based on the warning information generated by the detection unit 24, indicating that the event may affect the traveling of the vehicle 10.
[0079] Fig. 4(f) shows a state in which the rear side image generated by the generation unit 23 is displayed on the meter display 12M of the vehicle 10. As shown in Fig. 4(f), the meter display 12M digitally displays various meters such as a speedometer 121 and a fuel gauge 122, and is also provided with a display unit 123 such as an LCD that can display various information.
[0080] The display unit 123 of the meter display 12M displays the rear-side image generated by the generation unit 23, and the following vehicle in the rear-side image is marked with a triangular mark 29m based on the warning information generated by the detection unit 24, indicating that this is an event that may affect the driving of the vehicle 10.
[0081] When the rear side image is displayed on the CID 12C or the meter display 12M, a warning sound or the like may be issued based on the warning information generated by the detection unit 24 as described above.
[0082] 4 is also applied when the generating unit 23 generates an image for monitoring the right rear side based on a right side image captured by the side camera 11R on the right side of the vehicle 10. In this case, the right rear side image is generated as if it were a mirror image seen through the right door mirror.
[0083] That is, the generation unit 23 mirror-inverts the right side image of the vehicle 10 captured by the side camera 11R, cuts out the area toward the right rear side of the vehicle 10, and corrects distortion caused by the wide-angle lens.
[0084] This generates a right rear side monitoring image in which the right rear lateral portion of the vehicle 10 is included in a range of, for example, 20% to 30%, and the rear position of the vehicle 10 in the side image on a line horizontal to a line connecting the front and rear ends of the vehicle 10 at the mounting height of the side camera 11R is located within an area of 30% to 60% from the top end.
[0085] When an event that may affect the traveling of the vehicle 10 is detected in the right rear monitoring image, the detection unit 24 generates warning information linked to the right rear monitoring image.
[0086] FIG. 5 is a diagram showing how the ECU 20 according to the embodiment generates an image for monitoring the surroundings.
[0087] Fig. 5(a) is an example of a side image of the vehicle 10 captured by the side camera 11L. The side image in Fig. 5(a) is the same as the side image shown in Fig. 4(a) above. That is, the side image in Fig. 5(a) shows the vehicle's own lane in front of and behind the vehicle 10 traveling on the road, an adjacent lane to the left of the vehicle's own lane, and a following vehicle traveling in the adjacent lane, together with the body of the vehicle 10 located on the lower side of the side image.
[0088] 5(a), the generation unit 23 of the ECU 20 cuts out a cut-out area CAs for use in the periphery monitoring image that is narrower than, for example, the entire side image captured by the side camera 11L in both the horizontal and vertical directions. In addition, the generation unit 23 cuts out an area in the side image captured by the side camera 11L so that the position in the side image on the line connecting the left and right ends of the vehicle 10 is located in an area that is 10% to 20% from the upper end of the periphery monitoring image.
[0089] Fig. 5(b) is an example of a periphery monitoring image generated from the side image of the side camera 11L by the generation unit 23. As shown in Fig. 5(b), the periphery monitoring image is generated as is from the cut-out area CAs in Fig. 5(a).
[0090] FIG. 5( c ) shows how the surroundings monitoring image generated by the generation unit 23 is displayed on the CID 12C of the vehicle 10.
[0091] The method of generating an image for periphery monitoring shown in FIG. 5 is also applied to the case where the generating unit 23 generates an image for periphery monitoring based on a right side image captured by the side camera 11R on the right side of the vehicle 10.
[0092] As described above, for example, when a control signal is input from the turn switch 16 and the determination unit 22 determines that the vehicle 10 is about to change course, the generation unit 23 generates an image for rear side monitoring as shown in FIG. 4(d) based on the side images of the side cameras 11R, 11L on the side of the course of the vehicle 10, and the output unit 25 outputs the image to the meter display 12M, for example, and displays it as shown in FIG. 4(f).
[0093] However, when an operation to specify the display destination is performed, for example, using the function switch 12s of CID 12C, the output unit 25 may output the rear side monitoring image generated by the generation unit 23 to CID 12C in addition to the above-mentioned meter display 12M, and display it as shown in Figure 4(e).
[0094] Furthermore, even if a control signal, for example, is not input from the turn switch 16, when an operation to turn on the image display function is performed, for example, by the function switch 12s of CID12C, the output unit 25 may output the rear side monitoring image generated by the generation unit 23 to CID12C and display it as shown in Figure 4(e).
[0095] In this way, when the rear side image is displayed on the meter display 12M or CID 12C in conjunction with a change in course of the vehicle 10 or when the image display function is turned on, a display or a warning sound based on the warning information generated by the detection unit 24 may be presented to the driver.
[0096] In addition, for example, when the determination unit 22 determines that the vehicle 10 has decelerated to less than a predetermined speed based on the detection result of the wheel speed sensor 13, the generation unit 23 generates an image for monitoring the surroundings, for example as shown in Figure 5(b), and the output unit 25 outputs it to CID 12C and displays it as shown in Figure 5(c).
[0097] (Example of ECU processing) Next, an example of processing by the ECU 20 of the embodiment will be described with reference to Fig. 6. Fig. 6 is a flow chart showing an example of a procedure of image generation processing by the ECU 20 according to the embodiment.
[0098] As shown in FIG. 6, the determination unit 22 of the ECU 20 monitors the input of a control signal from, for example, the turn switch 16 to the input unit 21 (step S101).
[0099] When the judgment unit 22 judges that the vehicle 10 is about to change lane, for example because a control signal from the turn switch 16 is input to the input unit 21 (step S101: Yes), the input unit 21 starts acquiring side images of the side of the lane on which the vehicle 10 is heading, from among the side images captured by the side cameras 11R, 11L (step S102).
[0100] The generating unit 23 generates a rear side monitoring image from a side image captured by either of the side cameras 11R, 11L (step S103). That is, as described above, the generating unit 23 mirror-inverts the side image captured by either of the side cameras 11R, 11L, cuts out a cut-out area CAb used for the rear monitoring image, and corrects image distortion in the cut-out area CAb.
[0101] The detection unit 24 attempts to detect an event that may affect the traveling of the vehicle 10 from the rear side monitoring image generated by the generation unit 23 (step S104). If an event that may affect the traveling of the vehicle 10 is detected (step S104: Yes), the detection unit 24 generates warning information linked to the rear side monitoring image (step S105). If an event that may affect the traveling of the vehicle 10 is not detected (step S104: No), the generation process of warning information by the detection unit 24 is skipped.
[0102] The determination unit 22 determines whether or not a signal indicating an operation for designating a display destination of an image has been input to the input unit 21 from the function switch 12s provided on the CID 12C (step S106).
[0103] If no instruction to specify the display destination has been given (step S106: No), the output unit 25 outputs the rear side monitoring image generated by the generation unit 23 to the meter display 12M (step S107). If an instruction to switch the display destination has been given (step S106: Yes), the output unit 25 outputs the rear side monitoring image generated by the generation unit 23 to both the meter display 12M and the CID 12C (step S108). At this time, in the processes of steps S107 and S108, if the warning information generated by the detection unit 24 is linked, display based on the warning information is also performed.
[0104] Thereafter, the turn switch 16 is turned off, for example, and the course change of the vehicle 10 is completed (step S109).
[0105] If there is no control signal input from the turn switch 16 (step S101: No), the judgment unit 22 monitors the input of a control signal to the input unit 21, for example from the function switch 12s, which turns on the display function of the image from either of the side cameras 11R, 11L (step S201).
[0106] When the judgment unit 22 judges that the image display function has been turned on, for example by inputting a control signal from the function switch 12s to the input unit 21 (step S201: Yes), the input unit 21 starts acquiring the side image of the side camera 11R, 11L that is indicated by the control signal of the function switch 12s (step S202).
[0107] The generating unit 23 generates a rear side monitoring image from the side image captured by either of the side cameras 11R, 11L (step S203). That is, as described above, the generating unit 23 mirror-inverts the side image captured by either of the side cameras 11R, 11L, cuts out a cut-out area CAb used for the rear monitoring image, and corrects image distortion in the cut-out area CAb.
[0108] The detection unit 24 attempts to detect an event that may affect the traveling of the vehicle 10 from the rear side monitoring image generated by the generation unit 23 (step S204). If an event that may affect the traveling of the vehicle 10 is detected (step S204: Yes), the detection unit 24 generates warning information linked to the rear side monitoring image (step S205). If an event that may affect the traveling of the vehicle 10 is not detected (step S204: No), the generation process of warning information by the detection unit 24 is skipped.
[0109] The output unit 25 outputs the rear side monitoring image generated by the generation unit 23 to the CID 12C (step S207). At this time, if the warning information generated by the detection unit 24 is linked, the output unit 25 also performs display or the like based on the warning information.
[0110] Thereafter, the image display function is turned off by operating the function switch 12s or the like (step S209).
[0111] If there is no control signal input from the turn switch 16 (step S101: No), and there is also no control signal input from the function switch 12s to turn on the image display function (step S201: No), the judgment unit 22 judges whether or not the speed of the vehicle 10 is less than a predetermined speed based on the detection result of the wheel speed sensor 13 input to the input unit 21 (step S301).
[0112] When the determination unit 22 determines that the speed of the vehicle 10 is equal to or higher than the predetermined speed (step S301: No), the process from step S101 is repeated.
[0113] When the determination unit 22 determines that the speed of the vehicle 10 is less than the predetermined speed (step S301: Yes), the input unit 21 starts acquiring peripheral images including lateral images from the side cameras 11R, 11L (step S302).
[0114] The generating unit 23 generates a surroundings monitoring image from the surroundings image including the side images of the side cameras 11R and 11L (step S303).
[0115] That is, as described above, the generating unit 23 generates the periphery monitoring image by cutting out the cutout area CAs used for the periphery monitoring image from the side images of the side cameras 11R and 11L. In addition, the generating unit 23 may generate the periphery monitoring image based on other periphery images of the front camera 11F and the back camera 11B, or an overhead image obtained by combining these, as necessary.
[0116] The output unit 25 outputs the surroundings monitoring image generated by the generation unit 23 to the CID 12C (step S307).
[0117] When the detection result of the wheel speed sensor 13 indicates that the vehicle 10 has reached zero or has exceeded a predetermined speed, the judgment unit 22 judges that the parking of the vehicle 10 has ended and the vehicle 10 has stopped, or that the vehicle 10 has left the parking area and started normal driving (step S309).
[0118] With the above, the image generation process by the ECU 20 according to the embodiment is completed.
[0119] (Overview) Because it is difficult for drivers to see the rear side of the vehicle, blind spot monitors (BSMs) have been developed that detect following vehicles using, for example, a laser irradiated toward the rear of the vehicle and notify the driver with an indicator or a warning sound. However, indicators or warning sounds cannot visually present the situation of following vehicles to the driver. In addition, BSMs using lasers are expensive.
[0120] Meanwhile, a technology has been used that displays an image from a camera attached to the side of a vehicle on an electronic side mirror or the like to enable the driver to monitor the rear side. In addition, when parking or starting the vehicle, driving assistance may be provided by displaying an image for monitoring the surroundings from a side camera or the like that captures the sides of the vehicle on a monitor inside the vehicle. However, it is necessary to install both a camera for monitoring the rear side and a side camera for monitoring the surroundings, which also increases costs.
[0121] According to the ECU 20 of the embodiment, a partial area is cut out from the side image of the side cameras 11R, 11L to generate an image for rear side monitoring to be displayed on the meter display 12M or the CID 12C, and an image for periphery monitoring to be displayed on the CID 12C. In this way, by using the side cameras 11R, 11L for periphery monitoring also for rear side monitoring, costs can be reduced.
[0122] According to the ECU 20 of the embodiment, when generating an image for monitoring the rear side from the lateral image of the side cameras 11R, 11L, a portion of the side image is cut out so that the center of the cut-out in the horizontal direction of the side image is to the rear side of the vehicle 10, starting from the mounting position of the side cameras 11R, 11L in the fore-and-aft direction of the vehicle 10, and the cut-out portion is mirror-inverted.
[0123] This allows the driver to monitor the rear side as if he were looking through an optical mirror such as a door mirror. Also, the situation behind the vehicle 10, which is likely to be a blind spot for the driver, can be visually presented to the driver, reducing the burden on the driver. There is no need to provide a laser or the like for rear side monitoring, which reduces costs.
[0124] According to the ECU 20 of the embodiment, when generating a rear side monitoring image from the side images of the side cameras 11R, 11L, the distortion of the side images caused by the wide-angle lenses of the side cameras 11R, 11L is corrected. This can further reduce the discomfort felt by the driver with respect to the rear side monitoring image.
[0125] According to the ECU 20 of the embodiment, when generating a periphery monitoring image from a side image of the side cameras 11R, 11L, a partial area of the side image is cut out so that the cut-out center in the horizontal direction of the side image coincides with a line perpendicular to a line connecting the front and rear ends of the vehicle 10, starting from the mounting positions of the side cameras 11R, 11L in the front-rear direction of the vehicle 10. This provides an image of a wide range including the front, rear and sides of the vehicle 10, making it easy to monitor the periphery of the vehicle 10.
[0126] According to the ECU 20 of the embodiment, when generating a rear side monitoring image from the side images of the side cameras 11R, 11L, a partial area of the side image is cut out so that the rear side portion of the vehicle 10 accounts for 20% to 30% of the rear side monitoring image. This makes it possible to make the rear side monitoring image closer to the mirror image seen through the door mirror, allowing the driver to check the rear side more comfortably.
[0127] According to the ECU 20 of the embodiment, when a rear side monitoring image is generated from the side images of the side cameras 11R, 11L, a partial area of the side image is cut out so that the rear position of the vehicle 10 in the side image on a line horizontal to a line connecting the front and rear ends of the vehicle 10 at the mounting height of the side cameras 11R, 11L is located within an area of 30% to 60% from the upper end of the rear side monitoring image. This makes it possible to make the rear side monitoring image closer to the mirror image seen through the door mirror, allowing the driver to check the rear side more comfortably.
[0128] According to the ECU 20 of the embodiment, when a periphery monitoring image is generated from the side image of the side cameras 11R, 11L, a partial area of the side image is cut out so that the side position of the vehicle 10 in the side image on a line horizontal to a line connecting the left and right ends of the vehicle 10 at the mounting height of the side cameras 11R, 11L is located within an area of 10% to 20% from the upper end of the periphery monitoring image. This makes it easy to check the side including the road surface near the side of the vehicle 10, for example, when parking or starting the vehicle 10.
[0129] According to the ECU 20 of the embodiment, a rear side monitoring image is generated based on a side image of the road side indicated by a signal from the turn switch 16, among the side images from the side cameras 11R, 11L. As a result, when changing the course of the vehicle 10, a rear side image showing the situation of the road ahead of the course change of the vehicle 10 can be displayed without the driver having to perform an operation such as turning on an image display function, thereby improving the convenience for the driver.
[0130] According to the ECU 20 of the embodiment, when the vehicle 10 is traveling normally (for example, traveling forward or backward) at a speed equal to or higher than a predetermined speed and a signal is input from the turn switch 16, a rear side monitoring image is output to the meter display 12M. Also, when it is detected that the vehicle 10 is traveling at a speed lower than the predetermined speed, a surroundings monitoring image is output to the CID 12C.
[0131] In this way, by switching the generated images between those for rear-side monitoring and those for surrounding monitoring depending on the speed of the vehicle 10, etc., it is possible to present the driver with an image suited to the driving conditions of the vehicle 10.
[0132] As described above, the meter display 12M displays meters showing information necessary for driving the vehicle 10, and it is considered that the driver appropriately watches the meter display 12M while driving the vehicle 10. For example, by displaying a rear-side monitoring image on such a meter display 12M at the timing when the turn switch 16 is turned on, visual information that is important when changing lanes of the vehicle 10 can be presented to the driver more effectively.
[0133] According to the ECU 20 of the embodiment, an event that may affect the traveling of the vehicle 10 is detected from the rear-side monitoring image, and warning information including a warning to the driver is generated. This makes it easier for the driver to grasp the situation on the rear side of the vehicle 10.
[0134] In the above embodiment, the function switch 12s provided on the CID 12C has a function as a display switch that starts displaying images from the side cameras 11R, 11L, etc. However, instead of or in addition to the function switch 12s, the display switch may be provided at another position in the vehicle interior where the driver can operate it, such as at another portion including the vicinity of the meter display 12M of the instrument panel.
[0135] In the above embodiment, the meter display 12M is made to display images for monitoring the rearward direction, and the CID 12C is made to display images for monitoring the surroundings, but the display forms of various images including images for monitoring the rearward and lateral directions on the meter display 12M and the CID 12C can be changed in various ways. Another example of the display of images on the CID 12C is shown in FIG. 7.
[0136] FIG. 7 is a diagram showing other examples of various images displayed on the CID 12C according to the embodiment.
[0137] 7, the display unit of CID 12C is divided into three parts, and three types of images, a rear image, a left rear side image, and a right rear side image of vehicle 10, are displayed. The rear image of vehicle 10 is displayed in the upper part of the display unit. The left rear side image of vehicle 10 is displayed on the lower left side of the left side image of vehicle 10, and the right rear side image of vehicle 10 is displayed on the lower right side of the left side image of vehicle 10.
[0138] Also, a triangular mark 29c is attached to the following vehicle in the left rear side image of the vehicle 10 based on the warning information generated by the detection unit 24. Meanwhile, a motorbike traveling in the lane adjacent to the vehicle's own lane to the right is reflected in the right rear side image of the vehicle 10. However, since the motorbike is far enough away from the vehicle 10, no mark based on the warning information is attached to the motorbike.
[0139] In this way, images of the left and right rear sides of the vehicle 10 may be displayed simultaneously based on the operation of the function switch 12s, etc. This allows the driver to monitor the left and right rear sides of the vehicle 10 as if looking through the left and right door mirrors. [Explanation of symbols]
[0140] 10 Vehicles 11B Rear camera 11L, 11R Side camera 11F Front Camera 12C CID 12M Meter Display 12s function switch 13 Wheel speed sensor 16 Turn Switch 20 ECU 21 Input section 22 Judgment section 23 Generation part 24 Detection unit 25 Output section 26 Memory section
Claims
1. an input unit to which a side image of the vehicle is input from a camera having a wide-angle lens provided on a side of the vehicle; a generation unit that generates a first image and a second image to be displayed on an in-vehicle monitor by cutting out a partial area from each of the side images, The generation unit is When generating the first image, the partial area is cropped so that the crop center in the horizontal direction of the side image is located to the rear side of the vehicle, starting from the mounting position of the camera in the front-rear direction of the vehicle, and the cropped partial area is mirror-inverted, and distortion of the side image caused by the wide-angle lens is corrected; When generating the second image, the partial area is cut out so that the cut-out center in the horizontal direction of the side image coincides with a line perpendicular to a line connecting the front and rear ends of the vehicle, starting from a mounting position of the camera in the front-rear direction of the vehicle. Vehicle control device.
2. The camera includes: A first camera provided on a right side of the vehicle; a second camera provided on a left side of the vehicle; The input unit includes: A signal is input from a turn switch for indicating the path of the vehicle to the surroundings, The generation unit is generating the first image based on a side image of a road side indicated by a signal from the turn switch, among the side images from the first and second cameras; The vehicle control device according to claim 1 .
3. an output unit that outputs the first and second images to the in-vehicle monitor; The input unit includes: a signal from a display switch that starts displaying the first or second image on the in-vehicle monitor; A detection result of a wheel speed from a wheel speed sensor provided in the vehicle is inputted, The output unit is outputting the first image when the vehicle is traveling at a predetermined speed or faster and a signal is input from the turn switch or a signal is input from the display switch; When it is detected that the vehicle is moving at a speed less than the predetermined speed, the second image is output. The vehicle control device according to claim 2 .
4. The in-vehicle monitor includes: a first monitor for displaying information not necessary for driving; A second monitor having a function of displaying a meter and an image on which information necessary for driving is shown, The output unit is when the vehicle is traveling at a predetermined speed or faster and a signal is input from the turn switch, the first image is output to at least the second monitor. The vehicle control device according to claim 3 .
Citation Information
Patent Citations
On-board camera device
JP1997202180A
Device for visually recognizing around vehicle
JP2006050246A
Display system, moving body, display method, display program and storage medium thereof
JP2007223338A
Movable body, imaging system, and imaging apparatus
JP2022168699A
Display control device
WO2013084317A1
