Control device and computer program

Abstract

The present application provides a control device and a computer program, the control device for controlling the posture determination of a camera mounted on a vehicle is configured to determine the posture of the vehicle, and when the posture of the vehicle is a specific posture, the posture determination of the camera is not performed. The specific posture is a posture in which the posture of the camera relative to the vehicle does not change, but the posture of the camera relative to the road surface changes.

Description

Technical Field

[0001] This invention relates to control devices and computer programs. Background Technology

[0002] In Japanese Patent Application Publication No. 2015-191548, a conventional vehicle exterior monitoring device was disclosed, which is configured to determine the optical axis deviation of the vehicle camera based on images captured by the vehicle camera. Summary of the Invention

[0003] However, conventional vehicle exterior monitoring devices do not take vehicle posture into account. For example, when the front or rear wheels of a vehicle are being towed while in a lifted position, the device determines whether optical axis misalignment of the vehicle camera has occurred. In this case, in conventional vehicle exterior monitoring devices, although the posture of the vehicle camera relative to the vehicle does not change, the posture of the vehicle camera relative to the road surface does change. Therefore, there is a possibility that, although no optical axis misalignment of the vehicle camera has occurred, it may be mistakenly determined that optical axis misalignment of the vehicle camera has occurred.

[0004] This invention was made with the aim of addressing such a problem, and its purpose is to suppress situations where an anomaly is mistakenly identified even when no anomaly in the camera's pose (optical axis) has occurred.

[0005] To address the aforementioned issues, one embodiment of the present invention provides a control device for determining the posture of a camera mounted on a vehicle, configured as follows:

[0006] Determine the vehicle's posture.

[0007] When the vehicle is in a specific posture, the camera's posture determination is not performed.

[0008] In addition, a computer program according to one aspect of the present invention causes the processor to perform the following processing:

[0009] Determine the vehicle's posture.

[0010] When the vehicle is in a specific posture, the posture determination of the camera mounted on the vehicle is not performed.

[0011] According to these methods of the present invention, it is possible to suppress situations where an anomaly is mistakenly identified even when no anomaly in the camera's posture (optical axis) has occurred. Attached Figure Description

[0012] The features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which the same symbols denote the same elements, wherein:

[0013] Figure 1 This is a schematic structural diagram of a vehicle according to one embodiment of the present invention;

[0014] Figure 2 This is a flowchart illustrating the posture determination process of an external camera according to one embodiment of the present invention. Detailed Implementation

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, in the following description, the same reference numerals will be used to denote the same constituent elements.

[0016] Figure 1 This is a schematic structural diagram of a vehicle 100 according to one embodiment of the present invention.

[0017] The vehicle 100 includes a peripheral sensor 1, a vehicle sensor 2, a driver sensor 3, an HMI (Human Machine Interface) 4, an actuator 5, and a control device 6. The peripheral sensor 1, vehicle sensor 2, driver sensor 3, HMI 4, actuator 5, and control device 6 are connected in a communicable manner via an in-vehicle network 9 that conforms to a standard controller area network.

[0018] The peripheral sensor 1 is a sensor used to generate peripheral data representing the surrounding conditions of the vehicle 100. In this embodiment, the vehicle 100 includes one or more external cameras 11 used as peripheral sensors 1 to capture images of the surroundings of the vehicle 100. The external cameras 11 capture images of the surroundings of the vehicle 100 at a predetermined frame rate (e.g., 10 Hz to 40 Hz), generating peripheral images of the surroundings of the vehicle 100 (e.g., images of the front of the vehicle 100, etc.). Whenever a peripheral image is generated, the external cameras 11 send the generated peripheral image as peripheral data to the control device 6.

[0019] In addition to the external camera 11, a range sensor that measures the distance to objects and ground objects around the vehicle 100 can also be included as the peripheral sensor 1. Examples of range sensors include LiDAR (Light Detection and Ranging) sensors that illuminate radar light and measure distance based on its reflected light, and millimeter-wave radar sensors that illuminate radio waves and measure distance based on their reflected waves.

[0020] Vehicle sensor 2 is a sensor used to acquire vehicle data representing the state of vehicle 100. In this embodiment, vehicle 100 includes wheel speed sensor 21, positioning sensor 22, steering sensor 23, acceleration sensor 24, and braking sensor 25 as vehicle sensor 2. Wheel speed sensor 21 acquires wheel speed data representing the rotational speed (wheel speed) of each wheel of vehicle 100. Positioning sensor 22 acquires current position data representing the current position of vehicle 100, such as latitude and longitude. Steering sensor 23 acquires data related to steering operation, such as steering grip, steering torque, and steering angle. Acceleration sensor 24 acquires data related to acceleration operation, such as the amount of accelerator pedal pressure. Braking sensor 25 acquires data related to braking operation, such as the detection of brake pedal operation input. However, vehicle sensor 2 is not limited to these sensors. The data acquired by each of sensors 21 to 25 is sent to control device 6 as vehicle data.

[0021] The driver sensor 3 is a sensor used to generate driver data representing the driver's state. In this embodiment, the vehicle 100 includes a driver monitoring camera 31 as the driver sensor 3, used to capture images of the driver's appearance, including the driver's face. The driver monitoring camera 31 captures the driver's appearance at a predetermined frame rate (e.g., 10 Hz to 40 Hz), generating an appearance image showing the driver's appearance. Whenever an appearance image of the driver is generated, the driver monitoring camera 31 sends the generated appearance image as driver data to the control device 6.

[0022] HMI4 is a user interface used for exchanging information between vehicle 100 and its occupants. HMI4 includes an output device 41 for notifying vehicle occupants through sensory inputs (e.g., visual, auditory, and tactile sensations) and an input device 42 for input and response operations by vehicle occupants. Output device 41 may be a display (e.g., instrument cluster display, central display, head-up display, etc.) or a speaker. Input device 42 may be a touch panel or a microphone.

[0023] HMI4 notifies the vehicle occupants of information corresponding to the output signals received from the control unit 6 via output device 41, and sends the data input by the vehicle occupants via input device 42 to the control unit 6.

[0024] The HMI4 can be pre-installed in vehicle 100, or it can be a smartphone or other terminal owned by vehicle occupants (driver and passengers). In the latter case, information can be exchanged, for example, via short-range wireless communication between vehicle 100 and the vehicle occupants' terminals. Alternatively, communication can occur between the vehicle occupants' terminals and an external server (not shown), allowing for indirect information exchange via the server.

[0025] Actuator 5 is a device for driving control of vehicle 100. In this embodiment, vehicle 100 includes an acceleration actuator 51, a braking actuator 52, and a steering actuator 53 as actuator 5. Acceleration actuator 51 is an actuator for accelerating vehicle 100, and may be, for example, at least one of an engine and a motor. Braking actuator 52 is an actuator for braking vehicle 100, and may be, for example, a hydraulic actuator. Steering actuator 53 is an actuator for steering vehicle 100, and may be, for example, a steering motor.

[0026] The control device 6 is an ECU (Electronic Control Unit) that includes a communication unit 61, a storage unit 62, and a processing unit 63.

[0027] The communication unit 61 includes an interface circuit for connecting the control device 6 to the in-vehicle network 9. The communication unit 61 provides various data received from the outside to the processing unit 63. Additionally, the communication unit 61 outputs various signals from the processing unit 63 to the outside.

[0028] The storage unit 62 has storage media such as HDD (Hard Disk Drive), SSD (Solid Disk Drive), and semiconductor memory, and stores various computer programs and data used in the processing of the processing unit 63.

[0029] The processing unit 63 has one or more CPUs (Central Processing Units) and their peripheral circuitry, executing various computer programs stored in the storage unit 62. The processing unit 63 is, for example, a processor. The processing unit 63 may also have other arithmetic circuitry such as a logic unit, a numerical operation unit, or a graphics processing unit. The processing unit 63 functions as a recognition unit 71, a driving assistance unit 72, and a camera posture determination unit 73 by executing processing according to the computer program, and also serves as a functional unit (module) that implements predetermined functions. In the following description, when the processing is described with each functional unit 71-73 as the subject, it indicates that the processing unit 63 executes the program that implements each functional unit 71-73.

[0030] The specific processing performed in the control device 6 will be described below. Specifically, the contents of each functional unit 71 to functional unit 73, which are implemented by the processing unit 63 according to the computer program, will be explained.

[0031] The recognition unit 71 identifies landmarks and ground objects surrounding the vehicle 100. For example, the recognition unit 71 sequentially inputs surrounding images received from the external camera 11 into the recognizer to identify other vehicles, bicycles, pedestrians, and / or similar objects such as curbs or fences (hereinafter referred to as "dividing objects"), road markings (e.g., lane markings), and other ground objects within the surrounding images. The recognizer can be configured, for example, as a convolutional neural network (CNN) with multiple convolutional layers connected in series from the input side to the output side. Furthermore, the recognition unit 71 calculates the distance from the vehicle 100 to the landmarks and ground objects, and calculates the positions of the landmarks and ground objects, for example, using the standard dimensions of the landmarks and ground objects and the dimensions of the identified landmarks and ground objects. The standard dimensions of the landmarks and ground objects are stored in the storage unit 62 according to the type of each landmark and ground object. The dimensions of the landmarks and ground objects are identified in the surrounding images. Furthermore, the method for recognizing landmarks and ground objects is not limited to this method; recognition can be performed using any known method.

[0032] The driver assistance unit 72 controls the actuators 5 based on the landmarks and ground objects identified by the recognition unit 71, and performs driver assistance for driving control of the vehicle 100. In this embodiment, the driver assistance unit 72 can perform driver assistance for driving control of the vehicle 100 at a driving control level of Level 3 as defined by the SAE (Society of Automotive Engineers). Level 3 driving control refers to a driving control level that does not require the driver to operate each actuator 51 to actuator 53 or monitor the surroundings. The driver assistance unit 72 can perform driver assistance for driving control of the vehicle 100 at a driving control level in which the driver participates in driving the vehicle 100 (e.g., Level 1 or Level 2 as defined by SAE). In addition, the driver assistance unit 72 provides appropriate displays and warnings to the driver via the HMI4 based on the landmarks and ground objects identified by the recognition unit 71.

[0033] The camera posture determination unit 73 determines whether the posture (optical axis) of the external camera 11 is normal. If the posture of the external camera 11 relative to the vehicle 100 changes due to poor assembly, detachment, or other reasons, the posture of the external camera 11 relative to the road surface also changes. Therefore, it may be impossible to capture the area that should be captured by the external camera 11, and the recognition unit may be unable to identify objects around the vehicle 100 and objects on the ground. Therefore, the camera posture determination unit 73 of this embodiment calculates the position of the FOE (Focus of Expansion) of the external camera 11, which is the intersection of the translational axis representing the direction of the movement vector of the external camera 11 and the projection image plane of the external camera 11. Then, the camera posture determination unit 73 determines whether the position of the FOE converges within a predetermined normal range (e.g., the central area of ​​the projection image plane) on the projection image plane.

[0034] The translational axis of the external camera 11 fixed to the vehicle 100 is substantially aligned with the direction of travel of the vehicle 100 (front and rear axles). Therefore, when traveling on essentially flat ground, the position of the field of view (FOE) on the projected image plane is essentially constant when the external camera 11 is in a normal orientation. Although the position of the FOE on the projected image plane may vary to some extent due to road surface inclination, if the external camera 11 is in a normal orientation, the position of the FOE converges within a predetermined range on the projected image plane. Therefore, by determining whether the position of the FOE converges within a predetermined normal range on the projected image plane, it is possible to determine whether the orientation of the external camera 11 is normal.

[0035] However, when towed by a trailer or other towing vehicle, the front or rear wheels of vehicle 100 are in a suspended position (hereinafter referred to as the "vehicle towing position"). Therefore, even if the external camera 11 is in a normal position relative to the vehicle, its position relative to the road surface will change. Consequently, the position of the FOE on the projected image plane changes, and the position of the FOE may deviate from the normal range.

[0036] Therefore, in this embodiment, when the posture of the vehicle 100 becomes a specific posture in which the posture of the external camera relative to the vehicle 100 does not change but the posture of the external camera relative to the road surface changes, such as the vehicle traction posture, the determination of whether the posture (optical axis) of the external camera 11 is normal is not performed.

[0037] Figure 2 This is a flowchart illustrating the posture determination processing of the external camera 11 in this embodiment, implemented by the control device 6. During the recording process of the external camera 11, the control device 6 repeatedly executes this routine at predetermined operation cycles.

[0038] In S1, the control device 6 determines whether the posture of the vehicle 100 is a specific posture in which the posture of the external camera relative to the vehicle 100 does not change but the posture of the external camera relative to the road surface changes.

[0039] In this embodiment, the control device 6 determines whether the posture of the vehicle 100 is a vehicle traction posture in which the front or rear wheels of the vehicle 100 are lifted. The method for determining whether the posture of the vehicle 100 is a vehicle traction posture is not particularly limited. For example, if the wheel speed sensors 21 of the left and right front wheels and the rear wheels do not detect any abnormalities, and the wheel speeds of each of the left and right front wheels and the rear wheels are above a predetermined speed VTH, it can be determined that the posture of the vehicle 100 is not a vehicle traction posture. In other words, when the wheel speed sensors 21 of the left and right front wheels and the rear wheels do not detect any abnormalities, even if the wheel speed of one of the front wheels and the rear wheels is above the predetermined speed VTH, if the wheel speed of the other of the front wheels and the rear wheels is less than the predetermined speed VTH, it can be determined that the posture of the vehicle 100 is a vehicle traction posture. Furthermore, the wheel speed sensors 21 are determined to be abnormal, for example, if their output values ​​continuously indicate abnormal values.

[0040] If the vehicle's posture becomes a specific posture (in this embodiment, the vehicle towing posture), the control device 6 will end the current processing without determining whether the posture of the external camera 11 is normal. On the other hand, if the vehicle's posture does not become a specific posture, the control device 6 will enter the processing of S2.

[0041] In S2, the control device 6 calculates the position of the field of view (FOE) of the external camera 11 and determines whether the position of the FOE converges within a predetermined normal range on the projection image plane. The method for calculating the position of the FOE is not particularly limited. For example, the position of the FOE can be calculated as the intersection point on the projection image plane when parallel straight lines such as the dividing lines identified by the recognition unit 71 are extended indefinitely. Alternatively, multiple consecutive surrounding images (time-series images) can be used to calculate the optical flow of an object on the translational axis of the external camera 11, and the intersection point of the straight lines extending that optical flow can be calculated as the position of the FOE. If the position of the FOE converges within a normal range on the projection image plane (e.g., the central region of the projection image plane), the control device 6 proceeds to the processing in S3. On the other hand, if the position of the FOE does not converge within a normal range on the projection image plane, the control device 6 proceeds to the processing in S4.

[0042] In S3, the control device 6 does not stop implementing driving assistance based on the driving assistance unit 72, but instead implements driving assistance based on the driving assistance unit 72 as needed. The driving assistance based on the driving assistance unit 72 is a driving assistance that uses the surrounding images generated by the external camera 11 and the recognition results of landmarks and ground objects around the vehicle 100.

[0043] In S4, the control device 6 temporarily suspends the implementation of driving assistance based on the driving assistance unit 72, and implements a notification or warning via the HMI4 to inform the vehicle occupants that the posture of the external camera 11 may be abnormal. The driving assistance based on the driving assistance unit 72 is a driving assistance that uses the surrounding images generated by the external camera 11 and corresponds to the recognition results of landmarks and ground objects around the vehicle 100.

[0044] The control device 6, which determines the posture (optical axis determination) of the external camera 11 (camera) mounted on the vehicle 100 in the above-described embodiment, determines the posture of the vehicle 100. Furthermore, the control device 6 is configured such that when the posture of the vehicle 100 is a specific posture, the posture determination of the external camera 11 is not performed.

[0045] Therefore, when the posture of the vehicle 100 is one that might be misjudged when the posture determination by the external camera 11 is performed, the posture determination by the external camera 11 can be omitted. Thus, it is possible to suppress misjudgments that occur when the posture determination by the external camera 11 is performed.

[0046] The specific posture is one in which the posture of the external camera 11 relative to the vehicle 100 remains unchanged, but the posture of the external camera 11 relative to the road surface changes.

[0047] In this embodiment, the control device 6 is configured such that if the wheel speed sensor 21, which detects the wheel speeds of the front and rear wheels of the vehicle 100, does not detect any abnormality, and the wheel speed of one of the front and rear wheels is above a predetermined speed VTH, while the wheel speed of the other of the front and rear wheels is below the predetermined speed VTH, then the vehicle 100 is determined to be in a specific posture.

[0048] Therefore, it is possible to suppress the behavior of the vehicle 100 when the vehicle traction posture is such that the external camera 11 is prone to misjudgment during the posture determination of the vehicle 100.

[0049] Furthermore, the control device 6 in this embodiment is configured to determine the posture of the external camera 11 based on whether the position of the FOE of the external camera 11 (camera) converges within a predetermined normal range on the projection image plane of the external camera 11 when the posture of the vehicle 100 is not a specific posture.

[0050] Therefore, when the posture of the vehicle 100 is such that the possibility of misjudgment when the posture determination of the external camera 11 is low, the posture (optical axis) of the external camera 11 can be determined with high accuracy.

[0051] The embodiments of the present invention have been described above, but the above embodiments only show a part of the application examples of the present invention, and the technical scope of the present invention is not limited to the specific structure of the above embodiments.

[0052] For example, in the above-described embodiments, in Figure 2 In flowchart S1, it is determined whether either the front wheel or the rear wheel speed sensor 21 has detected an abnormality. If either the front wheel or the rear wheel speed sensor 21 detects an abnormality, the process can also end without determining whether the posture of the external camera 11 is normal, regardless of the wheel speed of either the front or the rear wheel.

[0053] Furthermore, as in the embodiments described above, the computer program executed in the control device 6 may also be provided in the form of a computer-readable, portable recording medium such as a semiconductor memory, magnetic recording medium, or optical recording medium. For example, as in the embodiments described above, the computer program executed in the control device 6 may also be provided as a computer program product.

Claims

1. A control device for determining the posture of a camera mounted on a vehicle, said control device being configured to, Determine the posture of the vehicle. When the vehicle is in a specific posture, the posture determination of the camera is not performed.

2. The control device according to claim 1, The specific posture is one in which the posture of the camera relative to the vehicle remains unchanged, but the posture of the camera relative to the road surface changes.

3. The control device according to claim 1, configured as follows: If the wheel speed sensors that detect the wheel speeds of the front and rear wheels of the vehicle do not detect any abnormalities, and the wheel speed of one of the front and rear wheels is above a predetermined speed while the wheel speed of the other of the front and rear wheels is below the predetermined speed, then the vehicle is determined to be in the specific posture.

4. The control device according to any one of claims 1 to 3, configured as follows: When the vehicle's posture is not the specific posture, the posture determination of the camera is performed based on whether the position of the camera's FOE on the camera's projection image plane is within a predetermined normal range.

5. A computer program that causes a processor to perform the following processing: Determine the vehicle's posture. When the vehicle is in a specific posture, the posture determination of the camera mounted on the vehicle is not performed.

Images