Object detection system

Abstract

An object detection system according to the present invention comprises: a camera (20) that captures images of the surroundings of a vehicle (10); and a main controller (30) that performs object detection processing on the basis of the images captured by the camera. The camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to capture images. The main controller comprises a parameter acquisition unit that acquires vehicle-related parameters that indicate the situation of the vehicle and the surroundings of the vehicle at the time that images are being captured by the camera. The camera controller comprises a parameter reception unit that receives the vehicle-related parameters from the main controller, an exposure condition setting unit that sets exposure conditions for subsequent image capture on the basis of the vehicle-related parameters received from the main controller, and an image capture execution unit that causes the image sensor to capture images on the basis of the exposure conditions set by the exposure condition setting unit.

Description

Object Detection System CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Japanese Application No. 2024-031707 filed on March 1, 2024 and Japanese Application No. 2025-006892 filed on January 17, 2025, the contents of which are incorporated herein by reference.

[0002] The disclosure herein relates to object detection systems.

[0003] Conventionally, there has been known an object detection system that performs object detection using images captured by a camera mounted on a vehicle, etc. Also, there is known a configuration of the object detection system that includes a camera having an image sensor and a processor that can communicate with the camera, and the processor controls the image capture by the camera based on the situation around the vehicle, etc. (for example, Patent Document 1).

[0004] US Patent Application Publication No. 2023 / 249611

[0005] However, existing object detection systems do not seem to optimize the division of roles when it comes to camera control, and in this respect, there is room for improvement in existing object detection systems.

[0006] The present disclosure has been made in consideration of the above circumstances, and aims to provide an object detection system that optimizes camera control and thereby enables object detection processing to be performed appropriately.

[0007] Each means for solving the problem will be described below.

[0008] The first means is an object detection system comprising: a camera mounted on a vehicle that photographs the area around the vehicle at a predetermined interval; and a main controller that is capable of communicating with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor and a camera controller that causes the image sensor to photograph, and the main controller comprises a parameter acquisition unit that acquires vehicle-related parameters that indicate the situation of the vehicle and its surroundings at the time of camera photography, and the camera controller comprises: a parameter receiving unit that receives the vehicle-related parameters from the main controller, an exposure condition setting unit that sets exposure conditions for the next photograph based on the vehicle-related parameters received from the main controller, and an photography execution unit that causes the image sensor to photograph based on the exposure conditions set by the exposure condition setting unit.

[0009] When detecting objects using camera images, it is conceivable that the exposure conditions of the image sensor are set based on the conditions of the vehicle and its surroundings. However, these exposure conditions depend on the camera characteristics, the camera mounting position, and other factors. Therefore, the exposure conditions must be set according to the camera characteristics. In a system configuration in which these exposure conditions are set on the main controller side, changing the camera or using multiple cameras with different camera characteristics requires changing the software of the main controller. In this regard, in the above configuration, the camera controller sets the exposure conditions for the next image capture based on vehicle-related parameters received from the main controller, and then controls the image sensor to capture an image based on those exposure conditions. In this case, by assigning the exposure condition setting function to the camera side, the software design burden on the main controller and the processing burden on the main controller can be reduced. As a result, camera control can be optimized, and ultimately object detection processing can be performed appropriately.

[0010] In a system in which multiple cameras are installed on a vehicle, the multiple cameras can share vehicle-related parameters sent from the main controller and take appropriate photographs according to the condition and optical characteristics of each camera.

[0011] The second means is an object detection system comprising: a camera mounted on a vehicle and capturing images of the area around the vehicle at a predetermined interval; and a main controller capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images captured by the camera, wherein the camera comprises an image sensor and a camera controller that controls the image sensor to capture images, and the main controller comprises a parameter acquisition unit that acquires information regarding at least one of the ambient color of the area around the vehicle when capturing images with the camera and the incident light entering the camera as adjustment parameters for white balance adjustment, and the camera controller comprises: a parameter receiving unit that receives the adjustment parameters from the main controller, and a white balance adjustment unit that adjusts the white balance of the captured images to be sent to the main controller based on the adjustment parameters received from the main controller.

[0012] The camera controller acquires white balance adjustment parameters from the main controller, based on information about at least one of the ambient color around the vehicle and the incident light entering the camera, and performs white balance adjustment on the captured image to be sent to the main controller based on the adjustment parameters. This allows for white balance adjustment appropriate for each camera, even if each on-board camera performs different white balance adjustments depending on the camera's characteristics, camera installation position, etc. In this case, the main controller does not need to have a white balance adjustment function (white balance adjustment software) tailored to the camera's characteristics for each camera. In other words, even if the main controller does not have white balance adjustment software tailored to the camera's characteristics, each camera can perform white balance control tailored to the scene. As a result, the object detection system can achieve appropriate white balance adjustment for each image while adapting to the conditions around the vehicle.

[0013] In a system in which multiple cameras are installed on a vehicle, the multiple cameras can share vehicle-related parameters sent from the main controller and perform appropriate white balance adjustments according to the condition and optical characteristics of each camera.

[0014] The third means is an object detection system comprising: a camera mounted on a vehicle and capturing images of the area around the vehicle at a predetermined interval; and a main controller capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images captured by the camera, wherein the camera comprises an image sensor and a camera controller that causes the image sensor to capture images, the main controller comprises a parameter acquisition unit that acquires vehicle-related parameters that indicate the situation of the vehicle and its surroundings at the time of camera capture, and the camera controller comprises: a parameter receiving unit that receives the vehicle-related parameters from the main controller, and a mask processing unit that performs mask processing on areas in the image captured by the camera that are excluded from the detection target of the main controller, based on the vehicle-related parameters received from the main controller.

[0015] In order for the main controller to monitor the periphery of the vehicle extensively and appropriately, it is desirable to reduce the processing load of image recognition. Furthermore, depending on the conditions of the vehicle and its surroundings, the captured image may contain information (unnecessary features) that is unnecessary for object detection by the main controller. In consideration of this, the camera controller acquires vehicle-related parameters, which are information about the vehicle or its surroundings, from the main controller, and, based on the vehicle-related parameters, performs masking processing on areas in the captured image to be excluded from detection targets, which are transmitted to the main controller. In other words, the masking processing on the camera side reduces the processing load on the main controller, which receives the image.

[0016] In this case, the degree of masking required is determined depending on the optical characteristics and performance of the camera, but the masking that reflects the optical characteristics, etc. is performed by the camera controller of each camera. In this case, the main controller does not need to perform masking that matches the specifications of the optical characteristics, etc. of each camera. With the above configuration, captured images that have been subjected to appropriate masking can be transmitted to the main controller. As a result, object detection can be performed appropriately in the main controller.

[0017] The fourth means is an object detection system comprising: a camera mounted on a vehicle for capturing images of the periphery of the vehicle at a predetermined interval; and a main controller capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images captured by the camera, wherein the camera comprises an image sensor and a camera controller for causing the image sensor to capture images, the camera controller comprises: a noise information acquisition unit for acquiring noise amount information indicating the amount of noise contained in images captured by the image sensor; and a noise information transmission unit for transmitting the noise amount information to the main controller, the main controller comprises: a noise level determination unit for determining a noise level at which noise is to be removed from the captured image, based on the noise amount information received from the camera controller and the captured image, and the camera controller comprises: a noise level receiving unit for receiving the noise level from the main controller; and a noise removal unit for performing noise removal processing on the captured image to be transmitted to the main controller, based on the noise level received from the main controller and the acquired noise amount information.

[0018] The main controller is configured to determine the noise level for noise removal in the captured image based on the noise amount information received from the camera controller and the captured image. That is, the main controller determines the amount of noise in the captured image from the information received from the camera, and also identifies objects, etc. in the captured image from the captured image, and determines the noise level to be removed based on these. Meanwhile, the camera controller performs noise removal processing on the captured image to be sent to the main controller based on the noise level received from the main controller and the noise amount information of the captured image. This configuration enables noise level adjustment for noise occurring in the captured image in response to instructions from the main controller, and reduces the processing load on the main controller by allocating the noise removal function to the camera. This allows for optimal camera control, and ultimately allows for optimal object detection processing.

[0019] Furthermore, when a vehicle is equipped with multiple cameras, it is conceivable that the noise generation conditions for each camera will differ, but it is possible to perform appropriate noise reduction in accordance with the state and optical characteristics of each camera. In other words, even when multiple different cameras are used, the main controller does not need to correct the captured images by itself in accordance with the noise information of each camera. Furthermore, when performing noise correction on the camera side, it is possible to control the noise level to be suppressed to the amount desired by the main controller, regardless of the camera.

[0020] The fifth means is an object detection system comprising: a camera mounted on a vehicle that photographs the area around the vehicle at a predetermined interval; and a main controller that is capable of communicating with the camera via wired or wireless communication and performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor and a camera controller that causes the image sensor to photograph, and the camera controller comprises: a vanishing point information acquisition unit that acquires vanishing point information, which is the position of a vanishing point in an image photographed by the image sensor or corresponding information corresponding to the position of the vanishing point; a readout range determination unit that determines the readout range of the photographed image based on the vanishing point information; and an image transmission unit that transmits an image of the readout range determined by the readout range determination unit to the main controller.

[0021] The camera controller determines the readout range of the captured image based on the position of the vanishing point (vanishing point information) within the captured image, and transmits the image of that readout range to the main controller. Here, if the camera determines the position of the vanishing point and reads the image according to that vanishing point, the main controller does not need to specify the readout position based on the camera's attitude, etc. This eliminates the need for software for such processing on the main controller. Furthermore, by outputting the maximum image size that the image sensor can output on the camera and executing a series of processes from vanishing point search to image readout on the camera, even if temporary image misalignment occurs, proper image readout can be performed without waiting for the main controller to detect the vanishing point. This allows for optimal camera control, and ultimately proper object detection processing.

[0022] The sixth means is an object detection system comprising: a camera mounted on a vehicle that photographs the area around the vehicle at a predetermined interval; and a main controller that is capable of communicating with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the main controller comprises: an abnormality detection unit that detects the occurrence of an abnormal situation in the vehicle or around the vehicle; and a command transmission unit that transmits an image write command to the camera when the abnormality detection unit detects the occurrence of the abnormal situation; the camera comprises an image sensor, an image storage unit that is capable of storing images photographed by the image sensor, and a write processing unit that writes the photographed images to the image storage unit; and when the write processing unit receives the image write command from the main controller, it writes the photographed images to the image storage unit.

[0023] When the main controller detects the occurrence of an abnormality in or around the vehicle, the camera writes the captured image to the image storage unit based on an image write command from the main controller. That is, the camera stores the captured image at the time of the abnormality in the image storage unit. This allows the camera to properly store the captured image showing the abnormality, even when the main controller is under heavy processing load due to abnormality response processing when an abnormality occurs. This allows for proper camera control and therefore proper object detection processing.

[0024] The seventh means is an object detection system comprising: a camera mounted on a vehicle that photographs the area around the vehicle at a predetermined interval; and a main controller that is capable of communicating with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor, a camera controller that causes the image sensor to take images, and a memory unit that stores application software that can be executed by the camera controller, and the application software stored in the memory unit can be written by either the main controller or the camera controller, and the main controller comprises an authentication unit that certifies that the application software written to the memory unit is genuine.

[0025] When application software is stored on the camera, updating (updating) the application software requires accessing and editing the camera's software storage area from outside the camera. In this case, the camera may need to be removed from the vehicle to perform the update, and then remounted and calibrated after the update. This places a burden on both the user and the vehicle provider. This burden increases even more when multiple cameras are used. To eliminate this burden, the main controller is provided with a means for accessing the camera's software storage area, and the camera software can be rewritten by the main controller or camera controller. This allows the main controller to function as a hub, allowing software to be rewritten for multiple different cameras, reducing the burden on users and vehicle providers. It also enables software version checking for both the main controller and the camera, enabling compatibility verification simultaneously.

[0026] Furthermore, malicious rewriting of the camera's software storage area can adversely affect the camera's characteristics and potentially result in unintended vehicle behavior. In this regard, providing a function to verify that rewriting of the software storage area is legitimate and intended can prevent malicious rewriting. As a result, camera control can be optimized, and object detection processing can be performed appropriately.

[0027] The eighth means is an object detection system comprising: a camera mounted on a vehicle and taking images of the area around the vehicle at a predetermined interval; and a main controller capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images taken by the camera, wherein the camera comprises an image sensor and a camera controller that causes the image sensor to take images, and the camera controller comprises: a sleep-time photography unit that causes the image sensor to take images in a system sleep state in which object detection by the main controller is paused; a transmission determination unit that, when photography is performed by the sleep-time photography unit, determines whether or not the taken images need to be sent to the main controller; an event notification unit that, when the transmission determination unit determines that image transmission to the main controller is required, issues an event notification to the main controller indicating that image transmission to the main controller will begin; and an image transmission unit that begins image transmission to the main controller after the event notification.

[0028] For example, when the vehicle's power is off and the main controller's object detection is paused, if the camera controller determines that image transmission to the main controller is necessary based on images captured in the system's paused state, it notifies the main controller in advance of the image transmission and then starts image transmission to the main controller. In this case, while the object detection system is paused, image transmission can be started as soon as possible at the camera controller's discretion while minimizing communication between the main controller and the camera. As a result, images captured in the object detection paused state (system paused state) can be properly saved. This allows for proper camera control and, ultimately, proper object detection processing.

[0029] The ninth means is an object detection system comprising: a camera mounted on a vehicle and taking images of the surroundings of the vehicle at a predetermined interval; and a main controller capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images taken by the camera, wherein the camera comprises an image sensor and a camera controller that causes the image sensor to take images, the communication unit of the main controller and the communication unit of the camera are capable of time-division two-way communication, and the camera controller comprises a communication control unit that can change the proportion of time allocated to communication from the main controller to the camera controller.

[0030] For example, during the initialization process associated with camera startup, the camera controller starts transmitting captured images after the initialization process is completed. In this case, by temporarily increasing the proportion of time allocated to communication from the main controller to the camera controller during the initialization process on the camera side, the time required for the initialization process can be shortened, enabling the object detection process to start as quickly as possible. This allows for optimization of camera control, and ultimately allows the object detection process to be performed appropriately.

[0031] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. 1 is a diagram showing the schematic configuration of an object detection system for a vehicle, FIG. 2 is a diagram showing the configuration of a camera and a main controller in the object detection system, FIG. 3 is a diagram showing the configuration of a camera and a main controller in the object detection system, FIG. 4 is a functional block diagram related to exposure condition setting processing, FIG. 5 is a diagram showing the relationship between vehicle speed and exposure time, FIG. 6 is a diagram showing the relationship between attitude parameters and exposure time, FIG. 7 is a diagram showing the relationship between vehicle speed, weather information and exposure time, FIG. 8 is a diagram showing the relationship between vehicle speed, image brightness information and exposure time, FIG. 9 is a diagram showing the relationship between low speed / high speed states as transition parameters, vehicle speed and image brightness information, FIG. 10 is a flowchart for explaining the procedure for setting camera exposure conditions, FIG. 11 is a system configuration diagram when two cameras are provided, FIG. 12 is a diagram showing the relationship between front and rear cameras and exposure time, FIG. 13 is a diagram showing the wiper wiping range on the vehicle windshield, and FIG. 14 is a diagram showing the processing procedure executed by the camera controller of each camera. 24 is a functional block diagram relating to image readout processing; FIG. 25 is a diagram showing the readout range of a captured image; FIG. 26 is a flowchart illustrating the image readout procedure; FIG. 27 is a sequence diagram showing the flow of a series of camera controls by a main controller and a camera controller; FIG. 28 is a functional block diagram relating to a configuration for performing image storage processing when an abnormality occurs;31 is a functional block diagram relating to a configuration for rewriting application software; FIG. 32 is a flowchart illustrating the procedure for rewriting application software; FIG. 33 is a flowchart illustrating the procedure for the main controller when rewriting application software; FIG. 34 is a flowchart illustrating the procedure for the main controller when rewriting application software; FIG. 35 is a flowchart illustrating the procedure for determining whether application software has been tampered with; FIG. 47 is a functional block diagram relating to a configuration for performing emergency startup when the system is inactive, FIG. 48 is a flowchart explaining the processing procedure of the main controller when the system is inactive, FIG. 49 is a functional block diagram relating to a configuration for speeding up camera startup, FIG. 50 is a flowchart explaining the processing procedure of the camera controller when the camera is started, and FIG.1 is a time chart showing the communication state when the camera is started up.

[0032] Hereinafter, embodiments embodying the present disclosure will be described with reference to the drawings. The present embodiments are directed to an object detection system that uses a camera mounted on a vehicle to detect objects around the vehicle.

[0033] FIG. 1 shows a schematic configuration of an object detection system in a vehicle 10. The object detection system includes multiple cameras 20 that capture images of the vehicle's surroundings and a main controller 30 that performs object detection processing based on images captured by the cameras 20. Each camera 20 is separate from the main controller 30, and in this embodiment, one main controller 30 is provided for the multiple cameras 20. The cameras 20 are installed, for example, at the front, rear, left, and right sides of the vehicle 10. Each camera 20 has a different imaging direction. Note that the number of cameras 20 in the vehicle 10 is not limited to four, and may be three or less, or five or more. Each camera 20 captures an image of an area extending over a predetermined angular range (predetermined angle of view) in the forward imaging direction.

[0034] The front camera at the front of the vehicle captures an area in front of the vehicle as its imaging range. The rear camera at the rear of the vehicle captures an area behind the vehicle as its imaging range. The side cameras on both the left and right sides of the vehicle capture images of areas to the left and right of the vehicle 10 as their imaging range. In this embodiment, each camera 20 can capture images of the surroundings of the vehicle in a full horizontal range (approximately a 360-degree range) surrounding the vehicle 10.

[0035] For example, the front camera is installed on the top of the windshield of the vehicle 10 and has a wide shooting range (several hundred meters) in front of the vehicle. The rear camera is installed on the top of the rear window of the vehicle 10 and has a wide shooting range (several hundred meters) behind the vehicle. The shooting ranges of the front camera and the rear camera may be different, and the shooting range of the front camera may be wider than that of the rear camera. The side camera is installed integrally with the side mirror of the vehicle 10 and has a relatively narrow shooting range (several meters) to the side of the vehicle. Furthermore, each camera 20 has different camera characteristics, such as different lens characteristics depending on the lens type and different gradation characteristics (photoelectric conversion characteristics). Furthermore, each camera 20 may have a different horizontal angle of view.

[0036] 2 is a diagram showing the configuration of the camera 20 and the main controller 30 in the object detection system. The camera 20 includes an image sensor 21, a camera controller 22, an image storage unit 23, and a communication unit 24.

[0037] The image sensor 21 is, for example, a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor. As is well known, the image sensor 21 has a function of converting light incident through a lens unit into an electrical signal. Images captured by the image sensor 21 are continuously transmitted to the camera controller 22 at a predetermined time interval.

[0038] The image sensor 21 has a brightness measurement unit 25 that measures the brightness of a captured image. The brightness measurement unit 25 measures the brightness of a captured image using brightness measurement software, and the software is stored in a software storage unit 26 provided in the image sensor 21. The storage unit 26 is, for example, a non-volatile memory such as an electrically rewritable flash memory. Note that the brightness measurement unit 25 may measure the brightness of an image using a photometric sensor (illuminance sensor). The storage unit 26 may be disposed in a logic operation circuit stacked on the image sensor 21.

[0039] The camera controller 22 has a control unit 27 and a storage unit 28. The control unit 27 includes an image processing arithmetic chip (IC) and executes exposure control of the image sensor 21, correction processing of captured images, and the like using image processing software stored in the storage unit 28. In the camera controller 22, the control unit 27 is preferably an electronic control unit mainly composed of a microcomputer. The control unit 27 has a storage unit (program memory) that stores programs and a processor that executes the programs stored in the storage unit.

[0040] The image storage unit 23 is made of a storage medium to which image data can be written, and has an image storage area to store images captured by the image sensor 21. The image storage unit 23 is preferably made of a writable non-volatile memory that can store image data regardless of whether or not power is supplied. Although not shown in Figure 2, the multiple cameras 20 have the same configuration.

[0041] The main controller 30 includes a control unit 31, a communication unit 32, and an image storage unit 33. The main controller 30 is an electronic control unit (ECU) mainly configured with the control unit 31, which is a microcomputer. The control unit 31 has a storage unit (program memory) that stores programs and a processor that executes the programs stored in the storage unit. The storage unit is a storage medium that stores application software. The programs include an object detection processing program that detects objects around the vehicle and a vehicle driving assistance control program that is executed based on the object detection results. The driving assistance control includes, for example, collision avoidance assistance control, lane keeping control, and ACC (Adaptive Cruise Control). A method corresponding to the program is performed by executing a set of instructions that constitute the program.

[0042] The programs executed by the control unit 31 include various programs related to camera photography and image processing. The control unit 31 may have a function of generating an overhead image of the surroundings of the vehicle 10 using the captured images transmitted from each camera 20 of the vehicle 10.

[0043] Each camera 20 and the main controller 30 can communicate with each other via wired communication. Specifically, the communication unit 24 of the camera 20 and the communication unit 32 of the main controller 30 can communicate with each other. Examples of communication means include a Controller Area Network (CAN) and a Local Interconnect Network (LIN). The communication units 24 and 32 may be connected by, for example, a coaxial cable. However, each camera 20 and the main controller 30 may also communicate with each other via wireless communication. Captured images are transmitted from each camera 20 to the main controller 30 at a predetermined interval. Specifically, the captured images are transmitted at, for example, 40 fps (frames per second), i.e., every 25 milliseconds. Note that it is preferable that the image sensor 21 of the camera 20 captures images at a cycle shorter than the image transmission cycle.

[0044] The main controller 30 may be an in-vehicle terminal mounted on the vehicle, or may be a portable terminal that can be carried by the user (driver).

[0045] The image storage unit 33 is made of a storage medium to which image data can be written, and has an image storage area to store the captured images received from the camera 20. The image storage unit 33 is preferably made of a writable non-volatile memory that can store image data regardless of whether or not power is supplied. The control unit 31 detects objects based on the captured images received from the camera 20, and executes driving support control based on the detection results.

[0046] The object detection system may have the configuration shown in Fig. 3. The configuration in Fig. 3 differs from the configuration in Fig. 2 in that a camera controller 22 is provided integrally with the image sensor 21. Specifically, an image processing and calculation chip that constitutes the camera controller 22 may be mounted on the board of the image sensor 21. In this case, image processing software executed by the camera controller 22 is stored in a storage unit 26 within the image sensor 21.

[0047] <Photography control by camera controller> In this embodiment, for the photographed images sent from the cameras 20 to the main controller 30, each camera 20 sets the exposure conditions for camera photography and performs image processing on the photographed images according to the optical characteristics (camera characteristics) of each camera 20 and the camera installation position, and the details are described below.

[0048] In particular, in this embodiment, when setting exposure conditions and performing image processing based on vehicle-related parameters that indicate the situation of the vehicle 10 and its surroundings at the time of camera photography, the exposure condition setting function and image processing function are shared with the camera 20, thereby reducing the software design burden on the main controller 30 and the processing burden on the main controller 30.

[0049] In the camera 20, the processes executed by the camera controller 22 can be broadly categorized as follows: (1) The camera controller 22 receives vehicle-related parameters from the main controller 30 at the time of camera capture and sets exposure conditions for the next capture based on the vehicle-related parameters (exposure condition setting process). (2) The camera controller 22 receives vehicle-related parameters from the main controller 30 at the time of camera capture and adjusts the white balance of the captured image based on the vehicle-related parameters (white balance adjustment process). (3) The camera controller 22 receives vehicle-related parameters from the main controller 30 at the time of camera capture and performs masking of areas in the captured image that are to be excluded from detection targets of the main controller 30 based on the vehicle-related parameters (image masking process). (4) The camera controller 22 performs noise removal on the captured image based on the noise removal level set on the main controller 30 side (noise removal process). (5) The camera controller 22 determines the readout range of the image to be transmitted to the main controller 30 side (image readout process).

[0050] The above-mentioned (1) exposure condition setting process, (2) white balance adjustment process, (3) image masking process, (4) noise removal process, and (5) image reading process will be described in detail below.

[0051] (1) Exposure Condition Setting Process Fig. 4 is a functional block diagram relating to the exposure condition setting process. In Fig. 4, the main controller 30 includes a parameter acquisition unit 101 that acquires vehicle-related parameters during camera photography.

[0052] The vehicle-related parameters acquired by the parameter acquisition unit 101 may include the following parameters.

[0053] The parameter acquisition unit 101 acquires driving state parameters including at least one of the driving speed of the vehicle 10 and attitude information of the vehicle 10. In this case, the attitude information represents the longitudinal and lateral attitude of the vehicle 10 while the vehicle is driving, and is a parameter including at least one of pitching, yawing, and roll, for example. The vehicle speed and pitching, yawing, and roll information may be detected by a sensor or the like mounted on the vehicle 10. For example, the attitude of the vehicle 10 may be detected by a gyro sensor, a yaw rate sensor, a stroke sensor provided on the front and rear suspensions of the vehicle, or the like.

[0054] The parameter acquisition unit 101 also acquires ambient environment parameters that indicate the ambient environment of the vehicle 10. In this case, the ambient environment parameters are, for example, parameters that indicate the state of rainfall or snowfall. The state of rainfall or snowfall may be detected, for example, by a raindrop sensor provided on the windshield of the vehicle 10. In addition, weather information received by the vehicle 10 that indicates the state of rainfall or snowfall, or information that indicates the operating state of the wiper device of the vehicle 10, may also be used as the ambient environment parameters.

[0055] On the other hand, the camera controller 22 includes a brightness information acquisition unit 111, a parameter receiving unit 112, an exposure condition setting unit 113, an image capture execution unit 114, and an image transmission unit 115. The brightness information acquisition unit 111 acquires brightness information from the brightness measurement unit 25 of the image sensor 21. In this case, the brightness measurement unit 25 measures the brightness of the captured image at a cycle (e.g., 10 milliseconds) shorter than the transmission cycle of the captured image from the camera 20 to the main controller 30. The brightness information acquisition unit 111 acquires brightness information at the measurement cycle of the brightness measurement unit 25 or at a cycle longer than the measurement cycle of the brightness measurement unit 25 and shorter than the image transmission cycle to the main controller 30.

[0056] The parameter receiving unit 112 receives the driving state parameters and the surrounding environment parameters as the vehicle-related parameters from the main controller 30 .

[0057] The exposure condition setting unit 113 sets the exposure conditions for the next image capture based on the driving state parameters of the vehicle 10 and the brightness information of the image. The exposure condition setting unit 113 also sets the exposure conditions for the next image capture based on the ambient environment parameters around the vehicle and the brightness information of the image. The exposure condition setting unit 113 sets at least one of the exposure time (shutter speed) and gain of the image sensor 21 as the exposure condition.

[0058] When the traveling speed or attitude of the vehicle 10 changes, the degree of blur in the captured image changes. For example, as the vehicle speed increases, the pitching fluctuations caused by the vibrations of the vehicle 10 while traveling increase. In this case, it is advisable to set the exposure conditions of the image sensor 21 so as to eliminate blur in the captured image. Specifically, as shown in FIG. 5, for example, the higher the vehicle speed, the shorter the exposure time (i.e., the larger the shutter speed). If any of the pitching, yawing, or roll of the vehicle 10 is large, the blur in the captured image also increases. Therefore, as shown in FIG. 6, it is advisable to shorten the exposure time as the attitude parameters such as pitching, yawing, and roll of the vehicle 10 increase.

[0059] Furthermore, for example, when the vehicle speed is high or when the pitching, yawing, or rolling of the vehicle 10 is large, it is advisable to increase the gain for adjusting the brightness during camera capture. Both the exposure time and the gain may be set according to the driving state parameters, or only one of them may be set. As other exposure conditions, it is also possible to set the image synthesis ratio and brightness measurement coordinates in the captured image by the brightness measurement unit 25 according to the driving state parameters.

[0060] Furthermore, when the vehicle 10 is in a rainy or snowy environment, raindrops or snowflakes may appear in the image, and the movement of the raindrops or snowflakes may distort the image. In this case, the exposure conditions of the image sensor 21 should be set to reduce the effect of raindrops or snowflakes on the captured image. Specifically, when it is raining or snowing, the exposure time should be shortened (i.e., the shutter speed should be increased). Also, when it is raining or snowing, the gain should be increased. Both the exposure time and the gain may be set depending on the ambient environment parameters, or only one of them may be set. Other exposure conditions include the image composition ratio and the brightness measurement coordinates in the captured image by the brightness measurement unit 25, which can be set depending on the ambient environment parameters.

[0061] When setting exposure conditions based on the driving state parameters of the vehicle 10 and weather information, which is a surrounding environment parameter, it is advisable to set the exposure conditions using, for example, the relationship shown in Figure 7. In Figure 7, the relationship is such that the exposure time is shortened as the vehicle speed increases, and further, the exposure time is shortened in rainy weather (including rain and snow) compared to normal weather (including sunny and cloudy weather). The same applies when setting exposure time based on attitude parameters.

[0062] Furthermore, the exposure condition setting unit 113 shortens the exposure time as the brightness of the image increases. For example, the relationship between the vehicle speed and image brightness information, which are driving state parameters of the vehicle 10, and the exposure time may be as shown in Fig. 8. Fig. 8 shows the relationship in which the exposure time is shortened as the vehicle speed increases or the brightness of the image increases.

[0063] In addition, the exposure condition setting unit 113 may have only one of the functions of setting the exposure conditions for the next photographing in accordance with the driving state parameters of the vehicle 10 and the function of setting the exposure conditions for the next photographing in accordance with the ambient environment parameters around the vehicle.

[0064] The photography execution unit 114 causes the image sensor 21 to take a photograph based on the exposure conditions set by the exposure condition setting unit 113. The image transmission unit 115 transmits the captured image to the main controller 30 after the camera has taken the photograph.

[0065] In the main controller 30, the parameter acquisition unit 101 may determine, as the vehicle-related parameters, transition parameters that transition the exposure conditions (at least one of exposure time and gain) of the image sensor 21 depending on the situation of the vehicle 10 or the situation around the vehicle at the time of camera capture, and may cause the exposure conditions to be set on the camera 20 side based on these transition parameters. The transition parameters define the vehicle situation or the situation around the vehicle for transitioning (changing) the exposure conditions. The transition parameters are, for example, information indicating that the vehicle 10 is in a high-speed state or information indicating that the vehicle environment is in a rainy state.

[0066] The parameter receiving unit 112 of the camera controller 22 receives the transition parameters as vehicle-related parameters from the main controller 30. Then, the exposure condition setting unit 113 sets the exposure conditions for the next image capture based on the transition parameters and brightness information. In this case, the exposure time and gain are set according to changes in the vehicle speed, etc.

[0067] For example, the relationship shown in Fig. 9 indicates the relationship between the vehicle speed and the brightness information of the image when the camera 20 receives information indicating that the vehicle 10 is in a low-speed state or a high-speed state as a transition parameter. Fig. 9 shows the relationship in which the exposure time is shorter when the vehicle 10 is in a high-speed state than when it is in a low-speed state.

[0068] Additionally, the parameter acquisition unit 101 of the main controller 30 may acquire, as the driving state parameter, a road surface parameter indicating that the road on which the vehicle 10 is traveling is a bad road. Specifically, information indicating that the road is an unpaved road or that the road is a snow-covered road surface is acquired as the road surface parameter.

[0069] In this case, the exposure condition setting unit 113 of the camera controller 22 sets the exposure conditions for the next image capture based on the road surface parameters received from the main controller 30. Specifically, for example, when the road on which the vehicle 10 is traveling is rough, the exposure condition setting unit 113 executes at least one of a process of shortening the exposure time (i.e., a process of increasing the shutter speed) and a process of increasing the gain, compared to when the road is not rough.

[0070] Figure 10 is a flowchart for explaining the procedure for setting the exposure conditions of the camera 20 (image sensor 21). In Figure 10, (a) shows the processing of the main controller 30, and (b) shows the processing of the camera controller 22. Each of the processes in Figures 10(a) and 10(b) is repeatedly executed at a predetermined interval under an image capturing state in which the camera 20 is capturing an image. The main controller 30 and the camera controller 22 work together to set the exposure conditions of the camera 20 based on related parameters that indicate the situation around the vehicle when the camera is capturing an image, and cause the camera 20 to capture an image under those exposure conditions.

[0071] 10A, in step S1001, the main controller 30 acquires driving state parameters of the vehicle 10. The driving state parameters include at least one of the driving speed of the vehicle 10 and attitude information of the vehicle 10. In addition, in step S1002, ambient environment parameters around the vehicle are acquired. The ambient environment parameters include, for example, information on rainfall or snowfall.

[0072] In step S1003, parameter information including driving state parameters and surrounding environment parameters is transmitted to the camera 20. Note that in steps S1002 and S1003, transition parameters may be determined as information for transitioning (changing) the exposure conditions of the camera 20, and the transition parameters may be transmitted to the camera 20.

[0073] 10B, in step S1011, the camera controller 22 receives parameter information from the main controller 30. In step S1012, brightness information is acquired from the brightness measurement unit 25 of the image sensor 21.

[0074] Then, in step S1013, the exposure conditions for the next image capture are set based on the vehicle 10's driving state parameters, the surrounding environment parameters, and the image brightness information. At this time, the exposure time is set based on, for example, any of the relationships shown in Figures 5 to 8. Alternatively, the gain for the camera image capture is set based on the driving state parameters and the surrounding environment parameters.

[0075] Thereafter, in step S1014, based on the exposure conditions set in step S1013, an image is captured by the image sensor 21. In step S1015, the captured image is transmitted to the main controller 30.

[0076] In step S1004, the main controller 30 receives the captured image transmitted from the camera 20 side.

[0077] The multiple cameras 20 mounted on the vehicle 10 may have the following configuration. Fig. 11 is a system configuration diagram in which the cameras 20 include a first camera 20A and a second camera 20B. For example, the first camera 20A is a front camera located at the front of the vehicle, and the second camera 20B is a rear camera located at the rear of the vehicle.

[0078] 11, the first camera 20A has a first image sensor 21A and a first camera controller 22A. The second camera 20B has a second image sensor 21B and a second camera controller 22B. The image sensors 21A and 21B have different optical characteristics.

[0079] In the first camera controller 22A, the exposure condition setting unit 113 sets the exposure conditions for the first image sensor 21A based on the vehicle-related parameters and image brightness information received from the main controller 30. In the second camera controller 22B, the exposure condition setting unit 113 sets the exposure conditions for the second image sensor 21B based on the vehicle-related parameters and image brightness information received from the main controller 30. In this case, the processing modes of the exposure condition setting units 113 of the cameras 20A, 20B are different from each other. Specifically, the exposure condition setting units 113 of the cameras 20A, 20B set the exposure conditions for each image sensor 21A, 21B using a relationship between the vehicle-related parameters and the exposure conditions that is determined for each camera.

[0080] In an in-vehicle camera system having multiple cameras 20, if the optical characteristics of the image sensors 21 of the cameras 20 are different, it is conceivable that the exposure conditions and image correction contents will differ even if the vehicle driving information and environmental conditions transmitted from the main controller 30 are the same. In this regard, since the camera controller 22 of each camera 20 processes requests from the main controller 30 individually, the main controller 30 does not need to change the processing correspondence for each camera 20, and simplification of calculation processing can be realized.

[0081] For each camera 20 in the vehicle 10, even in the same driving environment, exposure conditions may be set individually according to the shooting direction or camera mounting position of each camera 20. The camera controller 22 of each camera 20 sets the exposure conditions based on the vehicle-related parameters (driving state parameters, surrounding environment parameters) received from the main controller 30 and the shooting direction or mounting position of each camera 20 in the vehicle 10.

[0082] When the vehicle 10 is traveling, the relative speed with respect to oncoming vehicles in front of the vehicle is high, and the relative speed with respect to following vehicles behind the vehicle is low. In this case, there is a concern that the image of the oncoming vehicle captured by the front camera is more likely to be blurred than the image of the oncoming vehicle captured by the rear camera. This difference in situation is thought to become more pronounced as the traveling speed of the vehicle 10 increases.

[0083] Therefore, when the vehicle 10 is traveling, the front camera and the rear camera use different relationships between the traveling speed of the vehicle 10 and the exposure conditions, and the exposure conditions of each camera 20 are set based on the traveling speed of the vehicle 10. In this case, it is preferable to set the exposure time (shutter speed) as the exposure condition using, for example, the relationship shown in Figure 12. In Figure 12, the relationship is set so that the exposure time of the front camera is shorter than that of the rear camera. The exposure speed of the side cameras may be the same as that of the rear camera.

[0084] In a dark environment, such as at night, the area in front of the vehicle is brightly captured with the vehicle headlights on, so there is no need to use an excessively long exposure time for the front camera. On the other hand, it is desirable to use a long exposure time for the rear camera to capture a bright image of the area behind the vehicle.

[0085] For example, in vehicle 10, the front camera among the multiple cameras 20 is mounted inside the vehicle interior on the inside of the vehicle windshield, and when the wipers operate on the vehicle windshield during rainy weather, the wiper blades may appear in the image captured by the front camera. In this case, the front camera is mounted inside the vehicle, and corresponds to an "interior-mounted camera."

[0086] 13 , wiper blades 52 are provided on the outer surface of windshield 51 of vehicle 10 (i.e., on the vehicle exterior side), and the wiping area of ​​wiper blade 52 is designated RA. Front camera 20F is provided inside the vehicle cabin. In this case, if the camera position overlaps with wiping area RA, wiper blade 52 will appear in the image captured by front camera 20F when the wipers are operating.

[0087] Therefore, when the wipers are operating in rainy weather, it is conceivable to shorten the exposure time (increase the shutter speed) of the front camera 20F. In contrast, the wiper blades 52 are not captured in the images captured by cameras 20 installed outside the vehicle, such as side cameras on the sides of the vehicle, even in the same rainy weather. Therefore, it is not necessary to shorten the exposure time as a countermeasure to prevent the wipers from being captured in the images in rainy weather.

[0088] Therefore, it is advisable to set different exposure conditions for the front camera and the side cameras in rainy weather. In this case, the front camera has a short exposure time to prevent the wiper from being captured in the image, and the side cameras have a longer exposure time than the front camera.

[0089] In addition, in a dark environment such as at night, it is recommended to shorten the exposure time of the front camera to prevent the wiper from being caught in the image, and to lengthen the exposure time of the side camera to obtain a brighter image.

[0090] If a rear camera at the rear of the vehicle is installed inside the vehicle cabin in the same way as the front camera and the wiper of the vehicle's rear window is reflected in the image, the rear camera should be controlled in the same way as the front camera.

[0091] 14 is a flowchart showing a processing procedure executed by the camera controller 22 in each camera 20. This processing may be executed, for example, in step S1013 in FIG.

[0092] 14, in step S1401, it is determined whether the vehicle 10 is in a traveling state. At this time, it is determined whether the vehicle speed is equal to or greater than a predetermined speed other than 0, for example. If step S1401 is positive, the process proceeds to step S1402. In step S1402, exposure conditions are set based on the traveling speed of the vehicle 10 and the shooting direction of the camera on the vehicle 10. At this time, if the camera is a front camera, the exposure time is set to a relatively short time. Furthermore, if the camera is other than a front camera (a side camera or a rear camera), the exposure time is set to a relatively long time.

[0093] In step S1403, it is determined whether information indicating a rainfall or snowfall state (rain / snow information) has been received as an ambient environment parameter. The rain / snow information may be information from the raindrop sensor of the vehicle 10, weather information, or information indicating wiper operation.

[0094] If step S1403 is positive, the process proceeds to step S1404. In step S1404, exposure conditions are set based on the ambient environment parameters and the fact that the camera is an indoor-mounted camera (e.g., a front camera). At this time, if the camera is a camera that captures the wiper, such as a front camera, the exposure time is set to a relatively short time to prevent the wiper from being captured. On the other hand, if the camera is a camera that does not capture the wiper, such as a side camera, the exposure time is set to a relatively long time.

[0095] According to the above-described configuration for the exposure condition setting process, the following effects can be obtained.

[0096] When detecting an object using camera images, it is conceivable that the exposure conditions of the image sensor 21 are set based on the conditions of the vehicle 10 and its surroundings. However, these exposure conditions depend on the camera characteristics, the camera mounting position, and other factors. Therefore, the exposure conditions must be set according to the camera characteristics. In a system configuration in which the exposure conditions are set on the main controller 30 side, software changes to the main controller 30 are necessary when the camera 20 is replaced or when multiple cameras 20 with different camera characteristics are used. In this regard, in the above configuration, the camera controller 22 sets the exposure conditions for the next image capture based on vehicle-related parameters received from the main controller 30, and then causes the image sensor 21 to capture an image based on these exposure conditions. In this case, by assigning the exposure condition setting function to the camera 20 side, the software design burden on the main controller 30 and the processing burden on the main controller 30 can be reduced. As a result, the control of the camera 20 can be optimized, and ultimately, the object detection process can be performed appropriately.

[0097] In a system in which multiple cameras 20 are installed on a vehicle 10, the multiple cameras 20 can share vehicle-related parameters transmitted from the main controller 30 and take appropriate photographs according to the condition and optical characteristics of each camera 20.

[0098] The camera controller 22 acquires driving state parameters including at least one of the driving speed and attitude information of the vehicle 10 from the main controller 30, and sets the exposure conditions of the image sensor 21 based on the driving state parameters. This allows the camera 20 to perform exposure control that suppresses blurring and the like, without the main controller 30 transmitting a control command for the exposure itself of the camera 20. In this case, even in a system having multiple cameras 20, there is no need to have exposure control software that is tailored to the optical characteristics of each camera 20, thereby increasing the degree of design freedom.

[0099] The camera controller 22 acquires ambient environment parameters indicating the ambient environment of the vehicle 10 from the main controller 30, and sets the exposure conditions of the image sensor 21 based on the ambient environment parameters. This allows the camera 20 to perform exposure control that suppresses the effects of raindrops and the like, without the main controller 30 transmitting a control command for the exposure of the camera 20 itself.

[0100] The camera 20 itself sets at least one of the exposure time and gain of the image sensor 21 in accordance with vehicle-related parameters, thereby enabling appropriate exposure control to be performed without waiting for an instruction on exposure conditions from the main controller 30.

[0101] The main controller 30 determines transition parameters for transitioning the exposure conditions of the image sensor 21 in response to changes in the situation of the vehicle 10 and its surroundings during camera photography, while the camera controller 22 sets the exposure conditions for the next photograph using the transition parameters received from the main controller 30. In this case, the camera controller 22 specifies the amount of transition, such as exposure time, so that it is possible to limit the influence on image quality caused by factors external to the camera 20 accumulated over time. This makes it possible to generate images with reduced noise.

[0102] The brightness measurement unit 25 of the camera 20 is configured to measure the brightness of a captured image at a cycle shorter than the cycle of transmitting the captured image to the main controller 30. In other words, the brightness in the camera image is measured more frequently than the frequency of transmitting the image to the main controller 30. This enables dynamic exposure control while following changes in the external environment of the camera image capture. As a result, images with stable brightness can be transmitted to the main controller 30.

[0103] The camera 20 is configured so that the image sensor 21 has a brightness measurement function, which enables brightness measurement processing without imposing restrictions on the frequency of communication between the image sensor 21 and the image processing calculation chip.

[0104] The camera controller 22 of each camera 20 is configured to set exposure conditions for each camera 20 based on the vehicle-related parameters received from the main controller 30 and the camera's shooting direction or installation position on the vehicle 10. This makes it possible to appropriately respond even when the influence of the driving state parameters and surrounding environment parameters differs for each camera 20 depending on the camera's shooting direction or installation position on the vehicle 10.

[0105] The camera controller 22 is configured to set exposure conditions based on the traveling speed of the vehicle 10 and the shooting direction of each camera 20 on the vehicle 10. In this case, it is possible to take appropriate measures while taking into consideration that the degree of influence of the vehicle traveling speed on camera shooting differs depending on the shooting direction of each camera.

[0106] The camera controller 22 is configured to set exposure conditions based on the ambient environment parameters and the fact that its own camera 20 is an indoor-mounted camera (e.g., a front camera). In this case, it is possible to take appropriate measures while taking into consideration that the degree of influence of the ambient environment parameters on camera photography varies depending on the mounting form of each camera.

[0107] (2) White Balance Adjustment Process FIG. 15 is a functional block diagram relating to a configuration for performing white balance adjustment process.

[0108] 15, the main controller 30 includes a parameter acquisition unit 121 that acquires, as adjustment parameters for white balance adjustment, information on at least one of the ambient color of the vehicle's surroundings when capturing an image with the camera and the incident light entering the camera 20. The adjustment parameters are information indicating the color tone of the ambient environment or information indicating the color tone of the light entering the camera 20.

[0109] The captured image received from the camera 20 shows the scenery around the vehicle, with the surrounding scenery colored vermilion at dusk and the camera 20 being backlit. It can also be seen that the captured image is colored by lighting in a tunnel, street lights, illumination, or the like. The main controller 30 may determine the driving scene based on the captured image and may include a color recognition unit that recognizes the color of the surrounding environment in the driving scene. That is, the parameter acquisition unit 121 acquires adjustment parameters corresponding to the scene captured by the camera from the color recognition unit, etc. Furthermore, because the color of the incident light changes depending on the weather in the driving area of ​​the vehicle 10, it may be possible to recognize whether the weather is sunny, cloudy, or rainy from the captured image and acquire adjustment parameters corresponding to the weather information.

[0110] For example, the parameter acquisition unit 121 detects that the scene is a twilight scene based on the captured image, and acquires a parameter indicating a "strong red level" as an adjustment parameter. The main controller 30 transmits the adjustment parameter acquired by the parameter acquisition unit 121 to the camera 20 at a predetermined interval.

[0111] On the other hand, the camera controller 22 includes a brightness information acquisition unit 131, a parameter receiving unit 132, a white balance adjustment unit 133 that performs white balance adjustment, and an image transmission unit 134. The brightness information acquisition unit 131 acquires brightness information from the brightness measurement unit 25 of the image sensor 21. In this case, the brightness measurement unit 25 measures the brightness of the captured image at a cycle (e.g., 10 milliseconds) shorter than the transmission cycle of the captured image from the camera 20 to the main controller 30. The brightness information acquisition unit 131 acquires the brightness information at the measurement cycle of the brightness measurement unit 25 or at a cycle longer than the measurement cycle of the brightness measurement unit 25 and shorter than the image transmission cycle to the main controller 30.

[0112] The parameter receiving unit 132 receives the adjustment parameters from the main controller 30 .

[0113] The white balance adjustment unit 133 adjusts the white balance of the captured image based on the adjustment parameters and the brightness information of the image. For example, when the ambient color around the vehicle is vermilion at dusk or when backlight is incident on the camera, the white balance adjustment is performed to correct the color balance of the image, taking into account that the color balance of the image is disrupted due to external light. At this time, it is preferable to perform color temperature correction, such as increasing or decreasing the redness or increasing or decreasing the blueness of the image, depending on the adjustment parameters.

[0114] For example, in a twilight scene, the adjustment parameter received from the main controller 30 is a parameter indicating a "strong red level." Based on this adjustment parameter, the white balance adjustment unit 133 adjusts the white balance so as to suppress red.

[0115] Each on-board camera 20 has a different mounting position on the vehicle 10, a different shooting direction, different camera characteristics, etc. In this regard, the camera controller 22 of each camera 20 is configured to have a white balance adjustment function for each camera according to the mounting position on the vehicle 10, the different shooting direction, the different camera characteristics, etc.

[0116] The white balance adjustment unit 133 may adjust the white balance of the captured image at a cycle shorter than the cycle of transmitting the captured image to the main controller 30 (for example, every 10 milliseconds).

[0117] The image transmission unit 134 transmits the captured image after the white balance adjustment to the main controller 30.

[0118] In the main controller 30, the parameter acquisition unit 121 may determine, as an adjustment parameter, a transition parameter that transitions the amount of white balance adjustment depending on the ambient color around the vehicle or the incident light entering the camera 20, and may cause the camera 20 to perform white balance adjustment based on this transition parameter. The transition parameter defines the color tone around the vehicle. Specifically, the transition parameter may be determined based on information about the color of the surrounding scenery, for example. Alternatively, the transition parameter may be configured to take into account color tone due to weather conditions in a complex manner.

[0119] In the camera controller 22, the parameter receiving unit 132 receives the transition parameters from the main controller 30. The white balance adjustment unit 133 adjusts the white balance of the captured image based on the transition parameters. Specifically, for example, when the sky becomes increasingly reddish at dusk, it is advisable to adjust the white balance according to the transition parameter that indicates the amount of reddishness.

[0120] Fig. 16 is a flowchart for explaining the procedure for adjusting the white balance of a captured image. In Fig. 16, (a) shows the processing of the main controller 30, and (b) shows the processing of the camera controller 22. Each of the processing in Fig. 16(a) and (b) is repeatedly executed at a predetermined interval under the shooting conditions in which the camera 20 is shooting.

[0121] 16A, in step S1601, the main controller 30 acquires adjustment parameters, which are information relating to at least one of the ambient color around the vehicle and the incident light entering the camera 20. In step S1602, the main controller 30 transmits the adjustment parameters to the camera 20. Note that in steps S1601 and S1602, transition parameters may be determined as information for causing the camera 20 to perform white balance adjustment, and the transition parameters may be transmitted to the camera 20.

[0122] 16B, in step S1611, the camera controller 22 receives adjustment parameters from the main controller 30. In step S1612, brightness information is acquired from the brightness measurement unit 25 of the image sensor 21. In step S1613, an image captured by the image sensor 21 is acquired.

[0123] Thereafter, in step S1614, the camera controller 22 adjusts the white balance of the captured image based on the adjustment parameters and the brightness information of the image. At this time, it is preferable that the camera controller 22 adjusts the white balance of the captured image at a cycle shorter than the cycle for transmitting the captured image to the main controller 30. In step S1615, the captured image after white balance adjustment is transmitted to the main controller 30.

[0124] In step S1603, the main controller 30 receives the captured image transmitted from the camera 20 side.

[0125] According to the above configuration in which the white balance adjustment process is performed, the following effects can be obtained.

[0126] The camera controller 22 acquires white balance adjustment parameters from the main controller 30, including information regarding at least one of the ambient color around the vehicle and the incident light entering the camera 20, and performs white balance adjustment on the captured image to be sent to the main controller 30 based on the adjustment parameters. This allows for white balance adjustment appropriate for each camera 20, even if different white balance adjustments are performed for each on-board camera 20 depending on the camera's characteristics, camera installation position, etc. In this case, the main controller 30 does not need to have a white balance adjustment function (white balance adjustment software) tailored to the camera's characteristics for each camera 20. In other words, even if the main controller 30 does not have white balance adjustment software tailored to the characteristics of each camera, white balance control tailored to the scene can be performed for each camera 20. As described above, the object detection system can achieve appropriate white balance adjustment for each image while adapting to the conditions around the vehicle.

[0127] Furthermore, by allocating the white balance adjustment function for the captured image to the camera 20, the software design burden on the main controller 30 and the processing burden on the main controller 30 can be reduced.

[0128] In a system in which multiple cameras 20 are installed on a vehicle 10, the multiple cameras 20 can share adjustment parameters transmitted from the main controller 30 and perform appropriate white balance adjustment according to the state and optical characteristics of each camera 20.

[0129] The camera controller 22 is configured to adjust the white balance of a captured image at a cycle shorter than the cycle of transmitting the captured image to the main controller 30. In other words, the camera 20 measures the white balance in the image more frequently than it transmits the image to the main controller 30. This enables dynamic white balance control that follows changes in the external environment of the camera image capture. As a result, it is possible to transmit an image to the main controller 30 with the influence of the external environment reduced.

[0130] The main controller 30 determines transition parameters for transitioning the amount of white balance adjustment for the captured image in response to changes in at least one of the ambient color around the vehicle and the color of the incident light entering the camera 20, while the camera controller 22 adjusts the white balance of the captured image based on the transition parameters. This allows for appropriate white balance adjustment even when the color of the surroundings of the vehicle changes, thereby preventing a decline in image recognition performance.

[0131] The image sensor 21 of the camera 20 is configured to have software for the brightness measurement unit 25. This allows brightness measurement processing without imposing restrictions on the frequency of communication between the image sensor 21 and the image processing calculation chip.

[0132] (3) Image Masking Processing Next, a configuration for performing masking processing on a captured image will be described. The main controller 30 has limitations in its processing performance, making it difficult to detect (recognize) all objects, for example, in a 360-degree range around the vehicle, from close to far. Therefore, in this embodiment, the camera 20 performs image masking on portions of the captured image that are not necessary for detecting objects around the vehicle or for driving assistance control of the vehicle 10, thereby reducing the processing load on the main controller 30. Image masking is a process for hiding or disabling specific portions of the captured image. The masking process reduces the feature values ​​of the image sent to the main controller 30. In this embodiment, the image masking process involves a smoothing process (filtering process) that smooths and blurs part or all of the captured image, or a filling process that fills part or all of the captured image.

[0133] FIG. 17 is a functional block diagram relating to a configuration for performing image mask processing.

[0134] 17, the main controller 30 includes a parameter acquisition unit 141 that acquires vehicle-related parameters during camera photography. The parameter acquisition unit 141 acquires, as the vehicle-related parameters, various pieces of information that indicate the conditions of the vehicle 10 and its surroundings.

[0135] The information indicating the status of the vehicle 10 is, for example, the traveling speed and the turning status of the vehicle 10. The information indicating the traveling speed and the turning status of the vehicle 10 may be acquired from the detection values ​​of a speed sensor and a steering angle sensor mounted on the vehicle 10.

[0136] The information indicating the situation around the vehicle is area information indicating the area in which the vehicle 10 is traveling. The information indicating the situation around the vehicle may be acquired, for example, from map information used in a navigation system. Specifically, the information indicating the situation around the vehicle includes information indicating whether the area in which the vehicle 10 is traveling is an urban area or a non-urban area, information indicating whether the area in which the vehicle 10 is traveling is an area near a school including a route to and from school such as an elementary or junior high school, or a non-school area, information indicating whether the road on which the vehicle is traveling is an expressway or a non-expressway, etc.

[0137] In addition, information indicating whether the area is an urban area, information indicating whether the area is near a school, and information indicating whether the area is on a highway correspond to area information indicating whether the area has a relatively large or relatively small number of objects to be detected near the vehicle.

[0138] The determination of whether or not the area is an urban area may be made based on the number of buildings and intersections within a predetermined distance around the vehicle. Urban areas may be distinguished from other areas in the map information in advance. The determination of whether or not the area is a school neighborhood may be made based on whether or not there are elementary or junior high schools, etc. within a predetermined distance around the vehicle. The map information may be distinguished in advance between school neighborhood areas and other areas.

[0139] The main controller 30 transmits the vehicle-related parameters acquired by the parameter acquisition unit 141 to the camera 20 at a predetermined interval.

[0140] On the other hand, the camera controller 22 includes a parameter receiving unit 151, a mask processing unit 152, and an image transmitting unit 153. The parameter receiving unit 151 receives vehicle-related parameters from the main controller 30.

[0141] The mask processing unit 152 performs mask processing on areas in the captured image of the camera 20 that are to be excluded from detection targets in the main controller 30, based on the vehicle-related parameters. The mask processing unit 152 has an area setting unit 154 that sets a mask area, which is an area in the captured image where mask processing is performed. The mask processing unit 152 also has either a smoothing processing unit 155 that performs smoothing processing on the captured image, or a filling processing unit 156 that performs filling processing on the captured image to be transmitted to the main controller 30.

[0142] The region setting unit 154 sets a mask region RM in the captured image based on the vehicle-related parameters received from the main controller 30. For example, as shown in Fig. 18 , in the image captured by the camera 20 in front of the vehicle, a predetermined region farther from the vehicle 10 is set as the mask region RM. The mask region RM is set in the upper part of the captured image that is farther from the vehicle 10.

[0143] In this case, the region setting unit 154 may variably set the vertical dimension LA of the mask region RM in the up-down direction according to, for example, the traveling speed of the vehicle 10. The vertical dimension LA is the dimension from the top edge of the captured image. For example, the vertical dimension LA of the mask region RM is determined using the relationship shown in FIG. 19. According to FIG. 19, the higher the traveling speed of the vehicle 10, the smaller the vertical dimension LA becomes, and the smaller the mask region RM is set. In other words, the higher the traveling speed of the vehicle 10, the larger the detection region in which object detection is performed in the captured image is set.

[0144] Furthermore, the region setting unit 154 may set the mask region RM according to information indicating the turning situation of the vehicle 10. For example, the mask region RM is variably set according to whether the vehicle 10 is turning right or left. When the vehicle 10 is turning, it may be turning right or left, or changing lanes. In this case, it is preferable to make the mask region RM different on the left and right sides of the center of the left-right direction of the captured image (i.e., the center of the vehicle 10 in the left-right direction) according to the turning direction of the vehicle 10. For example, as shown in FIG. 20 , when turning left, it is preferable to make the mask region RM smaller on the left side of the image and larger on the right side of the image.

[0145] Furthermore, the area setting unit 154 may set the mask area RM depending on whether the driving area of ​​the vehicle 10 is an urban area or a non-urban area. In this case, in an urban area, it is conceivable that many objects to be detected by the main controller 30 are present in areas relatively close to the vehicle 10. As the number of detection objects increases, the processing load on the main controller 30 increases, so it is desirable to mask on the screen objects that do not need to be detected for driving assistance control, etc. Therefore, in an urban area, the mask area RM is made larger than in a non-urban area to enable the main controller 30 to properly detect objects.

[0146] For example, in urban areas, areas more than 300 m away from the vehicle 10 are set as the mask area RM, whereas in non-urban areas, areas more than 400 m or 500 m away are set as the mask area RM.

[0147] Furthermore, the region setting unit 154 may set the mask region RM depending on whether the driving area of ​​the vehicle 10 is a school vicinity area or not. In this case, in a school vicinity area, it is necessary to properly detect children and students in a situation where they are present in large numbers. Considering the processing load of the main controller 30, it is desirable to prioritize object detection near the vehicle. Therefore, in a school vicinity area, the mask region RM is made larger than in an area not surrounding a school, so that the main controller 30 can properly detect objects.

[0148] Furthermore, the region setting unit 154 may set the mask region RM depending on whether the driving area of ​​the vehicle 10 is an expressway or a non-expressway. In this case, on an expressway, there are no intersections or traffic lights, and there are fewer detection targets near the vehicle compared to non-expressways. Therefore, on an expressway, the mask region RM is made smaller than on a non-expressway.

[0149] The smoothing processing unit 155 performs smoothing processing using a smoothing filter in the mask region RM set by the region setting unit 154. This smoothing processing blurs part of the captured image, making it difficult or impossible for the main controller 30 to detect an object.

[0150] In the smoothing processing unit 155, the smoothing processing as mask processing may be performed by image processing software. In this case, the image processing software is stored in the storage unit 26 in the image sensor 21 (see FIG. 3). Note that the storage unit 26 may be disposed in a logical operation circuit stacked on the image sensor 21. Also, in the configuration of FIG. 2, the image processing software for performing the smoothing processing may be stored in the storage unit 28 in the camera controller 22.

[0151] Furthermore, the smoothing processing unit 155 may be configured so that the smoothing processing is performed by a smoothing calculation circuit serving as an image mask calculation circuit. In this case, the smoothing calculation circuit may be disposed in the image sensor 21 or in a logical calculation circuit stacked on the image sensor 21.

[0152] The filling processing unit 156 performs filling processing in the mask area RM set by the area setting unit 154. This filling processing makes it difficult or impossible for the main controller 30 to detect an object in a part of the captured image.

[0153] In the fill processing unit 156, the fill processing as a mask processing may be performed by image processing software. In this case, the image processing software is stored in the storage unit 26 in the image sensor 21 (see FIG. 3). Note that the storage unit 26 may be disposed in a logic operation circuit stacked on the image sensor 21. Also, in the configuration of FIG. 2, the image processing software for performing the fill processing may be stored in the storage unit 28 in the camera controller 22.

[0154] Furthermore, the filling processing unit 156 may be configured so that the filling process is performed by a filling operation circuit serving as an image mask operation circuit. In this case, the filling operation circuit may be disposed in the image sensor 21 or in a logical operation circuit stacked on the image sensor 21.

[0155] The image transmission unit 153 transmits the captured image after the smoothing process by the smoothing processing unit 155 or the filling process by the filling processing unit 156 has been performed to the main controller 30 .

[0156] In addition, in the main controller 30, the parameter acquisition unit 141 may acquire information regarding the driving direction of the vehicle 10, i.e., information regarding whether the vehicle 10 is driving forward or backward, as a vehicle-related parameter during camera photography.

[0157] The region setting unit 154 sets the mask region RM based on the relationship between the traveling direction of the vehicle 10 and the imaging direction of the camera 20. Specifically, when the traveling direction of the vehicle 10 and the imaging direction of the camera 20 do not match, the region setting unit 154 makes the mask region RM larger than when the traveling direction of the vehicle 10 and the imaging direction of the camera 20 match. In other words, for the image captured by the front camera, when the vehicle 10 is traveling backward, the mask region RM is made larger than when the vehicle 10 is traveling forward.

[0158] In addition to masking a portion of the image captured by the camera 20 as a mask region RM, the mask processing unit 152 may also perform mask processing on the entire captured image as a mask region RM. For example, if the multiple cameras 20 mounted on the vehicle 10 include a high-resolution long-distance recognition camera used to recognize distant objects, it is preferable to perform mask processing on the entire image captured by the long-distance recognition camera as a mask region RM when the vehicle 10 is traveling at a low speed. The long-distance recognition camera may be mounted on the vehicle 10 as a front telephoto camera or a rear camera.

[0159] Fig. 21 is a flowchart for explaining the procedure for masking a captured image. In Fig. 21, (a) shows the processing by the main controller 30, and (b) shows the processing by the camera controller 22. Each of the processes in Fig. 21(a) and (b) is repeatedly executed at a predetermined interval under a shooting condition in which shooting is performed by the camera 20.

[0160] 21A, in step S2101, the main controller 30 acquires various information indicating the conditions of the vehicle 10 and its surroundings as vehicle-related parameters. In step S2102, the main controller 30 transmits the vehicle surroundings information to the camera 20.

[0161] As shown in Fig. 21(b), in step S2111, the camera controller 22 receives vehicle-related parameters from the main controller 30. In step S2112, a mask region RM in the captured image is set based on the vehicle-related parameters. In step S2113, mask processing is performed on the mask region RM in the captured image. In step S2113, either smoothing processing or filling processing is performed as the mask processing.

[0162] Thereafter, in step S2114, the captured image after the mask processing is transmitted to the main controller 30.

[0163] In step S2103, the main controller 30 receives the captured image transmitted from the camera 20 side.

[0164] According to the above configuration in which mask processing is performed on a captured image, the following effects can be obtained.

[0165] In order for the main controller 30 to monitor the periphery of the vehicle over a wide area and appropriately, it is desirable to reduce the processing load of image recognition. Furthermore, depending on the conditions of the vehicle 10 and its surroundings, the captured image may contain information (unnecessary features) that is unnecessary for object detection by the main controller 30. In consideration of this, the camera controller 22 is configured to acquire vehicle-related parameters, which are information about the vehicle 10 or its surroundings, from the main controller 30, and to perform masking processing on areas in the captured image to be excluded from detection targets based on the vehicle-related parameters. In other words, the masking processing on the camera 20 side reduces the processing load on the main controller 30, which is the image receiving side.

[0166] In this case, the degree of masking required is determined depending on the optical characteristics and performance of the camera 20, but masking that reflects the optical characteristics, etc. is performed by the camera controller 22 of each camera 20. In this case, the main controller 30 does not need to perform masking that matches the specifications, such as the optical characteristics, of each camera 20. With the above configuration, it is possible to transmit captured images that have been subjected to appropriate masking to the main controller 30. As a result, the main controller 30 can perform object detection appropriately.

[0167] The camera controller 22 is configured to perform masking processing by smoothing the captured image based on vehicle-related parameters. The degree of smoothing required for the captured image is determined by the specifications of the lens and image sensor in the camera 20. In this case, the main controller 30 transmits information associated with the object detection distance (angular resolution information, distance resolution information, etc.) to the camera 20, and the camera 20 determines the smoothing area based on the specifications of its own lens and image sensor and performs the smoothing processing. This makes it possible to create an intended detection area regardless of the specifications of each camera 20 mounted on the vehicle 10, thereby enabling appropriate object detection.

[0168] The camera controller 22 is configured to perform masking processing by filling in the captured image based on vehicle-related parameters. The areas in the captured image that require filling are determined by the number of pixels and size of the image sensor 21 of each camera 20. In this case, the main controller 30 transmits information (such as angle information) associated with the object detection area to the camera 20, and the camera 20 determines the area to be filled in based on the specifications of its own lens and image sensor, and then performs the filling process. This allows the intended detection area to be created regardless of the specifications of each camera 20 mounted on the vehicle 10, thereby enabling appropriate object detection.

[0169] The camera controller 22 sets a mask area RM in the captured image based on vehicle-related parameters, which is excluded from the detection target of the main controller 30. Specifically, the mask area RM is set based on the traveling speed or turning conditions of the vehicle 10. In this case, the detection target (i.e., the feature points to be recognized) will differ depending on the traveling condition of the vehicle 10, but the above configuration makes it possible to achieve appropriate masking processing.

[0170] The main controller 30 acquires area information (such as information about urban areas, areas around schools, and expressways) indicating whether the area has a relatively large or small number of objects to be detected near the vehicle as vehicle-related parameters, and the camera controller 22 sets a mask area RM based on the area information. This makes it possible to adjust the load according to the situation of the objects to be detected near the vehicle and achieve appropriate object detection.

[0171] The image sensor 21 is configured to have the smoothing process or filling process performed by image processing software, and further configured to have a storage unit 26 in which the image processing software is stored. Here, when transmitting an image from the image sensor 21 to the image processing arithmetic chip, the memory capacity (DRAM capacity) of the image processing arithmetic chip and the communication capacity during transmission become factors that limit processing. In this regard, by having software that performs mask processing (smoothing process or filling process) inside the image sensor, it is possible to share the RAM with the RAM used when capturing an image, and processing can be performed efficiently.

[0172] Alternatively, the mask processing is performed by an image mask calculation circuit, and the image mask calculation circuit is further provided in the image sensor 21. In this case, the mask processing (smoothing processing or filling processing) is performed by a calculation circuit rather than by software, and by having this circuit inside the image sensor, the processing can be performed almost in synchronization with image capture.

[0173] (4) Noise Removal Processing Next, the noise removal processing for the captured image will be described. Fig. 22 is a functional block diagram relating to the noise removal processing.

[0174] 22, the main controller 30 includes an image / noise information receiving unit 161 , a noise level determining unit 162 , a scene information acquiring unit 163 , a threshold information setting unit 164 , and a noise level transmitting unit 165 .

[0175] The camera controller 22 also includes a noise information acquisition unit 171, a noise level reception unit 172, a noise removal unit 173, a defect repair unit 174, and an image and noise information transmission unit 175. First, the configuration of the camera controller 22 will be described.

[0176] The noise information acquisition unit 171 acquires noise amount information indicating the amount of noise contained in the image captured by the image sensor 21. In this case, the noise information acquisition unit 171 detects the amount (level) of salt-and-pepper noise or spectral noise present throughout the captured image and acquires noise amount information indicating the amount of noise. Furthermore, since it is considered that the amount of noise increases as the temperature of the image sensor 21 increases, the noise information acquisition unit 171 may be configured to acquire noise amount information estimated based on the sensor temperature. The noise information acquisition unit 171 acquires noise amount information at a cycle (e.g., 10 milliseconds) shorter than the transmission cycle of the captured image from the camera 20 to the main controller 30.

[0177] The noise level receiving unit 172 receives the removed noise level determined by the noise level determining unit 162 of the main controller 30. The removed noise level is determined based on the noise amount information acquired by the noise information acquiring unit 171, and details thereof will be described later.

[0178] The noise removal unit 173 performs noise removal processing on the captured image based on the noise removal level received from the main controller 30 and the noise amount information acquired by the noise information acquisition unit 171. The noise removal unit 173 performs noise removal on the image using, for example, a noise removal filter (smoothing filter). At this time, if the noise removal level (the level of noise to be removed) is low, the degree of noise removal is relatively increased, and if the noise removal level is high, the degree of noise removal is relatively decreased. Note that the noise removal level can also be rephrased as the noise level (allowable noise level) permitted in the captured image. If the allowable noise level is low, i.e., if noise tolerance is low, the degree of noise removal is relatively increased. On the other hand, if the allowable noise level is high, i.e., if noise tolerance is high, the degree of noise removal is relatively decreased.

[0179] In the noise removal unit 173, the noise removal process may be performed by image processing software. In this case, the image processing software is stored in the storage unit 26 in the image sensor 21 (see FIG. 3). Note that the storage unit 26 may be disposed in a logical operation circuit stacked on the image sensor 21. In the configuration of FIG. 2, the image processing software for performing the noise removal process may be stored in the storage unit 28 in the camera controller 22.

[0180] In addition to noise, the captured image may also contain pixel defects that affect object detection. The defect repair unit 174 repairs pixel defects in the captured image based on threshold information for pixel defect repair received from the main controller 30. Specifically, the defect repair unit 174 searches for areas where image defects occur based on the threshold information from the main controller 30 and repairs the defects by image correction. At this time, the defect repair unit 174 changes the level at which pixel defects are searched in the captured image based on the threshold information. For example, when the threshold value used as threshold information is small, more pixel defects are searched for than when the threshold value is large. For example, in driving scenes during nighttime driving, it is preferable to search for more pixel defects than in driving scenes during daytime driving. Furthermore, in driving scenes during urban driving with many obstacles, it is preferable to search for more pixel defects than in driving scenes in places with few obstacles. In this way, pixel defects are corrected according to the driving scene of the vehicle 10.

[0181] The image / noise information transmission unit 175 transmits to the main controller 30 the captured image after noise removal by the noise removal unit 173 and image defects repaired by the defect repair unit 174, as well as the noise amount information acquired by the noise information acquisition unit 171.

[0182] Meanwhile, in the main controller 30, the image and noise information receiving section 161 acquires, from the camera controller 22, a captured image and noise amount information indicating the amount of noise contained in the entire image.

[0183] The noise level determination unit 162 determines the noise level (noise removal level) at which noise is removed from the captured image based on the noise amount information and the captured image received from the camera controller 22. When an object to be detected (another vehicle, a person, a fixed object, etc.) is present in the image, the noise level determination unit 162 determines the noise removal level based on noise present at and around the object. At this time, noise removal is performed on the camera 20 side based on the noise amount information, and then the noise removal level is determined as an index indicating the degree of noise removal that is still required.

[0184] Pixel defects in a captured image affect object detection, but the level of the pixel defects that affect the image varies depending on the driving scene of the vehicle 10. In the main controller 30, the scene information acquisition unit 163 acquires driving scene information relating to the driving scene of the vehicle 10 when capturing an image with the camera, and the threshold information setting unit 164 sets threshold information for recognizing pixel defects in the captured image based on the driving scene information.

[0185] The scene information acquisition unit 163 acquires, as driving scene information, at least one of information on the time of day, information on traffic conditions, and information on the driving location while the vehicle is driving. Specifically, the scene information acquisition unit 163 acquires information on whether the vehicle is driving at night, whether the vehicle is driving in an urban area with heavy traffic or pedestrian traffic, or whether the vehicle is driving on a highway.

[0186] The threshold information setting unit 164 sets threshold information for recognizing pixel defects in the captured image based on the driving scene information and transmits the threshold information to the camera controller 22. At this time, the threshold information setting unit 164 sets the threshold for recognizing pixel defects in the captured image based on information about the time of day, traffic conditions, and driving location. For example, when the vehicle is driving at night or in an urban area, the threshold for recognizing pixel defects in the captured image is reduced to actively repair pixel defects. This makes it easier to recognize pixel defects in the captured image. On the other hand, when the vehicle is driving on a highway at night, the threshold for recognizing pixel defects in the captured image is increased to prevent light sources such as taillights and headlights of other vehicles that are very far away from the vehicle from being deemed pixel defects and repaired unnecessarily. This makes it harder to recognize pixel defects in the captured image.

[0187] Instead of the above threshold value, the threshold information setting unit 164 may set, as the threshold information, information indicating whether the captured image is a defect repair scene in which a pixel defect should be repaired. The threshold information setting unit 164 sets defect repair scene information indicating that the captured image is a scene in which a pixel defect should be repaired, for example, when driving at night or in an urban area.

[0188] The noise level transmitting unit 165 transmits the removed noise level determined by the noise level determining unit 162 and the threshold information set by the threshold information setting unit 164 to the camera 20 side.

[0189] Fig. 23 is a flowchart for explaining the procedure for removing noise from a captured image. In Fig. 23, (a) shows the processing of the main controller 30, and (b) shows the processing of the camera controller 22. Each of the processes in Fig. 23(a) and (b) is repeatedly executed at a predetermined interval under a shooting condition in which shooting is performed by the camera 20.

[0190] 23(b), in step S2311, the camera controller 22 acquires an image captured by the image sensor 21, and in the subsequent step S2312, acquires noise amount information indicating the amount of noise contained in the captured image. At this time, the amount (level) of salt-and-pepper noise and spectral noise present in the captured image is detected, and noise amount information indicating the amount of noise is acquired. Alternatively, since the amount of noise is likely to increase as the temperature of the image sensor 21 increases, noise amount information estimated based on the sensor temperature is acquired.

[0191] Thereafter, in step S2313, the noise removal level and threshold information for pixel repair are received from the main controller 30.

[0192] In step S2314, noise removal processing for the captured image is performed based on the noise removal level received from main controller 30 and the salt-and-pepper noise and spectral noise contained in the captured image. In step S2315, pixel defects in the captured image are repaired based on the threshold information received from main controller 30.

[0193] In step S2316, the noise amount information acquired in step S2311 and the captured image after noise removal and pixel defect repair in steps S2314 and S2315 are transmitted to the main controller 30.

[0194] 23A, in step S2301, the main controller 30 receives noise amount information and a captured image after noise removal and pixel defect repair from the camera controller 22. In step S2302, based on the noise amount information and the captured image received in step S2301, the main controller 30 determines a noise removal level for performing noise removal on the captured image. At this time, the noise removal level is determined based on the noise amount information and the presence or absence of noise near the object to be detected in the captured image.

[0195] Then, in step S2303, information on the driving scene of the vehicle 10 at the time of capturing the image is acquired. At this time, the driving scene information acquired includes information on the time of day, traffic conditions, and driving location at the current time.

[0196] In step S2304, threshold information for recognizing pixel defects in the captured image is set based on the driving scene information, and in step S2305, the noise removal level and threshold information are transmitted to the camera 20.

[0197] According to the above configuration in which noise removal processing is performed, the following effects can be obtained.

[0198] The above configuration makes it possible to adjust the noise level of noise occurring in a captured image in response to instructions from the main controller 30, and also reduces the processing load on the main controller 30 by allocating the noise removal function to the camera 20. As a result, it is possible to optimize the control of the camera 20, and ultimately to perform object detection processing appropriately.

[0199] Furthermore, when multiple cameras 20 are mounted on the vehicle 10, it is conceivable that the noise generation conditions will differ for each camera 20, but it is possible to perform appropriate noise reduction in accordance with the state and optical characteristics of each camera 20. In other words, even when multiple different cameras 20 are provided, the main controller 30 does not need to correct the captured images by itself in accordance with the noise information of each camera. Furthermore, when performing noise correction on the camera 20 side, it is possible to control the noise level to be suppressed to a level desired by the main controller 30, regardless of the camera 20.

[0200] The camera controller 22 acquires noise amount information indicating the amount of noise contained in the entire captured image, and the main controller 30 determines the noise removal level based on the noise amount information and the presence or absence of noise near the object to be detected within the captured image received from the camera controller 22. This allows the camera 20 and the main controller 30 to share the role of noise judgment, while performing appropriate noise removal on the captured image.

[0201] The camera controller 22 is configured to acquire information about the amount of noise contained in a captured image at a cycle shorter than the cycle for transmitting the captured image to the main controller 30. In other words, the frequency at which the amount of noise within an image frame is measured is higher than the frequency at which images are transmitted to the main controller 30. This enables dynamic exposure control that follows changes in the external environment of camera photography. As a result, images with stable brightness can be transmitted to the main controller 30.

[0202] The noise removal process is performed by image processing software, and a storage unit 26 storing the image processing software is provided in the image sensor 21. When transmitting an image from the image sensor 21 to the image processing arithmetic chip, the memory capacity (DRAM capacity) of the image processing arithmetic chip and the communication capacity during transmission are factors that limit processing. In this regard, by having the software that performs the noise removal process inside the image sensor, it can be shared with the RAM used when capturing an image, and processing can be performed efficiently.

[0203] The pixel defect level that affects object detection in an image varies depending on the scene. Therefore, if the pixel defect detection level can be dynamically changed, object detection performance can be improved. In this regard, the main controller 30 sets threshold information for recognizing pixel defects in an image based on information about the driving scene of the vehicle 10 when the camera captures the image, and the camera controller 22 repairs pixel defects in the captured image based on the threshold information received from the main controller 30 and transmits the captured image to the main controller 30 after the defect repair. This allows pixel defects to be appropriately corrected.

[0204] (5) Image Readout Processing Considering the processing performance of the main controller 30 and the limitations of communication capacity when transmitting images, it is conceivable that the main controller 30 will read out a captured image at an image size smaller than the image size that the image sensor 21 can output. In this embodiment, the origin position of image readout is changed on the camera 20 side to read out the desired image. Figure 24 is a functional block diagram related to the image readout processing.

[0205] 24 , the main controller 30 includes an information acquisition unit 181 that acquires detection target information, which is information about an object that is present on the elevation angle side (i.e., above the horizontal direction) of the vehicle 10 and that is to be detected. The information acquisition unit 181 acquires, as the detection target information, information such as traffic lights, road signs, and destination guide signs around the vehicle. The detection target information is the presence and position of traffic lights and the like. The detection target information can be acquired from map information. The information acquisition unit 181 also acquires, as the detection target information, that a traffic light exists ahead of the vehicle as a detection target and that the vehicle 10 is waiting for the light.

[0206] On the other hand, the camera controller 22 includes an information receiving unit 191, a vanishing point information acquiring unit 192, a readout range determining unit 193, and an image transmitting unit 194. The information receiving unit 191 receives detection target information from the main controller 30. The camera 20 is, for example, a front camera.

[0207] The vanishing point information acquisition unit 192 acquires vanishing point information from the vanishing point search unit 195 of the image sensor 21. That is, the image sensor 21 is provided with the vanishing point search unit 195 that searches for the position of the vanishing point within the captured image. The vanishing point search unit 195 searches for the vanishing point in a cycle (for example, 10 milliseconds) shorter than the transmission cycle of the captured image from the camera 20 to the main controller 30. Software for searching for the vanishing point in the vanishing point search unit 195 is stored in the memory unit 26 within the image sensor 21 (see FIG. 3). Note that the memory unit 26 in which the vanishing point search software is stored may be disposed in a logic operation circuit stacked on the image sensor 21.

[0208] The readout range determination unit 193 determines the readout range of the captured image based on the vanishing point information. Specifically, the readout range determination unit 193 determines a portion of the entire range of the captured image as the readout range based on the vanishing point position in the image. In this case, for example, it is preferable that the portion of the captured image that is in front of the vanishing point be determined as the readout range.

[0209] In the vehicle 10, the position of the vanishing point in the image changes with a change in attitude. Specifically, when the pitch, which is the tilt of the vehicle 10 in the fore-and-aft direction, changes, the optical axis of the camera 20 changes in the up-and-down direction, and therefore the position of the vanishing point in the image changes upward or downward with this change in the optical axis. When the position of the vanishing point in the image changes with a change in attitude, the readout range determination unit 193 determines the readout range in accordance with the change in attitude.

[0210] It is also desirable to determine the readout range of the image according to the driving scene of the vehicle 10. Therefore, the readout range determination unit 193 determines the readout range of the image using information on detection targets around the vehicle (such as traffic lights and road signs around the vehicle) in addition to vanishing point information. Specifically, when the readout range determination unit 193 receives detection target information such as traffic lights and road signs, it extends the readout range toward the elevation angle so as to include the detection targets. The readout range determination unit 193 extends the readout range toward the elevation angle, for example, by adding a predetermined extension range above the readout range determined based on the vanishing point position in the image. Furthermore, when the readout range determination unit 193 receives information from the main controller 30 that the vehicle is waiting at a traffic light, it may extend the readout range toward the elevation angle.

[0211] A specific example of the readout range of a captured image is shown in Figure 25. In Figure 25(a), a readout range XA is determined in the captured image based on a vanishing point P1 in the captured image. In this case, the portion of the captured image below (toward the viewer) the vanishing point P1 is determined as the readout range XA. This enables proper object detection, including lane markings such as white lines on the road surface, by focusing on the range on the depression angle side in the captured image. Note that the readout range XA may also include a range a predetermined distance above the vanishing point P1 based on the vanishing point P1.

[0212] As shown in Figure 25 (b), when the vehicle 10 pitches (here, dives), the vanishing point P1 moves upward in the captured image, and the readout range XA is determined based on the moved vanishing point P1.

[0213] 25(c), when a traffic light S, which is the detection target, is detected in the captured image, the readout range XA is expanded toward the elevation angle side to include the traffic light S. Furthermore, the readout range XA may be expanded toward the elevation angle side based on information that the vehicle 10 is waiting at a traffic light. This enables proper object detection to be performed in the captured image, including the range toward the elevation angle side of the vanishing point P1.

[0214] The image transmission unit 194 transmits the image of the read range determined by the read range determination unit 193 to the main controller 30. The main controller 30 performs object detection and the like based on the image (image of the read range) received from the camera 20. It is also possible to perform camera calibration processing such as adjusting the position and angle of the camera 20 as aiming using the image received from the camera 20.

[0215] The main controller 30 may acquire parameters relating to the attitude of the vehicle 10 , and the camera controller 22 may determine the image readout range based on the attitude of the vehicle 10 .

[0216] Fig. 26 is a flowchart for explaining the procedure for reading an image. In Fig. 26, (a) shows the processing of the main controller 30, and (b) shows the processing of the camera controller 22. Each of the processing in Fig. 26(a) and (b) is repeatedly executed at a predetermined cycle under the shooting conditions in which the camera 20 is shooting.

[0217] 26(a), in step S2601, main controller 30 acquires detection target information relating to an object that is present on the elevation angle side of vehicle 10 and is to be detected. At this time, information on traffic lights, road signs, destination guide signs, etc. around the vehicle is acquired as the detection target information. Thereafter, in step S2602, the detection target information is transmitted to camera 20.

[0218] On the other hand, as shown in FIG. 26( b ), the camera controller 22 receives detection object information from the main controller 30 in step S 2611 .

[0219] In step S2612, vanishing point information is obtained from the vanishing point search unit 195 of the image sensor 21. In step S2613, the readout range of the captured image is determined based on the detection object information and the vanishing point information. At this time, the image readout range determined based on the vanishing point information is appropriately expanded toward the elevation angle side based on the detection object information. When the vehicle 10 is waiting at a traffic light, it is preferable that the readout range be expanded toward the elevation angle side based on information that the vehicle is waiting at a traffic light.

[0220] Thereafter, in step S2614, the image of the read range read in step S2613 is transmitted to the main controller 30.

[0221] In step S2604, the main controller 30 receives the image (image of the read range) acquired from the camera 20 side.

[0222] According to the above-described configuration in which the image readout process is executed, the following effects can be obtained.

[0223] The camera controller 22 determines the readout range of the captured image based on the position of the vanishing point (vanishing point information) in the captured image and transmits the image of that readout range to the main controller 30. Here, if the camera 20 determines the position of the vanishing point and reads the image according to that vanishing point position, the main controller 30 does not need to specify the readout position based on the attitude of the camera 20, etc. This eliminates the need for the main controller 30 to have software for such processing. Furthermore, the camera 20 outputs the maximum image size that the image sensor 21 can output and executes a series of processes from vanishing point search to image readout, so that even if a temporary image shift occurs, the image can be read out properly without waiting for the main controller 30 to detect the vanishing point. As a result, the control of the camera 20 can be optimized, and ultimately the object detection process can be performed properly.

[0224] The vanishing point search section 195 on the camera 20 side is configured to search for the vanishing point at a cycle shorter than the cycle for transmitting captured images to the main controller 30. In other words, the frequency with which the vanishing point position is searched is higher than the frequency with which images are transmitted to the main controller 30. This makes it possible to dynamically process images (image cropping, image size conversion) while following changes in the external environment of the camera capture. As a result, it is possible to transmit images to the main controller 30 with the influence of the external environment reduced.

[0225] The image sensor 21 of the camera 20 is configured to have software for the vanishing point search unit 195. This makes it possible to process the vanishing point search without placing restrictions on the frequency of communication between the image sensor 21 and the image processing calculation chip in the camera controller 22.

[0226] The position of the captured image relative to the vanishing point in the image that should be cropped varies depending on the driving scene of the vehicle 10. In this regard, the camera controller 22 receives detection target information, which is information about objects that are present on the elevation angle side of the vehicle 10 and that are to be detected, from the main controller 30, and determines the readout position of the captured image based on the detection target information and the vanishing point information, so that the captured image can be read out appropriately.

[0227] The camera controller 22 receives information from the main controller 30 that the vehicle 10 is waiting at a traffic light, and expands the readout range toward the elevation angle based on the waiting at a traffic light information. This allows the vehicle 10 to appropriately control vehicle start and other operations based on the signal display of the traffic light.

[0228] Next, a series of camera control processes including the above-mentioned (1) exposure condition setting process, (2) white balance adjustment process, (3) image masking process, (4) noise removal process, and (5) image readout process will be described. Figure 27 is a sequence diagram showing the flow of a series of camera control processes by the main controller 30 and the camera controller 22 of each camera 20. In Figure 27, steps S2701 to S2703 on the main controller 30 side and steps S2711 to S2719 on the camera controller 22 side are performed as a series of processes in synchronization with image transmission.

[0229] 27 , in step S2701, the main controller 30 transmits an image transmission command and various parameters to each camera 20. At this time, the main controller 30 transmits to the camera 20 the following to the camera 20: traveling state parameters including at least one of the traveling speed of the vehicle 10 and attitude information of the vehicle 10, surrounding environment parameters indicating the surrounding environment of the vehicle 10, adjustment parameters which are information regarding at least one of the ambient color around the vehicle and the incident light incident on the camera 20, information regarding the vehicle 10 or the surroundings of the vehicle, noise level to be removed in the captured image, and detection target information.

[0230] Then, in step S2702, the main controller 30 executes a parameter acquisition process to acquire various parameters used for camera control, such as driving state parameters, surrounding environment parameters, adjustment parameters, vehicle surroundings information, and noise removal level.

[0231] In step S2703, the main controller 30 detects an object around the vehicle based on the captured image received from the camera 20 in the previous cycle. Based on the object detection result, the main controller 30 executes driving assistance control for the vehicle 10. In step S2703, the main controller 30 receives the captured image and noise amount information from the camera 20.

[0232] Meanwhile, in step S2711, the camera controller 22 receives the image transmission command and various parameters transmitted from the main controller 30. Then, in step S2712, the camera controller 22 updates the exposure conditions of the image sensor 21 based on the parameters received from the main controller 30. At this time, the exposure time (shutter speed) and gain are updated as the exposure conditions of the image sensor 21.

[0233] In step S2713, the camera controller 22 causes the image sensor 21 to capture an image under the updated exposure conditions, and acquires the captured image captured by the image sensor 21.

[0234] In step S2714, the camera controller 22 determines the read position of the captured image based on vanishing point information indicating the position of the vanishing point within the image.

[0235] In step S2715, the camera controller 22 performs white balance adjustment on the captured image determined as the readout range in step S2714.

[0236] In step S2716, the camera controller 22 executes mask processing, which is a smoothing process or a filling process, in the mask region RM.

[0237] In step S2717, the camera controller 22 acquires noise amount information from the captured image.

[0238] In step S2718, the camera controller 22 performs noise removal processing on the captured image based on the noise removal level received from the main controller 30 and the noise amount information acquired from the image.

[0239] In step S2719, the camera controller 22 transmits the image and the noise amount information to the main controller 30.

[0240] Thereafter, the main controller 30 and the camera controller 22 repeatedly execute the above-described processing at the image transmission cycle.

[0241] <Image Storage Processing When an Abnormality Occurs> In the vehicle 10, for example, an abnormal situation may occur while traveling, such as contact with another vehicle, a person, or an object such as a ground structure. When an abnormal situation occurs, the main controller 30 detects the occurrence of the abnormal situation and is required to execute a process to deal with the abnormal situation. This temporarily increases the processing load on the main controller 30, raising concerns that the processing load may increase. In this case, in order to keep a record of the occurrence of the abnormal situation, it is desirable to store images captured at the time of the abnormal situation and immediately thereafter. However, there is a concern that this may restrict the image storage processing in the main controller 30. On the other hand, if priority is given to image storage in the main controller 30, there is a concern that this may restrict the processing of dealing with the abnormality in the main controller 30.

[0242] Therefore, in this embodiment, in a situation where the processing load of the main controller 30 increases due to the occurrence of an abnormal situation, the camera 20 is responsible for storing the captured images, and the captured images to be used for accident analysis, etc. Fig. 28 is a functional block diagram relating to the configuration for performing image storage processing when an abnormality occurs.

[0243] 28 , the main controller 30 includes an abnormality detection unit 201 and a command transmission unit 202. The abnormality detection unit 201 detects the occurrence of an abnormal situation in the vehicle 10 or in the vicinity of the vehicle. For example, the abnormality detection unit 201 detects that the vehicle 10 has come into contact with another object such as another vehicle, a person, or an obstacle, or that the vehicle 10 is about to come into contact with another object. The contact or near-contact with the other object may be detected based on a captured image, or may be detected from the detection results of a distance measurement sensor (laser sensor), a collision detection sensor, an acceleration sensor, or the like provided in the vehicle 10.

[0244] When the abnormality detection unit 201 detects that an abnormal situation has occurred, the command transmission unit 202 transmits an image writing command to the camera 20 side.

[0245] On the other hand, camera 20 includes image sensor 21, image storage unit 23 capable of storing images captured by image sensor 21, and write processing unit 211 that writes the captured images into image storage unit 23. Image storage unit 23 may be disposed within image sensor 21 or within a logical operation circuit stacked on image sensor 21.

[0246] When an image write command is received from the main controller 30, the write processing unit 211 writes the captured image to the image storage unit 23. When the camera 20 receives information that an abnormality has occurred in the vehicle 10, the image captured by the camera 20 at the time the abnormality occurred is written to the image storage unit 23.

[0247] In a normal vehicle driving scene, the camera 20 does not write the captured image to the image storage unit 23. In contrast, when an abnormal situation occurs in the vehicle 10 or in the vicinity of the vehicle, the camera 20 exceptionally writes the captured image to the image storage unit 23 based on a command from the main controller 30.

[0248] Upon receiving the image write command, the write processing unit 211 continuously writes the captured images at predetermined time intervals to the image storage unit 23. The interval at which the write processing unit 211 writes the captured images to the image storage unit 23 may be synchronized with the interval at which the captured images are transmitted to the main controller 30.

[0249] The write processing unit 211 may write the captured images to the image storage unit 23 at a cycle (for example, 10 milliseconds) shorter than the cycle at which the captured images are transmitted from the camera 20 to the main controller 30 .

[0250] The write processing unit 211 may be realized by the camera controller 22, or may be realized by a processing chip for write processing that enables image writing. In the write processing unit 211, software for performing the write processing is stored in the memory unit 26 in the image sensor 21 (see FIG. 3). Note that the memory unit 26 may be disposed in a logical operation circuit stacked on the image sensor 21.

[0251] The write processing unit 211 may write the captured image to the image storage unit 23 for a predetermined period of time that includes the timing of the occurrence of the abnormal situation and the subsequent response processing. The image writing period may include a period during which a collision avoidance operation is performed in the vehicle 10.

[0252] After transmitting the image write command, when a period of time has elapsed during which it is determined that the image needs to be saved on the camera 20 side, the main controller 30 may transmit a write end signal to the camera 20 side. In this case, the camera 20 side continuously writes image data to the image storage unit 23 during the period from when it receives the image write command to when it receives the write end signal.

[0253] 28 , the main controller 30 may have a camera identification unit 203. The camera identification unit 203 identifies which of the multiple cameras 20 in the vehicle 10 captured an image of contact with another object, etc. In this case, the camera identification unit 203 may identify the camera 20 that captured an image of contact with another object, etc., using a captured image, an on-board sensor, etc.

[0254] The camera 20 receives the camera identification result from the camera identification unit 203. Then, the identified camera 20 writes (saves) the captured image to the image storage unit 23 in response to an image write command from the main controller 30. However, the camera 20 may also store captured images from a camera other than the camera 20 identified as having captured an image of contact with another object in the vehicle 10. In this case, it is preferable that the number of images stored per hour for the camera other than the camera 20 identified as having captured an image of contact is smaller than that for the camera 20 identified as having captured an image of contact.

[0255] The configuration described in FIG. 28 can be realized in combination with any of the configurations described so far.

[0256] Figure 29 is a flowchart for explaining the processing procedures of the controllers 30, 22 when the vehicle 10 is in a power-on state (when the object detection system is in operation). In Figure 29, (a) shows the processing of the main controller 30, and (b) shows the processing of the camera controller 22. The processing in Figure 29(b) is executed by each of the on-board cameras 20. Each of the processing in Figures 29(a) and 29(b) is repeatedly executed at a predetermined cycle.

[0257] 29(a), the main controller 30 detects the occurrence of an abnormal situation in the vehicle 10 or in the vicinity of the vehicle in step S2901. At this time, for example, it is detected that the vehicle 10 has come into contact with another object such as another vehicle, a person, or an obstacle, or that the vehicle 10 is about to come into contact with another object. In step S2902, it is determined whether or not the occurrence of an abnormal situation was detected in step S2901. If the occurrence of an abnormal situation is detected, the process proceeds to step S2903.

[0258] In step S2903, among all the on-board cameras 20, a camera that is highly likely to have captured an image of the abnormal situation is identified. At this time, the location where the abnormal situation occurred in the vehicle 10 is identified using the captured image, on-board sensors, etc., and the camera 20 that is closest to the location where the abnormal situation occurred is identified as the camera that captured an image of the abnormal situation. As a result, among all the on-board cameras, the camera 20 that captured an image of the vehicle 10 contacting another object, etc. is identified. Note that if it is determined that an abnormal situation has occurred but the location where the abnormal situation occurred in the vehicle 10 is unknown, it is also possible to designate all the cameras 20 as cameras that captured an abnormal situation.

[0259] In step S2904, an image write command is sent to the camera that captured the abnormal image and the other cameras 20. At this time, camera identification information indicating whether the camera captured the abnormal image or not may also be sent to each camera 20. When the image write command is sent to the camera 20, the camera 20 executes the image write process.

[0260] After the image writing process on the camera 20 side has started, in step S2905 it is determined whether or not to end the image writing on the camera 20 side. For example, after an image writing command is sent to the camera 20 side, when the period required for image storage on the camera 20 side has elapsed, it is determined that the image writing on the camera 20 side should be ended, and the process proceeds to step S2906. In step S2906, a command signal to end writing is sent to the camera 20 side. The period required for image storage may be, for example, several tens of seconds to several minutes.

[0261] On the other hand, as shown in FIG. 29(b), in step S2911, the camera controller 22 determines whether or not an image write command has been received from the main controller 30, and if an image write command has been received, proceeds to the subsequent step S2912.

[0262] In step S2912, it is determined whether the camera itself is an abnormal camera. The determination in step S2912 may be made based on the camera identification information transmitted from the main controller 30.

[0263] If the camera itself is an abnormally photographed camera, the process proceeds to step S2913, where the photographed image is stored in the image storage unit 23. If the camera itself is not an abnormally photographed camera, the process proceeds to step S2914, where the photographed image is stored in the image storage unit 23. Steps S2913 and S2914 both correspond to image writing processing, but when comparing steps S2913 and S2914, it is preferable that the image writing cycle (number of images saved) is different. In this case, if the camera itself is an abnormally photographed camera, it is preferable to shorten the image writing cycle (increase the number of images saved) compared to cameras that are not abnormally photographed cameras.

[0264] Specifically, in a camera that has taken abnormal photographs, it is preferable that the photographed images be written to the image storage unit 23 at a cycle shorter than the image transmission cycle from the camera 20, for example, at 10 millisecond cycles. In addition, in a camera that has not taken abnormal photographs, it is preferable that the photographed images be written to the image storage unit 23 at a cycle longer than that of a camera that has taken abnormal photographs, for example, at the same cycle as the image transmission cycle from the camera 20 to the main controller 30 (for example, 25 millisecond cycles) or at a cycle shorter than the image transmission cycle.

[0265] However, it is also possible to configure the image storage unit 23 to store images captured by cameras that have taken abnormal photographs, regardless of whether the camera has taken such photographs or not.It is also possible to configure the image storage unit 23 to store images captured by cameras that have taken abnormal photographs only.

[0266] In steps S2913 and S2914, the images captured during the period including the timing of the abnormality occurrence are stored in the image storage unit 23 on the camera 20 side.

[0267] In step S2915, the image captured by camera 20 is transmitted to main controller 30. The image transmission in step S2915 is preferably executed at a longer cycle than the image writing in steps S2913 and S2914. This prevents an increase in the processing load on main controller 30.

[0268] Thereafter, in step S2916, it is determined whether or not a command signal to end writing has been received from main controller 30. Furthermore, in step S2917, it is determined whether or not a predetermined time has elapsed since the image writing command was received from main controller 30. Then, if either step S2916 or S2917 is positive, the process proceeds to step S2918, where image saving on the camera 20 side is terminated.

[0269] It is conceivable that an abnormality may occur in the driver of the vehicle 10 while the vehicle is traveling. In this case, the main controller 30 will need to deal with the abnormality upon detecting the occurrence of the driver abnormality, which will temporarily increase the processing load of the main controller 30. Furthermore, since there is a risk that the vehicle 10 may come into contact with another vehicle, it is desirable to monitor the condition of the vehicle 10 and appropriately store captured images.

[0270] Therefore, when the occurrence of a driver abnormality is detected, similarly to the above, the main controller 30 may send an image write command to the camera 20 side, and the camera 20 side may write the captured image to the image storage unit 23 based on the image write command from the main controller 30. Specifically, the main controller 30 may monitor the driver's condition based on, for example, an image captured by an in-vehicle camera that captures an image of the driver inside the vehicle, and may transmit an image write command to the camera 20 side if an abnormality occurs in the driver when the driver's level of alertness drops or when the driver loses consciousness.

[0271] FIG. 30 is a flowchart showing the procedure of processing by the main controller 30, and this processing is a partial modification of FIG. 29(a).

[0272] In FIG. 30 , in step S3001, the occurrence of an abnormal situation in the vehicle 10 or in the vicinity of the vehicle is detected. At this time, the main controller 30 detects that the vehicle 10 has come into contact with or is about to come into contact with another object, and also detects that an abnormal situation has occurred in the driver of the vehicle 10. Specifically, the main controller 30 acquires images captured by a monitoring camera inside the vehicle. Then, the captured images are used to analyze the driver's face, driver posture, etc., and determine whether the driver's alertness level has decreased or the driver has lost consciousness. It is assumed that an abnormal situation has occurred in the driver when the driver's alertness level has decreased or the driver has lost consciousness. Alternatively, the occurrence of an abnormal situation in the driver may be detected based on information from a wearable device worn by the driver.

[0273] Thereafter, if it is determined in step S3002 that an abnormal situation has occurred, the process proceeds to step S3003. In step S3003, it is determined whether the abnormal situation is an abnormal situation that has occurred to the driver. If the driver is not abnormal, the process proceeds to step S3004. In this case, in steps S3004 and S3005, similar to steps S2903 and S2904 in FIG. 29A, the camera that captured the abnormal image is identified, and an image write command is sent to each camera 20. As a result, the image write process is executed on the camera 20 side.

[0274] If the driver is in an abnormal state, the process proceeds to step S3006. In step S3006, images captured by all cameras 20 are stored in image storage unit 23. At this time, the image storage period in camera 20 may be shorter than the image transmission period from camera 20 to main controller 30, for example, 10 milliseconds.

[0275] Thereafter, in step S3007, it is determined whether or not to end image writing on the camera 20 side. If the result of step S3007 is affirmative, the process proceeds to step S3008, where a command signal to end writing is transmitted to the camera 20 side (similar to steps S3005 and S3006 in FIG. 29A).

[0276] According to the above-described configuration in which image saving processing is executed when an abnormality occurs, the following effects can be obtained.

[0277] When the main controller 30 detects the occurrence of an abnormal situation in or around the vehicle 10, the camera 20 writes a captured image to the image storage unit 23 based on an image write command from the main controller 30. This causes the camera 20 to write the captured image at the timing of the abnormality occurrence to the image storage unit 23. This allows the camera 20 to properly store the captured image showing the abnormality situation, etc., even when the processing load on the main controller 30 is heavy due to abnormality response processing when an abnormality occurs. As a result, the control of the camera 20 is optimized, and the object detection process can be properly performed.

[0278] The camera 20 is configured to write captured images continuously to the image storage unit 23 at a predetermined time interval upon receiving an image write command from the main controller 30. In this case, the camera 20 stores multiple image frames as a video. Therefore, in the event of a vehicle accident or the like, a series of image data from the dangerous scene to the evasive action or the occurrence of the accident can be stored. This makes it possible to use the data stored in the camera 20 for post-accident analysis.

[0279] The write processing unit 211 on the camera 20 side is configured to write captured images to the image storage unit 23 at a cycle shorter than the cycle at which captured images are transmitted from the camera 20 to the main controller 30. In other words, the image frame rate at which images are saved is configured to be higher than the frequency at which images are transmitted to the main controller 30. In this case, by synchronizing the image saving rate with the camera 20's capture frame rate and setting the capture frame rate higher than the transmission frame rate to the main controller 30, images can be saved at a higher frame rate, and useful data can be obtained for later analysis of danger or accident scenes.

[0280] When analyzing scenes where abnormal events such as accidents occur, the more images captured by the camera 20, the more appropriate the post-mortem analysis becomes. However, there is a concern that storing a large number of captured images in the main controller 30 immediately after an abnormality occurs may cause an overload. In this regard, in the above configuration, the process of storing a large number of captured images immediately after an abnormality occurs is shared by the camera 20. This makes it possible to achieve an appropriate post-mortem analysis.

[0281] When an abnormal situation is detected, the camera that is most likely to have captured the image of the abnormal situation is identified from among the multiple cameras 20, and the camera that captured the image writes the captured image to the image storage unit 23. This allows each camera 20 to store appropriate images without unnecessarily increasing the number of images stored.

[0282] Furthermore, when an abnormal situation is detected, the cameras 20 other than the camera that captured the abnormality also write captured images to the image storage unit 23. As a result, images are saved in the camera that captured the abnormality and the other cameras 20 in a redundant manner on the main controller 30 side. In this case, even during an emergency response to an abnormality, captured images showing the abnormality occurrence situation can be reliably saved.

[0283] When an abnormality is detected in the driver of the vehicle 10, all of the cameras 20 are configured to write captured images to the image storage unit 23. When an abnormality occurs in the driver, the behavior of the vehicle 10 becomes unpredictable, but even under such circumstances, appropriate image storage can be performed.

[0284] In the camera 20, a storage unit 26 storing writing software is provided in the image sensor 21. When transmitting an image from the image sensor 21 to the image processing arithmetic chip, the memory capacity (DRAM capacity) of the image processing arithmetic chip and the communication capacity during transmission are factors that limit processing. In this regard, by having the software that executes the writing process inside the image sensor, it can be shared with the RAM used when capturing an image, and processing can be executed efficiently.

[0285] When transmitting an image from the image sensor 21 to the image storage area on the camera 20 side, the communication capacity during transmission is also a factor in processing constraints. By providing the image storage unit 33 on the image sensor itself, images can be stored without being restricted by the transmission capacity on the board.

[0286] <Camera Software Update> In the object detection system of this embodiment, the camera 20 is configured to have application software such as image processing software. In this case, when rewriting (updating) the application software, it is necessary to access the software storage area of ​​the camera 20 from outside the camera and perform editing. If this is performed for each camera 20, it may be necessary to remove the camera 20 from the vehicle 10 to perform the update, and then remount the camera 20 on the vehicle 10 after the update and calibrate the camera 20. This also imposes a burden on the owner or user of the vehicle 10.

[0287] Therefore, in this embodiment, in order to eliminate the above-mentioned burden, a means is provided that enables the main controller 30 to access the software storage area on the camera 20 side, and software rewriting on the camera 20 is performed by the main controller 30 or the camera controller 22. Fig. 31 is a functional block diagram relating to the configuration for rewriting application software on the camera 20 side.

[0288] In FIG. 31 , the camera 20 includes a storage unit 26 that stores application software executable by the camera controller 22. The storage unit 26 corresponds to an application storage area that stores the application software. As described in FIG. 3 , the camera controller 22 is integrally provided with the image sensor 21. The storage unit 26 in the image sensor 21 stores image processing software as application software. However, the application software stored in the storage unit 26 may be evaluation software that evaluates the state of the camera itself. The storage unit 26 is located in the image sensor 21 or in a logic operation circuit stacked on the image sensor 21. However, the camera 20 may have the configuration shown in FIG. 2. In the configuration of FIG. 2 , the storage unit 28 in the camera controller 22 corresponds to the application storage area, and the application software is stored in the storage unit 28.

[0289] The application software stored in the storage unit 26 (application storage area) can be rewritten (updated) by application software transmitted from the main controller 30 .

[0290] The camera 20 includes a rewrite execution unit 231 that rewrites the application software in the storage unit 26. The rewrite execution unit 231 receives encrypted application software from a software transmission unit 223 of the main controller 30 (described later), and decrypts the application software before writing it to the storage unit 26. In this case, the main controller 30 cannot directly access the storage unit 26, and the application software is written to the storage unit 26 via the camera controller 22.

[0291] The camera controller 22 transitions to software rewrite mode when it receives a command to rewrite the application software from the main controller 30. At this time, the camera controller 22 verifies whether the rewrite command from the main controller 30 is legitimate. If the camera controller 22 receives the legitimate rewrite command from the main controller 30, it returns a response signal to the main controller 30 indicating that it has transitioned to software rewrite mode.

[0292] If the writing of application software to the storage unit 26 is interrupted, it is possible to perform a rollback to return the storage unit 26 to the state before the writing started. It is also preferable that the software storage area of ​​the storage unit 26 has redundancy with two or more sides.

[0293] Meanwhile, the main controller 30 includes components related to rewriting application software, such as a software receiving unit 221, an authentication unit 222, a software transmitting unit 223, and a rewriting completion confirmation unit 224. The software receiving unit 221 receives application software to be rewritten from an external device X, such as a data server. The application software may be transmitted from the external device X to the main controller 30, for example, over the air (OTA).

[0294] The authentication unit 222 authenticates that the application software received from the external device X is legitimate. The authentication unit 222 authenticates whether the external device X that transmitted the application software is a legitimate source of the application software, whether the application software received from the external device X is legitimate application software, etc.

[0295] If the authentication unit 222 authenticates that the application software used in this software rewriting is genuine, the software transmission unit 223 transmits the application software to the camera 20. The software transmission unit 223 encrypts the application software and transmits it to the camera 20.

[0296] After the application software is transmitted from the main controller 30, the rewrite execution unit 231 on the camera 20 side decrypts the application software received from the main controller 30 side and then writes it to the storage unit 26.

[0297] Here, the camera controller 22 (rewrite execution unit 231) receives a software write command from the main controller 30 and transitions to software write mode based on the software write command. At this time, the camera controller 22 confirms that the current write is a regular procedure based on the software write command from the main controller 30 and notifies the main controller 30 that regular writing will be performed. The main controller 30 transmits the application software to the camera 20, conditional on receiving a notification from the camera controller 22.

[0298] In the main controller 30, the rewrite completion confirmation unit 224 confirms whether the rewriting was performed properly after the application software has been rewritten on the camera 20 side. For example, the rewrite completion confirmation unit 224 confirms whether the rewriting was performed properly by comparing the application software transmitted from the main controller 30 side with the application software stored in the memory unit 26 (application memory area) of the camera 20.

[0299] Figure 32 is a flowchart for explaining the procedure for rewriting application software for the camera 20. In Figure 32, (a) shows the processing of the main controller 30, and (b) shows the processing of the camera controller 22. The processing of Figure 32(b) is executed by each of the on-vehicle cameras 20. Each of the processing of Figures 32(a) and 32(b) is repeatedly executed at a predetermined cycle.

[0300] As shown in Fig. 32(a), in step S3201, the main controller 30 receives application software to be rewritten from external device X. In step S3202, the application software received from external device X is authenticated to determine whether it is genuine. In step S3203, as a result of the authentication, it is determined whether the application software is genuine. If the application software is genuine, the process proceeds to step S3204. In step S3204, an application software rewrite command is sent to the camera 20.

[0301] 32(b), in step S3211, the camera controller 22 determines whether or not a command to rewrite the application software has been received from the main controller 30. If a command to rewrite the application software has been received, the process proceeds to step S3212, where the process enters software rewrite mode.

[0302] In step S3213, it is determined whether the rewrite command from the main controller 30 is valid and follows a normal procedure. If the rewrite command is valid, the process proceeds to step S3214. If the rewrite command from the main controller 30 is not valid and the process is not determined to follow a normal procedure, the process ends. In step S3214, a response signal indicating that the system has entered software rewrite mode is sent to the main controller 30.

[0303] 32A, the main controller 30 determines whether a response signal has been received from the camera 20, and if a response signal has been received, the process proceeds to step S3206. In step S3206, the main controller 30 transmits the application software to the camera 20. At this time, the application software is encrypted before being transmitted to the camera 20.

[0304] 32(b), the camera controller 22 determines whether or not application software has been received from the main controller 30, and if application software has been received, proceeds to step S3216. In step S3216, the application software received from the main controller 30 is decrypted, and then written to the storage unit 26. In the following step S3217, the application software written to the storage unit 26 is transmitted to the main controller 30.

[0305] 32(a), the main controller 30 compares the application software that it itself (main controller 30) previously transmitted with the application software already written to the memory unit 26 on the camera 20 side to confirm whether the writing was performed correctly. This confirmation completes the series of application software rewriting operations. If the software transmitted from the main controller 30 does not match the software written on the camera 20 side, it is preferable to determine that the application software rewriting has failed, disable the application software on the camera 20 side, and notify an error message.

[0306] In addition, in FIG. 31, the main controller 30 preferably includes a camera parameter receiving unit 225 and a parameter returning unit 226 .

[0307] When application software is to be rewritten, the camera parameter receiving unit 225 receives current camera parameters from the camera 20 whose application software is to be rewritten before the rewriting, and stores them in a predetermined storage area.

[0308] The camera parameters are parameters that indicate the mounting state of each camera 20 on the vehicle 10, such as the mounting position of the camera 20 on the vehicle 10 and the direction of the optical axis of the camera 20. More specifically, the camera parameters are the position (X coordinate, Y coordinate, Z coordinate) and angle (roll, pitch, yaw) of the camera 20, and distortion of the camera lens. The camera parameters are typically estimated by capturing images of a calibration pattern (marker) placed in front of the camera 20 or capturing images of the scenery for a certain period of time while the vehicle is traveling. The camera parameters are used, for example, when an overhead image is created using images captured by each camera 20 of the vehicle 10, when object detection is performed around the vehicle, or when driving assistance control is performed based on the object detection results.

[0309] The parameter return unit 226 temporarily receives camera parameters when the application software is rewritten, and returns them to the original camera 20 after the application software is rewritten. In this case, the camera parameters used before the application software is rewritten are maintained and available in a valid state before and after the application software is rewritten. As a result, even after the application storage area of ​​the camera 20 is cleared once due to the software rewriting, the camera parameters are properly reflected immediately after the software rewriting, and object detection and other operations are correctly performed.

[0310] FIG. 33 is a flowchart showing the processing procedure of the main controller 30 when the application software of the camera 20 is rewritten.

[0311] 33 , in step S3301, the main controller 30 determines whether it is time to rewrite the application software, and if so, proceeds to step S3302. In step S3302, before executing the software rewrite, the main controller 30 acquires current camera parameters for the cameras 20 whose application software is to be rewritten and stores them in a predetermined storage area. For each camera 20, the main controller 30 acquires parameters indicating the camera mounting position on the vehicle 10, the direction of the optical axis of the camera 20, and the like, as camera parameters. At this time, by storing (temporarily saving) the camera parameters in a storage area on the main controller 30 side, erasure of the camera parameters along with the old application software before rewriting is prevented when the application software is rewritten.

[0312] Thereafter, in step S3303, it is determined whether or not the rewriting of the application software has been completed. The rewriting of the application software conforms to the processes shown in Figures 32(a) and 32(b) above. If the rewriting of the application software has been completed, the process proceeds to step S3304.

[0313] In step S3304, the camera parameters acquired in step S3302 are returned to the original camera 20. This makes it possible to use the camera parameters after the application software is rewritten. That is, after the application software is rewritten, it becomes possible to create an overhead image, detect objects around the vehicle, and control driving assistance based on the object detection results, based on the camera parameters before the rewriting.

[0314] After the application software is rewritten, the camera parameters may be reacquired.

[0315] Furthermore, depending on fraudulent methods such as spoofing, it may be impossible to detect tampering with the application software during authentication processing when the software is received. In light of this, it may be possible to determine that no abnormalities have occurred in the writing of the application software, i.e., that the application software has been rewritten correctly, by comparing images taken immediately before and after the software rewrite.

[0316] 31, the main controller 30 includes an application abnormality determination unit 227. The application abnormality determination unit 227 acquires images of the camera 20 whose application software is to be rewritten immediately before and immediately after the software rewrite, and compares the images taken before and immediately after the software rewrite to determine whether an abnormality has occurred due to the rewriting of the application software. The images taken immediately before and immediately after the software rewrite are multiple images taken from the same position and the same attitude while the vehicle is stopped, and by comparing these images with each other, it is possible to confirm whether the application software was rewritten appropriately.

[0317] For example, the application abnormality determination unit 227 may calculate the amount of change in luminance in the captured image before and after software rewriting, and based on the amount of change in luminance, determine whether the application software was rewritten correctly, i.e., whether any abnormalities occurred during software rewriting. At this time, the amount of change in luminance in the captured image before and after software rewriting is calculated, and determine whether the amount of change in luminance is less than a predetermined threshold. If the amount of change in luminance is less than the threshold, the rewriting is determined to be normal, and if the amount of change in luminance is equal to or greater than the threshold, the rewriting is determined to be abnormal. The threshold used to determine the amount of change in luminance may be determined in advance based on an assumed allowable amount of change in luminance before and after software rewriting.

[0318] The application abnormality determination unit 227 can detect when application software has been rewritten with software that makes unauthorized changes to images. For example, it can detect software tampering such as embedding ghosts in images that may be mistaken for people. Furthermore, the application abnormality determination unit 227 can compare images before and after the software rewrite and determine whether an abnormality has occurred due to the rewrite of the application software by comparing the objects captured in the captured images.

[0319] FIG. 34 is a flowchart showing the processing procedure of the main controller 30 when the application software of the camera 20 is rewritten.

[0320] 34, in step S3401, main controller 30 determines whether it is time to rewrite application software, and if so, proceeds to step S3402. In step S3402, for the camera 20 whose application software is to be rewritten, an image captured immediately before the software is rewritten is acquired.

[0321] Thereafter, in step S3403, it is determined whether or not the rewriting of the application software has been completed. The rewriting of the application software conforms to the processes shown in Figures 32(a) and 32(b) above. If the rewriting of the application software has been completed, the process proceeds to step S3404.

[0322] In step S3404, images captured after software rewriting are acquired for the camera 20 whose application software is to be rewritten. Then, in step S3405, images captured immediately before and after the software rewriting are compared with each other, and in the subsequent step S3406, it is determined whether the images before and after the rewriting match.

[0323] At this time, the main controller 30 acquires multiple images taken at the same position and in the same posture while the vehicle is stopped as images taken before and after the software rewrite, and calculates the difference in brightness in the captured images before and after the software rewrite as the brightness change amount. For example, the image may be divided into multiple sections, and the brightness change amount may be calculated for each section. Then, it is determined whether the brightness change amount is less than a predetermined threshold.

[0324] If the captured images before and after the software rewrite match, i.e., if the amount of change in luminance is less than the threshold, the result in step S3406 is affirmative and the process proceeds to step S3407. If the captured images before and after the software rewrite do not match, i.e., if the amount of change in luminance is equal to or greater than the threshold, the result in step S3406 is negative and the process proceeds to step S3408.

[0325] In step S3407, it is determined that the application software was rewritten correctly without any abnormalities. In step S3408, it is determined that an abnormality occurred in the application software rewriting. If an abnormality occurred in the rewriting, it is preferable to disable the application software on the camera 20 side and notify an error message.

[0326] The main controller 30 may also include a tampering determination unit 228 that performs a tampering determination at a timing other than when the software is rewritten. The tampering determination unit 228 determines whether or not the application software stored in the storage unit 26 has been tampered with. The tampering determination unit 228 may perform the tampering determination, for example, when the camera 20 is started up. When the camera 20 is started up, the camera may be permitted to start up on the condition that the tampering determination unit 228 determines that the application software has not been tampered with. If the tampering determination unit 228 determines that the application software has been tampered with, the tampering determination unit 228 may store the tampering information as fail information or notify the vehicle user.

[0327] The tampering determination unit 228 may read the characteristics of the application software stored in the memory unit 26, such as the creator, creation management number, and creation date, and perform the tampering determination based on these characteristics.

[0328] It is also possible to provide the function of a tampering determination unit on the camera 20 side. In this case, tampering determination may be performed by comparing application software redundantly stored in the storage unit 26, for example.

[0329] 35 is a flowchart showing the procedure for determining whether application software has been tampered with. This process is executed by the main controller 30 at predetermined intervals.

[0330] 35, in step S3501, main controller 30 determines whether camera 20 is starting up, and if so, proceeds to step S3502. In step S3502, main controller 30 executes a tampering determination process for application software stored in storage unit 26 of camera 20. At this time, main controller 30 determines whether the application software has been tampered with based on the identity of the application software.

[0331] If it is not determined in the determination process of step S3502 that tampering has occurred, the result of step S3503 is negative and the process proceeds to step S3504. In step S3504, activation of the camera 20 is permitted. On the other hand, if it is determined in the determination process of step S3502 that tampering has occurred, the result of step S3503 is positive and the process proceeds to step S3505. In step S3505, activation of the camera 20 is not permitted. In this case, the main controller 30 stores the tampering information in a storage area and notifies the vehicle user.

[0332] Furthermore, for example, when application software is rewritten for verification or repair of the object detection system at a vehicle dealer or repair shop, the application authentication function by authentication unit 222 may interfere with the work. In view of this, main controller 30 may be configured to include authentication invalidation unit 229A and rewrite permission unit 229B.

[0333] The authentication invalidation unit 229A invalidates the authentication performed by the authentication unit 222 based on a predetermined invalidation command. The invalidation command may be input from a management device, for example, during system verification or repair.

[0334] In a state where the authentication by the authentication invalidation unit 229A is invalidated, i.e., in a state where the authentication function for authenticating whether or not the application software is to be rewritten is invalidated, the rewrite permission unit 229B permits the rewriting of the application software and the taking of photographs by the camera 20. In this case, it is preferable that, for example, application software for verification is temporarily written to the memory unit 26 of the camera 20.

[0335] FIG. 36 is a flowchart showing the processing procedure of the main controller 30 when the application software of the camera 20 is rewritten.

[0336] As shown in FIG. 36 , in step S3601, the main controller 30 determines whether or not the current work is authorized at a vehicle dealer or repair shop. Authorized work refers to authorized system verification or repair work performed at a vehicle dealer or repair shop. If step S3601 is positive, the process proceeds to step S3602. For example, during authorized work, work authentication information (such as an ID) is transmitted from an authenticated external management device to the main controller 30, and the current work is determined to be authorized based on the work authentication information.

[0337] In step S3602, it is determined whether an authentication invalidation command has been received from the external management device. The authentication invalidation command is a command to invalidate the function of the main controller 30 to perform rewrite authentication of application software (the authentication function by the authentication unit 222). If step S3602 is positive, the process proceeds to step S3603.

[0338] In step S3603, authentication for rewriting the application software is invalidated. In step S3604, rewriting of the application software and permission to take pictures with the camera 20 are permitted. After this, the main controller 30 receives, for example, application software for verification from the external management device, and writes the software to the memory unit of the main controller 30 or the camera 20.

[0339] When the authentication invalidation command from the external management device is discontinued, the authentication invalidation state is released in the main controller 30. Alternatively, when the main controller 30 receives a signal from the external management device to stop authentication invalidation, the authentication invalidation state is released. This returns the system to the original state in which application software rewrite authentication is performed.

[0340] Alternatively, the main controller 30 may be configured to be able to access the storage unit 26 (application storage area) on the camera 20 side, and the application software in the storage unit 26 may be rewritten by the main controller 30. This configuration will be described with reference to Fig. 37 .

[0341] In the configuration shown in Fig. 37, the main controller 30 includes, as components related to rewriting of application software, a software receiving unit 241, an authentication unit 242, and a rewriting execution unit 243. As with the configuration shown in Fig. 31 above, the software receiving unit 241 receives application software to be rewritten from an external device X such as a data server, and the authentication unit 242 certifies that the application software received from the external device X is authentic.

[0342] The rewrite execution unit 243 directly accesses the storage unit 26 on the camera 20 side and executes the rewrite of the application software stored in the storage unit 26. At this time, the rewrite execution unit 243 executes the rewrite of the application software in the storage unit 26 with the authenticated application software, on the condition that the application software has been authenticated by the authentication unit 222.

[0343] FIG. 38 is a flowchart for explaining the procedure for rewriting the application software of the camera 20 by the main controller 30.

[0344] As shown in Fig. 38, in step S3801, the main controller 30 receives application software to be rewritten from the external device X. In step S3802, the application software received from the external device X is authenticated to determine whether it is genuine. In step S3803, as a result of the authentication, it is determined whether the application software is genuine. If the application software is genuine, the process proceeds to step S3804. In step S3804, the memory unit 26 on the camera 20 side is directly accessed, and the application software stored in the memory unit 26 is rewritten.

[0345] It is also possible to use the system configuration shown in Fig. 39. In Fig. 39, the main controller 30 is able to access the memory unit 26 on the camera 20 side, and each time the object detection system is started, the main controller 30 writes application software into the memory unit 26 on the camera 20 side. The camera 20 remains operational even when the system is inactive.

[0346] 37 , the main controller 30 has a configuration related to rewriting of application software, which includes a software receiving unit 241, an authentication unit 242, and a rewriting execution unit 243. Furthermore, the main controller 30 has an application storage area 244 that stores application software used on the camera 20 side, and a software writing unit 245 that writes the application software to the storage unit 26 on the camera 20 side.

[0347] The application storage area 244 is a storage medium made of a nonvolatile memory, and stores application software received from the external device X. In this case, the application storage area 244 is configured to write application software that has been received from the external device X by the software receiving unit 241 and authenticated by the authentication unit 242.

[0348] Then, each time the system is started, the software writing unit 245 reads the application software from the application storage area 244 and executes a writing process to write the read application software to the storage unit 26 on the camera 20 side. After the system is started, the camera 20 side executes the application using the application software written by the main controller 30.

[0349] The memory unit 26 on the camera 20 preferably has a nonvolatile memory area 26a made of nonvolatile memory and a temporary memory area 26b made of volatile memory such as RAM. The nonvolatile memory area 26a is preferably a memory area made of electrically rewritable flash memory or the like. When the system is started up, the main controller 30 writes application software to the temporary memory area 26b of the memory unit 26 in the camera 20. When the power supply is cut off upon the end of operation of the camera 20, the application software stored in the temporary memory area 26b is erased. This prevents unauthorized access to the camera 20 while the power supply is cut off.

[0350] In an object detection system configured such that the camera 20 is also put into a sleep state along with the main controller 30 when the system is in a sleep state, it is preferable that application software that is only required while the system is in operation is stored in the temporary storage area 26b.

[0351] Furthermore, the memory unit 26 of the camera 20 may be configured so that different application software is stored in both the nonvolatile memory area 26a and the temporary memory area 26b. In this configuration, in consideration of the possibility that the camera 20 may be in operation even when the system is inactive, i.e., when the main controller 30 is inactive, the object detection system may store application software required throughout system operation and system inactivity in the nonvolatile memory area 26a, while application software required only when the system is in operation, i.e., when the main controller 30 is in operation, in the temporary memory area 26b. The application software stored in the temporary memory area 26b is erased when the system is inactive.

[0352] 40 is a flowchart for explaining the procedure for writing application software for camera 20 by main controller 30. Note that, as a premise, application software for camera 20 received from external device X is stored in application storage area 244 of main controller 30.

[0353] 40 , in step S4001, the main controller 30 determines whether the system is currently booting up, and if so, proceeds to step S4002. In step S4002, the main controller 30 reads application software from the application storage area 244 and executes a write process to write the read application software to the storage unit 26 of the camera 20.

[0354] After the system is started up, the camera 20 executes applications using application software written by the main controller 30. The application software stored in the camera 20 at system start-up realizes various processes required for object detection and driving support control on the main controller 30 side. Specifically, the application software stored in the camera 20 at system start-up includes software for executing the above-mentioned exposure condition setting process, white balance adjustment process, image masking process, noise removal process, image reading process, and image saving process when an abnormality occurs.

[0355] The configurations described with reference to FIGS. 31, 37, and 39 can be realized in combination with any of the configurations described so far.

[0356] The above-described configuration for updating the software of the camera 20 provides the following advantages.

[0357] The main controller 30 is provided with a means for accessing the software storage area on the camera 20 side, and the software on the camera 20 is rewritten by the main controller 30 or the camera controller 22. This allows the main controller 30 to function as a hub, making it possible to rewrite software for a plurality of different cameras 20, thereby reducing the burden on the owner or user of the vehicle 10.

[0358] Furthermore, software version checks can be performed on both the main controller 30 side and the camera 20 side, and compatibility can be confirmed at the same time.

[0359] Furthermore, malicious rewriting of the software storage area of ​​the camera 20 may adversely affect the characteristics of the camera 20, potentially resulting in unintended vehicle behavior. In this regard, the main controller 30 has a function for verifying that rewriting of the software storage area is legitimate and intended, thereby preventing malicious rewriting. As a result, the control of the camera 20 can be optimized, and the object detection process can be performed appropriately.

[0360] The camera 20 is configured to have a function for writing application software and a function for decrypting encrypted application software transmitted from the main controller 30. Direct access from the main controller 30 to the storage unit 26 (application storage area) is also prohibited. With this configuration, even if a cyber attack occurs in communication between the main controller 30 and the camera 20, unauthorized tampering or the like can be detected.

[0361] To correctly detect objects around the vehicle, information on camera parameters indicating the camera installation status of the vehicle 10 is required. In consideration of this, before rewriting the application software, the main controller 30 receives the camera parameters from the camera 20 whose software is to be rewritten, and after rewriting the application software, returns the camera parameters to the original camera 20. This makes it possible to correctly perform object detection while properly reflecting the camera parameters immediately after the software rewrite, even after the application storage area of ​​the camera 20 is temporarily cleared due to the software rewrite. Therefore, object detection and the driving assistance system of the vehicle 10 can be properly executed immediately after the software rewrite.

[0362] Furthermore, when generating an overhead image of the vehicle's surroundings using images from multiple cameras, information on the posture of each camera 20 can be retained even when the application software for some or all of the cameras 20 is rewritten. Therefore, the desired overhead image can be generated even immediately after the software is rewritten.

[0363] The main controller 30 acquires images of the camera 20 immediately before and immediately after the software is rewritten, and compares the images taken before and after the software is rewritten to determine whether or not an abnormality has occurred due to the rewriting of the application software. This makes it possible to detect unauthorized rewriting of the application software.

[0364] The main controller 30 or the camera 20 is configured to have a function for determining whether or not the application software stored in the storage unit 26 has been tampered with. This allows appropriate measures (appropriate fail-safe operation) to be taken even if unauthorized rewriting occurs.

[0365] The main controller 30 is configured to disable the authentication function that authenticates whether or not the application software is to be rewritten based on an external command, etc., and, with the authentication disabled, to permit the application software to be rewritten and the camera 20 to take pictures. In this case, excessive operational restrictions can be prevented, for example, when verifying or repairing the object detection system, improving convenience in manufacturing plants and repair plants.

[0366] The image sensor 21 is configured to have a storage unit 26 for storing application software. In this case, the image sensor 21 has a storage area for application software, and the software can be edited on the main controller 30 side. This makes it possible to link, for example, exposure control software to the image sensor itself, and to manage the software by linking it to the hardware type of the image sensor 21. This makes it easy to check the compatibility of the application software.

[0367] For example, compatibility checks can be performed by storing model and version information of the target image sensor 21 in the application software, and obtaining and comparing the model and version information from the image sensor's own register when rewriting the software.

[0368] The main controller 30 receives application software from the external device X and uses the received application software to rewrite the software in the storage unit 26 of the camera 20 (FIG. 37). In this case, even if the main controller 30 does not initially store application software for the camera 20 in its own storage area, it is possible to rewrite (edit) the application software on the camera 20 by receiving it from the external device X.

[0369] On the camera 20 side, application software is written by the main controller 30 each time the system is started, and the application is executed by that application software (Fig. 39). In this case, the main controller 30, which has high-performance computing capabilities, can verify whether the application software has been tampered with. This makes it possible to restore the camera 20 even if it is subjected to a cyber attack, for example.

[0370] When the system is started, the main controller 30 writes the application software to the temporary storage area 26b of the storage unit 26 on the camera 20 side. In this case, when the system is put into a sleep state in the camera 20, the application software stored in the temporary storage area 26b is erased. This makes it possible to prevent suspicious operation of the camera 20 while the system is in a sleep state.

[0371] In the camera 20, the nonvolatile storage area 26a of the storage unit 26 stores application software (first software) required during both system operation and system pause, while the temporary storage area 26b stores application software (second software) required only during system operation. When the system is started, the main controller 30 writes the second software to the temporary storage area 26b. This allows the camera 20 to operate properly both during system operation and system pause.

[0372] <Emergency Start-up of Controller When System is Hibernated> For example, when the vehicle 10 is powered off and object detection by the main controller 30 is paused, if an emergency system startup is required, such as when some abnormality occurs in the vehicle 10 or around the vehicle, it is desirable to be able to quickly save captured images in the main controller 30. Therefore, in this embodiment, the main controller 30 is properly emergency started and captured images are properly saved when the system is in a hibernated state. Figure 41 is a functional block diagram related to the configuration for performing emergency startup when the system is in a hibernated state.

[0373] 41, the camera 20 and the main controller 30 can communicate with each other via their respective communication units 24, 32. In the vehicle 10, the object detection system is configured to be inactive when the power switch is turned off. In this system inactive state, the main controller 30 is inactive, but the camera controller 22 remains active, allowing the image sensor 21 to take images and perform processing operations.

[0374] The camera controller 22 includes a rest time photographing unit 261 , a transmission determination unit 262 , an event notification unit 263 , and an image transmission unit 264 .

[0375] The rest time photographing unit 261 causes the image sensor 21 to photograph when object detection by the main controller 30 is paused. The rest time photographing unit 261 repeatedly photographs images at a predetermined cycle. The rest time photographing unit 261 repeatedly photographs the surroundings of the vehicle 10 while the vehicle 10 is parked.

[0376] The transmission determination unit 262 determines whether or not it is necessary to transmit the captured image to the main controller 30, based on the captured image captured by the pause-time capture unit 261. The determination of whether or not it is necessary to transmit the captured image to the main controller 30 is, in other words, a determination of whether or not it is necessary to store the image in the main controller 30.

[0377] At this time, the transmission determination unit 262 detects whether an abnormal situation or a situation that may be an abnormal situation has occurred in the vehicle 10 or around the vehicle based on the captured image taken by the resting state capture unit 261. If it is detected that an abnormal situation or a situation that may be an abnormal situation has occurred, the transmission determination unit 262 determines that image transmission to the main controller 30 is necessary. The transmission determination unit 262 uses the captured image to detect that an abnormal situation or a situation that may be an abnormal situation has occurred when it is deemed that an object is present in a position very close to the parked vehicle 10. For example, when another vehicle or a person approaches the parked vehicle 10, it is preferable to detect that an abnormal situation or a situation that may be an abnormal situation has occurred based on a change in brightness of the captured image.

[0378] By monitoring the surroundings using such images while the vehicle is at a standstill, it is determined that images need to be sent to the main controller 30, for example, if the parked vehicle 10 is hit by another vehicle, if the door of an adjacent vehicle hits the vehicle, if a person passing near the vehicle 10 commits fraud, or if any of these possibilities arise.

[0379] Here, a part of the vehicle 10 on which the camera 20 is mounted may appear in the image captured by the camera 20. Specifically, as shown in Fig. 42, a vehicle body B (e.g., a hood portion) may appear in the image captured by the camera 20 at the front of the vehicle. In this case, if the light reflectance of the vehicle body B is high, the area captured in the captured image will experience large changes in brightness due to the passing of an object or illumination by the headlights of other vehicles. This brightness change may cause the camera 20 to frequently request the main controller 30 to save images even when the scene does not require image saving.

[0380] Therefore, as a countermeasure to prevent unnecessary event notifications from occurring, the transmission determination unit 262 determines whether or not it is necessary to transmit the captured image to the main controller 30, using the image range excluding the area in which part of the vehicle 10 is captured in the captured image taken by the rest time capture unit 261.

[0381] In this case, the main controller 30 analyzes the image captured by the camera 20 in advance and provides the camera 20 with information about areas where a part of the vehicle is captured. The camera 20 can reduce unnecessary event notifications by reviewing its own image transmission start conditions (in other words, image storage conditions in the main controller 30) based on the information about the captured area. The camera controller 22 does not use the area in the captured image where the vehicle 10 is captured when determining whether to start image transmission.

[0382] While the vehicle 10 is traveling, the main controller 30 may determine whether or not a part of the vehicle 10 is reflected in the captured images, which are successive in time series, captured by the camera 20. In this case, if an object of the same shape is continuously captured in the same area in the captured images while the vehicle is traveling, the main controller 30 determines that a part of the vehicle 10 is reflected in the captured images, and identifies the area where the object is reflected.

[0383] When the camera 20 is attached to the vehicle 10, if it is known that part of the vehicle 10 will be reflected in the captured image, the transmission determination unit 262 may use the image range excluding the reflected area to determine whether or not the captured image needs to be transmitted to the main controller 30.

[0384] It is possible that an object mounted on the roof of the vehicle 10 may appear in the captured image. If the object has a high reflectivity, there is a concern that unnecessary event notifications may be generated due to changes in brightness caused by the passing of an object or the illumination of headlights from other vehicles. Similar measures can be taken in this case as well. In this embodiment, in addition to the body of the vehicle 10, the object mounted in a fixed state on the vehicle 10 is also considered to be part of the vehicle 10.

[0385] When the transmission determination unit 262 determines that image transmission to the main controller 30 is required, the event notification unit 263 issues an event notification indicating that image transmission to the main controller 30 will begin. Prior to image transmission to the main controller 30, the event notification unit 263 issues a prior notification to the main controller 30 to prompt initialization for startup.

[0386] In this system, a first communication mode in which the communication volume per unit time is relatively high and a second communication mode in which the communication volume per unit time is relatively low are defined as communication modes between the communication unit 32 of the main controller 30 and the communication unit 24 of the camera 20. The first communication mode is a communication mode in a normal power state when the main controller 30 is operating (when the system is operating), and the second communication mode is a communication mode in a low power state when the main controller 30 is inactive (when the system is inactive). It is also possible to say that the first communication mode is a normal power consumption mode when the system is operating, and the second communication mode is a low power consumption mode when the system is inactive.

[0387] When the transmission determination unit 262 determines that image transmission to the main controller 30 is required, the event notification unit 263 notifies the main controller 30 in the second communication mode to start image transmission. After receiving the notification that image transmission will be performed in the second communication mode, the image transmission unit 264 switches the communication mode from the second communication mode to the first communication mode and starts image transmission. This causes images captured by the image sensor 21 to be transmitted to the main controller 30. In this case, it is possible to start image transmission as soon as possible after the notification from the main controller 30, while causing the camera 20 to be activated in a low-power state.

[0388] In the camera controller 22 , switching from the first communication mode to the second communication mode is performed spontaneously when an event notification is sent to the main controller 30 , even without an instruction from the main controller 30 .

[0389] Alternatively, a simpler configuration is possible in which the camera 20 is connected to a signal input terminal provided on the main controller 30 via a signal line, and an H level signal is input from the camera 20 to the signal input terminal of the main controller 30, thereby providing advance notification to the main controller 30.

[0390] On the other hand, the main controller 30 includes a mode transition unit 271 , a validity determination unit 272 , an image storage determination unit 273 , and an event stop notification unit 274 .

[0391] When an event notification is received from camera controller 22 while object detection is paused, mode transition unit 271 transitions to an image storage mode in which the captured image received from camera 20 is stored in image storage unit 33 on the main controller 30 side. Note that, in the paused state, main controller 30 has enabled at least a function for recognizing event notifications from camera controller 22. Then, main controller 30 starts up upon receiving an event notification.

[0392] After transitioning to image storage mode, the validity determination unit 272 performs image analysis of the captured image used by the transmission determination unit 262 in the camera controller 22 to determine whether or not image transmission is necessary, and based on the analysis results, determines the validity of the transmission determination unit 262's determination that image transmission is necessary.

[0393] Specifically, the validity determination unit 272 performs object recognition by pattern matching of objects in the captured image received from the camera 20, for example, and determines that another vehicle or the like has collided with the vehicle 10. Alternatively, the transmission determination unit 262 determines, based on the captured image, that there is a suspicious person around the vehicle 10. In this case, the main controller 30 has higher processing performance than the camera controller 22, and is capable of object recognition with higher accuracy than the camera controller 22. For example, the main controller 30 is capable of more detailed and diverse pattern matching than the camera controller 22.

[0394] After transitioning to the image storage mode, the image storage determination unit 273 determines whether or not it is necessary to continuously store the image received from the camera 20 side, based on the result of the determination of validity by the validity determination unit 272. In this case, if the determination of the transmission determination unit 262 is deemed valid, it determines that it is necessary to store the image received from the camera 20 side, and if the determination of the transmission determination unit 262 is deemed invalid, it determines that it is not necessary to store the image received from the camera 20 side.

[0395] When the image storage determination unit 273 determines that it is not necessary to store the received image, the event stop notification unit 274 transmits a notification of the event stop to the camera controller 22. This prevents the main controller 30 from transmitting, receiving, or recording unnecessary image data.

[0396] The main controller 30 may include an exposure condition determination unit 275 and an image capture command unit 276. When the exposure condition determination unit 275 receives an event notification from the camera controller 22 and transitions to image storage mode while object detection is paused, the exposure condition determination unit 275 determines the exposure conditions of the camera 20. In this case, the exposure condition determination unit 275 sets the exposure conditions while taking into account the vehicle's surroundings at the time of the event notification. Information about the vehicle's surroundings may include ambient environmental information, such as brightness around the vehicle and weather information, such as rain or snow. The exposure time (shutter speed) and gain of the image sensor 21 are determined as exposure conditions based on the external environmental information. As a result, the captured images stored and held on the main controller 30 side are switched from images captured using exposure conditions set as a function of the camera alone during system pause to images captured using more precisely set exposure conditions on the main controller 30 side. This enables the capture of appropriate captured images suited to each shooting scene.

[0397] The photography command unit 276 transmits the exposure conditions determined by the exposure condition determination unit 275 to the camera controller 22, and causes the camera to take a photograph under the exposure conditions.

[0398] The main controller 30 may also include a judgment condition setting unit 277. After transition to the image storage mode, the judgment condition setting unit 277 variably sets the judgment conditions used for the necessity determination of image transmission by the transmission judgment unit 262 on the camera 20 side. In this case, the judgment condition setting unit 277 performs image analysis of the captured image used for the necessity determination of image transmission, and variably sets the judgment conditions used for the transmission judgment unit 262's determination of image transmission based on the analysis results.

[0399] For example, when the vehicle 10 is parked and it is raining or there is a lot of traffic or pedestrians, the captured image may change frequently, which may result in unnecessary event notifications. In other words, there is a concern that event notifications may be repeatedly issued due to false detections. In this regard, by analyzing the scene in the received image on the main controller 30 side, excessively frequent saving of captured images while the system is inactive is suppressed.

[0400] Specifically, the determination condition setting unit 277 sets the determination conditions to have a relatively low determination sensitivity, i.e., to make it difficult to determine that image transmission is required, when there are many other vehicles or people around the vehicle, taking into consideration the possibility of unnecessary event notifications occurring, whereas when there are few other vehicles or people around the vehicle, the determination conditions to have a relatively high determination sensitivity, i.e., to make it easy to determine that image transmission is required.

[0401] The camera controller 22 may also include a periodic transmission unit 265. That is, when object detection by the main controller 30 is paused, the camera controller 22 performs image capture using the pause image capture unit 261, but does not transmit the captured images to the main controller 30. The periodic transmission unit 265 then periodically transmits to the main controller 30 the determination result of whether the image sensor 21 is normal or abnormal. The periodic transmission cycle to the main controller 30 may be shorter than the periodic image capture cycle by the pause image capture unit 261. However, the periodic transmission cycle may be longer than the periodic image capture cycle during pause, or the same as the periodic image capture cycle during pause.

[0402] The periodic transmission unit 265 may compare the images captured in time series by the image sensor 21 to determine whether or not there is an abnormality in the image sensor 21. For example, if a sudden change in brightness or a change in the presence of an object occurs in the images captured in time series that would not occur due to the movement of an object in reality, it may be determined that an abnormality has occurred in the image sensor 21.

[0403] In this case, the main controller 30 can know that the camera 20 is continuing to monitor the image around the vehicle based on the minimum necessary information received from the camera 20, and can also know whether there is a malfunction in the camera 20. When the main controller 30 receives information from the periodic transmission unit 265 of the camera controller 22 that the image sensor 21 is abnormal, it is preferable that the main controller 30 notify the vehicle user of the occurrence of the abnormality.

[0404] The configuration shown in Fig. 43 may also be used. In Fig. 43, the image storage unit 33 of the main controller 30 has a temporary storage area 33a and a non-volatile storage area 33b, and the image received from the camera 20 side can be written to either the temporary storage area 33a or the non-volatile storage area 33b.

[0405] 41 , the main controller 30 also includes a first image storage unit 281 and a second image storage unit 282. The first image storage unit 281 stores received images in the temporary storage area 33a immediately after switching to the image storage mode. The second image storage unit 282 stores a series of received images, including the captured image used by the transmission determination unit 262 to determine whether to transmit the images, in the non-volatile storage area 33b when the image storage determination unit 273 determines that continued storage of received images is required.

[0406] In addition, when the received image is continuously stored by the second image storage unit 282, the image stored in the temporary storage area 33a may be transferred to the non-volatile storage area 33b immediately after switching to the image storage mode.

[0407] Figure 44 is a flowchart for explaining the processing procedures of the controllers 30, 22 when the vehicle 10 is in a power-off state (when the object detection system is in a dormant state). Here, the processing procedures corresponding to the configurations of Figures 41 and 43 will be explained. In Figure 44, (a) shows the processing of the main controller 30, and (b) shows the processing of the camera controller 22. Each of the processes in Figures 44(a) and (b) is repeatedly executed at a predetermined cycle.

[0408] The main controller 30 can operate in normal mode and in sleep mode when the system is in sleep mode. In sleep mode, the processes that can be executed are limited compared to normal mode, thereby reducing power consumption. In sleep mode, it is possible to receive signals from at least the camera controller 22. Furthermore, the camera controller 22 is capable of taking pictures even in system sleep mode. However, in system sleep mode, the communication mode with the main controller 30 is different from that in the system operating state, and is set to a second communication mode that uses less power than normal.

[0409] 44(b), in step S4421, the camera controller 22 determines whether the system is in a hibernation state. If step S4421 is YES, the process proceeds to the subsequent step S4422. In step S4422, image capture during the hibernation state is performed by the image sensor 21. At this time, for example, while the vehicle 10 is parked, the surroundings of the vehicle are repeatedly captured.

[0410] In step S4423, based on the captured image taken in step S4422, it is determined whether or not the captured image needs to be transmitted to the main controller 30. Specifically, based on the image taken during rest, it is detected whether or not an abnormal situation or a situation that may be abnormal has occurred in the vehicle 10 or around the vehicle, and based on the detection result, it is determined whether or not the image needs to be transmitted to the main controller 30. At this time, if an abnormal situation or a situation that may be abnormal has occurred, it is determined that the image needs to be transmitted to the main controller 30, and if no abnormal situation or a situation that may be abnormal has occurred, it is determined that the image does not need to be transmitted to the main controller 30. For example, when an object approaches in a position extremely close to the parked vehicle 10, it is detected that an abnormal situation or a situation that may be abnormal has occurred.

[0411] In step S4423, it is preferable to determine whether or not the captured image needs to be transmitted to the main controller 30 using the image range excluding the area in which part of the vehicle 10 is captured in the captured image in step S4422.

[0412] In step S4424, it is determined whether or not it is determined in the detection process of step S4423 that image transmission to the main controller 30 side is required.

[0413] If step S4424 is negative, the process proceeds to step S4425. In step S4425, the result of the determination as to whether the image sensor 21 is normal or abnormal is periodically (at a predetermined interval) transmitted to the main controller 30. In a state in which the system is in a hibernation state and no abnormality or possible abnormality has occurred, steps S4422 to S4425 are repeatedly executed.

[0414] If step S4424 is affirmative, the process proceeds to step S4426. In step S4426, an event notification is sent to the main controller 30 to indicate that image transmission to the main controller 30 has begun. At this time, the event notification is sent to the main controller 30 in the second communication mode (low power mode). The event notification from the camera 20 to the main controller 30 is sent prior to transmission of the captured image from the camera 20 to the main controller 30.

[0415] Thereafter, in step S4427, the communication mode is switched from the second communication mode to the first communication mode. In the following step S4428, transmission of the resting image to the main controller 30 is started.

[0416] On the other hand, as shown in Fig. 44(a), in step S4401, the main controller 30 determines whether or not an event notification has been received from the camera 20 while the system is in a hibernation state. After the event notification is sent in step S4416 of Fig. 44(b), the camera controller 22 judges yes in step S4401 and proceeds to the subsequent step S4402. In step S4402, initialization processing is performed for startup to enable operation in normal mode. In step S4403, the mode transitions to image storage mode.

[0417] Thereafter, in step S4404, the captured image is received from the camera 20 side, and in the following step S4405, the captured image received from the camera 20 side is stored in the first area (temporary storage area 33a).

[0418] In step S4406, the camera controller 22 performs image analysis on the captured image used to determine whether or not an event notification was necessary, i.e., the captured image used to determine whether or not image transmission was necessary, and determines the appropriateness of the determination that image transmission was necessary based on the analysis results. In the following step S4407, it is determined from the result of step S4406 whether or not the event notification (image transmission) was appropriate. In step S4407, it is determined whether or not it is necessary to continuously store images received from the camera 20. If it is determined that the event notification was appropriate, the process proceeds to step S4408.

[0419] In step S4408, the main controller 30 continues to store the images received from the camera 20. At this time, the main controller 30 stores in the second area (non-volatile storage area 33b) a series of received images including the captured image used by the camera 20 to determine whether or not image transmission is necessary.

[0420] In step S4409, the exposure conditions of the camera 20 are determined and transmitted to the camera 20. At this time, it is preferable to set the exposure conditions taking into consideration the situation around the vehicle at the time of the event notification, etc.

[0421] If it is determined that the event notification is not valid, the process proceeds to step S4410, where it is determined that the received image does not need to be stored, and a notification to the effect that the event has been stopped is sent to the camera controller 22.

[0422] Thereafter, in step S4411, the determination conditions used to determine whether or not image transmission from the camera 20 is necessary are variably set. A situation in which step S4407 is negative means that an event notification has been sent from the camera 20, but the main controller 30 has determined that the event is unnecessary, and it may be better for the camera 20 to change the conditions for determining whether or not image transmission is necessary (the determination in step S4423 in FIG. 44(b)). Therefore, in step S4411, taking into consideration the possibility that an unnecessary event notification may be generated in the current vehicle environment, a command to change the conditions is sent to the camera 20 so that the determination sensitivity of the determination conditions (anomaly detection conditions) used to determine whether or not image transmission is necessary on the camera 20 side is set to be relatively low. As a result, for example, when the current vehicle environment is one in which there are many other vehicles and people around the vehicle, or one in which bad weather such as rain or snow is occurring, the judgment conditions are changed to ones with relatively low judgment sensitivity in order to suppress unnecessary event notifications on the camera 20 side.

[0423] Steps S4410 and S4411 suppress the transmission, reception, and recording of unnecessary image data in main controller 30. In step S4411, it is determined whether a part of vehicle 10 is reflected in the captured image received from camera 20, and if vehicle 10 is reflected in the captured image, a determination condition may be set based on information about the part of the vehicle that is reflected in the captured image.

[0424] 44B, when a notification of event stop is sent from the main controller 30 in step S4410, the event notification is stopped at that point. Also, when a command to change the determination conditions is issued in step S4411, that command is reflected in the next and subsequent processing.

[0425] FIG. 45 is a time chart for explaining a specific procedure for requesting the camera 20 to store an image in the main controller 30 when the object detection system is in a resting state.

[0426] In Figure 45, before timing t1, the system is in a sleep state, i.e., the main controller 30 is in a sleep state. In this system sleep state, the communication mode between the camera 20 and the main controller 30 is the second communication mode, which is a low-power consumption mode. Also, in the system sleep state, the camera 20 takes sleep-time photographs at a predetermined interval. However, in this case, images are not transmitted from the camera 20 to the main controller 30.

[0427] At timing t1, an image captured while the vehicle is at rest detects that an abnormality or a possible abnormality has occurred in or around the vehicle 10. Then, at timing t2, the camera 20 notifies the main controller 30 of the event.

[0428] Thereafter, at timing t3, the main controller 30 performs a startup process in response to the reception of the event notification. Furthermore, the camera 20 switches the communication mode from the second communication mode to the first communication mode, which is the normal mode, and starts transmitting images to the main controller 30. Furthermore, the main controller 30 switches to the image storage mode, and the images captured by the camera 20 are stored in the image storage unit 33.

[0429] Thereafter, at timing t4, the main controller 30 performs an analysis process of the captured image used to determine whether or not to notify the event in the camera controller 22. If the result of the image analysis indicates that the event notification (image transmission) is appropriate, the storage of the image received from the camera 20 side continues.

[0430] Although not shown in the figure, it is preferable that image saving by the main controller 30 be terminated when a predetermined time has elapsed since the event notification. Furthermore, if the event notification is invalid, the main controller 30 notifies the camera 20 that the event has been stopped. In this case, the event notification from the camera 20 is canceled. This causes the main controller 30 to return to its original sleep state.

[0431] Furthermore, during system inactivity, the camera 20 operates in a low-power consumption mode, which limits various processes on the camera 20 side. Therefore, it is difficult to perform analysis across a wide dynamic range, from bright to dark, and across the entire viewing angle of the camera 20. Under these conditions, to detect abnormalities in the surroundings regardless of the surrounding environment, it is considered preferable to monitor changes in brightness over time rather than brightness values ​​(absolute brightness values). Time changes in brightness enable scene analysis of abnormalities, etc., regardless of the brightness of the surrounding environment. Furthermore, by setting a threshold for determining brightness changes and limiting scene analysis of abnormalities, etc., to areas where the brightness change is above a certain level, it is not necessary to monitor the entire viewing angle of the camera 20.

[0432] The amount of change in luminance may be calculated from the differential value of the luminance detected by the image sensor itself, or may be calculated from the differential value of the luminance in the captured image.

[0433] 46 is a flowchart for explaining the processing procedure of the camera controller 22 in the system hibernation state. This processing may be executed by replacing the processing in FIG. 44(b).

[0434] In FIG. 46, if it is determined in step S4601 that the system is in a hibernation state, the process proceeds to step S4602, where hibernation photography is performed by the image sensor 21.

[0435] In step S4603, the luminance change amount, which is the amount of change in luminance per unit time within the viewing angle of the image sensor 21 (camera 20), is calculated. In the following step S4604, scene analysis is performed on areas where the luminance change amount is equal to or greater than a predetermined value. At this time, by analyzing the areas where the luminance change occurs, it is detected whether an abnormal situation or a situation that may be an abnormal situation has occurred in the vehicle 10 or around the vehicle. For example, it is preferable to detect an abnormality or a possible abnormality based on the number of points where the luminance change occurs or the size of the area.

[0436] In step S4605, based on the detection result of the abnormality or possibility of abnormality, it is determined whether or not it is necessary to transmit the captured image to the main controller 30. If a positive judgment is made in step S4605, the process proceeds to step S4606. In step S4606, an event notification is issued indicating that image transmission to the main controller 30 will begin. In step S4607, the communication mode is switched from the second communication mode to the first communication mode.

[0437] Then, in step S4608, the brightness values ​​within the viewing angle of the image sensor 21 (camera 20) are converted into an image, and the image is generated. In the following step S4609, transmission of the resting image to the main controller 30 is started.

[0438] In the system hibernation state, when the main controller 30 starts saving an image, the necessity of transmission may be determined based on the amount of change in brightness, and image analysis may not be performed. In this case, the image may be transmitted from the camera 20 to the main controller 30, and scene analysis is not performed.

[0439] It is also possible to adopt the configuration shown in Fig. 47. Fig. 47 shows a plurality of cameras 20 mounted on a vehicle 10. The main controller 30 includes an image transmission command unit 291, a storage processing unit 292, and an image storage unit 33.

[0440] When an event notification is received from any one of the cameras 20 and the camera 20 switches to the image storage mode, the image transmission command unit 291 commands the remaining cameras 20 other than the camera 20 that sent the event notification to transmit their captured images. The storage processing unit 292 stores the captured images transmitted from the remaining cameras 20 in the image storage unit 33 on the main controller 30 side.

[0441] In this case, the image transmission command unit 291 may be configured to command a specific camera 20 among the remaining cameras 20 other than the camera 20 that issued the event notification to transmit captured images. Specifically, depending on whether the camera 20 that issued the event notification is a front, rear, left, or right camera 20 of the vehicle 10, the image transmission command unit 291 may be configured to selectively command a specific camera to transmit captured images, depending on whether the camera 20 that issued the event notification is a front, rear, left, or right camera 20 of the vehicle 10. For example, when an event notification is transmitted from the front camera of the vehicle 10, the image transmission command unit 291 commands the nearby cameras 20 on both the left and right sides to transmit captured images. Furthermore, when an event notification is transmitted from the left side camera of the vehicle 10, the image transmission command unit 291 commands the nearby cameras 20 on the front and rear sides to transmit captured images.

[0442] Fig. 48 is a flowchart for explaining the processing procedure of the main controller 30 when the system is in a hibernation state. This processing is repeatedly executed at a predetermined cycle by the main controller 30. Note that, like Fig. 44, the processing of Fig. 48 includes image storage processing in the image storage unit 33 (storage areas 33a, 33b) in the main controller 30 and processing for determining the validity of the image transmission decision on the camera 20 side, but for convenience, overlapping processing is omitted.

[0443] In Figure 48, in step S4801, it is determined whether an event notification has been received from the camera 20 while the system is in a hibernation state. If step S4801 is YES, the process proceeds to the subsequent step S4802. In step S4802, initialization processing is performed to enable operation in normal mode. At this time, if an event notification has been received from any of the multiple cameras 20 in the vehicle 10, the main controller 30 is started. Thereafter, in step S4803, the process transitions to image storage mode.

[0444] Then, in step S4804, it is determined which of all the on-board cameras the remaining cameras 20 other than the camera 20 that issued the event notification are. In the following step S4805, a command is issued to transmit captured images to specific cameras 20 other than the camera 20 that issued the event notification. As a result, captured images are transmitted to the main controller 30 from the specific cameras other than the camera 20 that issued the event notification. In this case, the specific cameras are some or all of the cameras other than the camera 20 that issued the event notification.

[0445] In step S 4806 , captured images are received from the camera 20 that has issued the event notification among the plurality of cameras 20 and from other specific cameras, and the received images are stored in the image storage unit 33 .

[0446] The configurations described in FIGS. 41, 43, and 47 can be realized in combination with any of the configurations described so far.

[0447] The above-described configuration for emergency controller startup during system shutdown provides the following advantages.

[0448] When object detection by the main controller 30 is paused, if the camera controller 22 determines that image transmission to the main controller 30 is necessary based on images captured in the system pause state, the camera 20 issues a prior notification before transmitting the images to the main controller 30, and then starts transmitting the images to the main controller 30. In this case, when the object detection system is paused, image transmission can be started as soon as possible at the discretion of the camera controller 22, while minimizing communication between the main controller 30 and the camera 20. As a result, images captured in the object detection pause state (system pause state) can be properly saved. As a result, the control of the camera 20 can be optimized, and ultimately the object detection process can be properly performed.

[0449] When it is determined that image transmission from the camera controller 22 to the main controller 30 is required in an object detection pause state, the camera controller 22 notifies the main controller 30 of the start of image transmission in the low-speed communication mode (second communication mode), and after the notification, automatically starts image transmission in the high-speed communication mode (first communication mode). In this case, for example, in a scene where a suspicious vehicle or suspicious person is present near a parked vehicle, image transmission can be started as soon as possible while the camera 20 is activated in a low-power state.

[0450] When the camera controller 22 detects an abnormal situation or a situation that may be an abnormal situation in the vehicle 10 or around the vehicle based on an image captured while the system is in a hibernation state, the camera controller 22 determines that an image needs to be sent to the main controller 30. In this case, when an abnormal situation or a situation that may be an abnormal situation occurs, it becomes possible to save an image containing information about the occurrence, which can be used for subsequent abnormality analysis, etc.

[0451] If a part of the vehicle 10 is captured in an image captured by the camera 20, there is a concern that the camera 20 may unnecessarily request the main controller 30 to transmit the image (save the image). In consideration of this, the camera controller 22 is configured to use the image range of the captured image excluding the area in which a part of the vehicle 10 is captured to determine whether or not to transmit the captured image to the main controller 30. This allows the camera controller 22 to appropriately determine whether or not to transmit the image.

[0452] When the camera 20 determines whether to issue an event notification (i.e., whether to send images to the main controller 30) while the system is in a hibernation state, it is possible that an unnecessary event notification may be issued. For example, the camera controller 22 may determine that a potentially abnormal event has occurred, but it may simply be that another vehicle has approached the vehicle 10 or that a pedestrian has simply passed by the side of the vehicle. In such cases, it is possible that the camera controller 22 may erroneously determine that an event notification is required. In consideration of this, when the main controller 30 receives an event notification from the camera controller 22, it temporarily switches to an image storage mode in which the received image is stored in the image storage unit 33 of the main controller 30. However, if it is determined after switching to the image storage mode that storing the received image is unnecessary, it sends a notification to the camera controller 22 indicating that the event has been stopped. This prevents unnecessary transmission, reception, and recording of image data.

[0453] The main controller 30 determines the validity of the decision that image transmission is required by the camera controller 22 through image analysis of the captured image used to determine whether image transmission is required, and based on this validity, determines whether or not it is necessary to continuously store images received from the camera 20. In this case, the main controller 30 can perform high-performance and appropriate image analysis, and can optimize image storage even when the system is in a hibernation state.

[0454] When the main controller 30 receives an event notification from the camera controller 22 while object detection is paused and transitions to image storage mode, it determines the exposure conditions for the camera 20 and causes the camera to capture images under those exposure conditions. In this case, a configuration in which the exposure conditions are determined by the main controller 30 makes it possible to perform exposure control that takes into account the situation around the vehicle at the time of the event notification. In other words, when exposure control is performed by the camera alone, limited exposure control is possible due to software size constraints, etc., whereas exposure control by the main controller 30 makes it possible to perform appropriate exposure control tailored to the scene being captured each time. This allows appropriate image storage to be achieved when the main controller 30 is urgently started up from a system pause state.

[0455] The main controller 30 is configured to variably set the judgment conditions used to determine whether or not image transmission is necessary by image analysis of the captured image used to determine whether or not image transmission is necessary in the camera controller 22. For example, if event notifications due to erroneous detection are repeated, the scene in the received image is analyzed on the main controller 30 side and fed back to the camera 20 side. This prevents captured images from being saved too frequently while the system is inactive, and ultimately prevents an increase in unnecessary image data.

[0456] Immediately after switching to the image storage mode, the main controller 30 first stores the received image in the temporary storage area 33a, and if it determines that continued storage of the received image is required after switching to the image storage mode, it stores a series of received images, including the captured image used to determine whether or not image transmission is required, in the non-volatile storage area 33b. This allows image data to be stored more appropriately when an abnormality occurs, and enables the abnormality to be analyzed appropriately.

[0457] When the main controller 30 receives an event notification from any of the cameras 20 and transitions to the image storage mode, it commands the remaining cameras 20 other than the camera 20 that sent the event notification to transmit their captured images, and the captured images transmitted from the remaining cameras 20 are stored in the image storage unit 33 on the main controller 30 side. In this case, by storing the captured images from the other cameras 20 in response to the event notification from one camera 20, it is possible to simultaneously store the situation around the vehicle at the time of the event notification.

[0458] The camera controller 22 determines whether or not an image needs to be transmitted to the main controller 30 based on the amount of change in brightness within the viewing angle of the camera 20 while the system is in a sleep state, and after an event notification, transmits an image created by converting the brightness values ​​within the viewing angle of the camera 20 into an image to the main controller 30. This reduces the processing load on the camera 20. In this case, because the camera 20 is in a low power consumption mode while the system is in a sleep state, appropriate image storage can be performed even when the processing of the camera 20 is restricted.

[0459] The camera controller 22 is configured to capture images while the system is in a hibernation state, but not to transmit the captured images to the main controller 30, and to periodically transmit to the main controller 30 the results of its determination of whether the image sensor 21 is normal or abnormal. This allows the main controller 30 to know that the camera 20 is continuing to monitor the images around the vehicle based on the minimum necessary information received from the camera 20, and also to know whether the camera 20 is malfunctioning. In this case, when the main controller 30 is started up in response to an event notification from the camera 20, it is possible to omit a malfunction check on the camera 20, thereby minimizing the transition time at start-up.

[0460] <Faster Camera Startup> In an object detection system, when communication is performed between the camera 20 and the main controller 30, the amount of communication data sent from one side to the other changes depending on the scene. For example, in the camera 20, the amount of communication data sent or received between the camera 20 and the main controller 30 differs between when the camera 20 is started up and when the camera 20 is in normal operation after start-up. In this case, during normal operation, captured images are sent from the camera 20, so more data is sent to the main controller 30, whereas during start-up, more data is received from the main controller 30 due to the need for camera initialization. In this embodiment, the communication mode is changed depending on the scene in order to speed up camera startup.

[0461] FIG. 49 is a functional block diagram relating to a configuration that enables high-speed camera startup.

[0462] 49, the camera 20 and the main controller 30 each have a communication unit 24, 32. Each communication unit 24, 32 is capable of two-way communication via wired or wireless communication, but here the communication units 24, 32 are shown connected to each other via a communication line 41. Each communication unit 24, 32 is capable of time-division two-way communication. In this embodiment, in a configuration in which half-duplex communication is performed as two-way communication, it is possible to switch the time ratio between transmission and reception within one frame.

[0463] The camera controller 22 includes a communication control unit 301. During initialization processing accompanying startup of the camera 20, the communication control unit 301 temporarily makes the proportion of time allocated to communication from the main controller 30 to the camera controller 22 larger than the proportion of time allocated to communication from the camera controller 22 to the main controller 30.

[0464] Based on the completion of the initialization process, the communication control unit 301 cancels the process in which the proportion of time allocated to communication from the main controller 30 to the camera controller 22 has been temporarily increased.

[0465] Each of the communication units 24, 32 preferably includes a serializer-deserializer (SERDES) as a communication device. That is, each of the communication units 24, 32 includes a circuit (serializer) that serializes parallel data and a circuit (deserializer) that conversely parallelizes serial data. Each of the communication units 24, 32 is capable of bidirectional conversion using the SERDES. Mutual communication between the serializer and deserializer in each of the communication units 24, 32 preferably complies with various SERDES standards. The communication standard may be defined by the Ethernet standard.

[0466] The communication units 24 and 32 are connected to each other using a coaxial cable, in which a conducting wire is covered with a cylindrical conductor, as the communication line 41. However, an optical fiber cable or the like can also be used as the communication line.

[0467] In the communication units 24, 32 on the camera 20 side and the main controller 30 side, parallel data is serialized (converted into a serial signal) on the transmitting side, and serial data is parallelized (restored to the original data) on the receiving side. For example, when image data is transmitted from the camera 20 side to the main controller 30 side, the image data, which is parallel data, is serialized on the camera 20 side, and the serial data is transmitted to the main controller 30 side. The main controller 30 side restores the original image data from the serial data.

[0468] The configuration described in FIG. 49 can be realized in combination with any of the configurations described so far.

[0469] FIG. 50 is a flowchart for explaining the processing procedure of the camera controller 22 when the camera is started up.

[0470] 50, in step S5001, it is determined whether or not the camera 20 is starting up, and if so, the process proceeds to step S5002. In step S5002, the proportion of time allocated to communication from the main controller 30 to the camera controller 22 is set to be greater than the proportion of time allocated to communication from the camera controller 22 to the main controller 30. This allows information required for initialization processing at camera 20 startup to be quickly obtained from the main controller 30.

[0471] Then, in step S5003, it is determined whether the initialization process on the camera 20 side has been completed. If the initialization process has been completed, the process proceeds to step S5004. In step S5004, the process of step S5002, i.e., the process of increasing the proportion of time allocated to communication from the main controller 30 to the camera controller 22, is canceled.

[0472] According to the above process, the communication mode on the camera 20 side is switched as shown in Fig. 51. During the initialization period when initialization is performed at the time of camera startup, the time ratio of reception is greater than the time ratio of transmission in each successive frame in chronological order. Furthermore, after the initialization period ends, the time ratio of transmission is greater than the time ratio of reception in each frame.

[0473] During the initialization process associated with the startup of the camera 20, the camera controller 22 executes the initialization process and then starts transmitting captured images. In this case, by temporarily increasing the proportion of time allocated to communication from the main controller 30 to the camera controller 22 during the initialization process on the camera 20 side, the time required for the initialization process is shortened, enabling the object detection process to start as quickly as possible. As a result, the object detection system can perform proper object detection. As a result, the control of the camera 20 can be optimized, and ultimately the object detection process can be performed properly.

[0474] According to this configuration, in order to speed up the startup time of the camera 20, an independent communication means for speeding up communication from the main controller 30 to the camera 20 is not required, and advantages can also be expected in terms of system construction.

[0475] The communication unit 24 of the camera 20 and the communication unit 32 of the main controller 30 each have a serializer / deserializer, which allows for even faster communication between the camera 20 and the main controller 30.

[0476] The present embodiment described above addresses and solves the following problems, which will now be explained once again.

[0477] Problem 1. When a processor with large-scale computing power is used, it becomes necessary to separate the image sensor (camera) from the processor. In this case, the image sensor and processor are connected by a cable, and images are sent and received through the cable. In this case, the processor is limited in frame rate by the communication capacity of the cable. As a result, the image sensor's positional deviation determination and image changes due to changes in the external environment can only be perceived at the frame rate and the processor's calculation cycle. For example, assuming that the image sensor's positional deviation due to posture deviation and continuous changes in the external environment occur in a vehicle, there is a concern that communication and perception delays will make it difficult for the system to keep up. In other words, there is a concern about perception delays due to communication and the processor processing cycle.

[0478] Problem 2. When a vehicle uses multiple cameras to monitor its surroundings, not just the forward view, different cameras are deployed around the vehicle. Prior art approaches involve a processor analyzing image data, but the captured scene varies depending on the camera's position. For example, one camera may be backlit while another is frontlit, or one may be facing downward while another is facing forward. In such a system configuration, the processor receives images from all the different cameras and provides different control for each camera. This requires independent software to control each camera, and software updates on the processor are required every time the camera is changed. For example, upgrading the camera's image sensor without changing the processor or repurposing the same processor for a vehicle equipped with a different camera requires software development for the processor itself, which reduces development efficiency. In other words, having the processor control the cameras raises concerns about software inflexibility and reduced efficiency.

[0479] Problem 3: When a processor with large-scale computing power is used, it becomes necessary to separate the image sensor (camera) from the processor. In this case, the image sensor and processor are connected by a cable, and images are sent and received through the cable. For the processor to perform the desired control of the image sensor, a predetermined sequence and setting values ​​must be updated for the image sensor when the camera is started. However, this startup time is limited by the communication capacity of the cable. Therefore, it takes time for the vehicle and processor to start up and for the camera to send the images required by the processor, which raises concerns that the system may not be able to keep up with the driver's application execution request or the detection of a dangerous scene. In other words, there is a concern that sensor startup delays may occur due to the communication capacity of the cable.

[0480] Problem 4: Processors with large computing power consume a lot of power, making it impossible to perceive the surroundings unless they receive a large amount of power from the vehicle. Therefore, when the vehicle is turned off, the battery is low, or the battery is not connected, the ability to perceive the surroundings is lost. In other words, there is a concern that the main-sub relationship between the processor and the camera may result in functional limitations.

[0481] According to the above-described embodiment, it is possible to appropriately address and solve the above-mentioned problems.

[0482] In the above embodiment, the vehicle 10 is equipped with multiple cameras 20, and the object detection system is configured using the multiple cameras 20 and the main controller 30. However, this may be modified to configure the object detection system using one camera 20 and the main controller 30.

[0483] The technical concepts extracted from the above-described embodiments are described below. [Configuration 1] An object detection system comprising: a camera (20) mounted on a vehicle (10) for capturing images of the periphery of the vehicle at a predetermined interval; and a main controller (30) capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images captured by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) for causing the image sensor to capture images, the main controller comprises a parameter acquisition unit for acquiring vehicle-related parameters that indicate the status of the vehicle and its surroundings at the time of camera capture, the camera controller comprises: a parameter receiving unit for receiving the vehicle-related parameters from the main controller, an exposure condition setting unit for setting exposure conditions for the next capture based on the vehicle-related parameters received from the main controller, and an image capture execution unit for causing the image sensor to capture images based on the exposure conditions set by the exposure condition setting unit. [Configuration 2] The object detection system according to Configuration 1, wherein in the main controller, the parameter acquisition unit acquires, as the vehicle-related parameters, driving state parameters including at least one of the driving speed of the vehicle and attitude information of the vehicle, and in the camera controller, the parameter reception unit receives the driving state parameters from the main controller, and the exposure condition setting unit sets the exposure conditions based on the driving state parameters. [Configuration 3] The object detection system according to Configuration 1 or 2, wherein in the main controller, the parameter acquisition unit acquires, as the vehicle-related parameters, ambient environment parameters indicating an environment surrounding the vehicle, and in the camera controller, the parameter reception unit receives the ambient environment parameters from the main controller, and the exposure condition setting unit sets the exposure conditions based on the ambient environment parameters.[Configuration 4] The object detection system of any one of configurations 1 to 3, wherein in the camera controller, the exposure condition setting unit sets at least one of an exposure time and a gain of the image sensor as the exposure condition based on the vehicle-related parameters. [Configuration 5] The object detection system of any one of configurations 1 to 4, wherein in the main controller, the parameter acquisition unit determines transition parameters for transitioning the exposure conditions of the image sensor in accordance with changes in the situation of the vehicle and its surroundings during camera photography, and acquires the transition parameters as the vehicle-related parameters, and wherein in the camera controller, the parameter reception unit receives the transition parameters from the main controller, and the exposure condition setting unit sets the exposure conditions based on the transition parameters. [Configuration 6] The object detection system of any one of configurations 1 to 5, wherein the camera controller has a brightness information acquisition unit that acquires brightness information from a brightness measurement unit (25) that measures the brightness of an image captured by the image sensor, and the brightness measurement unit measures the brightness of the captured image at a cycle shorter than the cycle of transmitting the captured image to the main controller. [Configuration 7] The object detection system according to any one of configurations 1 to 6, wherein the camera controller has a brightness information acquisition unit that acquires brightness information from a brightness measurement unit (25) that measures the brightness of an image captured by the image sensor, and wherein software for measuring the brightness of the captured image in the brightness measurement unit is stored in a storage unit (26), and the storage unit is provided in the image sensor.[Configuration 8] The object detection system according to any one of Configurations 1 to 7, wherein the cameras include a first camera and a second camera, wherein the first camera has a first image sensor as the image sensor and a first camera controller as the camera controller, and the second camera has a second image sensor as the image sensor and a second camera controller as the camera controller, wherein the first image sensor and the second image sensor have different optical characteristics, and in the first camera controller and the second camera controller, the exposure condition setting unit sets the exposure conditions of each of the image sensors using a relationship between the vehicle-related parameters and the exposure conditions determined for each camera. [Configuration 9] The object detection system according to any one of Configurations 1 to 8, wherein the vehicle is equipped with a plurality of cameras each having a different shooting direction or mounting position, and the exposure condition setting unit in the camera controller of each of the cameras sets the exposure conditions based on the vehicle-related parameters received from the main controller and the shooting direction or mounting position of each of the cameras on the vehicle. [Configuration 10] The object detection system described in Configuration 9, wherein the plurality of cameras include a front camera that captures images ahead in the direction of travel of the vehicle and a rear camera that captures images behind the direction of travel of the vehicle, wherein in the main controller, the parameter acquisition unit acquires the vehicle's traveling speed as the vehicle-related parameter, and in the camera controller of each of the cameras, the exposure condition setting unit sets the exposure conditions based on the vehicle's traveling speed and the shooting direction of each of the cameras.[Configuration 11] The object detection system described in Configuration 9, wherein the vehicle is provided with wiper blades on the outside of a glass that separates the interior and exterior of the vehicle, and an interior-mounted camera is provided as one of the multiple cameras on the inside of the glass at a position that overlaps with the wiper wiping area of ​​the wiper blade, and in the main controller, the parameter acquisition unit acquires ambient environment parameters that indicate the ambient environment of the vehicle as the vehicle-related parameters, and in the camera controller of each of the cameras, the exposure condition setting unit sets the exposure conditions based on the ambient environment parameters and the fact that the camera is the interior-mounted camera. [Configuration 12] An object detection system comprising: a camera (20) mounted on a vehicle (10) that captures images of the area around the vehicle at a predetermined interval; and a main controller (30) capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images captured by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that controls the image sensor to capture images, the main controller comprising a parameter acquisition unit that acquires information regarding at least one of the ambient color of the area around the vehicle at the time of camera capture and incident light entering the camera as adjustment parameters for white balance adjustment, the camera controller comprising: a parameter reception unit that receives the adjustment parameters from the main controller, and a white balance adjustment unit that adjusts the white balance of the captured images to be transmitted to the main controller based on the adjustment parameters received from the main controller. [Configuration 13] The object detection system according to Configuration 12, wherein the white balance adjustment unit in the camera controller adjusts the white balance of the captured images at a interval shorter than the transmission interval of the captured images to the main controller.[Configuration 14] The object detection system according to any one of Configurations 12 to 14, wherein in the main controller, the parameter acquisition unit determines a transition parameter for transitioning an adjustment amount of the white balance of the captured image in accordance with a change in at least one of an ambient color around the vehicle and a color of incident light entering the camera, and acquires the transition parameter as the adjustment parameter, and in the camera controller, the parameter reception unit receives the transition parameter from the main controller, and the white balance adjustment unit adjusts the white balance of the captured image based on the transition parameter. [Configuration 15] The object detection system according to any one of Configurations 12 to 14, wherein the camera controller has a brightness information acquisition unit that acquires brightness information from a brightness measurement unit (25) that measures the brightness of the image captured by the image sensor, and the white balance adjustment unit adjusts the white balance of the captured image based on the adjustment parameter and the brightness information, and wherein the brightness measurement unit stores software for measuring the brightness of the captured image in a storage unit (26), and the storage unit is provided in the image sensor. [Configuration 16] An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the periphery of the vehicle at a predetermined interval; and a main controller (30) that can communicate with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to photograph, and the main controller comprises a parameter acquisition unit that acquires vehicle-related parameters that indicate the situation of the vehicle and the periphery of the vehicle at the time of camera photography, and the camera controller comprises: a parameter receiving unit that receives the vehicle-related parameters from the main controller, and a mask processing unit that performs mask processing on areas in the image photographed by the camera that are excluded from the detection target of the main controller, based on the vehicle-related parameters received from the main controller.[Configuration 17] The object detection system of Configuration 16, wherein, in the camera controller, the mask processing unit performs smoothing processing of the captured image to be transmitted to the main controller based on the vehicle-related parameters as the mask processing. [Configuration 18] The object detection system of Configuration 16, wherein, in the camera controller, the mask processing unit performs filling processing of the captured image to be transmitted to the main controller based on the vehicle-related parameters as the mask processing. [Configuration 19] The object detection system of any of Configurations 16 to 18, wherein, in the camera controller, the mask processing unit includes a region setting unit that sets a mask region in the captured image of the camera to be excluded from detection targets in the main controller based on the vehicle-related parameters. [Configuration 20] The object detection system of Configuration 19, wherein, in the main controller, the parameter acquisition unit acquires information indicating the speed or turning status of the vehicle as the vehicle-related parameters, and, in the camera controller, the region setting unit sets the mask region based on the information indicating the traveling speed or turning status of the vehicle. [Configuration 21] The object detection system of Configuration 19 or 20, wherein in the main controller, the parameter acquisition unit acquires, as the vehicle-related parameter, area information indicating that the area near the vehicle has a relatively large number or a relatively small number of objects to be detected, and in the camera controller, the area setting unit sets the mask area based on the area information. [Configuration 22] The object detection system of any of Configurations 16 to 21, wherein the masking is performed by image processing software, and a storage unit (26) storing the image processing software is provided in the image sensor. [Configuration 23] The object detection system of any of Configurations 16 to 22, wherein the masking is performed by an image mask calculation circuit, and the image mask calculation circuit is provided in the image sensor.[Configuration 24] An object detection system comprising: a camera (20) mounted on a vehicle (10) for capturing images of the periphery of the vehicle at a predetermined interval; and a main controller (30) capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images captured by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) for causing the image sensor to capture images, the camera controller comprising: a noise information acquisition unit for acquiring noise amount information indicating the amount of noise contained in images captured by the image sensor; and a noise information transmission unit for transmitting the noise amount information to the main controller, the main controller comprising: a noise level determination unit for determining a noise level at which noise is to be removed from the captured image, based on the noise amount information received from the camera controller and the captured image, the camera controller comprising: a noise level receiving unit for receiving the noise level from the main controller; and a noise removal unit for performing noise removal processing on the captured image to be transmitted to the main controller, based on the noise level received from the main controller and the acquired noise amount information. [Configuration 25] The object detection system of Configuration 24, wherein in the camera controller, the noise information acquisition unit acquires information indicating the amount of noise contained in the entire captured image as noise amount information, and in the main controller, the noise level determination unit determines the noise level based on the presence or absence of noise near the object to be detected in the captured image received from the camera controller and the noise amount information. [Configuration 26] The object detection system of Configuration 24 or 25, wherein in the camera controller, the noise information acquisition unit acquires the noise amount information at a cycle shorter than the cycle of transmitting the captured image to the main controller. [Configuration 27] The object detection system of any of Configurations 24 to 26, wherein the noise removal process is performed by image processing software, and a memory unit (26) in which the image processing software is stored is provided in the image sensor.[Configuration 28] An object detection system according to any one of configurations 24 to 27, wherein the main controller comprises: a scene information acquisition unit that acquires scene information relating to the driving scene of the vehicle when photographing with the camera; and a threshold information setting unit that sets threshold information for recognizing pixel defects in the photographed image based on the scene information; and the camera controller comprises: a defect repair unit that repairs pixel defects in the photographed image based on the threshold information received from the main controller; and the photographed image is transmitted to the main controller after the defect repair unit has repaired the defect. [Configuration 29] An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at a predetermined interval; and a main controller (30) that is capable of communicating with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to photograph, and the camera controller comprises: a vanishing point information acquisition unit that acquires vanishing point information, which is the position of a vanishing point in an image photographed by the image sensor or corresponding information corresponding to the position of the vanishing point; a readout range determination unit that determines the readout range of the photographed image based on the vanishing point information; and an image transmission unit that transmits an image of the readout range determined by the readout range determination unit to the main controller. [Configuration 30] The object detection system of Configuration 29, wherein the camera has a vanishing point search unit (195) that searches for the position of a vanishing point in an image from a captured image, and the vanishing point search unit searches for the vanishing point at a cycle shorter than the cycle of transmitting the captured image to the main controller. [Configuration 31] The object detection system of Configuration 29 or 30, wherein the camera has a vanishing point search unit (195) that searches for the position of a vanishing point in an image from a captured image, and software for searching for a vanishing point in the vanishing point search unit is stored in a memory unit (26), and the memory unit is provided in the image sensor.[Configuration 32] The object detection system of any of Configurations 29 to 31, wherein the main controller comprises an information acquisition unit that acquires detection target information, which is information about an object that is present at a position on the elevation angle side of the vehicle and that is to be detected, the camera controller comprises an information receiving unit that receives the detection target information from the main controller, and the readout range determination unit determines the readout range based on the detection target information and the vanishing point information received from the main controller. [Configuration 33] The object detection system of Configuration 32, wherein the camera is a front camera that captures an image of the area in front of the vehicle, and in the main controller, the information acquisition unit acquires, as the detection target information, that a traffic light is present in front of the vehicle as the detection target and that the vehicle is waiting for the traffic light to change, and in the camera controller, the readout range determination unit expands the readout range to the elevation angle side when it receives information from the main controller that the vehicle is waiting for the traffic light to change. [Configuration 34] An object detection system comprising: a camera (20) mounted on a vehicle (10) that captures images of the area around the vehicle at a predetermined time interval; and a main controller (30) capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images captured by the camera, wherein the main controller comprises: an abnormality detection unit that detects the occurrence of an abnormality in the vehicle or its surroundings; and a command transmission unit that transmits an image write command to the camera when the abnormality detection unit detects the occurrence of the abnormality, the camera comprising an image sensor (21), an image storage unit (23) capable of storing images captured by the image sensor, and a write processing unit that writes the captured images to the image storage unit when the write processing unit receives the image write command from the main controller. [Configuration 35] The object detection system according to Configuration 34, wherein the write processing unit writes the captured images to the image storage unit continuously at a predetermined time interval upon receiving the image write command.[Configuration 36] The object detection system of Configurations 34 or 35, wherein the write processing unit writes the captured images to the image storage unit at a cycle shorter than the cycle of transmitting the captured images to the main controller. [Configuration 37] The object detection system of any of Configurations 34 to 36, wherein software for performing the write processing in the write processing unit is stored in a software storage unit (26), and the storage unit is provided in the image sensor. [Configuration 38] The object detection system of any of Configurations 34 to 37, wherein the image storage unit is provided in the image sensor. [Configuration 39] The object detection system of any of Configurations 34 to 38, wherein the vehicle is equipped with a plurality of the cameras, and further comprises a camera identification unit that, when the abnormality detection unit detects the occurrence of the abnormal situation, identifies one of the plurality of cameras that is likely to have captured an image of the occurrence of the abnormal situation, and wherein, after receiving the image write command, the write processing unit writes the captured image to the image storage unit for the camera identified as the abnormal camera among the plurality of cameras. [Configuration 40] The object detection system according to any one of Configurations 34 to 39, wherein the vehicle is equipped with a plurality of the cameras, the abnormality detection unit is capable of detecting that an abnormal situation has occurred with a driver of the vehicle, and when the abnormality detection unit detects that an abnormal situation has occurred with the driver, all of the plurality of cameras write captured images to the image storage unit.[Configuration 41] An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the periphery of the vehicle at predetermined intervals; and a main controller (30) that can communicate with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21), a camera controller (22) that causes the image sensor to photograph, and a memory unit (26) that stores application software that can be executed by the camera controller, the application software stored in the memory unit can be written by either the main controller or the camera controller, and the main controller comprises an authentication unit that certifies that the application software written to the memory unit is genuine. [Configuration 42] The object detection system of Configuration 41, wherein the main controller comprises a software sending unit that encrypts the application software and sends it to the camera when rewriting the application software, and the camera controller comprises a rewriting execution unit that rewrites the application software stored in the storage unit, and the rewriting execution unit receives the encrypted application software from the main controller and decrypts the application software before writing it to the storage unit. [Configuration 43] The object detection system of Configuration 41 or 42, wherein the main controller comprises: a camera parameter receiving unit that receives, from the camera whose application software is to be rewritten, camera parameters indicating the installation status of the camera in the vehicle before the application software is rewritten, and a parameter returning unit that returns the camera parameters received by the camera parameter receiving unit after the application software is rewritten.[Configuration 44] The object detection system of any one of Configurations 41 to 43, wherein the main controller comprises an application abnormality determination unit that acquires images taken immediately before and immediately after software rewriting for the camera that is the target of software rewriting, and determines whether or not an abnormality has occurred due to the rewriting of the application software by comparing the images taken before and after the rewriting. [Configuration 45] The object detection system of any one of Configurations 41 to 44, wherein the main controller or the camera comprises a tampering determination unit that determines whether or not the application software stored in the memory unit has been tampered with. [Configuration 46] The object detection system of any one of Configurations 41 to 45, wherein the main controller comprises: a disabling unit that disables the authentication by the authentication unit based on a predetermined disabling command; and a rewrite permission unit that permits the rewriting of application software and the camera to take photos while authentication is disabled by the disabling unit. [Configuration 47] The object detection system of Configuration 41, wherein the main controller comprises: a software receiving unit that receives the application software from an external device; and a rewriting unit that accesses the memory unit on the camera side and rewrites the software in the memory unit with the application software received by the software receiving unit. [Configuration 48] The object detection system of any of Configurations 41 to 47, wherein the memory unit is provided in the image sensor. [Configuration 49] The object detection system of Configuration 41, wherein the main controller is able to access the memory unit on the camera side, and comprises a software writing unit that writes the application software to the memory unit each time the system is started up, and wherein the camera side executes an application using the application software written by the main controller after system start-up.[Configuration 50] The object detection system of Configuration 49, wherein in the camera, the memory unit includes a temporary memory area that is a temporary memory area, and in the main controller, the software writing unit writes the application software to the temporary memory area of ​​the memory unit at system startup. [Configuration 51] The object detection system of Configuration 49, wherein the camera remains operational even when the system is inactive, and the application software includes first software that is application software required throughout system operation and system inactivity, and second software that is application software required only when the system is in operation, the memory unit has a non-volatile memory area and a temporary memory area that is a temporary memory area, and the first software is stored in the non-volatile memory area and the second software is stored in the temporary memory area, and in the main controller, the software writing unit writes the second software to the temporary memory area at system startup. [Configuration 52] An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the periphery of the vehicle at predetermined intervals; and a main controller (30) that can communicate with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to photograph, and the camera controller comprises: a sleep-time photographing unit that causes the image sensor to photograph in a system sleep state in which object detection by the main controller is paused; a transmission determination unit that, when photographing is performed by the sleep-time photographing unit, determines whether or not the photographed image needs to be transmitted to the main controller; an event notification unit that, when it is determined by the transmission determination unit that image transmission to the main controller is required, issues an event notification to the main controller indicating that image transmission to the main controller will begin; and an image transmission unit that begins image transmission to the main controller after the event notification.[Configuration 53] The object detection system of Configuration 52, wherein communication modes between the communication unit of the main controller and the communication unit of the camera include a first communication mode in which the communication volume per unit time is relatively high and a second communication mode in which the communication volume per unit time is relatively low, and wherein in the camera controller, the event notification unit notifies the main controller in the second communication mode to start image transmission when the transmission determination unit determines that image transmission to the main controller is required, and the image transmission unit starts image transmission in the first communication mode after notifying that image transmission in the second communication mode. [Configuration 54] The object detection system of Configuration 52 or 53, wherein in the camera controller, the transmission determination unit detects the occurrence of an abnormal situation or a situation that may be abnormal in the vehicle or around the vehicle based on images captured by the idle time capture unit, and determines that image transmission to the main controller is required when the occurrence of an abnormal situation or a situation that may be abnormal is detected. [Configuration 55] The object detection system of any of Configurations 52 to 54, wherein in the camera controller, the transmission determination unit determines whether or not it is necessary to transmit the captured image captured by the pause image capture unit to the main controller side, using an image range excluding an area in which part of the vehicle is captured in the captured image. [Configuration 56] The object detection system of any of Configurations 52 to 55, wherein the main controller comprises: a mode transition unit that, when receiving the event notification from the camera controller in a system pause state, transitions to an image storage mode in which the captured image received from the camera side is stored in an image storage unit (33) on the main controller side; an image storage determination unit that, after transitioning to the image storage mode, determines whether or not it is necessary to continuously store the image received from the camera side; and an event stop notification unit that sends a notification to the camera controller that the event has stopped when it is determined by the image storage determination unit that it is not necessary to store the received image.[Configuration 57] In the camera controller, the transmission determination unit determines whether or not it is necessary to transmit the captured image to the main controller based on the captured image taken by the rest time capture unit, and the main controller includes a validity determination unit that performs image analysis of the captured image used by the transmission determination unit in the camera controller to determine whether or not it is necessary to transmit the image, and determines the validity of the transmission determination unit's determination that image transmission is necessary based on the analysis result, and the image storage determination unit determines whether or not it is necessary to continuously store the image received from the camera based on the validity determination result by the validity determination unit. This is an object detection system described in Configuration 56. [Configuration 58] The object detection system of Configuration 56 or 57, wherein the main controller has, as storage modes for storing images received from the camera side, a storage mode for storing image data in a temporary storage area that is a temporary storage area, and a storage mode for storing image data in a non-volatile storage area, a first storage processing unit that stores the received image in the temporary storage area immediately after switching to the image storage mode, and a second storage processing unit that stores a series of the received images including the captured image used for the image transmission determination unit's image transmission determination in the non-volatile storage area when the image storage determination unit determines that continuous storage of the received image is required. [Configuration 59] The object detection system of any of Configurations 52 to 58, wherein the main controller comprises: an exposure condition determination unit that determines exposure conditions for the camera after receiving the event notification from the camera controller in a system hibernation state, and an imaging command unit that transmits the exposure conditions determined by the exposure condition determination unit to the camera controller and causes the camera to take an image under the exposure conditions.[Configuration 60] The object detection system of any of Configurations 52 to 59, wherein in the camera controller, the transmission determination unit determines whether or not it is necessary to transmit a captured image to the main controller based on an image captured by the pause image capture unit, and the main controller comprises a determination condition setting unit that performs image analysis of the captured image used by the transmission determination unit in the camera controller to determine whether or not it is necessary to transmit an image, and variably sets a determination condition used by the transmission determination unit to determine whether or not it is necessary to transmit an image based on the analysis result. [Configuration 61] The object detection system of Configuration 60, wherein in the main controller, the determination condition setting unit determines whether a part of the vehicle is captured in the captured image received from the camera, and, if the vehicle is captured, sets the determination condition based on information about the part of the vehicle that is captured. [Configuration 62] The object detection system of any of Configurations 52 to 61, wherein in the camera controller, the transmission determination unit monitors a luminance change amount, which is a change in luminance per unit time within the field of view of the camera while the system is in a hibernation state, and determines whether or not the captured image needs to be transmitted to the main controller based on the luminance change amount, and the image transmission unit converts the luminance values ​​within the field of view of the camera into an image after the event notification and transmits the image to the main controller. [Configuration 63] The object detection system of any of Configurations 52 to 62, wherein a plurality of the cameras are mounted on the vehicle, and the main controller comprises: an image transmission command unit that, when receiving an event notification from any of the plurality of cameras and transitioning to the image storage mode, commands the remaining cameras other than the camera that issued the event notification to transmit captured images, and a storage processing unit that stores the captured images transmitted from the remaining cameras in a storage unit on the main controller side.[Configuration 64] The object detection system according to any one of Configurations 52 to 63, wherein the camera controller, in a system hibernation state, performs image capture using the hibernation image capture unit, but does not transmit the captured images to the main controller, and includes a periodic transmission unit that periodically transmits to the main controller a determination result of whether the image sensor is normal or abnormal. [Configuration 65] An object detection system comprising: a camera (20) mounted on a vehicle (10) that captures images of the periphery of the vehicle at a predetermined period; and a main controller (30) that is capable of communicating with the camera via wired or wireless communication and performs object detection processing based on the images captured by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that controls the image sensor to capture images, a communication unit of the main controller and a communication unit of the camera are capable of time-division bidirectional communication, and the camera controller includes a communication control unit that can change the proportion of time allocated to communication from the main controller to the camera controller. [Configuration 66] The object detection system of Configuration 65, wherein the communication control unit temporarily increases the proportion of time allocated to communication from the main controller to the camera controller compared to the proportion of time allocated to communication from the camera controller to the main controller when executing initialization processing accompanying startup of the camera. [Configuration 67] The object detection system of Configuration 66, wherein the communication control unit cancels the process of temporarily increasing the proportion of time allocated to communication from the main controller to the camera controller based on completion of the initialization processing. [Configuration 68] The object detection system of any of Configurations 65 to 67, wherein the communication unit of the main controller and the communication unit of the camera each have a serializer / deserializer, serializing parallel data on the transmitting side and parallelizing serial data on the receiving side. [Configuration 69] The object detection system of any of Configurations 65 to 68, wherein the camera side and the main controller side are connected using a coaxial cable as a communication line.

[0484] Although the present disclosure has been described with reference to the embodiments, it is understood that the present disclosure is not limited to the embodiments or structures. The present disclosure also encompasses various modifications and equivalent modifications. In addition, various combinations and forms, including only one element, more than one element, or less than one element, are also within the scope and spirit of the present disclosure.

Claims

1. An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at a predetermined interval; and a main controller (30) that is capable of communicating with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to photograph, and the main controller comprises a parameter acquisition unit that acquires vehicle-related parameters that indicate the situation of the vehicle and its surroundings at the time of camera photography, and the camera controller comprises: a parameter reception unit that receives the vehicle-related parameters from the main controller, an exposure condition setting unit that sets exposure conditions for the next photograph based on the vehicle-related parameters received from the main controller, and an photography execution unit that causes the image sensor to photograph based on the exposure conditions set by the exposure condition setting unit.

2. The object detection system of claim 1, wherein in the main controller, the parameter acquisition unit acquires driving state parameters including at least one of the vehicle's driving speed and attitude information of the vehicle as the vehicle-related parameters; and in the camera controller, the parameter reception unit receives the driving state parameters from the main controller; and the exposure condition setting unit sets the exposure conditions based on the driving state parameters.

3. The object detection system of claim 1, wherein in the main controller, the parameter acquisition unit acquires ambient environment parameters indicating the ambient environment of the vehicle as the vehicle-related parameters; and in the camera controller, the parameter reception unit receives the ambient environment parameters from the main controller; and the exposure condition setting unit sets the exposure conditions based on the ambient environment parameters.

4. An object detection system according to any one of claims 1 to 3, wherein in the camera controller, the exposure condition setting unit sets at least one of the exposure time and gain of the image sensor as the exposure condition based on the vehicle-related parameters.

5. An object detection system as described in any one of claims 1 to 3, wherein in the main controller, the parameter acquisition unit determines transition parameters that transition the exposure conditions of the image sensor in accordance with changes in the situation of the vehicle and its surroundings when photographing with the camera, and acquires these transition parameters as the vehicle-related parameters; and in the camera controller, the parameter reception unit receives the transition parameters from the main controller; and the exposure condition setting unit sets the exposure conditions based on the transition parameters.

6. An object detection system according to any one of claims 1 to 3, wherein the camera controller has a brightness information acquisition unit that acquires brightness information from a brightness measurement unit (25) that measures the brightness of an image captured by the image sensor, and the brightness measurement unit measures the brightness of the captured image at a cycle shorter than the cycle of transmitting the captured image to the main controller.

7. An object detection system according to any one of claims 1 to 3, wherein the camera controller has a brightness information acquisition unit that acquires brightness information from a brightness measurement unit (25) that measures the brightness of an image captured by the image sensor, and wherein software for measuring the brightness of the captured image in the brightness measurement unit is stored in a memory unit (26), and the memory unit is provided in the image sensor.

8. An object detection system according to any one of claims 1 to 3, comprising a first camera and a second camera as the cameras, the first camera having a first image sensor as the image sensor and a first camera controller as the camera controller, the second camera having a second image sensor as the image sensor and a second camera controller as the camera controller, the first image sensor and the second image sensor having different optical characteristics, and in the first camera controller and the second camera controller, the exposure condition setting unit sets the exposure conditions of each of the image sensors using the relationship between the vehicle-related parameters and the exposure conditions defined for each camera.

9. An object detection system as described in any one of claims 1 to 3, wherein the vehicle is equipped with a plurality of cameras each having a different shooting direction or mounting position, and in the camera controller of each of the cameras, the exposure condition setting unit sets the exposure conditions based on the vehicle-related parameters received from the main controller and the shooting direction or mounting position of each of the cameras on the vehicle.

10. The object detection system described in claim 9, wherein the plurality of cameras include a front camera that takes pictures in front of the vehicle in the direction of travel and a rear camera that takes pictures in the rear of the vehicle in the direction of travel, and in the main controller, the parameter acquisition unit acquires the vehicle's traveling speed as the vehicle-related parameter, and in the camera controller of each camera, the exposure condition setting unit sets the exposure conditions based on the vehicle's traveling speed and the shooting direction of each camera.

11. The object detection system described in claim 9, wherein the vehicle is provided with wiper blades on the outside of a glass that separates the interior and exterior of the vehicle, and an interior-mounted camera is provided as one of the multiple cameras on the inside of the glass in a position that overlaps with the wiper wiping area of ​​the wiper blade, and in the main controller, the parameter acquisition unit acquires ambient environment parameters that indicate the ambient environment of the vehicle as the vehicle-related parameters, and in the camera controller of each camera, the exposure condition setting unit sets the exposure conditions based on the ambient environment parameters and the fact that the camera is an interior-mounted camera.

12. An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at a predetermined interval; and a main controller (30) that can communicate with the camera via wired or wireless communication and performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that controls the image sensor to photograph; the main controller comprises a parameter acquisition unit that acquires information regarding at least one of the ambient color of the area around the vehicle when photographing with the camera and the incident light entering the camera as adjustment parameters for white balance adjustment; and the camera controller comprises: a parameter reception unit that receives the adjustment parameters from the main controller; and a white balance adjustment unit that adjusts the white balance of the photographed image to be sent to the main controller based on the adjustment parameters received from the main controller.

13. The object detection system according to claim 12, wherein in the camera controller, the white balance adjustment unit adjusts the white balance of the captured image at a cycle shorter than the cycle at which the captured image is transmitted to the main controller.

14. The object detection system described in claim 12, wherein in the main controller, the parameter acquisition unit determines a transition parameter that transitions the amount of adjustment of the white balance of the captured image in accordance with changes in at least one of the ambient color around the vehicle and the color of the incident light entering the camera, and acquires this transition parameter as the adjustment parameter; and in the camera controller, the parameter receiving unit receives the transition parameter from the main controller, and the white balance adjustment unit adjusts the white balance of the captured image based on the transition parameter.

15. An object detection system as described in any one of claims 12 to 14, wherein the camera controller has a brightness information acquisition unit that acquires brightness information from a brightness measurement unit (25) that measures the brightness of an image captured by the image sensor, the white balance adjustment unit adjusts the white balance of the captured image based on the adjustment parameters and the brightness information, and in the brightness measurement unit, software that measures the brightness of the captured image is stored in a memory unit (26), and the memory unit is provided in the image sensor.

16. An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at a predetermined interval; and a main controller (30) that can communicate with the camera via wired or wireless communication and performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to photograph, and the main controller comprises a parameter acquisition unit that acquires vehicle-related parameters that indicate the situation of the vehicle and its surroundings at the time of camera photography, and the camera controller comprises: a parameter receiving unit that receives the vehicle-related parameters from the main controller, and a mask processing unit that performs mask processing on areas in the image photographed by the camera that are excluded from detection targets of the main controller, based on the vehicle-related parameters received from the main controller.

17. The object detection system according to claim 16, wherein in the camera controller, the mask processing unit performs, as the mask processing, smoothing processing of the captured image to be transmitted to the main controller based on the vehicle-related parameters.

18. The object detection system according to claim 16, wherein in the camera controller, the mask processing unit performs, as the mask processing, a filling-in process of the captured image to be transmitted to the main controller based on the vehicle-related parameters.

19. An object detection system according to any one of claims 16 to 18, wherein in the camera controller, the mask processing unit includes an area setting unit that sets a mask area within the image captured by the camera that is excluded from detection targets in the main controller based on the vehicle-related parameters.

20. An object detection system as described in claim 19, wherein in the main controller, the parameter acquisition unit acquires information indicating the speed or turning status of the vehicle as the vehicle-related parameter, and in the camera controller, the area setting unit sets the mask area based on information indicating the traveling speed or turning status of the vehicle.

21. The object detection system described in claim 19, wherein in the main controller, the parameter acquisition unit acquires area information indicating that the area near the vehicle has a relatively large number or a relatively small number of objects to be detected as the vehicle-related parameter, and in the camera controller, the area setting unit sets the mask area based on the area information.

22. An object detection system according to any one of claims 16 to 18, wherein the masking process is performed by image processing software, and a memory unit (26) in which the image processing software is stored is provided in the image sensor.

23. The object detection system according to any one of claims 16 to 18, wherein the mask processing is performed by an image mask calculation circuit, and the image mask calculation circuit is provided in the image sensor.

24. An object detection system comprising: a camera (20) mounted on a vehicle (10) that captures images of the area around the vehicle at predetermined intervals; and a main controller (30) that is capable of communicating with the camera via wired or wireless communication and performs object detection processing based on images captured by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that controls the image sensor to capture images, the camera controller comprising: a noise information acquisition unit that acquires noise amount information indicating the amount of noise contained in images captured by the image sensor; and a noise information transmission unit that transmits the noise amount information to the main controller, the main controller comprising: a noise level determination unit that determines a noise level to perform noise reduction on the captured image based on the noise amount information received from the camera controller and the captured image, the camera controller comprising: a noise level reception unit that receives the noise level from the main controller; and a noise reduction unit that performs noise reduction processing on the captured image to be transmitted to the main controller based on the noise level received from the main controller and the acquired noise amount information.

25. An object detection system as described in claim 24, wherein in the camera controller, the noise information acquisition unit acquires information indicating the amount of noise contained in the entire captured image as noise amount information, and in the main controller, the noise level determination unit determines the noise level based on the presence or absence of noise near the object to be detected in the captured image received from the camera controller and the noise amount information.

26. The object detection system according to claim 24, wherein in the camera controller, the noise information acquisition unit acquires the noise amount information at a cycle shorter than the cycle of transmitting captured images to the main controller.

27. The object detection system according to claim 24, wherein the noise removal process is performed by image processing software, and a memory unit (26) in which the image processing software is stored is provided in the image sensor.

28. An object detection system as described in any one of claims 24 to 27, wherein the main controller comprises: a scene information acquisition unit that acquires scene information related to the driving scene of the vehicle when photographing with the camera; and a threshold information setting unit that sets threshold information for recognizing pixel defects in the photographed image based on the scene information; and the camera controller comprises: a defect repair unit that repairs pixel defects in the photographed image based on the threshold information received from the main controller; and the photographed image is transmitted to the main controller after the defects have been repaired by the defect repair unit.

29. An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at predetermined intervals; and a main controller (30) that can communicate with the camera via wired or wireless communication and performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to photograph, and the camera controller comprises: a vanishing point information acquisition unit that acquires vanishing point information, which is the position of a vanishing point in an image photographed by the image sensor or corresponding information corresponding to the position of the vanishing point; a readout range determination unit that determines the readout range of the photographed image based on the vanishing point information; and an image transmission unit that transmits an image of the readout range determined by the readout range determination unit to the main controller.

30. The object detection system of claim 29, wherein the camera has a vanishing point search unit (195) that searches for the position of a vanishing point in an image from a captured image, and the vanishing point search unit searches for the vanishing point at a cycle shorter than the cycle at which the captured image is transmitted to the main controller.

31. An object detection system as described in claim 29, wherein the camera has a vanishing point search unit (195) that searches for the position of a vanishing point in an image from a captured image, and software for searching for a vanishing point in the vanishing point search unit is stored in a memory unit (26), and the memory unit is provided in the image sensor.

32. An object detection system as described in any one of claims 29 to 31, wherein the main controller comprises an information acquisition unit that acquires detection target information, which is information relating to an object that is located at an elevation angle from the vehicle and is to be detected; the camera controller comprises an information receiving unit that receives the detection target information from the main controller; and the readout range determination unit determines the readout range based on the detection target information and the vanishing point information received from the main controller.

33. The object detection system described in claim 32, wherein the camera is a forward camera that captures images in front of the vehicle, and in the main controller, the information acquisition unit acquires, as the detection target information, that a traffic light is present in front of the vehicle as the detection target and that the vehicle is waiting at a traffic light, and in the camera controller, the readout range determination unit expands the readout range toward the elevation angle when it receives information from the main controller that the vehicle is waiting at a traffic light.

34. An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at predetermined intervals; and a main controller (30) that is capable of communicating with the camera via wired or wireless communication and that performs object detection processing based on images photographed by the camera, wherein the main controller comprises: an abnormality detection unit that detects the occurrence of an abnormality in the vehicle or its surroundings; and a command transmission unit that transmits an image write command to the camera when the abnormality detection unit detects the occurrence of the abnormality; and the camera comprises an image sensor (21), an image storage unit (23) that can store images photographed by the image sensor, and a write processing unit that writes the photographed image to the image storage unit, and when the write processing unit receives the image write command from the main controller, it writes the photographed image to the image storage unit.

35. The object detection system according to claim 34, wherein the write processing unit writes the captured images to the image storage unit continuously at predetermined time intervals in response to receiving the image write command.

36. The object detection system according to claim 34, wherein the write processing unit writes the captured images to the image storage unit at a cycle shorter than the cycle at which the captured images are transmitted to the main controller.

37. The object detection system according to claim 34, wherein in the write processing section, software for performing the write processing is stored in a memory section (26) for storing software, and the memory section is provided in the image sensor.

38. The object detection system according to claim 34, wherein the image storage unit is provided in the image sensor.

39. An object detection system as described in any one of claims 34 to 38, wherein the vehicle is equipped with a plurality of the cameras, and a camera identification unit is provided that, when the abnormality detection unit detects that an abnormal situation has occurred, identifies, from among the plurality of cameras, a camera that is likely to have captured an image of the occurrence of the abnormal situation, and after receiving the image write command, the write processing unit writes the captured image to the image storage unit for the camera identified as the camera that is the abnormally captured image.

40. An object detection system as described in any one of claims 34 to 38, wherein the vehicle is equipped with a plurality of the cameras, the abnormality detection unit is capable of detecting that an abnormal situation has occurred with the driver of the vehicle, and when the abnormality detection unit detects that an abnormal situation has occurred with the driver, all of the plurality of cameras write captured images to the image storage unit.

41. An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at predetermined intervals; and a main controller (30) that can communicate with the camera via wired or wireless communication and performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21), a camera controller (22) that causes the image sensor to photograph, and a memory unit (26) that stores application software executable by the camera controller, the application software stored in the memory unit can be written by either the main controller or the camera controller, and the main controller comprises an authentication unit that certifies that the application software written in the memory unit is genuine.

42. The object detection system described in claim 41, wherein the main controller includes a software transmission unit that encrypts the application software and transmits it to the camera when the application software is rewritten, and the camera controller includes a rewrite execution unit that rewrites the application software stored in the memory unit, and the rewrite execution unit receives the encrypted application software from the main controller and decrypts the application software before writing it to the memory unit.

43. The object detection system described in claim 41, wherein the main controller comprises: a camera parameter receiving unit that receives, before the application software is rewritten, camera parameters indicating the installation status of the camera in the vehicle from the camera whose application software is to be rewritten; and a parameter returning unit that returns the camera parameters received by the camera parameter receiving unit after the application software is rewritten.

44. The object detection system of claim 41, wherein the main controller is provided with an application abnormality determination unit that acquires images taken immediately before and immediately after the software rewrite for the camera to be rewritten with the application software, and compares the images taken before and after the rewrite to determine whether or not an abnormality has occurred due to the rewriting of the application software.

45. The object detection system according to claim 41, wherein the main controller or the camera includes a tampering determination unit that determines whether or not the application software stored in the memory unit has been tampered with.

46. ​​The object detection system of claim 41, wherein the main controller comprises: an invalidation unit that invalidates the authentication by the authentication unit based on a predetermined invalidation command; and an overwrite permission unit that permits rewriting of application software and photography by the camera when authentication is invalidated by the invalidation unit.

47. The object detection system of claim 41, wherein the main controller comprises: a software receiving unit that receives the application software from an external device; and a rewriting unit that accesses the memory unit on the camera side and rewrites the software in the memory unit with the application software received by the software receiving unit.

48. The object detection system according to claim 41, wherein the memory unit is provided in the image sensor.

49. An object detection system as described in any one of claims 41 to 48, wherein the main controller is capable of accessing the memory unit on the camera side and is provided with a software writing unit that writes the application software to the memory unit each time the system is started up, and wherein, on the camera side, after the system is started up, an application is executed using the application software written by the main controller.

50. The object detection system described in claim 49, wherein in the camera, the memory unit includes a temporary memory area that is a temporary storage area, and in the main controller, the software writing unit writes the application software to the temporary memory area of ​​the memory unit when the system is started up.

51. The object detection system described in claim 49, wherein the camera remains operational even when the system is inactive; the application software includes first software, which is application software required both when the system is in operation and when the system is inactive, and second software, which is application software required only when the system is in operation; the memory unit has a non-volatile memory area and a temporary memory area, which is a temporary memory area, and the first software is stored in the non-volatile memory area and the second software is stored in the temporary memory area; and in the main controller, the software writing unit writes the second software to the temporary memory area when the system is started.

52. An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at predetermined intervals; and a main controller (30) that can communicate with the camera via wired or wireless communication and performs object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to photograph, and the camera controller comprises: a sleep-time photographing unit that causes the image sensor to photograph when the system is in a sleep state in which object detection by the main controller is paused; a transmission determination unit that, when photographing is performed by the sleep-time photographing unit, determines whether or not the photographed image needs to be transmitted to the main controller; an event notification unit that, when the transmission determination unit determines that image transmission to the main controller is required, issues an event notification to the main controller indicating that image transmission to the main controller will begin; and an image transmission unit that begins image transmission to the main controller after the event notification.

53. An object detection system as described in claim 52, wherein the communication modes between the communication unit of the main controller and the communication unit of the camera include a first communication mode in which the communication volume per unit time is relatively high and a second communication mode in which the communication volume per unit time is relatively low, and in the camera controller, when the transmission determination unit determines that image transmission to the main controller side is required, the event notification unit notifies the main controller side in the second communication mode that image transmission will begin, and the image transmission unit begins image transmission in the first communication mode after being notified that image transmission will be performed in the second communication mode.

54. An object detection system as described in claim 52, wherein in the camera controller, the transmission determination unit detects the occurrence of an abnormal situation or a situation that may be abnormal in the vehicle or around the vehicle based on the captured image taken by the idle capture unit, and when the occurrence of an abnormal situation or a situation that may be abnormal is detected, determines that an image needs to be transmitted to the main controller.

55. An object detection system as described in claim 52, wherein in the camera controller, the transmission determination unit determines whether or not to transmit the captured image to the main controller side using an image range excluding an area in which part of the vehicle is captured in the captured image taken by the resting time capture unit.

56. The object detection system of claim 52, wherein the main controller comprises: a mode transition unit that, when receiving the event notification from the camera controller in a system hibernation state, transitions to an image storage mode in which the captured image received from the camera side is stored in an image storage unit (33) on the main controller side; an image storage determination unit that, after transitioning to the image storage mode, determines whether or not it is necessary to continuously store the image received from the camera side; and an event stop notification unit that, when it is determined by the image storage determination unit that it is not necessary to store the received image, sends a notification to the camera controller that the event has stopped.

57. The object detection system described in claim 56, wherein in the camera controller, the transmission determination unit determines whether or not it is necessary to transmit the captured image to the main controller based on the captured image taken by the resting time capture unit, the main controller comprises a validity determination unit that performs image analysis of the captured image used by the transmission determination unit in the camera controller to determine whether or not it is necessary to transmit the image, and determines based on the analysis results whether or not the transmission determination unit has determined that image transmission is necessary, and the image storage determination unit determines whether or not it is necessary to continuously store the image received from the camera based on the validity determination result by the validity determination unit.

58. The object detection system of claim 56, wherein the main controller has, as storage modes for storing images received from the camera side, a storage mode for storing image data in a temporary storage area that is a temporary storage area, and a storage mode for storing image data in a non-volatile storage area; and further comprises: a first storage processing unit that stores the received images in the temporary storage area immediately after transitioning to the image storage mode; and a second storage processing unit that stores a series of the received images, including the captured image used for the image transmission determination unit's image transmission determination, in the non-volatile storage area when the image storage determination unit determines that continuous storage of the received images is required.

59. An object detection system as described in any one of claims 52 to 58, wherein the main controller comprises: an exposure condition determination unit that determines the exposure conditions of the camera after receiving the event notification from the camera controller while the system is in a hibernation state; and an imaging command unit that transmits the exposure conditions determined by the exposure condition determination unit to the camera controller and causes the camera to take an image under those exposure conditions.

60. An object detection system as described in any one of claims 52 to 58, wherein in the camera controller, the transmission determination unit determines whether or not it is necessary to transmit a captured image to the main controller based on the captured image captured by the idle capture unit, and the main controller is provided with a determination condition setting unit that performs image analysis of the captured image used by the transmission determination unit in the camera controller to determine whether or not it is necessary to transmit an image, and variably sets the determination conditions used by the transmission determination unit to determine whether or not it is necessary to transmit an image based on the analysis results.

61. An object detection system as described in claim 60, wherein in the main controller, the judgment condition setting unit determines whether a part of the vehicle is reflected in the captured image received from the camera, and, if the vehicle is reflected, sets the judgment condition based on information about the part of the vehicle that is reflected.

62. An object detection system as described in any one of claims 52 to 58, wherein in the camera controller, the transmission determination unit monitors the amount of change in brightness, which is the amount of change in brightness per unit time within the field of view of the camera while the system is in a hibernation state, and determines whether or not the captured image needs to be transmitted to the main controller based on the amount of change in brightness, and the image transmission unit, after receiving the event notification, converts the brightness values ​​within the field of view of the camera into an image and transmits the image to the main controller.

63. An object detection system as described in any one of claims 52 to 58, wherein the vehicle is equipped with a plurality of the cameras, and the main controller comprises: an image transmission command unit that, after receiving an event notification from one of the plurality of cameras, commands the remaining cameras other than the camera that issued the event notification to transmit captured images; and a memory processing unit that stores the captured images transmitted from the remaining cameras in a memory unit on the main controller side.

64. An object detection system as described in any one of claims 52 to 58, wherein the camera controller, when in a system hibernation state, performs image capture using the hibernation capture unit, but does not transmit the captured image to the main controller, and is provided with a periodic transmission unit that periodically transmits to the main controller the result of determining whether the image sensor is normal or abnormal.

65. An object detection system comprising: a camera (20) mounted on a vehicle (10) that photographs the area around the vehicle at a predetermined interval; and a main controller (30) capable of communicating with the camera via wired or wireless communication and performing object detection processing based on images photographed by the camera, wherein the camera comprises an image sensor (21) and a camera controller (22) that causes the image sensor to photograph; the communication unit of the main controller and the communication unit of the camera are capable of time-division two-way communication; and the camera controller comprises a communication control unit that can change the proportion of time allocated to communication from the main controller to the camera controller.

66. An object detection system as described in claim 65, wherein the communication control unit temporarily increases the proportion of time allocated to communication from the main controller to the camera controller compared to the proportion of time allocated to communication from the camera controller to the main controller when executing initialization processing accompanying startup of the camera.

67. The object detection system described in claim 66, wherein the communication control unit cancels the process of temporarily increasing the proportion of time allocated to communication from the main controller to the camera controller based on the completion of the initialization process.

68. An object detection system described in any one of claims 65 to 67, wherein the communication unit of the main controller and the communication unit of the camera each have a serializer / deserializer, serializing parallel data on the transmitting side and parallelizing serial data on the receiving side.

69. An object detection system according to any one of claims 65 to 67, wherein the camera side and the main controller side are connected using a coaxial cable as a communication line.

Images