CAMERA SYSTEM, MOBILE BODY, CAMERA SYSTEM CONTROL METHOD, AND COMPUTER PROGRAM

Abstract

To provide a camera system that can perform optimal gradation compression processing according to an imaging mode.SOLUTION: A camera system comprises: an imaging system that picks up an image of a subject to generate an image signal, performs gradation compression processing on the image signal based on two or more input-output conversion parameters, and having imaging system control means that sets the input-output conversion parameter; and an image processing system that acquires an image from the imaging system in two or more imaging modes. The control means of the imaging system sets the input-output conversion parameter to gradation compression means based on the imaging mode in the image processing system.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a camera system, a moving object, a control method for a camera system, a computer program, and the like. [Background technology]

[0002] Many recent in-vehicle camera systems achieve dynamic range expansion (HDR) by synthesizing multiple images taken under different exposure conditions. A typical example of dynamic range expansion is a technology that synthesizes multiple images taken with different exposure times to generate a single image with expanded dynamic range. Specifically, a single composite image is generated by synthesizing low-luminance pixels from an image taken with a long exposure time and high-luminance pixels from an image taken with a short exposure time. By performing this type of processing, it is possible to obtain a high dynamic range image that retains the gradation in the dark areas and can also express bright subjects without blown-out highlights.

[0003] However, when multiple images are synthesized using this dynamic range expansion technology, the bit depth of the luminance value becomes large, and the transmission rate becomes enormous. Since the transmission bandwidth of the camera output is limited, it is essential to reduce the data amount of the synthesized image in order to output the synthesized image generated using the dynamic range expansion technology. Furthermore, such an increase in data amount is a problem not only due to the dynamic range expansion but also due to the high resolution of the camera.

[0004] Patent Document 1 proposes a technology for compressing image data by setting a partial area within an image capture area based on the vehicle's running state and lowering the resolution outside the partial area. Patent Document 2 also proposes a technology for detecting moving objects early and reducing the load of detection processing by generating multiple reduced images by compressing an original image captured by a camera in stages and detecting moving objects in the reduced images in order of the compression level. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-56838 A [Patent Document 2] JP 2014-146135 A Summary of the Invention [Problem to be solved by the invention]

[0006] By reducing the resolution using the techniques of Patent Document 1 and Patent Document 2, it is possible to reduce the amount of data in the output image, but the reduction in resolution reduces image quality performance for viewing and object recognition performance for sensing, etc. Therefore, there is a possibility that the performance may be adversely affected with respect to the original purpose (visual recognition of images, object recognition, etc.).

[0007] In response to this, it is possible to compress the gradations by, for example, allocating more gradations to low-luminance pixels and less gradations to high-luminance pixels in accordance with the characteristics of human vision. However, for example, when an autonomous vehicle performs sensing using an on-board camera, if many gradations are assigned to low-luminance pixels in the same way as the input / output conversion parameters during viewing, fewer gradations are assigned to the middle to high luminance range. This may result in a decrease in sensing performance for objects in the middle to high luminance range. In particular, if the autonomous vehicle misrecognizes objects in the middle to high luminance range, such as traffic lights and road signs, during autonomous driving, it may erroneously estimate its own position or cause an accident.

[0008] SUMMARY OF THE PRESENT EMBODIMENTS An object of the present invention is to provide a camera system capable of performing optimal tone compression processing according to the imaging mode. [Means for solving the problem]

[0009] In order to achieve the above object, the camera system according to the present application comprises: an imaging means for imaging an object and generating an image signal; a gradation compression means for performing a gradation compression process on the image signal based on two or more input / output conversion parameters; an imaging system having an imaging system control means for setting the input / output conversion parameters in the gradation compression means; A first imaging mode for displaying the captured image to a user driving in a manual driving mode and a second imaging mode for performing recognition using image features in an automatic driving mode. and an image processing system for acquiring an image from the imaging system; The imaging system control means In the gradation compression means, the input / output conversion parameters for viewing are set in the first imaging mode, and the input / output conversion parameters for sensing are set in the second imaging mode. The present invention is characterized by the above. Effect of the Invention

[0010] According to the present invention, a camera system capable of performing optimal tone compression processing according to an image capture mode can be realized. [Brief description of the drawings]

[0011] [Figure 1] 1 is a functional block diagram of an in-vehicle camera system 100 according to a first embodiment. [Diagram 2] 2 is a conceptual diagram for explaining an exposure control method according to the first embodiment. FIG. [Diagram 3] FIG. 1A is a functional block diagram showing an outline of the configuration of an HDR synthesis unit 105 in the first embodiment, FIG. 1B is a diagram for explaining processing in a luminance correction unit 303, and FIG. 1C is a diagram for explaining a method for generating a synthetic image by an image synthesis unit 304 in the first embodiment. [Figure 4] FIG. 1A is a functional block diagram showing an outline of the configuration of the gradation compression unit 106 of the first embodiment, and FIG. 1B is a diagram showing an example of parameter characteristics of input / output conversion parameters stored in a viewing parameter storage unit 401. FIG. 1C is a diagram showing an example of parameter characteristics of input / output conversion parameters stored in a sensing parameter storage unit 402. [Diagram 5] 4 is a flowchart for explaining a control flow of the image processing system according to the first embodiment. [Figure 6] 13 is a conceptual diagram of a case where a transition parameter is used when switching between input / output conversion parameters used in the gradation compressor 106 of the first embodiment. FIG. [Figure 7] FIG. 7 is a functional block diagram of an in-vehicle camera system 700 according to a second embodiment. [Figure 8] 10 is a flowchart for explaining a control flow of an image processing system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0013] <First embodiment> 1 is a functional block diagram of an in-vehicle camera system 100 in the embodiment 1. The in-vehicle camera system 100 is composed of two systems, an imaging system 101 and an image processing system 102. In this embodiment, the in-vehicle camera system 100 is mounted on an automobile as a moving body, but the moving body is not limited to an automobile.

[0014] 1 are realized by causing a computer (not shown) included in the vehicle-mounted camera system 100 to execute a computer program stored in a memory (not shown) serving as a storage medium. However, some or all of these may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP), etc., may be used.

[0015] The functional blocks in the imaging system 101 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. Similarly, the functional blocks in the image processing system 102 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

[0016] First, a description will be given of the imaging system 101. The imaging system 101 is composed of an imaging system control unit 103, an imaging unit 104, an HDR synthesis unit 105, a gradation compression unit 106, and the like. The imaging system control unit 103 controls the entire imaging system 101. It mainly controls the imaging unit 104, HDR synthesis unit 105, and gradation compression unit 106, and exchanges information with the image processing system 102. The imaging system control unit 103 also has a built-in CPU as a computer, and functions as a control unit that controls the operation of each unit of the imaging system based on a computer program stored in a memory as a storage medium. Details will be described later.

[0017] The imaging unit 104 includes an optical system (not shown) and an imaging element such as a CMOS image sensor that captures an optical image of a subject formed by the optical system. The imaging element includes a plurality of pixels, and each pixel includes a photoelectric conversion element (photodiode) that photoelectrically converts incident light. The imaging unit 104 has an AD conversion function that converts an analog electrical signal output from the photoelectric conversion element into a digital signal. The imaging unit 104 also has a function of capturing an image of a subject and generating an image signal, as well as controlling an exposure time according to an exposure control signal input from the imaging system control unit 103. The gain of the imaging element may be configured to be switchable. In this embodiment, the image capturing unit 104 is configured with a plurality of cameras arranged, for example, at the front, rear, left and right of an automobile as a moving body, for capturing images of subjects around the automobile.

[0018] FIG. 2 is a conceptual diagram for explaining the exposure control method of the first embodiment, and the exposure control method in this embodiment will be explained with reference to FIG. 2, VD is a signal indicating the start timing of each frame. A VD control signal is transmitted from the imaging system control unit 103 to the imaging unit 104, and when the VD signal rises, control such as resetting and reading for acquiring one frame of image is started.

[0019] The Reset signal is a control signal for connecting the photoelectric conversion element to a reset power supply and setting the potential of the photoelectric conversion element to a reset (reference) potential, and reset control is executed while the Reset signal is High. The Read signal is a control signal for reading out the electric charge generated in the photoelectric conversion element, and the charge read control starts at the timing when the Read signal rises.

[0020] The exposure period is the period from the falling edge of Reset to the rising edge of Read, and indicates the period during which charge is accumulated while the photoelectric conversion element is irradiated with light. The amount of charge generated in the photoelectric conversion element changes depending on the length of the exposure time. In other words, the level of output luminance for incident light changes depending on the length of the exposure time. In this embodiment, the first frame is photographed with a short exposure time, and the following second frame is photographed with a long exposure time.

[0021] From the third frame onwards, exposure control is performed by alternating between short and long exposure for each frame. In this embodiment, two types of exposure conditions, long and short exposure, are used for explanation, but three or more types of exposure conditions may be combined. By increasing the number of exposure conditions, the function of the dynamic range expansion process can be improved.

[0022] The period shown in the graph of the sensor output in FIG. 2 indicates a period during which the imaging unit 104 converts the electrical signal read from the photoelectric conversion element in accordance with the read control into a digital luminance signal, and further outputs the digital luminance signal to the HDR synthesis unit 105.

[0023] The HDR synthesis unit 105 is for performing HDR synthesis processing using two image signals acquired under two different exposure conditions from the imaging unit 104. The HDR synthesis method in this embodiment will be described with reference to FIG.

[0024] 3A is a functional block diagram showing an outline of the configuration of the HDR synthesis unit 105 of the embodiment 1. The HDR synthesis unit 105 includes a long-second exposure image memory 301, a short-second exposure image memory 302, a luminance correction unit 303, an image synthesis unit 304, and the like.

[0025] Of the image signals output from the imaging unit 104, images acquired with long-second exposure are stored in a long-second exposure image memory 301. Images acquired with short-second exposure are stored in a short-second exposure image memory 302. A luminance correction unit 303 performs correction to match the luminance level of the image acquired with short-second exposure to a luminance level equivalent to that of an image acquired with long-second exposure. This correction makes it possible to seamlessly stitch together and synthesize the image acquired with long-second exposure and the image acquired with short-second exposure.

[0026] 3B is a diagram for explaining the processing in the luminance correction unit 303. In FIG. 3B, the horizontal axis indicates the amount of light of the subject, and the vertical axis indicates the output luminance of the imaging unit 104. 3A is a graph showing the output luminance versus the amount of light on a subject during long-second exposure, 3B is a graph showing the output luminance versus the amount of light on a subject during short-second exposure, and 3B' is a graph showing the output luminance of 3B after luminance correction processing has been performed by luminance correction unit 303. Also, Sat on the vertical axis indicates the saturation level of the photoelectric conversion element.

[0027] Since 3A has a long exposure time, it reaches the Sat level immediately when the amount of light on the subject increases. Since 3B has a short exposure time, it reaches the Sat level after the amount of light on the subject increases more than 3A. The luminance correction unit 303 matches the luminance outputs of 3A and 3B by multiplying the luminance output of 3B by a predetermined gain. Specifically, the gain is calculated by dividing the exposure time during long exposure by the exposure time during short exposure, and the luminance output of the short exposure image is multiplied by this gain to make the luminance output of the image acquired with long exposure and the luminance output of the image acquired with short exposure equivalent in luminance value.

[0028] The image synthesis unit 304 synthesizes the luminance output of 3A and the luminance output of 3B' to generate a single image with expanded dynamic range by synthesizing the long-exposure image and the short-exposure image. Fig. 3(C) is a diagram for explaining a method for generating a composite image by the image synthesis unit 304 in the first embodiment. In Fig. 3(C), the horizontal axis represents input luminance, and the vertical axis represents the synthesis ratio of the long second exposure image and the short second exposure image. Furthermore, the luminances α to β on the horizontal axis are luminances near the switching point between the long second exposure image and the short second exposure image, and ideally, it is preferable to set the luminance of β to a luminance near Sat in Fig. 3(B).

[0029] When the input luminance value is equal to or less than α, the luminance value 3A of the long exposure image is used at 100%. When the luminance value is in the range of α to β, the luminance value 3A of the long exposure image and the luminance value 3B' of the short exposure image are each multiplied by a blending ratio as shown in the graph in Figure 3(C), and the results are added together. When the luminance value is equal to or greater than β, the luminance value 3B' of the short exposure image is used at 100%. By performing this type of processing, it is possible to combine an image acquired by long exposure and an image acquired by short exposure to generate a single composite image with an expanded dynamic range.

[0030] The dynamic range expanded image generated by the HDR synthesis unit 105 is output to the tone compression unit 106. The gradation compression unit 106 performs gradation compression processing on the dynamic range expanded image output by the HDR synthesis unit 105 based on input / output conversion parameters for compressing gradations.

[0031] 2 and 3, the image with expanded dynamic range is generated by outputting an image signal that combines images obtained by capturing images of the subject with different exposure times. However, the image with expanded dynamic range may also be generated by outputting an image signal that combines images obtained by capturing images of the subject with different sensitivities (gains).

[0032] The tone compression processing method in this embodiment will be described with reference to FIG. 4A is a functional block diagram showing an outline of the configuration of the gradation compression unit 106 of the embodiment 1. The gradation compression unit 106 includes a viewing parameter storage unit 401, a sensing parameter storage unit 402, a gradation compression processing unit 403, and the like.

[0033] The viewing parameter storage unit 401 stores input / output conversion parameters suitable for visually recognizing an image used in the gradation compression unit 106 when viewing. Fig. 4(B) is a diagram showing an example of parameter characteristics of input / output conversion parameters stored in the viewing parameter storage unit 401. The horizontal axis of Fig. 4(B) shows input luminance, and the vertical axis shows output luminance. In this embodiment, a graph showing conversion of 24-bit input luminance to 12-bit output luminance is shown, but the number of bits is not limited to this example.

[0034] In the parameter characteristic of Fig. 4(B), based on the visual characteristic that the human eye is more sensitive to dark areas than bright areas, more gradations than the sensing parameters are assigned to luminance values ​​below a certain value. That is, the viewing parameters are set so that the slope of the output luminance below a certain luminance value is larger than that of the sensing parameters shown in Fig. 4(C), for example. On the other hand, the slope of the output luminance for intermediate luminances larger than the certain luminance value is set so that it is smaller than that of the sensing parameters. By using such parameter characteristics, it is possible to improve the expressiveness of gradation in dark areas and to display an image of suitable quality for viewing.

[0035] The sensing parameter storage unit 402 stores input / output conversion parameters suitable for performing image recognition used in the gradation compression unit 106 when performing sensing. Fig. 4(C) is a diagram showing an example of parameter characteristics of the input / output conversion parameters stored in the sensing parameter storage unit 402. The horizontal axis of Fig. 4(C) indicates input luminance, and the vertical axis indicates output luminance. Note that Fig. 4(C) also shows a graph for converting 24-bit input luminance to 12-bit output luminance, but the number of bits is not limited to this example.

[0036] As shown in the graph of FIG. 4C, when performing object recognition, the feature amount is often detected mechanically using the luminance difference of the object without relying on the luminance value, so in this embodiment, the gradation is assigned approximately equally to all luminances.

[0037] That is, the sensing parameters are set so that the slope of the output luminance for input luminance values ​​below a certain value is smaller than that of the viewing parameters, and less gradation is allocated for luminance values ​​below the certain value than that of the viewing parameters. On the other hand, the slope of the output luminance for intermediate luminances greater than the certain value is set to be larger than that of the viewing parameters. This is because sufficient gradation of intermediate luminance is required for image recognition. In this embodiment, the gradation is allocated approximately evenly, resulting in a linear characteristic (curve), but the characteristics of the input / output conversion parameters may be changed as appropriate depending on the method, characteristics, algorithm specifications, etc. of object recognition.

[0038] Here, a method for setting input / output conversion parameters in the two parameter storage units will be described. In this embodiment, when starting up the imaging system 101, the imaging system control unit 103 acquires input / output conversion parameters for both viewing and sensing from the image processing system 102. Of the acquired input / output conversion parameters, the input / output conversion parameters for viewing as shown in Fig. 4(B) are stored in the viewing parameter storage unit 401. Also, the input / output conversion parameters for sensing as shown in Fig. 4(C) are stored in the sensing parameter storage unit 402.

[0039] In this embodiment, the input / output conversion parameters for both viewing and sensing are acquired at startup, but other timing may be used as well, and the parameters may be set at appropriate timing in accordance with the system control specifications.

[0040] In addition, in this embodiment, the description is given on the premise that information on input / output conversion parameters is acquired from the image processing system 102, but as shown in Fig. 4A, an imaging control ROM 404 may be provided separately in the imaging system 101 and automatically read out. By doing so, it is possible to complete the setting of input / output conversion parameters only in the imaging system 101, without communicating information with other systems.

[0041] 4A, a switch may be used to select whether the information on the input / output conversion parameters is to be acquired from the image processing system 102 or from the imaging control ROM 404. The image processing system 102 may also be configured to acquire the input / output conversion parameters from an external server or the like via a network, for example. This allows the input / output conversion parameters to be easily updated.

[0042] In addition, in this embodiment, two parameter storage units, one for viewing and one for sensing, may be provided, but only one or three or more may be provided. When only one is provided, the circuit scale of the gradation compression unit 106 is small. However, when switching the setting of the gradation compression process, it is necessary to acquire the input / output conversion parameters from the imaging system control unit 103 each time and reset the parameters, so that switching takes time. When three or more are provided, the setting can be switched quickly, but the circuit scale of the gradation compression unit 106 increases. The number of parameter storage units may be determined according to the allowable system scale, the required switching speed, and the like.

[0043] The gradation compression processing unit 403 of this embodiment is configured to perform gradation compression processing based on two or more input / output conversion parameters, and selects the input / output conversion parameters based on information from the imaging system control unit 103. That is, the gradation compression processing unit 403 selects either the input / output conversion parameters for viewing set in the parameter storage unit for viewing 401 or the input / output conversion parameters for sensing set in the parameter storage unit for sensing 402. Based on the selected input / output conversion parameters, the gradation compression processing is performed by outputting an output luminance value corresponding to the luminance value of the composite image output from the HDR synthesis unit 105. After the gradation compression processing is performed by the gradation compression unit 106, it is output to the image processing system 102.

[0044] The image processing system 102 includes a receiving unit 107, a decompressing unit 108, an image processing unit 109, an image processing system control unit 110, an operation mode detection unit 111, an object detection unit 112, a display unit 113, and the like.

[0045] The image processing system 102 in this embodiment has two operation modes corresponding to the imaging modes, and acquires images from the imaging system 101 in different imaging modes in the two operation modes. That is, the image processing system 102 has an image processing step of acquiring images from the imaging system in two or more imaging modes.

[0046] In this embodiment, the first driving mode is a manual driving mode in which the driver mainly drives manually while looking at the electronic mirror, and the second driving mode is an automatic driving mode in which the image processing system 102 automatically drives the vehicle while performing image recognition of objects around the vehicle. In the manual driving mode, the system control for viewing is performed, and in the automatic driving mode, the system control for sensing is performed. Note that the automatic driving mode is not limited to a mode in which the driving of a vehicle as a moving body (steering, braking, starting and stopping, etc.) is performed completely automatically, but also includes a mode in which the brakes are applied automatically or a warning is issued using object recognition, etc. to assist manual driving.

[0047] The receiving unit 107 is for receiving the image signal output from the gradation compression unit 106. The received image signal is output to the expansion unit 108, where expansion processing is performed. Specifically, the luminance value compressed by the gradation compression unit 106 is expanded so that the luminance value becomes, for example, substantially the same as the luminance value before the gradation compression processing. The expansion unit 108 has two types of expansion parameters, one for viewing and one for sensing, similar to the gradation compression unit 106, and sets the appropriate expansion parameter according to the operation mode.

[0048] The decompression parameters for viewing and sensing have characteristics obtained by reversing the vertical and horizontal axes of the compression characteristics for viewing and sensing shown in Figures 4(B) and (C), respectively. In this embodiment, since the compression characteristics for sensing are linear as shown in Figure 4(C), the decompression parameters for sensing are also linear. After being decompressed by the decompression unit 108 , the image is output to the image processing unit 109 .

[0049] The image processing unit 109 performs development processing, white balance processing, and other image processing required depending on the driving mode, etc. For example, when the driving mode is a manual driving mode, image processing is performed so that the image quality is suitable for viewing by the driver. After performing various image processing, processing for displaying the image on the display unit 113 where the driver can view the image is also performed to generate an image for viewing. Note that the image for viewing output from the image processing unit 109 may also be displayed on the display unit 113 in the automatic driving mode.

[0050] On the other hand, when the driving mode is the autonomous driving mode, the image processing system 102 performs image processing suitable for sensing. That is, before the object detection unit 112 detects an object, image enhancement processing such as contour enhancement processing and color enhancement processing is performed in advance. Note that, since images that have been subjected to such image enhancement processing suitable for sensing are not optimal for viewing by humans to observe the image, the display unit 113 may display the viewing image output from the image processing unit 109 as described above.

[0051] In that case, the image processing unit 109 will simultaneously perform image enhancement processing suitable for sensing and processing for images for viewing, which increases the load. Therefore, the image processing unit may be provided with two separate image processing units: an image processing unit for sensing that performs image enhancement processing suitable for sensing, and an image processing unit for viewing that performs processing for images for viewing.

[0052] The image processing system control unit 110 controls the entire image processing system 102, such as detecting the operation mode, controlling the operation, controlling the receiving unit 107, the decompression unit 108, and the image processing unit 109, and transmitting information related to the operation mode to the imaging system 101. The image processing system control unit 110 also has a built-in CPU as a computer, and functions as a control unit that controls the operation of each unit of the image processing system 102 based on a computer program stored in a memory as a storage medium.

[0053] The driving mode detection unit 111 detects the driving mode and obtains driving mode information from a moving body drive control device provided in the moving body. The moving body drive control device is a device with a built-in computer for controlling the drive of a motor or an engine as a driving source for moving the moving body, and has multiple driving modes.

[0054] Specifically, the driving mode detection unit 111 detects whether the current driving mode is a manual driving mode or an automatic driving mode. Based on the detection result, the image processing system control unit 110 determines whether to perform control for viewing or control for sensing. The image processing system control unit 110 then transmits information indicating whether the current state is sensing or viewing to the imaging system control unit 103. The imaging system control unit 103, having received this information, sets the input / output conversion parameters of the gradation compression unit 106 to the appropriate parameter of FIG. 4(B) or FIG. 4(C).

[0055] The object detection unit 112 performs object detection when the driving mode is the automatic driving mode. Specifically, the object detection unit 112 performs an object detection process by performing image recognition on the output image from the image processing unit 109 based on information such as the color, outline, and feature amount of the object. For example, a template matching method may be used as the object detection process by performing image recognition. The image processing system control unit 110 transmits the object detection result to a mobile object drive control device (not shown), and the mobile object drive control device performs automatic driving control such as steering, acceleration, and braking based on the object detection result.

[0056] The display unit 113 is for displaying images acquired from the imaging system, and is used for the driver to visually confirm (view) the situation around the vehicle when the driving mode is a manual driving mode, etc. Specifically, the display unit 113 displays an image for viewing output from the image processing unit 109. As described above, even in the automatic driving mode, the image for viewing may be generated in the image processing unit 109 and displayed on the display unit 113. In this embodiment, the display unit 113 has the form of, for example, an electronic mirror, but this is not limited as long as the driver can view the image. Also, the image may be recorded in a recording unit such as a drive recorder, separately from the display unit 113 for the driver to view in real time.

[0057] Furthermore, the image processing system 102 may be provided with a communication unit for two-way communication with the outside of the mobile body via a network, for example, transmitting images and recording the images in a recording unit such as an external server. Note that the images recorded in these recording units are not limited to images for viewing, and may be images for sensing, or images whose gradations have been compressed before being decompressed by the decompression unit 108.

[0058] Fig. 5 is a flowchart for explaining a control flow of the image processing system according to the first embodiment, and the control flow of the image processing system according to the first embodiment will be explained with reference to Fig. 5. Note that the operation of each step of the flowchart in Fig. 5 is performed by the computer inside the vehicle-mounted camera system 100 executing a computer program stored in the memory.

[0059] In step S501, the computer of the in-vehicle camera system 100 performs initial setting of input / output conversion parameters for the gradation compression unit 106. Specifically, of the two input / output conversion parameters previously stored in the gradation compression unit 106, the input / output conversion parameters for sensing (FIG. 4(C)) are selected as initial setting values. In this embodiment, sensing is prioritized and set as the initial value as the environment is suitable for the autonomous driving mode. However, if the environment is unsuitable for autonomous driving, such as there are few subjects to be recognized, the input / output conversion parameters for viewing (FIG. 4(B)) may be set as the initial value.

[0060] In step S502, the image processing system control unit 110 uses the driving mode detection unit 111 to detect whether the current driving mode is the automatic driving mode or the manual driving mode.

[0061] In step S503, the image processing system control unit 110 determines whether the driving mode detected in step S502 is the autonomous driving mode. If the determination result is that the driving mode is the autonomous driving mode, the process proceeds to step S504 to perform system control for sensing. If the determination result is not that the driving mode is the autonomous driving mode, the process proceeds to step S505 to perform system control for viewing.

[0062] In step S504, input / output conversion parameters for sensing are set in the gradation compression unit 106. Specifically, the image processing system control unit 110 transmits information to the imaging system control unit 103 that sensing is currently being performed. Then, upon receiving the information, the imaging system control unit 103 sets the gradation compression unit 106 to use the input / output conversion parameters of Fig. 4(C). After setting the input / output conversion parameters and causing gradation compression to be performed, the process returns to step S502.

[0063] In step S505, input / output conversion parameters for viewing are set. Specifically, information that the image processing system control unit 110 is not in the automatic driving mode is transmitted to the imaging system control unit 103. Then, upon receiving the information, the imaging system control unit 103 sets the gradation compression unit 106 to use the input / output conversion parameters for viewing in FIG. 4(B). After setting the input / output conversion parameters and performing gradation compression, the process returns to step S502. In this way, this embodiment has a gradation compression step in which gradation compression processing is performed on the image signal based on two or more input / output conversion parameters.

[0064] As described above, the imaging system control unit 103 executes an imaging system control step of setting input / output conversion parameters in the gradation compression unit 106 based on the operation mode (imaging mode) in the image processing system 102. This makes it possible to perform optimal gradation compression processing according to the intended use and application of the captured image, and suppresses deterioration of image quality for viewing and object recognition performance for sensing that accompanies gradation compression processing.

[0065] In this embodiment, two input / output conversion parameters for sensing and viewing are used as parameters to be stored in the gradation compression unit 106, but other parameters may be used in addition to these. For example, a night vision mode may be provided as an imaging mode (driving mode), and the input / output conversion parameters for the night vision mode may be switched to those for the night vision mode in the night vision mode. That is, for example, a night vision viewing mode for nighttime use may be provided as an operating mode, and new input / output conversion parameters for night vision viewing that allocate more gradations to dark areas than the normal viewing mode may be provided.

[0066] In addition, a night vision sensing mode for nighttime use may be further provided as an operation mode, and new input / output conversion parameters for night vision sensing may be provided that allocate more gradations to dark areas than in the normal sensing mode. The more types of input / output conversion parameters there are, the more appropriate gradation compression processing can be performed according to the purpose or use.

[0067] Furthermore, when switching between three or more input / output conversion parameters, the parameters may be set in the gradation compression unit 106 in descending order of priority, and when the priority changes, the input / output conversion parameters to be set in the gradation compression unit 106 may be updated while being overwritten. For example, two types of input / output conversion parameters, that is, normal sensing and viewing, may be stored in the gradation compression unit 106 during the day, and input / output conversion parameters for night vision sensing and night vision viewing may be overwritten in the gradation compression unit 106 at night. By performing such control, it is possible to instantly switch between multiple types of input / output conversion parameters while suppressing the circuit scale of the gradation compression unit 106.

[0068] In addition, in this embodiment, the input / output conversion parameters of Figures 4(B) and 4(C) are explained under the assumption that they are switched directly, but it is also possible to have several transition parameters between Figures 4(B) and 4(C) and gradually switch via those transition parameters.

[0069] FIG. 6 is a conceptual diagram of a case where a transition parameter is used when switching between input / output conversion parameters used in the tone compressor 106 according to the first embodiment. For example, as shown in Fig. 6, three types of transition parameters (6a, 6b, 6c) are provided between the input / output conversion parameters of Fig. 4(B) and Fig. 4(C). When switching from the sensing parameters to the viewing parameters, the sensing parameters may be gradually changed over time to the viewing parameters after passing through the transition parameters 6a, 6b, and 6c. By performing such control, it is possible to reduce the possibility of abnormal driving (sudden acceleration, sudden braking, sharp turns, etc.) occurring due to a sudden change in gradation performance.

[0070] Furthermore, when gradually changing the input / output conversion parameters, the switching method of the input / output conversion parameters may be changed according to the psychological state of the driver, the operating status of the moving object, the speed of the moving object, the brightness distribution (histogram) of the subject, etc. For example, when switching the driving mode from the manual driving mode to the automatic driving mode, the driver may feel psychologically uneasy and not want to switch to the automatic driving mode yet. Such a psychological state of the driver can be determined by AI based on the driver's facial image, etc.

[0071] It is also possible to determine the psychological state of the driver from operation history information, such as steering being performed periodically even if the frequency is low. In cases where the psychological state of the driver is known, or there is an operation history, or the vehicle speed is high and switching to the automatic driving mode poses a high risk, the switching may be delayed for a predetermined time or more.

[0072] Furthermore, the input / output conversion parameter curve may be changed based on the luminance distribution so that the gradation of the high frequency part of the luminance histogram is higher than that of the low frequency part. Also, when the deviation of the frequency is equal to or higher than a predetermined level, the input / output conversion parameter curve for sensing may be switched to detect the gradation of only the part where the frequency is higher than a predetermined value. In this way, by changing the switching method of the input / output conversion parameters according to at least one of the psychological state of the driver of the moving body, the operating situation of the moving body, the speed of the moving body, and the luminance distribution of the subject, more optimal gradation characteristics can be obtained.

[0073] Also, for example, when switching from the normal sensing mode to the night vision sensing mode, if the change in brightness of the subject is slow, such as at sunset, the switching can be made slower than normal, and if the change is sudden, such as at the entrance or exit of a tunnel, the switching can be made earlier. In this way, the delay time for switching the input / output conversion parameters can be appropriately controlled according to the driving situation and environmental conditions.

[0074] Furthermore, the switching response time when switching from the sensing parameters to the viewing parameters may be made different from the switching response time when switching from the viewing parameters to the sensing parameters.

[0075] As described above, by setting appropriate input / output conversion parameters in the gradation compression unit 106 according to the imaging mode, such as the driving mode, it is possible to minimize the degradation of image quality performance for viewing and object recognition performance for sensing that is caused by the gradation compression process.

[0076] <Second embodiment> Hereinafter, a second embodiment of the present invention will be described. In the first embodiment, only one image transmission path is provided from the gradation compression unit to the image processing unit, whereas in the second embodiment, two are provided. The purpose is to perform optimal gradation compression processing on the image for viewing and the image for sensing when viewing and sensing are to be performed simultaneously, and to output both of them simultaneously.

[0077] In this embodiment, the two paths are path A and path B, and when two images are not output simultaneously, path A is used preferentially, and when two images are output simultaneously, path B is used in addition to path A. However, the present invention is not limited to the above, and other methods may be used as long as they allow switching between one image output and two image output.

[0078] Fig. 7 is a functional block diagram of an in-vehicle camera system 700 in the second embodiment. The configurations of 700 to 713 in Fig. 7 in this embodiment correspond to the configurations of 100 to 113 in Fig. 1, and have equivalent functions. (However, for the gradation compression unit A706, the receiving unit A707, the decompression unit A708, and the image processing unit A709, in order to distinguish between path A and path B, the letter A is added after the name of each unit to indicate that it is path A.)

[0079] Hereinafter, the contents overlapping with FIG. 1 will be omitted, and only the differences in this embodiment will be described. 7 are realized by causing a computer (not shown) included in the vehicle-mounted camera system 700 to execute a computer program stored in a memory (not shown) serving as a storage medium. However, some or all of these may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP), etc., may be used.

[0080] The functional blocks in the imaging system 701 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. Similarly, the functional blocks in the image processing system 702 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

[0081] In the second embodiment, in addition to the configuration of the first embodiment, a gradation compression unit B714, a receiving unit B715, an expansion unit B716, and an image processing unit B717 are further provided. Also, their functions are equivalent to those of the gradation compression unit A706 (106), the receiving unit A707 (107), the expansion unit A708 (108), and the image processing unit A709 (109).

[0082] In this embodiment, the image for sensing is used preferentially, and the image for viewing is also used as necessary. That is, the input / output conversion parameters for sensing (FIG. 4(C)) are stored in advance in the gradation compression unit A706 (first gradation compression means), and the input / output conversion parameters for viewing (FIG. 4(B)) are stored in advance in the gradation compression unit B714 (second gradation compression means). If it is desired to use the image for viewing preferentially, the parameters for viewing and sensing may be reversed, or the priorities of path A and path B may be swapped without changing the parameters.

[0083] The receiving unit B715, like the receiving unit A707, performs processing to receive the image signal output from the gradation compression unit B714. Decompression parameters for sensing are set in decompression section A708, and decompression parameters for viewing are set in decompression section B716.

[0084] The image processing unit A709 performs image processing suitable for sensing by the image processing system 102. In addition, before the object detection unit 712 detects an object, image enhancement processing and the like are performed in advance. The image processor B717 performs image processing so that the image quality becomes suitable for viewing by the driver. After performing various image processing, it also performs processing for displaying the image on the display unit 713 so that the driver can view the image.

[0085] Fig. 8 is a flowchart for explaining a control flow of an image processing system according to the second embodiment, and the control flow of the image processing system according to the second embodiment will be explained with reference to Fig. 8. Note that the operation of each step of the flowchart in Fig. 8 is performed by the internal computer of the vehicle-mounted camera system 700 executing a computer program stored in the memory.

[0086] In step S801, the computer inside the vehicle-mounted camera system 700 initializes input / output conversion parameters for the gradation compressor A706 and the gradation compressor B714 (second gradation compression means). Specifically, the input / output conversion parameters for sensing (FIG. 4(C)) stored in advance in the gradation compressor A706 are set as initial values. Also, the input / output conversion parameters for viewing (FIG. 4(B)) stored in advance in the gradation compressor B714 are set as initial values ​​in the gradation compressor B714.

[0087] Step S802 is similar to step S502 in the first embodiment, and therefore a description thereof will be omitted. In step S803, the image processing system control unit 710 determines whether or not both the sensing and viewing functions are required based on the determination result of the driving mode. If it is determined that the driving mode requires both functions, the image processing system control unit 710 transmits that information to the imaging system control unit 703. In that case, in step S804, the imaging system control unit 703 that has received the information enables both path A and path B, and causes the HDR processing unit 705 to output a composite image to both the gradation compression unit A 706 and the gradation compression unit B 714.

[0088] If the image processing system control unit 710 determines in step S803 that the driving mode does not require both the sensing and viewing functions, then in step S805, only path A is enabled. Then, the imaging system control unit 703 causes the HDR processing unit 705 to output the composite image only to the gradation compression unit A706. After steps S804 and S805 are executed, the process returns to step S802.

[0089] As described above, in an operation mode that requires both images for sensing and images for viewing, two different gradation compression units are used to simultaneously generate an image that has been subjected to gradation compression processing suitable for sensing and an image that has been subjected to gradation compression processing suitable for viewing. This makes it possible to simultaneously output images for both viewing and sensing with minimal performance degradation associated with gradation compression processing.

[0090] In the above embodiment, an example has been described in which a camera system is mounted on a moving object such as an automobile. However, the moving object in this embodiment is not limited to a vehicle such as an automobile, and may be any moving device that moves, such as a train, a ship, an airplane, a robot, or a drone. This embodiment can also be applied to the case where a moving object is remotely controlled.

[0091] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. A computer program for implementing all or part of the control in this embodiment may be supplied to a camera system or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the camera system or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0092] 100: In-vehicle camera system 101: Imaging system 102: Image processing system 103: Imaging system control unit 104: Imaging unit 105: HDR synthesis section 106: Gradation compression section 107: Receiving unit 108: Extension part 109: Image processing unit 110: Image processing system control unit 111: Operation mode detection unit 112: Object detection unit 113: Display section 301: Long exposure image memory 302: Short exposure image memory 303: Brightness correction unit 304: Image synthesis unit 401: Viewing parameter storage unit 402: Sensing parameter storage unit 403: Gradation compression processing unit 404: ROM for image capture control 700: In-vehicle camera system 701: Imaging system 702: Image Processing System 703: Imaging system control unit 704: Imaging section 705: HDR synthesis section 706: Gradation Compression Section A 707: Receiving section A 708: Extension part A 709: Image processing unit A 710: Image processing system control unit 711: Operation mode detector 712: Object detection unit 713: Display section 714: Gradation compression section B 715: Receiving section B 716: Extension part B 717: Image processing section B

Claims

1. an imaging means for imaging an object and generating an image signal; a gradation compression means for performing a gradation compression process on the image signal based on two or more input / output conversion parameters; an imaging system having an imaging system control means for setting the input / output conversion parameters in the gradation compression means; an image processing system that acquires images from the imaging system in a first imaging mode for displaying a captured image to a user who is driving in a manual driving mode and in a second imaging mode for performing recognition using image features in an automatic driving mode; A camera system characterized in that the imaging system control means, in the gradation compression means, sets the input / output conversion parameters for viewing in the first imaging mode, and sets the input / output conversion parameters for sensing in the second imaging mode.

2. 2. The camera system according to claim 1, wherein the input / output conversion parameters for the viewing mode have a greater number of gradations below a predetermined input luminance value than the input / output conversion parameters for the sensing mode.

3. 3. The camera system according to claim 1, wherein the imaging system control means switches the input / output conversion parameters to be set in the gradation compression means via transition parameters.

4. The camera system according to any one of claims 1 to 3, characterized in that, when gradually changing the input / output conversion parameters, the imaging system control means changes the manner in which the input / output conversion parameters are switched depending on at least one of the psychological state of a driver of a moving body on which the imaging means is mounted, the operating status of the moving body, the speed of the moving body, and the brightness distribution of a subject.

5. 5. The camera system according to claim 1, wherein the image processing system further has a night vision mode, and in the night vision mode, the image processing system switches to input / output conversion parameters for the night vision mode.

6. The camera system according to claim 1, characterized in that the gradation compression means includes a first gradation compression means for performing gradation compression processing using the input / output conversion parameters for the sensing, and a second gradation compression means for performing gradation compression processing using the input / output conversion parameters for the viewing.

7. The camera system according to any one of claims 1 to 6, characterized in that the imaging means outputs the image signal which is a composite of images obtained by imaging a subject with different exposure times or sensitivities.

8. A camera system according to any one of claims 1 to 7; A moving body comprising a display means for displaying an image acquired from the imaging system.

9. an imaging step of imaging an object and generating an image signal; a tone compression step of performing tone compression processing on the image signal based on two or more input / output transformation parameters; an imaging system control step of setting the input / output conversion parameters in the tone compression step; and an image processing step of acquiring images from the imaging system control step in a first imaging mode for displaying the captured image to a user who is driving in a manual driving mode and in a second imaging mode for performing recognition using image features in an automatic driving mode, A camera system control method characterized in that, in the imaging system control step, in the gradation compression step, the input / output conversion parameters are set for viewing in the first imaging mode, and the input / output conversion parameters are set for sensing in the second imaging mode.

10. A computer program for controlling each means of the camera system according to any one of claims 1 to 7 or the moving body according to claim 8 by a computer.

Images