Image processing device, information processing system, information processing method, and program

Abstract

To easily identify a fault location when a fault occurs in video display.SOLUTION: An image processing device processes and outputs images from multiple on-board cameras to an on-board information processing device that has a function for detecting fixation of input images. When it is determined that one of the multiple on-board cameras is malfunctioning, a controller of the image processing device determines which of multiple states is present, defined by a combination of the state of the vehicle's shift lever and the installation location of the malfunctioning on-board camera. Then, when a predetermined specific state is present among the multiple states, the controller instructs the on-board information processing device to disable the function for detecting fixation.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] A fault diagnosis processing unit has been proposed for a vehicle-mounted camera device that diagnoses whether or not a data line signal of an imaging element unit is in a fixed state (see, for example, Patent Document 1 below). In this technology, image data acquired by the imaging element unit has an entire imaging area that is divided into an effective image area and an invalid image area. A diagnostic data area containing fixation diagnostic data for diagnosing whether or not a data line signal of the imaging element unit is in a fixed state is provided within the invalid image area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2016 / 117401 Summary of the Invention [Problem to be solved by the invention]

[0004] Image processing devices such as a camera system called a Surround View Monitor (hereinafter referred to as SVM) that synthesizes images from cameras mounted on the front, rear, left and right sides of a vehicle and displays the synthesized image are being used. When an image from such an image processing device is displayed on an information processing device such as an IVI, a problem arises in the coordination between the image processing device and the in-vehicle information processing device.

[0005] For example, an in-vehicle information processing device may have a mechanism for detecting image freeze. On the other hand, an image processing device exemplified by an SVM has a mechanism for detecting whether an individual connected camera is disconnected or malfunctioning. If a connected camera is disconnected or malfunctioning, the image processing device outputs, for example, a monochrome image (e.g., solid blue) instead of the image from the malfunctioning camera.

[0006] As a result, the synthesized image generated by SVM is a monochromatic image with no change in part of the screen, and is input to the in-vehicle information processing device. This may result in the in-vehicle information processing device determining that the input image has become frozen. If the in-vehicle information processing device determines that the input image has become frozen, it may output, for example, an all-black image on the display as a fail-safe, thereby informing the driver of the frozen image.

[0007] When an in-vehicle information processing device outputs an image such as an all-black screen on its display, it becomes difficult to identify the location of the failure, i.e., it becomes difficult to determine whether the image processing device itself is faulty, which camera connected to the image processing device is faulty, or whether the in-vehicle information processing device itself is faulty.

[0008] An aspect of the disclosed embodiment is to facilitate identification of the location of a failure in an image processing device that synthesizes and displays images from multiple cameras, and an in-vehicle information processing device that receives and displays images from this image processing device, if a failure occurs in one of the cameras. [Means for solving the problem]

[0009] One aspect of the disclosed embodiment is exemplified by an image processing device including a controller. The image processing device processes and outputs images from multiple onboard cameras to an onboard information processing device having a function for detecting stuckness in the input image. When the controller determines that one of the multiple onboard cameras is malfunctioning, it determines whether the malfunctioning camera is in one of multiple states defined by a combination of the state of the vehicle's shift lever and the installation location of the malfunctioning onboard camera. Then, when a predetermined specific state is detected among the multiple states, the controller instructs the onboard information processing device to disable the stuckness detection function. [Effects of the Invention]

[0010] This image processing device processes and outputs images from multiple in-vehicle cameras to an in-vehicle information processing device that has a function for detecting whether the input image is frozen. When the in-vehicle information processing device detects a freeze, it outputs an image of a specific color (e.g., a black image) on the entire screen. However, if one of the multiple in-vehicle cameras malfunctions and the entire screen displays an image of the specific color, it becomes difficult to determine which in-vehicle camera has malfunctioned.

[0011] In such an image processing device and an in-vehicle information processing device, if one of the multiple in-vehicle cameras malfunctions, a fixed portion may appear in the input image to the information processing device processed by the image processing device. However, if the fixed portion is not dominant in the input image, the in-vehicle information processing device may not detect the fixation. On the other hand, if the fixed portion is dominant in the input image, the in-vehicle information processing device is more likely to detect the fixation.

[0012] Thus, the extent to which a failure of any one of the onboard cameras affects the input image processed by the image processing device depends on how the image from each onboard camera is processed by the image processing device. However, how the image from each onboard camera is processed by the image processing device usually depends on the state of the shift lever. This is because the image that is desirable to be provided to the driver differs depending on the state of the shift lever (e.g., shift position). Here, the desirable image depends on the installation location of each onboard camera on the vehicle (e.g., front side, rear side, etc.). Therefore, the extent to which the stuck portion affects the processed image can be determined to some extent depending on multiple states defined by the combination of the state of the vehicle's shift lever and the installation location of the failed onboard camera.

[0013] Therefore, in a predetermined specific state among the multiple states, the controller instructs the in-vehicle information processing device to disable the function for detecting sticking. By disabling the function, the in-vehicle information processing device can suppress the detection of sticking when there is a high possibility of detecting sticking, and can prevent the entire screen from being output as an image of a specific color. As a result, in an image processing device that combines and displays images from multiple in-vehicle cameras, and in an in-vehicle information processing device that receives and displays images from the image processing device, if a malfunction occurs in one of the in-vehicle cameras, the controller can easily identify the location of the malfunction. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating the details of the image processing device. [Figure 3] FIG. 3 is a diagram illustrating details of the information processing device. [Figure 4] FIG. 4 is a table showing examples of commands from the image processing device that are sent to the information processing device in response to the position of the vehicle's shift lever when the camera malfunctions. [Figure 5] FIG. 5 is an example of a front view screen displayed on the display of an information processing device. [Figure 6] FIG. 6 is an example of a rearview screen displayed on the display of an information processing device. [Figure 7] FIG. 7 is a flowchart illustrating the processing of the image processing device. [Figure 8] FIG. 8 is a flowchart illustrating the processing of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0015] Below, with reference to the drawings, an information processing system 100 including an image processing device 1 and an information processing device 2 according to one embodiment, an information processing method executed by the image processing device 1, and a computer program (hereinafter simply referred to as the program) installed in the image processing device 1 will be described.

[0016] <Configuration> 1 is a diagram illustrating an example of the configuration of an information processing system 100 according to this embodiment. The information processing system 100 is mounted on, for example, a vehicle, and provides an occupant of the vehicle (hereinafter also referred to as a user) with entertainment functions such as sound and video, and driving assistance functions such as navigation.

[0017] The information processing system 100 includes cameras C1 to C4, an image processing device 1 to which the cameras C1 to C4 are connected, and an information processing device 2 that cooperates with the image processing device 1. The information processing device 2 is provided with an operation unit such as a steering switch 5 and an operation button 6, and receives operations from a user.

[0018] In FIG. 1, a vehicle electric control unit (referred to as a vehicle ECU 3) is connected to the image processing device 1 and the information processing device 2. The vehicle ECU 3 is connected to an in-vehicle network N1. Therefore, the network N1 is connected to the image processing device 1 and the information processing device 2. The network N1 is, for example, a Controller Area Network (CAN) or an in-vehicle Local Area Network (LAN) such as FLEXRAY (registered trademark). In addition, in FIG. 1, the vehicle ECU 3 is connected to the image processing device 1 and the information processing device 2 by a signal line LR. The signal line LR is connected to the image processing device 1. The signal line LR is a signal line connected by an input / output interface such as a General Purpose Input / Output (GPIO). However, the signal line LR may be connected to an input / output interface such as a Serial Peripheral Interface (SPI), an Inter-Integrated Circuit (I2C), It may also be a signal line for serial communication such as the above.

[0019] The image processing device 1 has a function called a surround view system (SVM). A surround view system is also called a view-around system. For example, camera C1 is a photographing device that photographs the front of the vehicle, camera C2 the right side of the vehicle, camera C3 the rear of the vehicle, and camera C4 the left side of the vehicle. Cameras C1 to C4 are also called a front camera, a right camera, a rear camera, and a left camera, respectively. Cameras C1 to C4 are examples of multiple in-vehicle cameras.

[0020] The cameras C1 to C4 are connected to a network using, for example, Gigabit Multimedia Serial Link (GMSL), Low Voltage Differential Signaling (LVDS), Gigabit Video Interface (GVIF), ), Flat Panel Display Link (FPD-Link), etc. The images are input to the information processing device 1. The SVM generates, for example, a composite image of a vehicle viewed from above in a planar view based on the images from the cameras C1 to C4, and inputs the composite image as a video signal to the information processing device 2. The video signal input to the information processing device 2 is also a signal that complies with standards such as GMSL, LVDS, GVIF, and FPD-Link.

[0021] Furthermore, the image processing device 1 sends to the information processing device 2 a switching signal for switching the screen that the information processing device 2 outputs to the display 25 (see FIG. 3). The switching signal switches between outputting the images from the cameras C1 to C4, including the composite image obtained by SVM, to the display 25, and displaying a screen other than the images from the cameras C1 to C4. This switching signal will be further explained with reference to FIG. 2.

[0022] The image processing device 1 also receives a reverse signal from the vehicle ECU 3 via a signal line LR, indicating that the shift lever is in a position for backward travel (reverse position). Furthermore, the image processing device 1 receives signals indicating the positions of the vehicle's shift lever, including the reverse signal, from the vehicle ECU 3 via the network N1. For example, when the image processing device 1 receives a reverse signal from the signal line LR, indicating that the shift lever is in reverse, it instructs the information processing device 2 via a switching signal to output images from cameras C1 to C4 to the display 25 (see FIG. 3). These images include, for example, composite images obtained by SVM. Note that the information processing device 2 may output a rear monitor image captured by camera C3, which captures the area behind the vehicle, to the display 25, instead of the composite image obtained by SVM.

[0023] In this embodiment, when any of the cameras C1 to C4 fails, the image processing device 1 executes different processes depending on the position of the vehicle's shift lever. The different processes depending on the position of the shift lever will be described with reference to FIG.

[0024] The information processing device 2 is an example of an in-vehicle information processing device, and is also called, for example, In-Vehicle Infotainment (IVI), Display Audio (DA), or Head / Unit (H / U). That is, the information processing device 2 may have, for example, audio, visual, and navigation functions. However, the information processing device 2 is not limited to the above-mentioned in-vehicle video devices.

[0025] The steering switch 5 and the operation button 6 are a user interface that receives from the user an instruction to switch the content displayed on the display 25 by the information processing device 2. When the steering switch 5 is operated, the information processing device 2 displays the composite image from the image processing device 1 on the display 25. When the operation button 6 is operated while the composite image from the image processing device 1 is displayed on the display 25, the information processing device 2 outputs another display on the display 25 instead of the display of the composite image from the image processing device 1. The other display may be, for example, a television broadcast display, a screen for operating audio functions, a car navigation screen, or the like.

[0026] When the steering switch 5 and the operation button 6 are operated, the information processing device 2 notifies the image processing device 1 via the network N1 that it has accepted these operations. Then, the image processing device 1 instructs the information processing device 2 by a switching signal (camera ON / OFF) to output a screen corresponding to the operation of the steering switch 5 and the operation button 6 to the display 25.

[0027] The information processing device 2 is also connected to a vehicle ECU 3 via a network N1, and collects information on the vehicle state and the like from the vehicle ECU 3. The vehicle state includes, for example, the position of the shift lever, the speed, the acceleration, the steering angle, and the like.

[0028] FIG. 2 is a diagram illustrating the details of the image processing device 1. In addition, FIG. 2 also illustrates cameras C1 to C4, an information processing device 2, and a vehicle ECU 3. The image processing device 1 includes a central processing unit (hereinafter, referred to as CPU 11), a main storage unit 12, a deserializer 1, and a memory controller 13. 3A, a serializer 13B, an array of processor elements 14, and an input / output unit 16.

[0029] The deserializer 13A converts video signals such as GMSL input via serial communication from the cameras C1 to C4 into parallel signals and passes them on to the processor element array 14. The serializer 13B stores the video data (for example, one line) processed by the processor element array 14, converts it into a serial video signal, and supplies it to the information processing device 2.

[0030] Each processor element in the processor element array 14 has a plurality of arithmetic units that perform various types of calculations such as addition, comparison, multiplication, and multiply-and-accumulate. Each processor element uses the various arithmetic units to perform pipeline processing, and also performs image processing in parallel on frames input to the image processing device 1. Note that the image processing device 1 may have a single or multiple processors (for example, digital signal processors (DSPs)) instead of the processor element array 14.

[0031] The deserializer 13A and the processor element array 14 are connected by a network N2. The processor element array 14 and the serializer 13B are connected by a network N3. The networks N2 and N3 may be an integrated network. The networks N2 and N3 may include, for example, a crossbar switch. The networks N2 and N3 enable parallel access from the processor element array 14 to the deserializer 13A and the serializer 13B and parallel data transfer between the processor elements.

[0032] The CPU 11 executes a computer program deployed in an executable manner in the main memory unit 12 and controls each unit of the image processing device 1. That is, the CPU 11 executes parallelization of the video signals input from the cameras C1 to C4 by the deserializer 13A, initiation of image processing by the processor element array 14, and the like. Here, the image processing is, for example, generation of a composite image by SVM. Furthermore, the CPU 11 controls serialization by the serializer 13B of the composite image, etc., that has been image-processed by the processor element array 14. The CPU 11 is also called a processor. However, the CPU 11 is not limited to a single processor and may have a multi-processor configuration.

[0033] The main storage unit 12, which is also simply called memory, stores computer programs executed by the CPU 11, data processed by the CPU 11, etc. The main storage unit 12 may be a Dynamic Random Access Memory (DRAM), a Static Random Access Memory (SRAM), a Read Only Memory (READ ONLY MEMORY), etc. (ROM), etc. The CPU 11 and the main storage unit 12 can be referred to as a control unit 10. The control unit 10 is an example of a controller.

[0034] The input / output unit 16 is an interface for communicating with the outside of the image processing device 1, for example, with the information processing device 2, the vehicle ECU 3, etc. As already mentioned, the input / output unit 16 is, for example, an interface such as GPIO. However, the input / output unit 16 may also be an interface to a serial bus such as SPI or I2C. In this embodiment, the image processing device 1 transmits, via the input / output unit 16, to the information processing device 2 a signal instructing it to switch the screen to be output to the display 25 (see FIG. 3 ). The screen switching involves, for example, switching between displaying a composite image obtained by SVM and displaying a screen other than the composite image.

[0035] The input / output unit 16 may also include an interface for a signal line LR connected to the vehicle ECU 3. The input / output unit 16 may also include an interface for the network N1. The computer program executed by the CPU 11 may be loaded into the main storage unit 12 from an external device via the input / output unit 16, for example.

[0036] 3 is a diagram illustrating details of the information processing device 2. Note that the image processing device 1 is also illustrated in FIG. 3. The information processing device 2 has a main microcomputer controller (hereinafter referred to as main microcomputer 21), a sub-microcomputer 22, a video IC 23, a video interface (hereinafter referred to as video IF 24), and a display 25.

[0037] The main microcomputer 21 is, for example, a device called a System on a Chip (SoC). The main microcomputer 21 has, for example, a Central Processing Unit (CPU) and a memory. The CPU executes programs that are expanded in the memory in an executable manner. It executes computer programs and provides the functions of the main microcomputer 21. The CPU is not limited to a single processor and may have a multi-processor configuration. In addition to the CPU, the main microcomputer 21 may also have a graphics processing unit (GPU), a digital signal processor (DSP), etc.

[0038] Memory stores the computer programs executed by the CPU, the data processed by the CPU, etc. Memory is classified into Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Memory (SRAM), Read Only Memory (ROM), etc.

[0039] The main microcomputer 21 has an image synthesis unit 211, which is, for example, a module consisting of a CPU and a memory in which a program is executable. The image synthesis unit 211 generates a superimposed image by superimposing an image on a video (moving image) input from the image processing device 1 via the video IF 24 and the video IC 23, and returns the superimposed image to the video IC 23. The video input from the image processing device 1 is, for example, a synthesized video by SVM, a video of a rear monitor capturing an image behind the vehicle, etc. The superimposed image is, for example, a video input from the image processing device 1 with additional information for driving assistance or the like superimposed thereon.

[0040] The additional information is, for example, guide lines that indicate the positional relationship between the subject or components within the angle of view in the video input from the image processing device 1 and the vehicle on which the information processing device 2 is mounted. The guide lines are, for example, stored in advance as image data in the memory of the main microcomputer 21. The additional information may also be a graphics object, a character string, or the like that draws the driver's attention.

[0041] The main microcomputer 21 is also connected to the sub-microcomputer 22 by serial communication such as CAN. The main microcomputer 21 receives a switching signal (camera ON / OFF signal) from the image processing device 1 via the sub-microcomputer 22. The main microcomputer 21 also accesses the network N1 via the sub-microcomputer 22 and communicates with the image processing device 1, the vehicle ECU 3, etc.

[0042] The video IC 23 has the same components as the image processing device 1 in Figure 2. The video IC 23 has, for example, a CPU, memory, a processor element array, and an input / output unit. The CPU controls the processor element array using a computer program in the memory and provides the functions of the video IC 23. The video IC 23 receives video signals such as SVM composite video from the image processing device 1 via the video IF 24, decodes the received video signals, and converts them into video in a format that can be displayed on the display 25.

[0043] In a normal state where no abnormality such as sticking is detected, the video IC 23 transfers the video converted into a displayable format to the video synthesis unit 211 mounted on the main microcomputer 21, and acquires the above-mentioned superimposed image from the video synthesis unit 211. Then, the video IC 23 outputs the acquired superimposed image to the display 25 and displays it.

[0044] In this embodiment, the video IC 23 has a sticking detection unit 231. The sticking detection unit 231 detects abnormalities in the main microcomputer 21 and the like. That is, the sticking detection unit 231 determines whether or not the superimposed image (video) transmitted from the video synthesis unit 211 of the main microcomputer 21 is a stuck image, and detects sticking. A stuck image is an image in which each frame that makes up the video is the same image without any change. When the sticking detection unit 231 detects that the video is stuck, it executes a fail-safe process. The fail-safe process is, for example, a process of outputting a black image on the entire surface of the display 25. The video IC 23 uses the black image to notify the user that the main microcomputer 21 is stuck. Make it known that there is a malfunction in 1.

[0045] The sub-microcomputer 22 has various input / output interfaces and provides communication functions with the outside of the information processing device 2. The sub-microcomputer 22 has an interface for receiving, for example, a switching signal (camera ON / OFF signal). The sub-microcomputer 22 is also connected to each of the video IC 23 and the video IF 24 via serial communication, for example, I2C, SPI, etc. The sub-microcomputer 22 also has an interface connected to the network N1.

[0046] The video IF 24 is an interface that receives video signals from the image processing device 1. The video IF 24 receives signals that comply with standards such as GMSL, LVDS, GVIF, and FPD-Link. The display 25 outputs information such as video sent from the video IC 23. The display 25 is, for example, a liquid crystal display, an electroluminescence panel, or an organic light emitting diode (OLED).

[0047] (Example of screen display control) Fig. 4 is a table illustrating commands from the image processing device 1 that are sent to the information processing device 2 in accordance with the position of the vehicle's shift lever when one of the cameras C1 to C4 fails. The table in Fig. 4 illustrates the relationship between the failed cameras C1 to C4 (failed cameras), the output screen by the image processing device 1 (SVM) based on the position of the shift lever when the camera fails, and the commands that the image processing device 1 sends to the information processing device 2. In this embodiment, as already described, it is assumed that the camera C1 is the front camera, the camera C2 is the right camera, the camera C3 is the rear camera, and the camera C4 is the left camera.

[0048] For example, if the malfunctioning camera is the front camera (C1), and the shift lever is in D (drive, forward travel) or N (neutral), the image processing device 1 issues a command to the information processing device 2 to "not detect stuck gear." When the shift lever is in D or N, the SVM image generated by the image processing device 1 displays a front view on the display 25, in which the image from the front camera (C1) occupies most of the screen (see FIGS. 5A and 5B). As illustrated in FIG. 5, the SVM screen is a combination of an overall image (V1) of the vehicle's surroundings viewed from above in a plane and a camera image (V2) captured in a specific direction (for example, the direction ahead of the vehicle captured by the front camera (C1)). The camera image (V2) captured in the specific direction occupies a larger proportion of the screen than the overall image (V1).

[0049] In such a screen, if the front camera (C1) fails, the image processing device 1 outputs the image from the front camera (C1) as a monochrome image (for example, a solid blue image). As a result, the monochrome image becomes dominant on the SVM screen. In this case, the information processing device 2 is likely to determine that the image input from the image processing device 1 is frozen and turn the entire screen of the display 25 into a black image.

[0050] In this case, if the information processing device 2 does not detect the sticking in accordance with the command, the image input from the image processing device 1 is displayed as is on the display 25. As a result, the user can easily determine which of the cameras C1 to C4 is malfunctioning. Each row in FIG. 4 illustrates a plurality of states defined by a combination of the state of the vehicle's shift lever and the installation location of the malfunctioning camera C1 to C4. In addition, in FIG. 4, the state in which the front camera (C1) is malfunctioning and the shift lever is in the D or N position is an example of a predetermined specific state among the plurality of states.

[0051] In addition, if the malfunctioning camera is the front camera (C1), when the shift lever is in the P (parking) position, the image processing device 1 sends a command to the information processing device 2 to "detect sticking." When the shift lever is in the P position, in the SVM image generated by the image processing device 1, a screen other than the front view, for example, a side view in which images of both the left and right sides of the vehicle's traveling direction make up the majority, is displayed on the display 25. In this case, even if the front camera (C1) malfunctions and the image processing device 1 outputs the image from the front camera (C1) as a monochrome image (for example, a solid blue), the monochrome image occupies a small proportion of the screen. Therefore, the information processing device 2 is unlikely to determine that the image input from the image processing device 1 is stuck. Therefore, a command to "detect sticking" is sent to the information processing device 2.

[0052] Similarly, if the malfunctioning camera is the front camera (C1), and the shift lever is in the R (reverse, backward travel) position, the image processing device 1 issues a command to the information processing device 2 to "detect stuck." When the shift lever is in the R position, the SVM image generated by the image processing device 1 displays a rear view, with the majority of the image being of the rear of the vehicle, on the display 25. Therefore, just like when the shift lever is in the P (parking) position, the information processing device 2 is unlikely to determine that the image input from the image processing device 1 is stuck. Therefore, the command to "detect stuck" is issued to the information processing device 2.

[0053] For example, if the malfunctioning camera is the rear camera (C3), when the shift lever is in the R (reverse, backward travel) position, the image processing device 1 issues a command to the information processing device 2 to "not detect stuck." When the shift lever is in the R position, the SVM image generated by the image processing device 1 displays a rear view on the display 25, in which the image from the rear camera (C3) occupies most of the screen (see FIGS. 6A and 6B). If the rear camera (C3) malfunctions, the image processing device 1 outputs the image from the rear camera (C3) as a monochrome image (for example, solid blue). This likely causes the information processing device 2 to determine that the image input from the image processing device 1 is stuck and to turn the entire screen of the display 25 into a black image. Therefore, the command to "not detect stuck" is issued to the information processing device 2. In FIG. 4, the state in which the rear camera (C3) malfunctions and the shift lever is in the R position is an example of a predetermined specific state among multiple states defined by a combination of the state of the vehicle's shift lever and the installation location of the malfunctioning in-vehicle camera.

[0054] However, if the malfunctioning camera is the rear camera (C3), when the shift lever is in the D, N, or P position, the image processing device 1 will issue a "detect stuck" command to the information processing device 2. In this case, even if the rear camera (C3) malfunctions and the image processing device 1 outputs the image from the rear camera (C3) as a monochrome image (for example, a solid blue), the monochrome image will occupy a small proportion of the screen. Therefore, the information processing device 2 is unlikely to determine that the image input from the image processing device 1 is stuck. Therefore, the "detect stuck" command is issued to the information processing device 2.

[0055] Similarly, if the faulty camera is the right camera (C2) or the left camera (C4), the proportion of the screen occupied by a single-color image (e.g., a solid blue image) is small, regardless of the position of the shift lever. For example, when the shift lever is in the P (parking) position, the SVM image generated by the image processing device 1 displays a screen other than a front view on the display 25, for example, a side view in which images on both the left and right sides of the vehicle's traveling direction are the majority. However, even if either the right camera (C2) or the left camera (C4) fails, the proportion of the screen occupied by a single-color image is not large. Therefore, if the faulty camera is the right camera (C2) or the left camera (C4), the image processing device 1 issues a "sticking detection" command to the information processing device 2, regardless of the position of the shift lever. That is, in FIG. 5, if the faulty camera is the right camera (C2) or the left camera (C4), the image processing device 1 issues a "sticking detection" command to the information processing device 2 for all camera screens.

[0056] FIG. 5 is an example of a front view screen displayed on the display 25 of the information processing device 2. That is, FIG. 5 is an example of a screen when the shift lever is in the D (drive, forward travel) or N (neutral) position.

[0057] FIG. 5(A) illustrates an example of an SVM screen output to the display 25 by the image processing device 1 and the information processing device 2 when none of the cameras C1 to C4 are malfunctioning. In this example, the SVM screen is a combination of an overall image (V1) displaying the surroundings of the vehicle and a camera image (V2) taken in a specific direction. The camera image (V2) taken in the specific direction is larger than the overall image (V1) and occupies approximately two-thirds of the SVM screen. In FIG. 5(A), the shift lever is in the D or N position, and the camera image (V2) taken in the specific direction is a front view, capturing an image of the area ahead of the vehicle.

[0058] FIG. 5(B) illustrates an example of an SVM screen when the front camera (C1) in FIG. 5(A) fails. In this case, the image processing device 1 arranges a monochromatic (e.g., solid blue) image in the area A1 in front of the vehicle and in the front view (camera image V2) of the overall image (V1) displaying the vehicle's surroundings, and outputs the image to the information processing device 2. Note that normal images are acquired from the rear camera (C3), right camera (C2), and left camera (C4). In this case, the information processing device 2 does not perform sticking detection according to FIG. 4. Therefore, the information processing device 2 outputs the SVM image (FIG. 5(B)) input from the image processing device 1 to the display 25 as is. Note that in FIGS. 5 and 6, the monochromatic (e.g., solid blue) image is illustrated by hatching.

[0059] Fig. 5(C) illustrates an example of an SVM screen when the rear camera (C3) in Fig. 5(A) fails. In this case, the image processing device 1 arranges a monochrome (e.g., solid blue) image in the area A3 behind the vehicle in the overall image (V1) displaying the surroundings of the vehicle, and outputs the image to the information processing device 2. Note that the front camera (C1), right camera (C2), and left camera (C4) output normal images.

[0060] In this case, the image from the normal front camera (C1) is output to the front view (camera image V2). Therefore, the proportion of monochromatic images in the entire SVM image is not large. Therefore, although the information processing device 2 detects sticking according to FIG. 4, there is a low possibility that it will erroneously determine that the image is stuck. Therefore, the information processing device 2 outputs the SVM image (FIG. 5(C)) input from the image processing device 1 to the display 25 as is.

[0061] FIG. 5(D) illustrates an example of an SVM screen when all cameras C1 to C4 in FIG. 5(A) fail. In this case, the image processing device 1 outputs to the information processing device 2 a monochromatic (e.g., solid blue) image in all areas A1 to A4 of the overall image (V1) displaying the vehicle's surroundings. Furthermore, because the front camera (C1) is malfunctioning, a monochromatic (e.g., solid blue) image is also displayed in the front view (camera image V2) and output to the information processing device 2. In this case, the information processing device 2 does not perform sticking detection according to FIG. 4. Therefore, the information processing device 2 outputs the SVM image (FIG. 5(D)) input from the image processing device 1 to the display 25 as is.

[0062] Fig. 6 is an example of a rear view screen displayed on the display 25 of the information processing device 2. That is, Fig. 6 is an example of a screen when the shift lever is in the R (reverse, backward travel) position.

[0063] FIG. 6(A) illustrates an SVM screen when the rear camera (C3) in the rear view fails. In this case, the image processing device 1 arranges a monochrome (e.g., blue) image in the area A3 behind the vehicle in the overall image (V1) displaying the surroundings of the vehicle and in the rear view (camera image V2), and outputs the image to the information processing device 2. In addition, the front camera (C1), the right camera (C3), Normal images are acquired from the left camera (C2) and the right camera (C4). In this case, the information processing device 2 does not perform fixation detection according to Fig. 4. Therefore, the information processing device 2 outputs the SVM image (Fig. 6(A)) input from the image processing device 1 to the display 25 as is.

[0064] FIG. 6(B) illustrates an example of an SVM screen in FIG. 6(A) when the front camera (C1) fails, instead of the rear camera (C3). In this case, the image processing device 1 arranges a monochromatic (e.g., solid blue) image in the area A1 in front of the vehicle in the overall image (V1) displaying the vehicle's surroundings, and outputs the image to the information processing device 2. Normal images are also acquired from the right camera (C2), rear camera (C3), and left camera (C4). However, the rear view (camera image V2) is output from the normal rear camera (C3). Therefore, the proportion of monochromatic images in the entire SVM image is not large. Therefore, although the information processing device 2 detects sticking according to FIG. 4, there is a low possibility that the image will be erroneously determined to be stuck. Therefore, the information processing device 2 outputs the SVM image (FIG. 6(B)) input from the image processing device 1 to the display 25 as is.

[0065] (Processing Procedure) FIG. 7 is a flowchart illustrating the processing of the image processing device 1. The control unit 10 of the image processing device 1 executes the processing of FIG. 7 as an example of a controller. The image processing device 1 starts the processing of FIG. 7, for example, when the power supply of an accessory of the vehicle is turned on. The image processing device 1 may also start the processing of FIG. 7 when the power supply of the information processing device 2 is turned on. The image processing device 1 may also start the processing of FIG. 7 when the driving assistance processing (for example, FIG. 8) by the information processing device 2 is started.

[0066] In this process, the image processing device 1 monitors the video signals of the cameras C1 to C4 (S1). Then, the image processing device 1 determines whether the front camera (C1) is malfunctioning (S2). If the front camera (C1) is malfunctioning, the image processing device 1 converts the image from the front camera (C1) into a monochrome image (for example, a solid blue image). Then, the image processing device 1 synthesizes an SVM image using the monochrome image and images from the other cameras C2 to C4, and provides the synthesized image to the information processing device 2.

[0067] The image processing device 1 then determines whether the position of the shift lever is D or N (S3). If the determination in S2 is that the front camera (C1) has failed and the determination in S3 is that the position of the shift lever is D or N, this is an example of a predetermined specific state among multiple states defined by a combination of the state of the vehicle's shift lever and the installation location of the failed in-vehicle camera. If the position of the shift lever is D or N, the image processing device 1 issues a command to the information processing device 2 not to perform sticking detection (S4). The processing in S4 is an example of instructing the information processing device 2 to disable the function of detecting sticking. On the other hand, if the position of the shift lever is other than D or N, the image processing device 1 issues a command to the information processing device 2 to perform sticking detection (S5).

[0068] Furthermore, if the determination in S2 indicates that the front camera (C1) is not malfunctioning, the image processing device 1 determines whether the rear camera (C3) is malfunctioning (S6). If the rear camera (C3) is malfunctioning, the image processing device 1 converts the image from the rear camera (C3) into a monochrome image. The image processing device 1 then synthesizes an SVM image using the monochrome image and images from the other cameras C1, C2, and C4, and provides the SVM image to the information processing device 2.

[0069] Then, the image processing device 1 determines whether the position of the shift lever is in R (S7). If the determination in S6 is that the rear camera (C3) has failed and the determination in S7 is that the position of the shift lever is in R, the combination of the state of the shift lever of the vehicle and the installation location of the failed in-vehicle camera is determined. This is an example of a predetermined specific state among a plurality of states defined by the combination. When the position of the shift lever is R, the image processing device 1 issues a command to the information processing device 2 not to perform sticking detection (S8). The processing of S8 is an example of instructing the information processing device 2 to disable the function of detecting sticking. On the other hand, when the position of the shift lever is other than R, the image processing device 1 issues a command to the information processing device 2 to perform sticking detection (S9).

[0070] Then, the image processing device 1 determines whether or not to end the process (S10). The image processing device 1 ends the process of Fig. 7 when, for example, the power supply of the vehicle accessory is turned off. The image processing device 1 may also end the process of Fig. 7 when the power supply of the information processing device 2 is turned off. If the image processing device 1 does not end the process, it repeats the process from S1.

[0071] 8 is a flowchart illustrating the processing of the information processing device 2. The processing of FIG. 8 is included in, for example, driving assistance processing. The main microcomputer 21 of the information processing device 2 starts the driving assistance processing, for example, when the accessory power of the vehicle is turned on. However, the information processing device 2 may also start the driving assistance processing, for example, when the steering switch 5 is operated or the vehicle's shift lever is set to R.

[0072] In this process, the information processing device 2 receives a command from the image processing device 1 (S21). Then, the information processing device 2 determines whether or not a command not to perform sticking detection has been received (S22). If the information processing device 2 receives a command not to perform sticking detection, it executes the driving assistance process in a sticking detection-free mode (S23). In this case, the information processing device 2 outputs the image provided by the image processing device 1 to the display 25 as is without performing sticking detection.

[0073] On the other hand, if the information processing device 2 has not received a command not to perform sticking detection, it executes the driving assistance process in sticking detection mode (S24). In this case, the information processing device 2 performs sticking detection on the image provided from the image processing device 1. Therefore, for example, when the portion of the image where the screen is fixed becomes dominant as in FIGS. 5(B), 5(D), and 6(A), the information processing device 2 determines that the image provided from the image processing device 1 is stuck. Then, the image processing device 1 outputs a black image to the display 25.

[0074] Then, the information processing device 2 determines whether or not to end the process (S25). The information processing device 2 ends the process of Fig. 8 when, for example, the accessory power of the vehicle is turned off. If the information processing device 2 does not end the process, it repeats the process from S21.

[0075] (Effects of the embodiment) As described above, in this embodiment, when it is determined that any one of the multiple on-board cameras C1 to C4 has failed, the image processing device 1 executes the following process: That is, the image processing device 1 instructs the information processing device 2 to disable the function of detecting sticking in a predetermined specific state among multiple states defined by a combination of the state of the vehicle's shift lever and the installation location of the failed camera C1 or the like.

[0076] Therefore, even if a fixed portion of the image becomes dominant in the SVM image (video) provided from the image processing device 1, as in Figures 5(B), 5(D), and 6(A), the information processing device 2 does not output a black image due to fixation detection. That is, the information processing device 2 outputs the SVM image provided from the image processing device 1 as is to the display 25. Therefore, when the image processing device 1 displays an SVM image obtained by combining videos from multiple cameras C1 to C4, if a failure occurs in one of the multiple cameras C1 to C4, the user can easily identify the location of the failure.

[0077] Furthermore, when the front camera (C1) that takes pictures of the front of the vehicle when the vehicle's shift lever is in the D (forward drive) position or the N (neutral) position fails, the image processing device 1 instructs the in-vehicle information processing device 2 to disable the function of detecting sticking. Therefore, when the front camera (C1) fails, the image processing device 1 enables the user to easily identify the location of the failure.

[0078] Furthermore, when the rear camera (C3) that takes pictures of the rear of the vehicle breaks down while the shift lever of the vehicle is in the R (reverse) position, the image processing device 1 instructs the in-vehicle information processing device 2 to disable the function of detecting sticking. Therefore, when the rear camera (C3) breaks down, the image processing device 1 enables the user to easily identify the location of the failure.

[0079] (Computer-readable recording medium) A program that causes a computer or other machine or device (hereinafter referred to as a computer, etc.) to realize any of the above functions can be recorded on a computer-readable recording medium. Then, by having the computer, etc. read and execute the program from this recording medium, the function can be provided.

[0080] Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs electrically, magnetically, optically, mechanically, or chemically and can be read by a computer. Among such recording media, those that can be removed from a computer include, for example, flexible disks, magneto-optical disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and memory cards such as flash memory. Furthermore, examples of recording media that are fixed to a computer include hard disks and ROMs (Read Only Memory). Furthermore, SSDs (Solid State Drives) ) can be used as a recording medium that can be removed from a computer or the like, or as a recording medium that is fixed to a computer or the like. [Explanation of symbols]

[0081] 1. Image processing device 2. Communications equipment 10 Control Unit 11 CPU 12 Main memory 13A Deserializer 13B Serializer 16 Input / output section 21 Main microcomputer 22 Sub-microcomputer 23 Video IC 24 Video IF

Claims

1. An image processing device that processes and outputs images from a plurality of vehicle-mounted cameras to an in-vehicle information processing device having a function of detecting fixation of input images, and a controller that instructs the in-vehicle information processing device to disable the function of detecting the stuck state when it is determined that any one of the plurality of in-vehicle cameras is malfunctioning and in a predetermined specific state among a plurality of states defined by a combination of the state of the vehicle's shift lever and the installation location of the malfunctioning in-vehicle camera. Image processing device.

2. 2. The image processing device according to claim 1, wherein the controller instructs the in-vehicle information processing device to disable the function of detecting the sticking when a front camera that captures images in front of the vehicle fails when the shift lever of the vehicle is in a forward drive position or a neutral position.

3. 2. The image processing device according to claim 1, wherein the controller instructs the in-vehicle information processing device to disable the function of detecting the sticking when a rear camera that takes pictures of the rear of the vehicle when the shift lever of the vehicle is in a reverse drive position fails.

4. An information processing system including an in-vehicle information processing device having a function of detecting image fixation, an image processing device that processes and outputs images from a plurality of in-vehicle cameras, and the in-vehicle information processing device, The image processing device includes: and a controller that instructs the in-vehicle information processing device to disable the function of detecting the stuck state when it is determined that any one of the plurality of in-vehicle cameras is malfunctioning and in a predetermined specific state among a plurality of states defined by a combination of the state of the vehicle's shift lever and the installation location of the malfunctioning in-vehicle camera. Information processing system.

5. An image processing device that processes and outputs images from multiple in-vehicle cameras for an in-vehicle information processing device having a function of detecting fixation of input images, An information processing method that instructs the in-vehicle information processing device to disable the function of detecting sticking when it is determined that any of the multiple in-vehicle cameras is malfunctioning, in a predetermined specific state from among multiple states defined by a combination of the state of the vehicle's shift lever and the installation location of the malfunctioning in-vehicle camera.

6. An in-vehicle information processing device has a function of detecting the fixation of input images, and an image processing device processes and outputs images from multiple in-vehicle cameras. A program for instructing the in-vehicle information processing device to disable the function of detecting sticking when it is determined that any of the multiple in-vehicle cameras is malfunctioning, in a predetermined specific state from among multiple states defined by a combination of the state of the vehicle's shift lever and the installation location of the malfunctioning in-vehicle camera.

Images