Image processing apparatus, information processing system, information processing method, and program product

Abstract

The invention relates to an image processing apparatus, an information processing system, an information processing method, and a program product. The present invention makes it possible to easily identify a fault location when a fault occurs in the display of an image. The image processing device processes images of a plurality of in-vehicle cameras and outputs the processed images to an in-vehicle information processing device having a function of sensing stagnation of an input image. When it is determined that any one of the plurality of in-vehicle cameras has failed, a controller of the image processing apparatus determines which of a plurality of states is defined by a combination of a state of a shift lever of a vehicle and an installation location of the failed in-vehicle camera. Moreover, when the controller is in a predetermined specific state among a plurality of states, the controller instructs the in-vehicle information processing device to invalidate the function of sensing the clamping stagnation.

Description

Technical Field

[0001] This invention relates to image processing apparatus, information processing system, information processing method, and program. Background Technology

[0002] Conventionally, in vehicle-mounted camera devices, a fault diagnosis processing unit has been proposed that diagnoses whether the data line signal of the imaging element is stuck (for example, see Patent Document 1 below). In this technology, the image data acquired by the imaging element has a complete imaging area divided into a valid image area and an invalid image area. Furthermore, a diagnostic data area is provided in the invalid image area, which contains stuck diagnostic data for diagnosing whether the data line signal of the imaging element is stuck.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2016 / 117401

[0006] Furthermore, an image processing device, such as a camera system called a Surround View Monitor (SVM), is used to synthesize images from cameras mounted on the front, rear, left, and right sides of the vehicle and display the synthesized images. Additionally, an in-vehicle information processing device, such as an In-Vehicle Infotainment (IVI), is used. When displaying images from this image processing device via an information processing device like an IVI, problems arise regarding the collaboration between the image processing device and the in-vehicle information processing device.

[0007] For example, in-vehicle information processing devices sometimes have mechanisms to sense image lag. On the other hand, the image processing device in the example of SVM has a mechanism to sense whether connected cameras are not connected or whether connected cameras have malfunctioned. Furthermore, in the event that a connected camera is not connected or has malfunctioned, the image processing device may, for example, output a monochrome (e.g., full blue) image instead of the image from the malfunctioning camera.

[0008] As a result, a portion of the composite image obtained by SVM becomes an unchanged monochrome image before being input to the vehicle's information processing unit. This may cause the vehicle's information processing unit to perceive a lag in the input image. Furthermore, when the vehicle's information processing unit determines that the input image is lag-prone, it will output, for example, a completely black image on the display as a fault protection mechanism, informing the driver of the lag.

[0009] Therefore, if the vehicle's information processing unit outputs a completely black image to the display, it becomes difficult to pinpoint the source of the fault. That is, it's difficult to determine whether the fault lies with the image processing unit itself, a camera connected to the image processing unit, or the vehicle's information processing unit. Summary of the Invention

[0010] In the disclosed embodiments, in an image processing apparatus that synthesizes and displays images from multiple cameras and an in-vehicle information processing apparatus that inputs and displays images from the image processing apparatus, the location of the fault can be easily determined if any one of the cameras malfunctions.

[0011] One aspect of the disclosed implementation is exemplified by an image processing apparatus equipped with a controller. This image processing apparatus processes images from multiple vehicle-mounted cameras and outputs them to an in-vehicle information processing apparatus having a function to sense lag in the input images. If it is determined that any one of the multiple vehicle-mounted cameras has malfunctioned, the controller determines which of several states is defined by a combination of the state of the vehicle's gear shift lever and the location of the malfunctioning vehicle-mounted camera. Furthermore, when a specific preset state is in place among the multiple states, the controller instructs the in-vehicle information processing apparatus to disable the lag sensing function.

[0012] Invention Effects

[0013] This image processing apparatus processes images from multiple vehicle-mounted cameras and outputs them to an in-vehicle information processing unit that has the function of sensing image lag. When the in-vehicle information processing unit senses lag, it outputs an image of the entire screen as a specific color (e.g., a black and white image). However, if any one of the multiple vehicle-mounted cameras malfunctions, it becomes difficult to determine which camera malfunctioned if the entire screen is displayed as a specific color.

[0014] In such image processing devices and vehicle information processing devices, when any one of the multiple vehicle cameras malfunctions, a stuttering portion may appear in the input image to the information processing device after processing by the image processing device. However, if the stuttering portion is not dominant in the input image, the vehicle information processing device may not detect the stuttering. On the other hand, if the stuttering portion is dominant in the input image, the likelihood of the vehicle information processing device detecting the stuttering increases.

[0015] Thus, the extent to which a malfunction of any vehicle camera affects the input image processed by the image processing unit depends on how the image processing unit processes the images from each vehicle camera. However, the processing method of the image processing unit for the images from each vehicle camera typically depends on the state of the gear shift lever. This is because the desired image provided to the driver differs depending on the state of the gear shift lever (e.g., gear position). Here, the desired image depends on the location of each vehicle camera on the vehicle (e.g., front, rear, etc.). Therefore, based on various states defined by the combination of the state of the vehicle's gear shift lever and the location of the malfunctioning vehicle camera, the degree to which the jammed portion will affect the processed image can be determined to some extent.

[0016] Therefore, when in a specific preset state among multiple states, this controller instructs the vehicle information processing unit to disable the function of sensing lag. By disabling this function, when the vehicle information processing unit is highly likely to sense lag, the sensing of lag can be suppressed, thereby preventing the entire screen from outputting an image of a specific color. As a result, in both image processing units that synthesize and display images from multiple vehicle cameras and vehicle information processing units that input and display images from those image processing units, this controller can easily determine the location of the fault if any one of the vehicle cameras malfunctions. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating the configuration of an information processing system according to one implementation method.

[0018] Figure 2 This is a diagram illustrating the details of an image processing device.

[0019] Figure 3 This is a diagram illustrating the details of an information processing device.

[0020] Figure 4 This table provides an example of instructions from the information processing unit and the image processing unit to be sent when the camera malfunctions, based on the position of the vehicle's gear shift lever.

[0021] Figure 5This is an example of a view shown from the front on the display of an information processing device.

[0022] Figure 6 This is an example of a view from the rear, displayed on the screen of an information processing device.

[0023] Figure 7 This is a flowchart illustrating the processing of an image processing device.

[0024] Figure 8 This is a flowchart illustrating the processing of an information processing device.

[0025] Explanation of reference numerals in the attached figures:

[0026] 1: Image processing device; 2: Communication device; 10: Control unit; 11: CPU; 12: Main memory unit; 13A: Deserializer; 13B: Serializer; 16: Input / output unit; 21: Main microcomputer; 22: Sub-microcomputer; 23: Image IC; 24: Image IF. Detailed Implementation

[0027] Hereinafter, with reference to the accompanying drawings, an information processing system 100 including an image processing device 1 and an information processing device 2, an information processing method executed by the image processing device 1, and a computer program (hereinafter referred to as the program) mounted on the image processing device 1 will be described.

[0028] <Composition>

[0029] Figure 1 This diagram illustrates the configuration of the information processing system 100 according to this embodiment. The information processing system 100 is, for example, installed in a vehicle and provides vehicle occupants (hereinafter also referred to as users) with entertainment functions, navigation, and other driving assistance functions realized through sound, images, etc.

[0030] The information processing system 100 includes: cameras C1 to C4, etc.; an image processing device 1 for connecting cameras C1 to C4, etc.; and an information processing device 2 that cooperates with the image processing device 1. It should be noted that the information processing device 2 is equipped with operating parts such as a turn switch 5 and operation buttons 6, which accept operations from the user.

[0031] In addition, Figure 1In this configuration, image processing unit 1 and information processing unit 2 are connected to a vehicle electronic control unit (ECU3). The vehicle ECU3 is connected to image processing unit 1 and information processing unit 2 via an onboard network N1. Network N1 is, for example, a controller area network (CAN), FLEXRAY (registered trademark), or other onboard local area networks (LANs). Furthermore, in... Figure 1 In this configuration, the vehicle ECU3 is connected to the image processing unit 1 via signal line LR. Signal line LR can be, for example, a signal line connected via a general purpose input / output (GPIO) interface. However, signal line LR can also be a serial communication signal line such as a serial peripheral interface (SPI) or an inter-integrated circuit (I2C).

[0032] Image processing unit 1 has a function called a surround view system (SVM). The surround view system is also called a view around system. For example, camera C1 is a camera that captures the view in front of the vehicle, camera C2 is a camera that captures the view to the right of the vehicle, camera C3 is a camera that captures the view behind the vehicle, and camera C4 is a camera that captures the view to the left of the vehicle. Cameras C1 to C4 are also referred to as the front camera, right camera, rear camera, and left camera, respectively. Cameras C1 to C4 are an example of multiple vehicle-mounted cameras.

[0033] Cameras C1 through C4, etc., input image signals to image processing device 1, for example, following standards such as Gigabit Multimedia Serial Link (GMSL), Low Voltage Differential Signaling (LVDS), Gigabit Video Interface (GVIF), and Flat Panel Display Link (FPD-Link). The SVM uses the images from cameras C1 through C4 as a basis to generate, for example, a composite image obtained from a top-down view of the vehicle, and inputs it as an image signal to information processing device 2. The image signal input to information processing device 2 also conforms to standards such as GMSL, LVDS, GVIF, and FPD-Link.

[0034] Furthermore, the image processing device 1 sends the output of the information processing device 2 to the display 25 (see reference). Figure 3 The switching signal is used to switch between displaying images from cameras C1 to C4, including composite images obtained by SVM, and displaying images other than those from cameras C1 to C4. Figure 2 Further explanation is provided below.

[0035] Furthermore, the image processing unit 1 receives a reverse gear signal from the vehicle ECU 3 via signal line LR, which indicates that the gear shift lever is in a rearward driving position (reverse gear position). Moreover, the image processing unit 1 receives signals indicating the various positions of the vehicle's gear shift lever, including the reverse gear signal, from the vehicle ECU 3 via network N1. For example, when the image processing unit 1 receives a reverse gear signal indicating that the gear shift lever is in reverse gear from signal line LR, it outputs images from cameras C1 to C4 to the display 25 (see reference 2) by switching signal instruction information processing unit 2. Figure 3 This image may include, for example, a composite image obtained by an SVM. It should be noted that the information processing unit 2 may also output the rear monitoring image obtained by the camera C3 that is filming the rear of the vehicle to the display 25 instead of the composite image obtained by the SVM.

[0036] Furthermore, in this embodiment, when any of the cameras C1 to C4 malfunctions, the image processing device 1 will perform different processing based on the position of the vehicle's gear shift lever. The different processing based on the gear shift lever position will be as follows: Figure 4 Please provide an explanation.

[0037] Information processing device 2 is an example of an in-vehicle information processing device, also referred to as, for example, an in-vehicle infotainment system (IVI), a display audio system (DA), or a head / unit (H / U). That is, information processing device 2 may have, for example, audio, video, and navigation functions. However, information processing device 2 is not limited to in-vehicle imaging equipment as described above.

[0038] The turn signal switch 5 and the operation button 6 are user interfaces that allow the user to receive switching instructions from the information processing device 2 to the display 25. When the turn signal switch 5 is operated, the information processing device 2 displays the composite image from the image processing device 1 on the display 25. When the composite image from the image processing device 1 is displayed on the display 25, if the operation button 6 is operated, the information processing device 2 outputs other displays to the display 25 instead of the composite image from the image processing device 1. These other displays may include, for example, television broadcasts, images for operating audio functions, or car navigation displays.

[0039] It should be noted that when the turn signal switch 5 and the operation button 6 are operated, the information processing device 2 notifies the image processing device 1 via the network N1 that these operations have been accepted. In this way, the image processing device 1 instructs the information processing device 2 to output the image corresponding to the operation of the turn signal switch 5 and the operation button 6 to the display 25 via a switching signal (camera ON / OFF).

[0040] In addition, the information processing device 2 is connected to the vehicle ECU 3 via network N1 and collects information such as the vehicle's status from the vehicle ECU 3. The vehicle's status includes, for example, the position of the gear shift lever, speed, acceleration, and steering angle.

[0041] Figure 2 This is a diagram illustrating details of the image processing apparatus 1. It should be noted that... Figure 2 The document also includes cameras C1 to C4, an information processing unit 2, and a vehicle ECU 3. The image processing unit 1 has a central processing unit (CPU 11), a main storage unit 12, a deserializer 13A, a serializer 13B, an array of processor elements 14, and an input / output unit 16.

[0042] The deserializer 13A converts the GMSL and other image signals input from cameras C1 to C4 via serial communication into parallel signals and transfers them to the processor element array 14. The serializer 13B stores the image data (e.g., the amount of one line) processed by the processor element array 14, converts it into a serial image signal, and supplies it to the information processing device 2.

[0043] Each processor element within the processor element array 14 has multiple arithmetic logic units (ALUs) that perform various operations such as addition, comparison, product, and product summation. Each processor element performs pipelined processing via its ALUs and performs image processing in parallel on the frames input to the image processing apparatus 1. It should be noted that the image processing apparatus 1 may have one or more processors (e.g., digital signal processors (DSPs)) instead of the processor element array 14.

[0044] Deserializer 13A is connected to processor element array 14 via network N2. Furthermore, processor element array 14 is connected to serializer 13B via network N3. It should be noted that networks N2 and N3 can be a single network. Networks N2 and N3 may include, for example, a crossbar switch. Networks N2 and N3 allow parallel access to deserializer 13A and serializer 13B from processor element array 14, as well as parallel data transmission between processor elements.

[0045] CPU 11 executes a computer program that is executable in the main storage unit 12 and controls various parts of the image processing apparatus 1. Specifically, CPU 11 executes the deserializer 13A to parallelize the image signals input from cameras C1 to C4 and initiates image processing performed by the processor element array 14. Here, image processing includes, for example, the generation of composite images using an SVM (Simultaneous Visualization Machine). Furthermore, CPU 11 controls the serializer 13B to serialize the composite images, etc., after image processing by the processor element array 14. CPU 11 is also referred to as a processor. However, CPU 11 is not limited to a single processor and can also be configured with multiple processors.

[0046] The main storage unit 12, also simply referred to as memory, stores computer programs executed by the CPU 11 and data processed by the CPU 11. The main storage unit 12 can be Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), etc. It should be noted that the CPU 11 and the main storage unit 12 can be collectively referred to as the control unit 10. The control unit 10 is an example of a controller.

[0047] The input / output unit 16 is an interface for communicating with external devices of the image processing device 1, such as the information processing device 2 or the vehicle ECU 3. As mentioned above, the input / output unit 16 is, for example, an interface such as GPIO. However, the input / output unit 16 can also be an interface to serial buses such as SPI or I2C. In this embodiment, the image processing device 1 sends instructions to the information processing device 2 via the input / output unit 16 to the display 25 (see reference 1). Figure 3 The output signal switches between different scenes. For example, the switching between displaying the composite image obtained from the SVM and displaying a scene other than the composite image.

[0048] Furthermore, the input / output unit 16 may include an interface for the signal line LR connected to the vehicle ECU3. Also, the input / output unit 16 may include an interface to the network N1. Additionally, computer programs executed by the CPU 11 can be loaded into the main storage unit 12 from an external device via the input / output unit 16, for example.

[0049] Figure 3 This is a diagram illustrating the details of the information processing device 2. It should be noted that... Figure 3 The image processing device 1 is also described in the text. The information processing device 2 includes a main microcomputer controller (hereinafter referred to as main microcomputer 21), a secondary microcomputer 22, an image IC 23, an image interface (hereinafter referred to as image IF 24), and a display 25.

[0050] The main microcomputer 21 is, for example, a device called a System on a Chip (SoC), which houses various components on a single chip. The main microcomputer 21, for example, has a central processing unit (CPU) and memory. The CPU executes computer programs that are expanded in an executable manner in memory, providing the functions of the main microcomputer 21. The CPU is not limited to a single processor; it can also be composed of multiple processors. Furthermore, in addition to the CPU, the main microcomputer 21 may also have a graphics processing unit (GPU), a digital signal processor (DSP), etc.

[0051] Memory stores computer programs executed by the CPU and data processed by the CPU. Memory types include Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), and Read-Only Memory (ROM).

[0052] The main microcomputer 21, for example, includes an image compositing unit 211 as a module consisting of a CPU and a memory that expands programs in an executable manner. The image compositing unit 211 generates a superimposed image and returns it to the image IC 23. This superimposed image is obtained by overlaying an image (moving image) input from the image processing device 1 via the image IF 24 and the image IC 23. The image input from the image processing device 1 is, for example, a composite image obtained by SVM, or a rear-view monitoring image obtained by capturing images behind a vehicle. The superimposed image is, for example, an image overlaid with additional information for driver assistance, etc., on the image input from the image processing device 1.

[0053] Additional information may include, for example, guide lines that show the positional relationship between the subject or constituent elements in the image input from the image processing device 1 and the vehicle equipped with the information processing device 2. These guide lines may be pre-stored as image data in the memory of the main computer 21. Furthermore, additional information may also include image objects or strings that attract the driver's attention.

[0054] Furthermore, the main microcomputer 21 is connected to the auxiliary microcomputer 22 via serial communication such as CAN. The main microcomputer 21 receives switching signals (camera ON / OFF signals) from the image processing device 1 via the auxiliary microcomputer 22. In addition, the main microcomputer 21 accesses the network N1 via the auxiliary microcomputer 22, thereby communicating with the image processing device 1, the vehicle ECU 3, etc.

[0055] The image IC23 has the same characteristics as Figure 2 The image processing device 1 has the same components. The image IC 23 includes, for example, a CPU, a memory, a processor element array, and an input / output unit. The CPU controls the processor element array via a computer program in the memory, providing the functions of the image IC 23. The image IC 23 receives image signals such as composite images of SVM from the image processing device 1 via the image IF 24, decodes the received image signals, and converts them into images in a format that can be displayed on the display 25.

[0056] In normal conditions where no abnormalities such as jamming are detected, the image IC 23 converts the image into a displayable format and transfers it to the image compositing unit 211 mounted on the main microcomputer 21, and acquires the aforementioned superimposed image from the image compositing unit 211. Furthermore, the image IC 23 outputs the acquired superimposed image to the display 25 for display.

[0057] In this embodiment, the image IC 23 includes a lag sensing unit 231. The lag sensing unit 231 senses abnormalities in the host computer 21, etc. Specifically, the lag sensing unit 231 determines whether the superimposed image (image) sent from the image synthesis unit 211 of the host computer 21 is a lag image, thereby sensing lag. A lag image is the same image in which each frame constituting the image remains unchanged. When the lag sensing unit 231 senses that lag has occurred in the image, fault protection processing is performed. Fault protection processing, for example, is the process of outputting a black image across the entire display 25. The image IC 23 uses the black image to make the user aware that the host computer 21 has malfunctioned.

[0058] The sub-microcomputer 22 has various input / output interfaces, providing communication capabilities with the external information processing device 2. For example, the sub-microcomputer 22 has an interface for receiving switching signals (camera ON / OFF signals). Furthermore, the sub-microcomputer 22 is connected to the image IC 23 and image IF 24 via serial communication, such as I2C or SPI. In addition, the sub-microcomputer 22 has an interface for connecting to network N1.

[0059] The image IF 24 is an interface for receiving image signals from the image processing device 1. The image IF 24 receives signals conforming to standards such as GMSL, LVDS, GVIF, and FPD-Link. The display 25 outputs information such as images sent from the image IC 23. The display 25 may be, for example, a liquid crystal display, an electroluminescent panel, or an organic light-emitting diode (OLED).

[0060] (Example of control displayed on the screen)

[0061] Figure 4 This is a table illustrating an instruction from the image processing unit 1, which notifies the information processing unit 2 of the position of the vehicle's gear shift lever in the event of a malfunction in cameras C1 through C4. Figure 4 The table illustrates the relationship between the malfunctioning cameras C1 to C4 (faulty cameras), the image output by the image processing unit 1 (SVM) based on the position of the shift lever when the camera malfunctions, and the instructions notified by the image processing unit 1 to the information processing unit 2. In this embodiment, as described above, camera C1 is designated as the front camera, camera C2 as the right camera, camera C3 as the rear camera, and camera C4 as the left camera.

[0062] For example, if the malfunctioning camera is the front camera (C1), and the gear shift lever is in the D (drive, forward) or N (neutral) position, the image processing unit 1 notifies the information processing unit 2 of an instruction to "not perform jam sensing". When the gear shift lever is in the D or N position, in the SVM image generated by the image processing unit 1, the display 25 will show a frontal view (refer to) where the image from the front camera (C1) occupies most of the screen. Figure 5 (A) and (B)). For example Figure 5 As illustrated in the example, the SVM image is a combination of an overall image (V1) obtained from a top-down view of the vehicle's surroundings and a camera image (V2) obtained from a specific direction (e.g., the direction in front of the vehicle as captured by the front camera (C1)). Furthermore, the camera image (V2) from the specific direction occupies a larger proportion of the image than the overall image (V1).

[0063] In such a scenario, if the front camera (C1) malfunctions, the image processing unit 1 will output the image from the front camera (C1) as a monochrome (e.g., full blue) image. As a result, the monochrome image will dominate in the SVM screen. Consequently, the information processing unit 2 will determine that the image input from the image processing unit 1 is stuck, and there is a high probability that the entire screen of the display 25 will turn black.

[0064] In this situation, if the information processing device 2 does not perform jam sensing as instructed, the image input from the image processing device 1 will be displayed on the display 25 as is. As a result, the user can easily determine which of the cameras C1 to C4 has malfunctioned. Figure 4 Each line of the example illustrates various states defined by the combination of the vehicle's gear shift lever state and the location of the malfunctioning cameras C1 through C4, etc. Furthermore, in Figure 4 The state where the front camera (C1) malfunctions and the gear shift lever is in the D or N position is an example of a specific preset state among the above multiple states.

[0065] Furthermore, if the malfunctioning camera is the front camera (C1), and the gear shift lever is in the P (Park) position, the image processing unit 1 notifies the information processing unit 2 of an instruction to "perform jam sensing". When the gear shift lever is in the P position, in the SVM image generated by the image processing unit 1, the display 25 will show the view other than the forward view, for example, the left and right sides of the vehicle's direction of travel, which occupy most of the side view. In this case, even if the front camera (C1) malfunctions, the image processing unit 1 outputs the image from the front camera (C1) as a monochrome (e.g., full blue) image, and the monochrome image occupies a small proportion of the screen. Therefore, the information processing unit 2 determines that the possibility of jamming in the image input from the image processing unit 1 is low. Therefore, it notifies the information processing unit 2 of an instruction to "perform jam sensing".

[0066] Similarly, if the malfunctioning camera is the front camera (C1), and the gear shift lever is in the R (reverse, reverse driving) position, the image processing unit 1 notifies the information processing unit 2 of an instruction to "perform jam sensing". When the gear shift lever is in the R position, the image displayed on the display 25 in the SVM image generated by the image processing unit 1 will show the rear view of the vehicle occupying most of the rearward perspective. Therefore, similar to the case when the gear shift lever is in the P (park) position, the information processing unit 2 determines that the possibility of jamming in the image input from the image processing unit 1 is low. Therefore, it notifies the information processing unit 2 of an instruction to "perform jam sensing".

[0067] Furthermore, for example, if the malfunctioning camera is the rear camera (C3), and the gear shift lever is in the R (reverse, reverse driving) position, the image processing unit 1 notifies the information processing unit 2 of an instruction to "not perform jam sensing". When the gear shift lever is in the R position, in the SVM image generated by the image processing unit 1, the display 25 will show a rear view (refer to) where the image from the rear camera (C3) occupies most of the frame. Figure 6(A)(B)). Furthermore, if the rear camera (C3) malfunctions, the image processing unit 1 will output the image from the rear camera (C3) as a monochrome (e.g., full blue) image. In this case, the information processing unit 2 determines that the image input from the image processing unit 1 is stuck, and there is a high probability that the entire screen of the display 25 will turn black. Therefore, it will notify the information processing unit 2 of an instruction to "not perform stuck detection". Figure 4 The state where the rear camera (C3) malfunctions and the gear shift lever is in the R position is an example of a specific preset state among a variety of states defined by the combination of the state of the vehicle's gear shift lever and the location of the malfunctioning vehicle camera.

[0068] However, if the malfunctioning camera is the rear camera (C3), and the gear shift lever is in the D, N, or P position, the image processing unit 1 notifies the information processing unit 2 of an instruction to "perform jam sensing." In this case, even if the rear camera (C3) malfunctions, the image processing unit 1 outputs the image from the rear camera (C3) as a monochrome (e.g., full blue) image, and the monochrome image occupies a smaller proportion of the screen. Therefore, the information processing unit 2 determines that the possibility of jamming in the image input from the image processing unit 1 is low. Therefore, it notifies the information processing unit 2 of an instruction to "perform jam sensing."

[0069] Similarly, when the malfunctioning camera is the right camera (C2) or the left camera (C4), the monochrome (e.g., full blue) image occupies a small proportion of the screen regardless of the gear shift lever's position. For example, if the gear shift lever is in the P (park) position, the SVM image generated by the image processing unit 1 will display a view on the display 25 other than the forward view, for example, images showing the left and right sides of the vehicle's direction of travel occupying most of the side view. However, even if either the right camera (C2) or the left camera (C4) malfunctions, the monochrome image will not occupy a large proportion of the screen. Therefore, when the malfunctioning camera is the right camera (C2) or the left camera (C4), the image processing unit 1 will notify the information processing unit 2 of an instruction to "perform jam sensing" regardless of the gear shift lever's position. That is, in Figure 5 In the case where the malfunctioning camera is the right camera (C2) or the left camera (C4), the image processing device 1 will notify the information processing device 2 of the instruction to "perform lag sensing" in all camera images.

[0070] Figure 5 This is an example of a front-view image displayed on the display 25 of the information processing device 2. That is, Figure 5This is an example of the screen when the gear shift lever is in the D (drive, moving forward) or N (neutral) position.

[0071] Figure 5 Example (A) illustrates the SVM screen output to the display 25 by the image processing unit 1 and the information processing unit 2 when cameras C1 through C4 are all functioning correctly. In this example, the SVM screen is a combination of an overall image (V1) showing the area around the vehicle and a camera image (V2) from a specific direction. Furthermore, the camera image (V2) from the specific direction is larger than the overall image (V1), occupying approximately two-thirds of the SVM screen. Figure 5 In (A), the gear shift lever is in the D or N position, and the camera image (V2) in a specific direction is a front view, which is an image of the front of the vehicle.

[0072] Figure 5 Example (B) Figure 5 The SVM image in case the front camera (C1) malfunctions (A). In this case, the image processing unit 1 configures a monochrome (e.g., full blue) image of the area A1 in front of the vehicle in the overall image (V1) displaying the vehicle's surroundings and the forward view (camera image V2), and outputs it to the information processing unit 2. It should be noted that normal images are acquired from the rear camera (C3), right camera (C2), and left camera (C4). In this case, according to Figure 4 Information processing device 2 does not perform lag sensing. Therefore, information processing device 2 will process the SVM image input from image processing device 1. Figure 5 (B) is output as is to the display 25. It should be noted that... Figure 5 , Figure 6 In the example, a single color (e.g., pure blue) is shown using a shading line.

[0073] Figure 5 Example (C) Figure 5 The SVM image in case the rear camera (C3) malfunctions (A). In this case, the image processing unit 1 displays a monochrome (e.g., full blue) image of the area A3 behind the vehicle in the overall image (V1) showing the area around the vehicle, and outputs it to the information processing unit 2. It should be noted that the front camera (C1), right camera (C2), and left camera (C4) will output normal images.

[0074] In this case, the image from the malfunctioning front camera (C1) is output to the front view (camera image V2). Therefore, the proportion of monochrome image occupying the entire SVM image is not large. Therefore, according to Figure 4Although the information processing device 2 performs lag sensing, the possibility of falsely determining that the image is lag-prone is low. Therefore, the information processing device 2 will process the SVM image input from the image processing device 1 ( Figure 5 The (C) is output as is to the display 25.

[0075] Figure 5 Example (D) Figure 5 The SVM image in case (A) all cameras C1 to C4 malfunction. In this case, image processing unit 1 outputs monochrome (e.g., full blue) images to information processing unit 2 in all areas A1 to A4 of the overall image (V1) displaying the vehicle's surroundings. Furthermore, since the front camera (C1) malfunctions, a monochrome (e.g., full blue) image is also configured in the forward view (camera image V2) and output to information processing unit 2. In this case, according to... Figure 4 Information processing device 2 does not perform lag sensing. Therefore, information processing device 2 will process the SVM image input from image processing device 1 ( Figure 5 The (D) is output as is to the display 25.

[0076] Figure 6 This is an example of a rear-view image displayed on the display 25 of the information processing device 2. That is, Figure 6 This is an example of a screen when the gear shift lever is in the R (reverse, moving backward) position.

[0077] Figure 6 Example (A) illustrates an SVM image in a rear view where the rear camera (C3) malfunctions. In this case, image processing unit 1 configures a monochrome (e.g., full blue) image for the area A3 behind the vehicle in the overall image (V1) displaying the vehicle's surroundings and the rear view (camera image V2), and outputs it to information processing unit 2. Furthermore, normal images are acquired from the front camera (C1), right camera (C2), and left camera (C4). In this case, according to... Figure 4 Information processing device 2 does not perform lag sensing. Therefore, information processing device 2 will process the SVM image input from image processing device 1 ( Figure 6 The (A) is output as is to the display 25.

[0078] Figure 6 Example (B) Figure 6The SVM image in case (A) is faulty, but not the rear camera (C3). In this case, the image processing unit 1 configures a monochrome (e.g., full blue) image for the area A1 in front of the vehicle in the overall image (V1) displaying the vehicle's surroundings, and outputs it to the information processing unit 2. Furthermore, normal images are acquired from the right camera (C2), rear camera (C3), and left camera (C4). However, the image from the undamaged rear camera (C3) is output in the rear view (camera image V2). Therefore, the proportion of the monochrome image occupying the entire SVM image is not large. Therefore, according to... Figure 4 Although the information processing device 2 performs lag sensing, the possibility of misjudging image lag is low. Therefore, the information processing device 2 will process the SVM image input from the image processing device 1 (… Figure 6 (B) is output as is to the display 25.

[0079] (Processing procedure)

[0080] Figure 7 This is a flowchart illustrating the processing of image processing apparatus 1. The control unit 10 of image processing apparatus 1 executes as an example of a controller. Figure 7 The image processing device 1, for example, begins processing when the vehicle's auxiliary power is turned on. Figure 7 The image processing device 1 can also begin processing at the same time as the power to the information processing device 2 is turned on. Figure 7 The image processing device 1 can also initiate driver assistance processing (e.g., when the information processing device 2 starts the process). Figure 8 (Start at the same time) Figure 7 The processing.

[0081] In this process, the image processing unit 1 monitors the image signals from each camera C1 to camera C4 (S1). Furthermore, the image processing unit 1 determines whether the front camera (C1) has malfunctioned (S2). If the front camera (C1) has malfunctioned, the image processing unit 1 converts the image from the front camera (C1) into a monochrome (e.g., full blue) image. Then, the image processing unit 1 synthesizes an SVM image based on the monochrome image and the images from the other cameras C2 to C4, and provides it to the information processing unit 2.

[0082] Furthermore, the image processing device 1 determines whether the gear shift lever is in position D or N (S3). The case where the front camera (C1) malfunctions in the determination in S2 and the gear shift lever is in position D or N in the determination in S3 is an example of a specific preset state among multiple states defined by the combination of the vehicle's gear shift lever state and the location of the malfunctioning vehicle camera. If the gear shift lever is in position D or N, the image processing device 1 notifies the information processing device 2 of an instruction not to perform jam sensing (S4). The processing in S4 is an example of instructing the information processing device 2 to disable the jam sensing function. On the other hand, if the gear shift lever is in a position other than D or N, the image processing device 1 notifies the information processing device 2 of an instruction to perform jam sensing (S5).

[0083] Furthermore, in the determination in S2, if the front camera (C1) is not malfunctioning, the image processing unit 1 determines whether the rear camera (C3) is malfunctioning (S6). If the rear camera (C3) is malfunctioning, the image processing unit 1 converts the image from the rear camera (C3) into a monochrome image. Then, the image processing unit 1 synthesizes an SVM image based on the monochrome image and images from the other cameras C1, C2, and C4, and provides it to the information processing unit 2.

[0084] Furthermore, the image processing device 1 determines whether the gear shift lever is in position R (S7). The case where the rear camera (C3) malfunctions in the determination in S6, and the gear shift lever is in position R in the determination in S7, is an example of a specific preset state among multiple states defined by combining the state of the vehicle's gear shift lever with the location of the malfunctioning vehicle camera. When the gear shift lever is in position R, the image processing device 1 instructs the information processing device 2 not to perform jam sensing (S8). The processing in S8 is an example of instructing the information processing device 2 to disable the jam sensing function. On the other hand, when the gear shift lever is not in position R, the image processing device 1 instructs the information processing device 2 to perform jam sensing (S9).

[0085] Furthermore, the image processing device 1 determines whether to end the processing (S10). For example, the image processing device 1 ends the processing when the vehicle's auxiliary power is turned off. Figure 7 The image processing device 1 can also terminate when the power to the information processing device 2 is turned off. Figure 7 The image processing device 1 will repeat the processing from S1 without ending the processing.

[0086] Figure 8 This is a flowchart illustrating the processing of information processing device 2. Figure 8The processing is included, for example, in driver assistance processing. The main microcomputer 21 of the information processing unit 2 starts driver assistance processing, for example, when the vehicle's auxiliary power is turned on. However, the information processing unit 2 can also start driver assistance processing, for example, when the turn signal switch 5 is operated or the vehicle's gear shift lever is turned to R.

[0087] In this process, the information processing device 2 receives an instruction from the image processing device 1 (S21). Furthermore, the information processing device 2 determines whether it has received an instruction not to perform lag sensing (S22). If an instruction not to perform lag sensing is received, the information processing device 2 performs driver assistance processing in a lag-free sensing mode (S23). In this case, the information processing device 2 does not perform lag sensing on the image provided by the image processing device 1, but instead outputs it to the display 25 as is.

[0088] On the other hand, if no instruction is received to not perform jam sensing, the information processing unit 2 will perform driver assistance processing (S24) in jam sensing mode. In this case, the information processing unit 2 performs jam sensing on the image provided by the image processing unit 1. So, for example, like Figure 5 (B) Figure 5 (D) Figure 6 As in (A), when a fixed portion of the image becomes dominant, the information processing device 2 determines that the image provided by the image processing device 1 has become stuck. Furthermore, the image processing device 1 outputs a black image to the display 25.

[0089] Furthermore, the information processing device 2 determines whether to terminate the processing (S25). For example, the information processing device 2 terminates the processing when the vehicle's auxiliary power is turned off. Figure 8 The processing continues. Without ending the processing, the information processing device 2 repeats the processing from S21.

[0090] (Effects of the implementation method)

[0091] As described above, in this embodiment, when the image processing device 1 determines that any one of the multiple vehicle-mounted cameras C1 to C4 has malfunctioned, it performs the following processing. That is, when the vehicle is in a specific state preset among a variety of states defined by the combination of the state of the vehicle's gear shift lever and the location of the malfunctioning camera C1, the image processing device 1 instructs the information processing device 2 to disable the sensing jamming function.

[0092] So, for example, like Figure 5 (B) Figure 5 (D) Figure 6As in (A), even if the fixed portion of the image in the SVM image (video) provided by the image processing device 1 becomes dominant, the information processing device 2 will not output a black image due to stuck sensing. That is, the information processing device 2 outputs the SVM image provided by the image processing device 1 to the display 25 as is. Therefore, in the display of the SVM image after combining the images of multiple cameras C1 to C4, if multiple cameras C1 to C4 malfunction, the image processing device 1 can make it easy for the user to identify the location of the fault.

[0093] Furthermore, when the vehicle's gear shift lever is in the D (Drive) or N (Neutral) position, if the front camera (C1) malfunctions, the image processing unit 1 will instruct the onboard information processing unit 2 to disable the sensing jamming function. Therefore, in the event of a malfunction in the front camera (C1), the image processing unit 1 can enable the user to easily identify the location of the fault.

[0094] Furthermore, when the vehicle's gear shift lever is in the R (reverse) position, if the rear camera (C3) malfunctions, the image processing unit 1 instructs the onboard information processing unit 2 to disable the sensing jamming function. Therefore, in the event of a malfunction in the rear camera (C3), the image processing unit 1 allows the user to easily identify the location of the malfunction.

[0095] (Computer-readable recording media)

[0096] A program that enables a computer or other device (hereinafter referred to as a computer, etc.) to perform any of the aforementioned functions can be recorded in a computer-readable recording medium. Furthermore, the function can be provided by having the computer, etc., read and execute the program on the recording medium.

[0097] Here, a computer-readable recording medium refers to a recording medium capable of storing information such as data and programs through electrical, magnetic, optical, mechanical, or chemical processes, and which can be read from a computer. Examples of such recording media that are removable from a computer include floppy disks, magneto-optical disks, optical discs (CDs, Compact Discs), digital versatile discs (DVDs, Digital Versatile Discs), Blu-ray discs, and flash memory cards. In addition, recording media that are fixed to a computer include hard disks and read-only memory (ROM). Furthermore, solid-state drives (SSDs) can be used as both removable recording media to a computer and fixed recording media.

Claims

1. An image processing apparatus for processing images from multiple vehicle-mounted cameras and outputting them to a vehicle-mounted information processing apparatus having a function of sensing input images. The image processing device includes a controller that, when it is determined that any one of the plurality of vehicle-mounted cameras has malfunctioned, and the device is in a specific preset state among a plurality of states defined by the combination of the state of the vehicle's gear shift lever and the location of the malfunctioning vehicle-mounted camera, the controller instructs the vehicle information processing device to disable the sensing jamming function.

2. The image processing apparatus according to claim 1, wherein, When the vehicle's gear shift lever is in the forward driving position or in neutral, and the front camera that captures images of the front of the vehicle malfunctions, the controller instructs the on-board information processing device to disable the sensing jamming function.

3. The image processing apparatus according to claim 1, wherein, When the vehicle's gear shift lever is in the reverse position and the rear camera that captures images of the rear of the vehicle malfunctions, the controller instructs the on-board information processing device to disable the sensing jamming function.

4. An information processing system comprising: an image processing device for processing images from multiple vehicle-mounted cameras and outputting them to an in-vehicle information processing device having a function of sensing image capture; and said in-vehicle information processing device, The image processing device includes a controller that, when it is determined that any one of the plurality of vehicle-mounted cameras has malfunctioned, and the device is in a specific preset state among a plurality of states defined by the combination of the state of the vehicle's gear shift lever and the location of the malfunctioning vehicle-mounted camera, the controller instructs the vehicle information processing device to disable the sensing jamming function.

5. An information processing method, executed by an image processing device, wherein the image processing device processes images from multiple vehicle-mounted cameras and outputs them to an vehicle-mounted information processing device having a function of sensing the input image. If it is determined that any one of the plurality of vehicle-mounted cameras has malfunctioned, and the camera is in a specific preset state among a plurality of states defined by the combination of the state of the vehicle's gear shift lever and the location of the malfunctioning vehicle-mounted camera, the image processing device instructs the vehicle information processing device to disable the sensing jamming function.

6. A computer program product comprising a program in an image processing apparatus for processing images from a plurality of vehicle-mounted cameras and outputting them to an in-vehicle information processing device having a function of sensing input images, the program being configured to cause the image processing apparatus to execute: If it is determined that any one of the plurality of vehicle-mounted cameras has malfunctioned, and the vehicle is in a specific state preset among a variety of states defined by the combination of the state of the vehicle's gear shift lever and the location of the malfunctioning vehicle-mounted camera, the vehicle information processing device is instructed to disable the sensing jamming function.

Images