In-vehicle information processing device, information processing method, information processing program, and in-vehicle system

Abstract

To ensure a more thorough inspection of the vehicle's interior. [Solution] This in-vehicle information processing device includes a control unit that generates a first image of the vehicle's interior as viewed from above in a plan view, based on images of the vehicle's interior captured by a camera, and outputs a second image to a display by superimposing icon images representing objects detected in the vehicle's interior onto the first image.

Description

Technical Field

[0001] [[ID=][4]] The present invention relates to an in-vehicle information processing device, an information processing method, an information processing program, and an in-vehicle system.

Background Art

[0002] Systems for assisting driving using images captured by a camera have been proposed. In Patent Document 1, a system that can change the viewpoint of an image overlooking the surroundings of a vehicle has been proposed. In Patent Document 2, a system that complements the blind spots of a driver with camera images has been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, due to the spread of privacy glass and the like, the visibility inside the vehicle cabin has decreased as the inside of the vehicle cabin has become darker. Therefore, a driver who gets off the vehicle without noticing that a child has been left in the vehicle may lock the vehicle, resulting in the child being trapped in the vehicle. In addition, a passenger may also get off the vehicle without noticing that they have left something in the vehicle. Even if an inspection of the inside of the vehicle cabin is performed when getting off the vehicle, an oversight may occur. Such an oversight in checking the inside of the vehicle cabin is not limited to when getting off the vehicle, but may also occur when the vehicle starts. Further, even when the vehicle does not have privacy glass, an oversight in checking the inside of the vehicle cabin may occur.

[0005] One aspect of the disclosed technology aims to provide an in-vehicle information processing device, an information processing method, and an information processing program that can more reliably check the inside of the vehicle cabin. [Means for solving the problem]

[0006] One aspect of the disclosed technology is exemplified by the following in-vehicle information processing device. This in-vehicle information processing device includes a control unit that generates a first image of the vehicle's interior as viewed from above, based on images of the interior captured by a camera, and outputs a second image to a display by superimposing icon images representing objects detected in the vehicle's interior onto the first image. [Effects of the Invention]

[0007] According to the disclosed technology, it will be possible to more reliably check the interior of the vehicle. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of a vehicle according to this embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of an in-vehicle information processing device according to the present invention. [Figure 3] Figure 3 shows an example of a processing block for an in-vehicle information processing device according to an embodiment. [Figure 4] Figure 4 shows an example of a contact management table held by the management unit in this embodiment. [Figure 5] Figure 5 shows an example of a weight sensor management table included in the management unit in this embodiment. [Figure 6] Figure 6 shows an example of an in-car composite image generated by the compositing unit in an embodiment. [Figure 7] Figure 7 shows an example of an external composite image generated by the composite unit in an embodiment. [Figure 8] Figure 8 shows an example of a guidance video output by the output unit to the display in this embodiment. [Figure 9] Figure 9 shows an example of a meter in which an object detection warning light, which is an example of a status lamp, is illuminated by the output unit in an embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a processing flow of an in-vehicle information processing apparatus according to an embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a processing block of an in-vehicle information processing apparatus according to a first modification. [Figure 12] FIG. 12 is a first diagram showing an example of a guidance video output by an output unit to a display in the first modification. [Figure 13] FIG. 13 is a second diagram showing an example of a guidance video output by an output unit to a display in the first modification. [Figure 14] FIG. 14 is a diagram showing an example of a vehicle according to a second modification. [Figure 15] FIG. 15 is a diagram showing an example of a processing block of an in-vehicle information processing apparatus according to a second modification. [Figure 16] FIG. 16 is a diagram showing an example of a synthesized video under the vehicle generated by a synthesizing unit in the second modification. [Figure 17] FIG. 17 is a diagram showing an example of a guidance video output by an output unit to a display in the second modification. [Figure 18] FIGS. 18A, 18B, and 18C are diagrams for explaining generation of a guidance video in a third modification. [Figure 19] FIG. 19 is a diagram showing an example of a processing block of an in-vehicle information processing apparatus according to a fourth modification. [Figure 20] FIG. 20 is a first diagram showing an example of a guidance video output by an output unit to a display in the fourth modification. [Figure 21] FIG. 21 is a second diagram showing an example of a guidance video output by an output unit to a display in the fourth modification. [Figure 22] FIG. 22 is a third diagram showing an example of a guidance video output by an output unit to a display in the fourth modification.

MODE FOR CARRYING OUT THE INVENTION

[0009] <Embodiment> Hereinafter, embodiments will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a vehicle 100 according to an embodiment. FIG. 1 is a view of the vehicle 100 seen from above. Also, in FIG. 1, for the purpose of explaining the interior of the vehicle 100, the illustration of the ceiling is omitted. The vehicle 100 is, for example, a passenger car. Note that the vehicle 100 is not necessarily limited to a passenger car. The vehicle 100 may be, for example, a bus.

[0010] The vehicle 100 includes a driver's seat 1, a front passenger seat 2, a right rear seat 3, a center rear seat 4, and a left rear seat 5 as seats for passengers. The driver's seat 1 and the front passenger seat 2 are arranged in a row along the width direction of the vehicle 100. Also, the right rear seat 3, the center rear seat 4, and the left rear seat 5 are arranged in a row along the width direction of the vehicle 100 behind the driver's seat 1 and the front passenger seat 2. When not distinguishing the driver's seat 1, the front passenger seat 2, the right rear seat 3, the center rear seat 4, and the left rear seat 5, they are also referred to as seat 6.

[0011] In front of the driver's seat 1 and the front passenger seat 2, a dashboard 10 is arranged. Between the driver's seat 1 and the front passenger seat 2, a shift lever 32 is arranged. In front of the driver's seat 1, a steering wheel 20 and a meter 31 are arranged. The meter 31 includes, for example, a status lamp for notifying an abnormality of the vehicle 100, a speedometer, and a tachometer.

[0012] Also, the vehicle 100 is provided with a right front door 51, a left front door 52, a right rear door 53, and a left rear door 54 as doors for getting on and off. The right front door 51 is arranged on the right side of the driver's seat 1. The left front door 52 is arranged on the left side of the front passenger seat 2. The right rear door 53 is arranged on the right side of the right rear seat 3. The left rear door 54 is arranged on the left side of the left rear seat 5.

[0013] A display 30 is positioned in the center of the width of the dashboard 10. The display 30 is, for example, a Liquid Crystal Display (LCD), a Plasma Display Panel (PDP), an inorganic Electroluminescence (EL) panel, or an organic EL panel. The display 30 may also have a touch panel superimposed on it for detecting touch operations by, for example, a user's finger.

[0014] A weight sensor 41 for the driver's seat, a weight sensor 42 for the passenger seat, a weight sensor 43 for the right rear seat, a weight sensor 44 for the center rear seat, and a weight sensor 45 for the left rear seat are positioned on the seat surface of each of the seats 6, respectively, to detect the weight applied to the seat 6. When not distinguishing between the driver's seat weight sensor 41, passenger seat weight sensor 42, right rear seat weight sensor 43, center rear seat weight sensor 44, and left rear seat weight sensor 45, they are also referred to as weight sensor 40. The weight sensor 40 is located inside the seat surface of seat 6 and is therefore not visible externally, but the position of the weight sensor 40 is illustrated with a dotted line in Figure 1. In Figure 1, the weight sensor 40 is located in a portion of the seat surface of seat 6, but the weight sensor 40 may be located across the entire seat surface of seat 6.

[0015] The following cameras are positioned inside the passenger compartment 110 of vehicle 100: a front right interior camera 61, a front left interior camera 62, a rear right interior camera 63, and a rear left interior camera 64. The front right interior camera 61 is positioned above the driver's seat 1 on the ceiling of vehicle 100. The front left interior camera 62 is positioned above the passenger seat 2 on the ceiling of vehicle 100. The rear right interior camera 63 is positioned above the rear right seat 3 on the ceiling of vehicle 100. The rear left interior camera 64 is positioned above the rear left seat 5 on the ceiling of vehicle 100. When not distinguishing between the front right interior camera 61, the front left interior camera 62, the rear right interior camera 63, and the rear left interior camera 64, they are also referred to as interior cameras 60. The arrangement of the in-vehicle cameras 60 within the vehicle compartment 110 is just one example; the positions and shooting directions of the multiple in-vehicle cameras 60 should be determined so that the entire interior of the vehicle compartment 110 is covered within the shooting range. In this embodiment, each of the in-vehicle cameras 60 is mounted on the ceiling, allowing the vehicle compartment 110 to be photographed from above. The in-vehicle camera 60 is just one example of a "camera".

[0016] The vehicle body 120 of vehicle 100 is equipped with a right front external camera 71, a left front external camera 72, a right rear external camera 73, and a left rear external camera 74. The right front external camera 71 is located on the right front of the vehicle body 120, and its shooting direction is directed towards the right front. The left front external camera 72 is located on the left front of the vehicle body 120, and its shooting direction is directed towards the left front. The right rear external camera 73 is located on the right rear of the vehicle body 120, and its shooting direction is directed towards the right rear. The left rear external camera 74 is located on the left rear of the vehicle body 120, and its shooting direction is directed towards the left rear. When not distinguishing between the right front external camera 71, the left front external camera 72, the right rear external camera 73, and the left rear external camera 74, they are also referred to as external cameras 70. The arrangement and shooting direction of the external camera 70 on the vehicle body 120 is just an example; the positions and shooting directions of multiple external cameras 70 should be determined so that a 360-degree range around the vehicle 100 is covered as the shooting range. The external camera 70 is an example of a "second camera".

[0017] The vehicle 100 is equipped with an in-vehicle information processing device that controls a display 30, a weight sensor 40, an in-vehicle camera 60, and an external camera 70. Figure 2 shows an example of the hardware configuration of the in-vehicle information processing device 200 according to the embodiment. The in-vehicle information processing device 200 comprises a CPU 21, a main memory unit 22, an auxiliary memory unit 23, a connection unit 24, a signal input unit 25, and a connection bus B1. The CPU 21, the main memory unit 22, the auxiliary memory unit 23, the connection unit 24, and the signal input unit 25 are interconnected by the connection bus B1.

[0018] The CPU 21 is also called a microprocessor unit (MPU) or processor. The CPU 21 is not limited to a single processor and may be in a multiprocessor configuration. Furthermore, a single CPU 21 connected via a single socket may have a multicore configuration. At least a portion of the processing performed by the CPU 21 may be performed by other processors, such as dedicated processors like a Digital Signal Processor (DSP), Graphics Processing Unit (GPU), numerical processor, vector processor, or image processing processor. Also, at least a portion of the processing performed by the CPU 21 may be performed by integrated circuits (ICs) or other digital circuits. Furthermore, at least a portion of the CPU 21 may include analog circuits. Integrated circuits include Large Scale Integrated Circuits (LSIs), Application Specific Integrated Circuits (ASICs), and Programmable Logic Devices (PLDs). PLDs include, for example, Field-Programmable Gate Arrays (FPGAs). The CPU 21 may be a combination of a processor and integrated circuits. The combination is called, for example, a microcontroller unit (MCU), system-on-a-chip (SoC), system LSI, or chipset. In the in-vehicle information processing device 200, the CPU 21 loads the program stored in the auxiliary storage unit 23 into the work area of ​​the main storage unit 22 and controls peripheral devices through program execution. This allows the in-vehicle information processing device 200 to perform processing that matches a predetermined purpose. The main storage unit 22 and the auxiliary storage unit 23 are recording media that can be read by the CPU 21. The CPU 21 is an example of a "control unit".

[0019] The main memory unit 22 is exemplified as a memory unit that is directly accessed by the CPU 21. The main memory unit 22 includes Random Access Memory (RAM) and Read Only Memory (ROM).

[0020] The auxiliary storage unit 23 stores various programs and data on a recording medium in a read / write manner. The auxiliary storage unit 23 is also called an external storage device. The auxiliary storage unit 23 stores the operating system (OS), various programs, various tables, etc. In addition, the auxiliary storage unit 23 stores various images used by each processing block, which will be described later. The OS includes an interface program that exchanges data with the display 30, weight sensor 40, in-vehicle camera 60, and external camera 70, etc., which are connected via the connection unit 24.

[0021] The auxiliary storage unit 23 may be, for example, an Erasable Programmable ROM (EPROM), a Solid State Drive (SSD), or a Hard Disk Drive (HDD). Alternatively, the auxiliary storage unit 23 may be a Compact Disc (CD) drive, a Digital Versatile Disc (DVD) drive, or a Blu-ray® Disc (BD) drive.

[0022] The connection section 24 is a connection terminal to which the display 30, weight sensor 40, in-vehicle camera 60, and exterior camera 70 are connected. For example, the connection section 24 is made of Universal S Examples include USB (USB), High-Definition Multimedia Interface (HDMI, registered trademark), and DisplayPort.

[0023] The signal input unit 25 receives signals indicating the status of the shift lever 32 and signals indicating the status of the vehicle 100's engine. For example, the signal input unit 25 receives a signal indicating the range selected by the shift lever 32. In addition, the signal input unit 25 receives signals indicating, for example, the engine speed of the vehicle 100 and whether the engine is running or stopped.

[0024] <Processing block of the in-vehicle information processing device 200> Figure 3 shows an example of a processing block of the in-vehicle information processing device 200 according to the embodiment. The in-vehicle information processing device 200 includes a synthesis unit 201, a detection unit 202, an output unit 203, and a management unit 204. The in-vehicle information processing device 200 performs processing as each unit, such as the synthesis unit 201, the detection unit 202, the output unit 203, and the management unit 204, by having the CPU 21 execute a computer program that has been expanded in executable form in the main memory unit 22.

[0025] The management unit 204 is a database that manages the contact information of the occupants, the location of the weight sensor 40, etc. The management unit 204 is built in, for example, the auxiliary storage unit 23. Figure 4 is a diagram showing an example of a contact management table 2041 that the management unit 204 has in an embodiment. The contact management table 2041 is a table that manages the contact information of the occupants of the vehicle 100. The contact management table 2041 has the fields "occupant name" and "email address". The "occupant name" field stores the name of the occupant. The "email address" field stores the email address of the occupant. Although the contact management table 2041 exemplified in Figure 4 stores the email address of the occupants, the contact management table 2041 may also store occupant contact information other than email addresses. Examples of occupant contact information other than email addresses include Social Networking Service (SNS) accounts and telephone numbers that can be used for Short Message Service (SMS). The contact management table 2041 may store these contacts in place of, or in addition to, email addresses. Furthermore, the contact management table 2041 only needs to store the contact information to which the guidance video 2031, which will be sent by the output unit 203 described later, will be sent. In other words, the contact management table 2041 does not need to store the contact information of some of the occupants of the vehicle 100.

[0026] Figure 5 shows an example of a weight sensor management table 2042 in the management unit 204 in an embodiment. The weight sensor management table 2042 is a table that manages the position of the weight sensors 40 in the in-vehicle composite image generated by the synthesis unit 201, which will be described later. The weight sensor management table 2042 has the items "Sensor" and "Position". "Sensor" stores information that identifies each of the weight sensors 40. "Position" stores information that indicates the position of the weight sensors 40 in the in-vehicle composite image. The position is represented, for example, by the XY coordinates in the XY coordinate space set on the in-vehicle composite image. In the example of Figure 5, it is assumed, for example, that the area occupied by the weight sensors 40 on the in-vehicle composite image is rectangular. The position of the weight sensors 40 is exemplified by a set of coordinates that indicate the positions of the corners located diagonally opposite each other in the rectangle. Note that the information indicating the position of the weight sensors 40 in the weight sensor management table 2042 is not limited to such a set of coordinates, and may be represented, for example, by the center coordinates of the rectangular area.

[0027] Returning to Figure 3, the synthesis unit 201 receives images captured by the in-vehicle camera 60 and the exterior camera 70 via the connection unit 24. The synthesis unit 201 synthesizes the images captured by the in-vehicle camera 60 to create an in-vehicle composite image of the passenger compartment 110 viewed from above (a top-down view). The system generates video. In other words, the composite in-car video can be described as a video that provides an overhead view of the passenger compartment 110. There are no limitations to the method used to generate the composite in-car video by combining the images of the passenger compartment 110 captured by each of the multiple in-car cameras 60; various known methods can be applied.

[0028] Figure 6 shows an example of an in-vehicle composite image 2011 generated by the compositing unit 201 in an embodiment. The in-vehicle composite image 2011 is an image of the inside of the vehicle compartment 110 viewed from above in a plan view. The in-vehicle composite image 2011 includes the vehicle compartment 110 and objects inside the vehicle compartment 110. In the example of Figure 6, the objects inside the vehicle compartment 110 include an occupant M11 seated in the driver's seat 1, an occupant M12 seated in the right rear seat 3, and a bag M13 placed between the passenger seat 2 and the left rear seat 5, which are included in the in-vehicle composite image 2011. When occupant M11, occupant M12, and bag M13 are not distinguished, they are also referred to as objects M10. The in-vehicle composite image 2011 is an example of the "first image".

[0029] The synthesis unit 201 may also synthesize images captured by the external cameras 70 to generate an external composite image showing a 360-degree view of the area around the vehicle 100 from above. The external composite image can also be described as an overhead view of the area around the vehicle 100 from above. There are no limitations on the method of generating the external composite image by synthesizing images of the area around the vehicle 100 captured by each of the multiple external cameras 70; various known methods can be applied.

[0030] Figure 7 shows an example of an external composite image 2012 generated by the composite unit 201 in an embodiment. The external composite image 2012 is, for example, an image in which a vehicle image G100 representing the vehicle 100 is superimposed on an image captured by an external camera 70. In the example in Figure 7, the case in which the vehicle 100 is parked in a parking lot is illustrated. Therefore, the external composite image 2012 includes the parking lot lanes P11 and P12 captured by the external camera 70. The vehicle image G100 is, for example, stored in advance in the auxiliary storage unit 23. The composite unit 201 can generate an image that looks like a top-down, plan view of the vehicle 100 by superimposing the vehicle image G100 acquired from the auxiliary storage unit 23 onto the image captured by the external camera 70. The external composite image 2012 is an example of a "third image".

[0031] Returning to Figure 3, the detection unit 202 detects the object M10 inside the vehicle compartment 110. The detection of the object M10 inside the vehicle compartment 110 by the detection unit 202 is performed, for example, when the vehicle 100 is stopped. When the vehicle 100 is stopped, for example, the shift lever 32 of the vehicle 100 is switched to the parking range and the engine is stopped. The detection unit 202 may detect that the shift lever 32 of the vehicle 100 has been switched to the parking range based on the signal from the shift lever 32 input to the signal input unit 25. Alternatively, the detection unit 202 may detect that the engine of the vehicle 100 has been stopped based on a signal indicating the engine status input to the signal input unit 25.

[0032] The detection unit 202 detects objects M10 inside the passenger compartment 110 based on the in-vehicle composite video 2011. Examples of objects M10 to be detected by the detection unit 202 include the occupants of the vehicle 100 and luggage such as bags placed inside the passenger compartment 110. The detection unit 202 may, for example, pre-store background video in the auxiliary storage unit 23 showing a state where no objects M10 exist inside the passenger compartment 110, and then detect the area where a difference occurs between the in-vehicle composite video 2011 and the said background video as an object M10.

[0033] Furthermore, the detection unit 202 may detect the object M10 inside the vehicle compartment 110 using a learning model constructed by machine learning with the background image and the image of the object M10 to be detected as training data. In such a case, the detection unit 202 can, for example, input the in-vehicle composite video 2011 into the learning model to detect the object M10 inside the vehicle compartment 110. When using a learning model for detection, the detection unit 202 may also notify the output unit 203 of the type of object M10 detected (for example, two types: a person and a non-person object).

[0034] Furthermore, the detection unit 202 may detect an object M10 inside the vehicle compartment 110 based on the detection signal from the weight sensor 40. In this case, the detection unit 202 receives the detection signal from the weight sensor 40 via the connection unit 24. The detection unit 202 may, for example, detect an object M10 if the weight indicated by the detection signal received from the weight sensor 40 is above a predetermined threshold. Based on the information identifying the weight sensor 40 included in the detection signal received from the weight sensor 40 and the weight sensor management table 2042, the detection unit 202 identifies the location of the weight sensor 40 that detected a weight above a predetermined threshold. The detection unit 202 may then determine that an object M10 exists at the identified location.

[0035] The detection unit 202 may detect object M10 based on the in-vehicle composite image 2011 without using the detection signal from the weight sensor 40. In such a case, the weight sensor 40 may be omitted from the vehicle 100.

[0036] The output unit 203 generates a guidance video by superimposing an icon image indicating the position of object M10 detected by the detection unit 202 onto the in-vehicle composite video 2011. The output unit 203 then outputs the generated guidance video to the display 30. For example, the output unit 203 outputs the guidance video to the display 30 when the vehicle 100 is stopped. Alternatively, if the output unit 203 receives a display instruction from an occupant of the vehicle 100, it may have the detection unit 202 detect object M10 in the passenger compartment 110 and then output the guidance video to the display 30. The display instruction is given, for example, via a touch panel on the display 30.

[0037] Figure 8 shows an example of a guidance video 2031 output by the output unit 203 to the display 30 in an embodiment. The guidance video 2031 is a video that guides the occupant to the location of object M10 inside the vehicle compartment 110. The output unit 203 superimposes the interior composite video 2011 onto the exterior composite video 2012 at the location where the vehicle image G100 is displayed. The output unit 203 then generates the guidance video 2031 by superimposing icon images F11, F12, and F13 onto the location where object M10 is detected by the detection unit 202. Icon image F11 is positioned to surround at least a portion of occupant M11. Icon image F12 is positioned to surround at least a portion of occupant M12. Icon image F13 is positioned to surround at least a portion of bag M13. Icon images F11, F12, and F13 are images that indicate the location of the detected object M10. Icon images F11, F12, and F13 are stored in advance, for example, in the auxiliary storage unit 23. Guidance video 2031 is an example of the "second video" and the "fourth video".

[0038] In the example in Figure 8, circular images were used as icon images F11, 12, and F13, but there are no limitations on the shape of icon images F11, 12, and F13. Icon images F11, 12, and F13 may be rectangles surrounding the detected crew member M12. Preferably, icon images F11, 12, and F13 do not obscure the detected object M10, as exemplified by the circular icon images F11, 12, and F13 in Figure 8. However, this does not exclude cases in which icon images F11, 12, and F13 obscure the detected object M10. Furthermore, the position of icon images F11, F12, and F13 is not limited to a position superimposed on object M10. Icon images F11, F12, and F13 may be placed, for example, in the vicinity of object M10.

[0039] Furthermore, the shapes and colors of the icon images F11, F12, and F13 may be changed according to the type of object M10 detected. In such cases, the output unit 203 can, for example, change the shape and color of the icon image according to the type of object M10 notified by the detection unit 202. The correspondence between the type of object M10 and the icon image is stored in advance in the auxiliary storage unit 23, and the output unit 203 refers to this correspondence and changes the shape and color of the icon image according to the type of object M10 notified by the detection unit 202. You just need to select the icon image.

[0040] In addition to outputting to the guidance video 2031, the output unit 203 may also notify the system that object M10 has been detected using the status lamp on the meter 31. Figure 9 shows an example of the meter 31 in which the object detection warning lamp L1, which is an example of a status lamp, is illuminated by the output unit 203 in this embodiment. When object M10 is detected by the detection unit 202, the output unit 203 illuminates the object detection warning lamp L1 in the meter 31.

[0041] Furthermore, the output unit 203 may send the guidance video 2031 via email or other means when an occupant disembarks from the vehicle 100. Here, it is preferable that the output unit 203 sends the guidance video 2031 via email or other means when the occupant, who is the driver seated in the driver's seat 1, disembarks. For example, the output unit 203 obtains the email address of the occupant of the vehicle 100 by referring to the contact management table 2041. Then, the output unit 203 can send an email with the guidance video 2031 attached to the obtained email address. The output unit 203 may also determine that an occupant has disembarked from the vehicle 100 when it detects that the engine of the vehicle 100 has stopped and then detects that the door 50 of the vehicle 100 has been locked. Alternatively, the output unit 203 may determine that the driver has disembarked from the vehicle 100 when it detects that the weight detected by the driver's seat weight sensor 41 provided in the driver's seat 1 is below a predetermined threshold and then detects that the door 50 has been locked.

[0042] Furthermore, the output unit 203 does not need to output the guidance video 2031 to the display 30 when the vehicle 100 is reversing. When the vehicle 100 is reversing, for example, the rear view of the vehicle 100 captured by the right rear external camera 73 and the left rear external camera 74 may be displayed on the display 30 to assist in reversing. By not outputting the guidance video 2031 to the display 30 when the vehicle 100 is reversing, the output unit 203 can prevent interference with such reversing assistance.

[0043] <Processing Flow> Figure 10 shows an example of the processing flow of the in-vehicle information processing device 200 according to the embodiment. Hereinafter, an example of the processing flow of the in-vehicle information processing device 200 will be described with reference to Figure 10.

[0044] In step S1, the interior of the vehicle (110) is photographed by the interior camera (60) and the exterior of the vehicle (100) is photographed by the exterior camera (70).

[0045] In step S2, the detection unit 202 determines whether the vehicle 100 is stopped or not. If it is stopped (YES in step S2), the process proceeds to step S4. If it is not stopped (NO in step S2), the process proceeds to step S3.

[0046] In step S3, the output unit 203 determines whether or not a display instruction has been received. If a display instruction has been received (YES in step S3), the process proceeds to step S4. If a display instruction has not been received (NO in step S3), the process proceeds to step S1.

[0047] In step S4, the detection unit 202 detects the object M10 inside the vehicle compartment 110 using the video footage captured by the in-vehicle camera 60 in step S1 and the detection signal from the weight sensor 40. Alternatively, as described above, the detection unit 202 may detect the object M10 inside the vehicle compartment 110 using the video footage captured by the in-vehicle camera 60 without using the detection signal from the weight sensor 40.

[0048] In step S5, the output unit 203 combines the images captured by the in-vehicle camera 60 and the exterior camera 70 in step S1. Then, the output unit 203 processes the combined image. Then, by superimposing icon images F11, F12, and F13 representing the object M10 detected in step S4, the guidance video 2031 is generated.

[0049] In step S6, the output unit 203 outputs the guidance video 2031 generated in step S5 to the display 30. Furthermore, the output unit 203 may also illuminate the object detection warning light L1 on the meter 31.

[0050] <Effects of the Embodiment> In this embodiment, icon images F11, F12, and F13 are added to the location where object M10 is detected in the guidance video 2031. In recent years, the adoption of privacy glass in automobiles has increased, so it is conceivable that the image inside the passenger compartment 110 may become dark. In this embodiment, by adding icon images F11, F12, and F13, even if the guidance video 2031 becomes dark and object M10 becomes difficult to see, the presence and location of object M10 can be clearly communicated to the occupants. In other words, according to this embodiment, confirmation inside the passenger compartment 110 becomes more reliable. Furthermore, in this embodiment, since the presence and location of object M10 can be communicated to the occupants, the likelihood of leaving items behind when alighting from the vehicle 100 is reduced. It is also reduced the likelihood of some occupants alighting from the vehicle 100 and locking the doors.

[0051] In this embodiment, the location of the object M10 detected within the passenger compartment 110 is indicated by a guidance video 2031 that provides a top-down, plan view of the interior of the passenger compartment 110. Therefore, it becomes easier to confirm the presence or absence of object M10 in the rear seats, such as the right rear seat 3, the center rear seat 4, and the left rear seat 5, which are difficult to see from the driver's seat 1. This feature is particularly effective in vehicles such as three-row seat vehicles and buses, where it is not easy to see the rear end of the passenger compartment 110 from the driver's seat 1.

[0052] Furthermore, if object M10 is detected at a location other than seat 6, it is possible that the occupants are not seated in seat 6. For example, if the occupants are children, they may not be seated in seat 6 when the vehicle 100 is about to start. According to this embodiment, the presence and location of object M10 can be clearly communicated, thus preventing the vehicle 100 from starting when not all occupants are seated in seat 6.

[0053] Furthermore, in this embodiment, if the icon image is changed according to the type of object M10 notified by the detection unit 202, the type of object M10 can be visually and clearly communicated to the occupants by the icon image placed in the guidance video 2031. For example, if an icon image representing a person exists in a location other than seat 6, the driver of the vehicle 100 can prompt that person to sit in seat 6.

[0054] In this embodiment, the guidance video 2031 is generated by superimposing the in-vehicle composite video 2011 and the exterior composite video 2012. That is, the guidance video 2031 includes both the video inside the vehicle compartment 110 and the video outside the vehicle. Therefore, according to this embodiment, the occupants can view both the conditions inside the vehicle compartment 110 and the conditions outside the vehicle at a glance.

[0055] In this embodiment, when the occupant, who is the driver seated in the driver's seat 1, gets out of the vehicle, the guidance video 2031 is sent via email or the like. Therefore, according to this embodiment, even after the occupant gets out of the vehicle 100, they can check for forgotten items in the passenger compartment 110 using the guidance video 2031.

[0056] In this embodiment, the guidance video 2031 was output to the display 30 when the vehicle 100 was stopped, but the guidance video 2031 may also be output to the display 30 when the vehicle 100 is not stopped.

[0057] <First variation> In the embodiments described above, the guidance video 2031 includes both the in-vehicle composite video 2011 and the out-of-vehicle composite video 2012. In the first modified example, a configuration in which the video output to the display 30 changes in response to zooming and zooming operations will be described. Components common to the embodiments are denoted by the same reference numerals, and their descriptions are omitted. The first modified example will be described below with reference to the drawings.

[0058] Figure 11 shows an example of a processing block of the in-vehicle information processing device 200A according to the first modified example. The in-vehicle information processing device 200A differs from the in-vehicle information processing device 200 according to the embodiment in that it includes an output unit 203A instead of an output unit 203.

[0059] The output unit 203A changes the image output to the display 30 in response to zoom-in and zoom-out operations from the occupant. Figure 12 is the first figure showing an example of a guidance image 2031A output by the output unit 203A to the display 30 in the first modified example. The guidance image 2031A has a slider bar T1 that accepts zoom-in and zoom-out operations. The guidance image 2031A illustrated in Figure 12 is an example of a "second image".

[0060] The output unit 203A enlarges and reduces the guidance video 2031A output to the display 30 in accordance with the movement of the slider T11 of the slider bar T1. In the example in Figure 12, the guidance video 2031A enlarged to its maximum extent by the slider bar T1 is shown. The guide video 2031A when enlarged to its maximum extent includes the slider bar T1 and the in-vehicle composite video 2011. When the guidance video 2031A is enlarged to its maximum extent, the in-vehicle composite video 2011, which is synthesized based on the video captured by the in-vehicle camera 60, is output to the display 30 as large as possible, while the exterior composite video 2012, which is synthesized based on the video captured by the exterior camera 70, is no longer output to the display 30. In the guide video 2031A when enlarged to its maximum extent, the in-vehicle composite video 2011 is output larger, which allows the occupants to more reliably grasp the position of the object M10 inside the passenger compartment 110.

[0061] Figure 13 is a second figure showing an example of a guidance video 2031A output by the output unit 203A to the display 30 in the first modified example. In the example of Figure 13, the guidance video 2031A is shown as being minimized by the slider bar T1. When the guidance video 2031A is minimized, the exterior composite video 2012, which includes the vehicle image G100, is output to the display 30 instead of the interior composite video 2011. That is, the minimized guidance video 2031A includes the slider bar T1 and the exterior composite video 2012. Since the interior composite video 2011 is no longer output in the minimized guidance video 2031A, the view outside the vehicle 100 can be output more prominently. Therefore, the occupants can grasp a wider area of ​​the view outside the vehicle 100. The guidance video 2031A shown in Figure 13 is an example of the "fifth video".

[0062] According to the first modified example, the range of the image output to the display 30 can be changed in response to zoom-in and zoom-out operations. Therefore, the range of the image output as the guidance video 2031A can be varied according to the occupant's request, such as when they want to check inside the passenger compartment 110 or check outside the vehicle 100. In the guidance video 2031A, zoom-in and zoom-out operations were accepted by the slider bar T1, but zoom-in and zoom-out operations may also be accepted by pinch-in and pinch-out operations on the touch panel provided on the display 30.

[0063] <Second variation> In the embodiments and the first modified example described above, the interior of the passenger compartment 110 is captured by the in-car camera 60. In this example, the exterior of the vehicle 100 was photographed by the external camera 70, but the underside of the vehicle 100 may also be photographed by the camera. Components common to the embodiment are denoted by the same reference numerals, and their descriptions are omitted. A second modified example will be described below with reference to the drawings.

[0064] Figure 14 shows an example of a vehicle 100A according to the second modified example. Figure 14 is a view of vehicle 100A from the left side. Vehicle 100A is equipped with a front bottom camera 81 and a rear bottom camera 82 on the bottom 130 of vehicle 100A to photograph the underside of vehicle 100A. The front bottom camera 81 is located on the front side of the bottom 130. The rear bottom camera 82 is located on the rear side of the bottom 130. The positions and shooting directions of the front bottom camera 81 and the rear bottom camera 82 are determined so that the area D11 in which vehicle 100A and the road surface D1 overlap when viewed from above in a plan view is covered as the shooting range. When the front bottom camera 81 and the rear bottom camera 82 are not distinguished, they are also referred to as the external bottom camera 80. In the example of Figure 14, two external bottom cameras 80 are provided, but there is no limit to the number of external bottom cameras 80. The exterior underbody camera 80 may be a single unit or three or more units, as long as it can cover the area D11 under the floor of the vehicle 100 as its shooting range. The images captured by the exterior underbody camera 80 are input to the in-vehicle information processing device 200B, for example, via the connection unit 24. The exterior underbody camera 80 is an example of a "third camera".

[0065] Figure 15 shows an example of a processing block of the in-vehicle information processing device 200B according to the second modified example. The in-vehicle information processing device 200B differs from the in-vehicle information processing device 200 according to the embodiment in that it includes a synthesis unit 201A instead of a synthesis unit 201, a detection unit 202A instead of a detection unit 202, and an output unit 203B instead of an output unit 203.

[0066] The synthesis unit 201A receives images captured by the exterior underbody camera 80 via the connection unit 24. The synthesis unit 201A synthesizes the images captured by the exterior underbody camera 80 to generate a composite image of the underside of the vehicle 100A, viewed from above in a plan view.

[0067] Figure 16 shows an example of a composite image 2011A of the underside of a vehicle generated by the composite unit 201A in the second modified example. The composite image 2011A of the underside of a vehicle is an image of the region D11 viewed from above in a plan view. In the example of Figure 16, the composite image 2011A includes a cat M21 that had crawled under the vehicle 100A and a stone M22 that was located under the vehicle 100A. When the cat M21 and the stone M22 are not distinguished, they are also referred to as object M20. The composite image 2011A of the underside of a vehicle may also include a rectangular frame R1 that schematically shows the region occupied by the vehicle 100A when viewed from above. Furthermore, the composite image 2011A of the underside of a vehicle may also include tire images R2, R2, R2, R2 that schematically show the tires of the vehicle 100A. The composite image 2011A of the underside of a vehicle is an example of the "sixth image".

[0068] Returning to Figure 15, the detection unit 202A detects object M20 located beneath vehicle 100A based on the composite image of the underside of the vehicle 2011A. The detection method for object M20 by the detection unit 202A is the same as that of the detection unit 202.

[0069] The output unit 203B generates a vehicle underbody guidance video by superimposing an icon image indicating the position of object M20 detected by the detection unit 202A onto the vehicle underbody composite video 2011A. The output unit 203B then outputs the generated vehicle underbody guidance video to the display 30.

[0070] Figure 17 shows an example of the guidance video 2031B that the output unit 203B outputs to the display 30 in the second modified example. The output unit 203B generates the guidance video 2031B by placing icon images F21 and F22 at the positions where object M20 is detected by the detection unit 202A in the vehicle underside composite video 2011A. In the example in Figure 17, icon image F21 is placed so as to be superimposed on the cat M21. Icon image F22 is placed so as to be superimposed on the stone M22. It is arranged in a circular pattern.

[0071] According to the second modification, the occupants of vehicle 100A can confirm the presence of object M20 beneath vehicle 100A without having to look underneath. Therefore, according to the second modification, it is possible to suppress the occurrence of accidents caused by starting vehicle 100A with an object such as an animal or a stone present beneath it.

[0072] In addition, a second modification may be made by combining it with the embodiment and the first modification so that guidance video 2031A and guidance video 2031B are output to the display 30 in response to zooming and zooming operations. For example, when the slider bar T1 is zoomed to the maximum, guidance video 2031A illustrated in Figure 13 is output to the display 30. When the slider bar T1 is reduced to the minimum, guidance video 2031B illustrated in Figure 17 is output to the display 30. Furthermore, when a zoom ratio between the maximum and minimum is specified by the slider bar T1, guidance video 2031A illustrated in Figure 12 is output to the display 30. In this way, by outputting guidance video 2031A and guidance video 2031B in response to zooming and zooming operations, the occupants can check the exterior of the vehicle 100A, the interior of the passenger compartment 110, and the underside of the vehicle 100A with the simple operation of sliding the slider T11 of the slider bar T1.

[0073] Furthermore, in the second modified example, the guidance video 2031 and guidance video 2031A may not be output to the display 30. In other words, the second modified example may be implemented independently of the embodiment and the first modified example.

[0074] <Third variation> In the second modified example, the guidance video 2031A was generated based on the image from the underside camera 80, but the guidance video 2031A can also be generated even if the underside camera 80 is omitted.

[0075] Figures 18A, 18B, and 18C illustrate the generation of the guidance video 2031A in a third modified example. The examples in Figures 18A, 18B, and 18C illustrate the generation of the guidance video 2031A when a vehicle 100 is moving forward on road D21. In Figure 18A, the area D22 on road D21 is captured by the right front external camera 71 and the left front external camera 72 of the vehicle 100.

[0076] Figure 18B illustrates a state where the vehicle 100 has moved further forward than in Figure 18A. In the state of Figure 18B, the region D22 captured by the right front external camera 71 and the left front external camera 72 in the stage of Figure 18A has moved to the underside of the vehicle 100A. The output unit 203B determines the positional relationship between region D22 and the vehicle 100 based on the speed of the vehicle 100, and when region D22 is located below the vehicle 100, it generates a guidance video 2031A based on the images of region D22 captured by the right front external camera 71 and the left front external camera 72. Also, in the stage of Figure 18B, images of region D23, which is located even further forward than region D22, are captured by the right front external camera 71 and the left front external camera 72.

[0077] Figure 18C illustrates a state where the vehicle 100 has moved further forward than in Figure 18B. In the state of Figure 18C, the area D22 captured in Figure 18A is located behind the vehicle 100, and the area D23 captured in Figure 18B has moved to the underside of the vehicle 100. When the area D23 is located under the vehicle 100, the output unit 203B generates a guidance video 2031A based on the images of area D23 captured by the right front external camera 71 and the left front external camera 72.

[0078] According to the third modified example, the output unit 203B can generate a guidance video 2031A that guides the occupants to objects under the floor of the vehicle 100, even when the underbody camera 80 is omitted. When vehicle 100 is moving in reverse, the output unit 203B can similarly generate the guidance video 2031A by using the right rear external camera 73 and the left rear external camera 74.

[0079] <Fourth variation> In the first modification, the image output to the display 30 was changed in response to zooming in and out operations. In the fourth modification, a configuration is described in which the image output to the display 30 is changed by specifying the line of sight, rather than by zooming in and out operations. Components common to the first modification are denoted by the same reference numerals, and their descriptions are omitted. The fourth modification will now be described with reference to the drawings.

[0080] Figure 19 shows an example of the processing block of the in-vehicle information processing device 200C according to the fourth modification. The in-vehicle information processing device 200C differs from the in-vehicle information processing device 200A according to the first modification in that it is equipped with an output unit 203C instead of an output unit 203B.

[0081] The output unit 203C switches the video output to the display 30 in response to an operation specifying the viewpoint from the occupant. Figures 20 to 22 show an example of the guidance video 2031C output by the output unit 203C to the display 30 in the fourth modified example. The guidance video 2031C has a viewpoint height specification area 2031R1 and a display area 2031R2. The viewpoint height specification area 2031R1 also has a slider bar T2 that accepts the specification of the viewpoint height.

[0082] In the viewpoint height specification area 2031R1, the viewpoint height for looking down on vehicle 100 is specified by the slider T21 of slider bar T2. In the example in Figure 20, three types of viewpoint heights can be specified: height C11, height C12, and height C13. Height C11 indicates, for example, the height above the ceiling of vehicle 100. Height C12 indicates, for example, the height of the ceiling of vehicle 100. Height C13 indicates, for example, the height of the floor of vehicle 100.

[0083] The vehicle image 2031C3 displayed in the display area 2031R2 is an image viewed from above, from the height specified by the slider T21 in the viewpoint height specification area 2031R1. The output unit 203C then switches the vehicle image 2031C3 output to the display area 2031R2 according to the position of the slider T21 on the slider bar T2.

[0084] For example, in the viewpoint height specification area 2031R1, when the slider T21 of the slider bar T2 is set to height C11, the output unit 203C outputs a vehicle image 2031C3, which shows the vehicle 100 as viewed from above the ceiling of the vehicle 100, to the display area 2031R2, as illustrated in Figure 21. Hereinafter in this specification, the vehicle image 2031C3, which shows the vehicle 100 as viewed from above the ceiling of the vehicle 100, will also be referred to as vehicle image G2. For example, vehicle image G100 may be used as vehicle image G2. In the example in Figure 21, since the position of the slider T21 of the slider bar T2 is set to height C11, vehicle image G2 is displayed in the display area 2031R2.

[0085] For example, in the viewpoint height specification area 2031R1, when the slider T21 of the slider bar T2 is set to height C12, the output unit 203C outputs a vehicle image 2031C3, which shows the view looking down into the passenger compartment 110 of the vehicle 100 from the ceiling of the vehicle 100, to the display area 2031R2, as illustrated in Figure 20. Hereinafter, in this specification, the vehicle image 2031C3, which shows the view looking down into the passenger compartment 110 of the vehicle 100 from the ceiling of the vehicle 100, will also be referred to as vehicle image G1. In the example in Figure 20, since the position of the slider T21 of the slider bar T2 is set to height C12, the vehicle image G1 is displayed in the display area 2031R2.

[0086] For example, in the viewpoint height specification area 2031R1, the slider T of slider bar T2 When 21 is set to height C13, the output unit 203C outputs a vehicle image 2031C3, which shows a view looking down from the floor of the vehicle 100 to the underside of the vehicle 100, to the display area 2031R2, as illustrated in Figure 22. Hereinafter in this specification, the vehicle image 2031C3, which shows a view looking down from the floor of the vehicle 100 to the underside of the vehicle 100, will also be referred to as vehicle image G3. For example, either the composite video of the underside of the vehicle 100 2011A (see Figure 16) or the guidance video 2031B (see Figure 17) may be used as vehicle image G3. In the example in Figure 22, the position of the slider T21 of the slider bar T2 is set to height C13, so the vehicle image G3 is displayed in the display area 2031R2.

[0087] According to the fourth modification, the video output to the display area 2031R2 can be switched between vehicle images G1, G2, and G3 in response to the operation of specifying the viewing height by operating the slider T21 of the slider bar T2. Therefore, the output unit 203C can switch the video output as guidance video 2031A in response to the occupant's request, such as when they want to check inside the passenger compartment 110 or check outside the vehicle 100. In other words, according to the fourth modification, it is possible to easily check the safety of the area around the vehicle 100 and inside the passenger compartment 110 in response to the occupant's request.

[0088] In the fourth modified example, the switching of the vehicle image 2031C3 according to the height of the slider T21 of the slider bar T2 may be performed by, for example, crossfading. When the vehicle image 2031C3 is switched from vehicle image G2 (see Figure 21) to vehicle image G1 (see Figure 20) in the output unit 203C, the ceiling of the vehicle 100 may gradually become transparent, and the passenger compartment 110 of the vehicle 100 may become visible through the ceiling of the vehicle 100.

[0089] Furthermore, in the fourth modified example, the switching of the vehicle image 2031C3 may be triggered by something other than user operation on the slider T21 of the slider bar T2. The output unit 203C may switch the vehicle image 2031C3, for example, when the adaptive cruise control is turned on or off, or when the shift lever 32 is switched to a predetermined range (for example, the reverse range). In other words, the output unit 203C may switch the vehicle image 2031C3 at a predetermined timing.

[0090] Furthermore, when the output unit 203C switches the vehicle image 2031C3 at a predetermined timing, it may switch the vehicle image 2031C3 from vehicle image G2 to vehicle image G1 and then to vehicle image G3. Alternatively, when the output unit 203C switches the vehicle image 2031C3 at a predetermined timing, it may switch the vehicle image 2031C3 from vehicle image G2 to vehicle image G1. Alternatively, when the output unit 203C switches the vehicle image 2031C3 at a predetermined timing, it may switch the vehicle image 2031C3 from vehicle image G1 to vehicle image G3. Switching the vehicle image 2031C3 at a predetermined timing makes it easier to check for safety around the vehicle 100 and inside the passenger compartment 110, for example, when starting the adaptive cruise control of the vehicle 100.

[0091] <Other variations> In the embodiments, first modified example, second modified example, and third modified example described above, multiple cameras were provided in each of the areas to be photographed, such as the exterior of the vehicle 100, the interior of the vehicle 100, and the underside of the vehicle 100. However, one camera may be provided in each of the areas to be photographed, such as the exterior of the vehicle 100, the interior of the vehicle 100, and the underside of the vehicle 100. If only one camera is used for each area, for example, a camera with a wide field of view, such as a 360-degree camera, can be used so that the area to be photographed can be covered by a single camera.

[0092] In the embodiment described above, a weight sensor 40 was used, but instead of the weight sensor 40, or in addition to the weight sensor 40, a seat belt sensor for detecting the fastening of a seat belt may be used. This may also be adopted. The seat belt sensor can also detect an occupant seated in seat 6.

[0093] The embodiments and variations disclosed above can be combined in any way.

[0094] <Computer-readable recording medium> An information processing program that enables a computer or other machine or device (hereinafter referred to as "computer, etc.") to perform any of the above functions can be recorded on a recording medium that the computer, etc. can read. By having the computer, etc. read and execute the program on this recording medium, it can be made to provide that function.

[0095] Here, a recording medium that can be read by a computer refers to a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical means and can be read by a computer. Examples of such recording media that can be removed from a computer include flexible disks, magneto-optical disks, Compact Disc Read Only Memory (CD-ROM), Compact Disc-Recordable (CD-R), Compact Disc-ReWriterable (CD-RW), Digital Versatile Disc (DVD), Blu-ray Disc (BD), Digital Audio Tape (DAT), 8mm tape, flash memory, external hard disk drives, and Solid State Drives (SSDs). In addition, recording media that are fixed to a computer include internal hard disk drives, SSDs, and ROMs. [Explanation of Symbols]

[0096] 1. Driver's seat 2...Passenger seat 3. Right rear seat 4. Center rear seat 5. Left rear seat 6 seats 10. Dashboard 20. Steering Wheel 21. CPU 22...Main memory 23...Auxiliary storage section 24. Connection part 25. Signal Input Section 30-inch display 31 meters 32. Shift lever 40. Weight sensor 41. Weight sensor for the driver's seat 42. Passenger seat weight sensor 43. Weight sensor for the right rear seat 44. Weight sensor for the central rear seat 45. Weight sensor for the left rear seat 50 doors 51. Right front door 52. Left front door 53. Right rear door 54. Left rear door 60. In-car camera 61. Right front interior camera 62. Left front interior camera 63. Right rear interior camera 64. Left rear interior camera 70. Exterior car camera 71. Right front exterior camera 72. Left front exterior camera 73. Right rear exterior camera 74. Left rear exterior camera 80. Exterior underbody camera 81. Front bottom camera 82. Rear bottom camera 100 vehicles 100A Vehicle 110...Car interior 120...vehicle body 130...bottom 200 · In-vehicle information processing unit 200A · In-vehicle information processing unit 200B - In-vehicle information processing unit 201 ··Synthesis Department 201A··Synthesis Department 202. Detection Unit 202A ··Detection Unit 203 Output Section 203A...Output section 203B...Output section 204...Management Department 2011...Composite video of the car interior 2011A · Composite video of the underside of the vehicle 2012 · Composite video of the car's exterior 2031... Informational video 2031A · Guidance video 2041 Contact Management Table 2042 ··Weight Sensor Management Table B1 connecting bus D1...Road surface F11 Icon Image F12... Icon image F13... Icon image F21... Icon image F22... Icon image G100 vehicle image L1 Object Detection Warning Light M10...object M11 Crew M12 Crew M13 bag M20...object M21 Cat M22 Stone T1 Slider Bar T11 Slider

Claims

1. The control unit generates a first image of the vehicle's interior viewed from above based on images of the interior captured by a camera, and outputs a second image to a display by superimposing icon images representing objects detected in the vehicle's interior onto the first image. In-vehicle information processing system.

2. The aforementioned icon image is superimposed on the first image so as to surround the detected object. The in-vehicle information processing device according to claim 1.

3. The aforementioned icon image is changed according to the type of object detected. The in-vehicle information processing device according to claim 1.

4. The camera includes a second camera that photographs the area outside the vehicle. The control unit generates a third image of the outside of the vehicle viewed from above in a plan view, based on the image captured by the second camera, and outputs a fourth image to the display by superimposing the second image onto the third image. The in-vehicle information processing device according to claim 1.

5. The control unit generates a fifth image in which a vehicle image, pre-stored in the memory unit and viewed from above in a plan view, is superimposed on the third image, and switches the image output to the display between the second image, the fourth image, and the fifth image according to the user's operation. The in-vehicle information processing device according to claim 4.

6. The camera includes a third camera that photographs the underside of the vehicle. The control unit generates a sixth image of the underside of the vehicle viewed from above based on the image captured by the third camera, and outputs the sixth image to the display. The in-vehicle information processing device according to claim 1.

7. The control unit switches between a vehicle image, which is stored in the memory unit in advance and is viewed from above in a plan view, the second video, and the sixth video, and outputs them to the display. The in-vehicle information processing device according to claim 6.

8. When the control unit detects the object inside the vehicle, it transmits the second video to a contact pre-stored in the memory unit. The in-vehicle information processing device according to any one of claims 1 to 6.

9. The information processing system installed in the vehicle The process involves generating a first image of the interior of the vehicle as seen from above, based on images of the interior captured by a camera, and then outputting a second image to a display, in which icon images representing objects detected inside the vehicle are superimposed on the first image. Information processing methods.

10. The information processing device installed in the vehicle, The system generates a first image of the vehicle's interior viewed from above, based on images captured by a camera, and then outputs a second image to a display by superimposing icon images representing objects detected inside the vehicle onto the first image. Information processing program.

11. A camera that films the interior of the vehicle, A display located inside the vehicle interior, The system includes an in-vehicle information processing device that generates a first image of the interior of the vehicle viewed from above based on images of the interior of the vehicle captured by the camera, and outputs a second image to the display by superimposing icon images representing objects detected in the interior of the vehicle onto the first image. In-vehicle systems.

Images