Onboard apparatus, onboard system, vehicle, and program for vehicles

Abstract

To provide a technology capable of predicting an action of a human being existing around an own vehicle even in a situation where an appearance of the human being cannot be imaged by a camera.SOLUTION: An onboard apparatus installed in an own vehicle (V1) acquires other vehicle temperature information signifying the temperature of an object component included in an exterior part of another vehicle which is at a halt, and predicts a possibility (abrupt emergence possibility) that a human being may emerge from the inside of another vehicle (V2) to outside of the other vehicle on the basis of the other vehicle temperature information.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an in-vehicle device, an in-vehicle system, a vehicle, and a vehicle program. [Background technology]

[0002] A technology has been proposed in which pedestrians around a vehicle are photographed with a camera and, based on the photographed images, a prediction is made as to whether a person will suddenly jump out into the road (see, for example, Patent Document 1 below). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5737397 Summary of the Invention [Problem to be solved by the invention]

[0004] This technology is based on the premise that the person's behavior is captured by a camera, and therefore does not function effectively when the person's behavior cannot be captured by the camera due to an obstacle (for example, the body of a parked vehicle). It would be beneficial if the behavior of people around the vehicle could be predicted even in situations where the person's behavior cannot be captured by the camera due to an obstacle.

[0005] An object of the present invention is to provide a technology that can predict the behavior of people around a vehicle even in a situation where the person's behavior cannot be captured by a camera. [Means for solving the problem]

[0006] The vehicle-mounted device of the present invention is an vehicle-mounted device mounted on a vehicle and equipped with a controller, wherein the controller acquires other vehicle temperature information indicating the temperature of a target component included in the exterior components of another vehicle that is parked, and predicts the possibility of a person appearing from inside the other vehicle toward the outside of the other vehicle based on the other vehicle temperature information. [Effects of the Invention]

[0007] By referencing the temperature of a target part included in the exterior parts of another parked vehicle, it is possible to estimate whether the other vehicle has just stopped. On the other hand, if the other vehicle has just stopped, there is a high possibility that an occupant of the other vehicle will emerge from inside the other vehicle to exit. Therefore, based on the temperature information of the other vehicle indicating the temperature of the target part, it is possible to accurately predict the possibility that a person will emerge from inside the other vehicle toward the outside of the other vehicle. Utilizing the results of this prediction supports the safe driving of the vehicle itself and the safety of people who exit the other vehicle. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a perspective view of an exterior of a vehicle envisioned in this embodiment. [Figure 2] FIG. 2 is a diagram illustrating the relationship between a user and other components according to an embodiment of the present invention. [Figure 3] 1 is a diagram illustrating an internal configuration of an in-vehicle system according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a shooting area of ​​a camera according to an embodiment of the present invention. [Figure 5] 1 is a diagram illustrating an internal configuration of an in-vehicle device according to an embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating an internal configuration of a vehicle sensor unit according to the embodiment of the present invention. [Figure 7] 1A and 1B are diagrams illustrating an example of an input image and a temperature map according to an embodiment of the present invention. [Figure 8] FIG. 2 is a functional block diagram of a controller according to a first example of an embodiment of the present invention. [Figure 9] FIG. 2 is a diagram showing how another vehicle is detected in an input image according to a first example pertaining to an embodiment of the present invention. [Figure 10] 1 is a top view of a host vehicle and a plurality of other vehicles in a scene to be predicted according to a first example belonging to an embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing the structure of auxiliary information according to a first embodiment of the present invention. [Figure 12] FIG. 2 is a diagram showing the internal configuration of a prediction unit according to a first example of an embodiment of the present invention. [Figure 13] 4 is a flowchart illustrating the operation of a controller according to a first example of an embodiment of the present invention. [Figure 14] FIG. 10 is a diagram showing a hardware configuration involved in machine learning according to a second example belonging to an embodiment of the present invention. [Figure 15] FIG. 10 is a top view of a cooperating vehicle involved in machine learning according to a second example of an embodiment of the present invention. [Figure 16] FIG. 10 is a functional block diagram of a controller according to a third example of an embodiment of the present invention. [Figure 17] FIG. 10 is a diagram showing the structure of auxiliary information according to a third embodiment of the present invention. [Figure 18] FIG. 10 is a diagram showing the internal configuration of a prediction unit according to a third example of the embodiment of the present invention. [Figure 19] FIG. 13 is a diagram showing a determiner that can be used as a prediction model according to a seventh example belonging to an embodiment of the present invention. [Figure 20] FIG. 13 is a top view of the host vehicle and other vehicles in a scene to be predicted according to a ninth example belonging to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, examples of embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are designated by the same reference numerals, and duplicate descriptions of the same parts will be omitted as a general rule. In this specification, for the sake of simplicity, symbols or signs referring to information, signals, physical quantities, functional units, circuits, elements, or components may be used, and the names of the information, signals, physical quantities, functional units, circuits, elements, or components corresponding to the symbols or signs may be omitted or abbreviated.

[0010] 1(a) and 1(b) show external perspective views of an arbitrary vehicle VV assumed in this embodiment. There are many types of vehicles, but in this embodiment, a vehicle VV having the structure shown in Fig. 1(a) and 1(b) is assumed. The vehicle VV is an automobile or the like that runs on a road surface.

[0011] A driver's seat (not shown) is installed inside the vehicle VV. In the vehicle VV, the direction from the driver's seat of the vehicle VV toward the steering wheel is defined as "forward," and the direction from the steering wheel of the vehicle VV toward the driver's seat is defined as "rearward." The direction perpendicular to the fore-aft direction and parallel to the road surface on which the vehicle VV is traveling is defined as the left-right direction. The direction perpendicular to the fore-aft direction and perpendicular to the left-right direction is defined as the up-down direction. The driver of the vehicle VV sits in the driver's seat of the vehicle VV facing forward. With respect to the vehicle VV, the fore-aft direction, left-right direction, and up-down direction correspond to the fore-aft direction, left-right direction, and up-down direction as seen from the driver of the vehicle VV. Figure 1(a) is an external perspective view of the vehicle VV when observed from diagonally forward of the vehicle VV. Figure 1(b) is an external perspective view of the vehicle VV when observed from diagonally rearward of the vehicle VV.

[0012] A vehicle VV is provided with various exterior parts, and the exterior parts of the vehicle VV form the body of the vehicle VV. The exterior parts of the vehicle VV include a hood BN, doors DR, trunk lid TL, front window FW, door windows DW, rear window RW, and tires TR. The front window, door windows, and rear window are also referred to as windshield, door glass, and rear glass, respectively. The front window FW, door windows DW, and rear window RW are all windows included in the exterior parts of the vehicle VV. The hood BN, doors DR, and trunk lid TL are made of steel plate. The front window FW, door windows DW, and rear window RW are made of glass. The tires TR are made of rubber (natural rubber, synthetic rubber), tire cord, etc.

[0013] The vehicle VV is provided with an engine compartment that houses the engine of the vehicle VV, and the hood BN is a lid that covers the engine compartment. Here, the engine compartment is located in front of the driver's seat in the vehicle VV, and the hood BN is located above the engine compartment. The doors DR are installed on the sides of the vehicle VV. The doors DR are installed so that they can be opened and closed freely relative to the vehicle VV, and when open, the occupants of the vehicle VV move between the outside of the vehicle VV and the interior of the vehicle VV through the doors DR. The doors DR include at least the front doors, and may also include rear doors. A trunk space capable of accommodating luggage is installed behind the driver's seat in the interior of the vehicle VV. The trunk lid TL is a lid that covers the trunk space, and is installed at the rear end of the body of the vehicle VV.

[0014] The front windshield FW is glass installed in front of the driver's seat. The door windows DW are glass arranged on the sides of each seat (driver's seat, passenger seat, etc.) of the vehicle VV. The door windows DW are fitted into the doors DR and, together with the doors DR, form part of the body of the vehicle VV (the side parts of the body). A vehicle VV may be provided with multiple door windows DW. The rear window RW is glass installed behind each seat (including the driver's seat) in the passenger compartment of the vehicle VV. The rear window RW is fitted into the trunk lid TR and, together with the trunk lid TR, form part of the body of the vehicle VV (the rear part of the body). A vehicle VV may be provided with multiple tires TR. The vehicle VV in the example of Figures 1(a) and (b) has two front tires and two rear tires as tires TR.

[0015] FIG. 2 shows the relationship between user U1 and other components assumed in an embodiment of the present invention. User U1 is an occupant of host vehicle V1. Host vehicle V1 is an example of vehicle VV. User U1 is the driver of host vehicle V1. However, user U1 may also be an occupant other than the driver (i.e., a passenger in host vehicle V1). An in-vehicle system 1 is mounted on host vehicle V1, and each component of the in-vehicle system 1 is installed in an appropriate location in host vehicle V1. A seat ST1 is installed in the cabin of host vehicle V1. User U1 sits in seat ST1 facing forward. Since user U1 is assumed to be the driver, seat ST1 is the driver's seat. Note that, for convenience of description, the reference symbols shown in FIGS. 1(a) and 1(b) are not used for exterior vehicle components shown as an example of vehicle VV. For example, the hood of host vehicle V1 is simply referred to as hood, and the reference symbol "BN" corresponding to the hood is not used.

[0016] 3 shows a schematic block diagram of the in-vehicle system 1. The in-vehicle system 1 includes an in-vehicle device 10, a cruise control device 20, an actuator unit 30, a vehicle sensor unit 40, a camera unit 50, an HMI 60, and a temperature detection unit 70. The components of the in-vehicle system 1 can transmit and receive any signals and information to and from each other through an in-vehicle network formed in the host vehicle V1. The in-vehicle network includes, for example, a CAN (Controller Area Network) and an AVCLAN (Audio Visual Communication Local Area Network).

[0017] The in-vehicle device 10 performs functions such as predicting whether a person outside the vehicle will suddenly jump out (details will be described later). The in-vehicle device 10 may be a drive recorder that works in cooperation with a camera unit 50 to record the situation outside or inside the vehicle. The driving control device 20 controls the driving of the host vehicle V1 using an actuator unit 30. The actuator unit 30 has various driving components such as a motor that realizes the driving of the host vehicle V1. Specifically, the actuator unit 30 includes an engine and motor that generate driving force for the host vehicle V1, a steering actuator that drives the steering of the host vehicle V1, and a brake actuator that drives the brakes of the host vehicle V1.

[0018] The vehicle sensor unit 40 has sensors that detect the details of the driving operation of the vehicle V1 by the driver of the vehicle V1 and sensors that detect various states of the vehicle V1. The vehicle sensor unit 40 generates and outputs vehicle sensor information containing the detection results. The driving control device 20 realizes driving control of the vehicle V1 by driving and controlling the actuator unit 30 in accordance with the vehicle sensor information.

[0019] The camera unit 50 consists of one or more unit cameras that capture images of the outside or inside of the vehicle V1. Each unit camera captures images at a predetermined frame rate. Some of the unit cameras provided in the camera unit 50 are exterior cameras. The exterior cameras have a capture area set outside the vehicle V1, and generate exterior camera images by capturing images of the scene within the capture area. The exterior camera images are images captured in the capture area of ​​the exterior camera. Data representing the content of any image is called image data.

[0020] In the following, attention will be focused mainly on camera 51, which is one of the unit cameras provided in the camera unit 50. Camera 51 is an exterior camera that captures an image of the area ahead of the host vehicle V1. Therefore, as shown in FIG. 4, the image capturing area of ​​camera 51 includes the area ahead of the host vehicle V1. In FIG. 4, a hatched area SR1 represents a portion of the image capturing area of ​​camera 51. However, the image capturing area of ​​camera 51 may also be the rear area, right side area, or left side area of ​​the host vehicle V1. The image capturing area of ​​camera 51 may also include all or part of the front area, rear area, right side area, and left side area of ​​the host vehicle V1. The front area, rear area, right side area, and left side area of ​​the host vehicle V1 are areas located in the external area of ​​the host vehicle V1, and in the front, rear, right side, and left side of the host vehicle V1, respectively. Image data of the image captured by camera 51 is sent to the in-vehicle device 10.

[0021] The HMI 60 is a human machine interface and is provided with a display device 61, a speaker 62, and an operation input unit 63.

[0022] The display device 61 has a display screen such as a liquid crystal display panel, and displays any video (image) under the control of the in-vehicle device 10, the driving control device 20, or a display control device not shown. The display device 61 is installed in an appropriate location in the cabin of the host vehicle V1 so that each occupant of the host vehicle V1 can see the display content of the display device 61. Multiple display devices 61 may be installed in the cabin of the host vehicle V1. The display device 61 may be a component of a car navigation system installed in the host vehicle V1. The car navigation system may be included in the in-vehicle system 1. The display device 61 may be a display device provided in an information terminal (smartphone, etc.) carried by the user U1.

[0023] The speaker 62 outputs any sound (message, warning sound, music, etc.) under the control of the in-vehicle device 10, the driving control device 20, or an audio device (not shown). The speaker 62 is installed in an appropriate location in the cabin of the host vehicle V1 so that each occupant of the host vehicle V1 can hear the output sound of the speaker 62. Multiple speakers 62 may be installed in the cabin of the host vehicle V1. The speaker 62 may be a speaker provided in the information terminal.

[0024] The operation input unit 63 receives arbitrary operations from each occupant of the host vehicle V1. The operation input unit 63 can be configured with operation buttons, a touch panel, or the like. A microphone may be provided in the HMI 60, and voice operations using the microphone may be input to the operation input unit 63. The operation input unit 63 may be an operation input unit provided in the information terminal (smartphone, etc.). In addition, a vibration device that applies vibrations to the occupants (particularly the driver) of the host vehicle V1 may be provided in the HMI 60.

[0025] The temperature detection unit 70 includes a thermal sensor 71, an outside air temperature sensor 72, and a temperature sensor 73 (a temperature sensor for the host vehicle). The thermal sensor 71 measures the temperature at each position within the measurement target area and generates and outputs thermal sense information indicating the measured temperature at each position within the measurement target area. The outside air temperature sensor 72 measures the outside air temperature, which is the temperature outside the host vehicle V1, and generates and outputs outside air temperature information indicating the measured outside air temperature. The temperature sensor 73 is attached to a specific exterior part of the host vehicle V1 and measures the temperature of the specific exterior part of the host vehicle V1. The temperature sensor 73 generates and outputs host vehicle temperature information indicating the measured temperature of the specific exterior part of the host vehicle V1. The specific exterior part is, for example, a hood and a front windshield, which will be described in detail later. Each sensor within the temperature detection unit 70 periodically updates the information it generates, and the latest thermal sense information, outside air temperature information, and host vehicle temperature information are sequentially output from the temperature detection unit 70 to the in-vehicle device 10.

[0026] 5 shows the internal configuration of the in-vehicle device 10. The in-vehicle device 10 includes a controller 11, a memory 12, a communication unit 13, and a recording medium 14.

[0027] The controller 11 includes a processing unit 11a including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) as hardware resources. The controller 11 may execute a program recorded in the memory 12 or any other recording medium to realize any function, operation, and process that should be realized by the controller 11. All or part of the operations performed by the controller 11 described below may be understood to be operations performed by the processing unit 11a.

[0028] The memory 12 is configured to include a non-volatile memory such as a ROM (Read Only Memory) or a flash memory, and a volatile memory such as a RAM (Random Access Memory). The memory 12 stores various data referenced by the controller 11 as well as various programs to be executed by the controller 11.

[0029] The communication unit 13 is a communication circuit (communication module) that transmits and receives any signal between the in-vehicle device 10 and a counterpart device different from the in-vehicle device 10. The counterpart device for the communication unit 13 includes components other than the in-vehicle device 10 among the components of the in-vehicle system 1 shown in FIG. 3. The communication unit 13 can communicate with the counterpart device via an in-vehicle network formed in the vehicle V1. The counterpart device for the communication unit 13 can include an external device (such as a server device) connected to an external network. The external network includes the Internet and an intranet. Note that the controller 11 can transmit and receive any information to and from the counterpart device using the communication unit 13, but the description of the communication unit 13 may be omitted below.

[0030] The recording medium 14 is a nonvolatile recording medium made up of a magnetic disk, a flash memory, or the like, and stores (records) any information in a nonvolatile manner. The controller 11 is capable of recording any information on the recording medium 14 and reading any information recorded on the recording medium 14. The recording medium 14 may be detachable from the in-vehicle device 10. The recording medium 14 may be external to the in-vehicle device 10 and installed within the in-vehicle system 1. If the in-vehicle device 10 has a drive recorder function, the controller 11 can record image data of an image captured by any unit camera in the camera unit 50 on the recording medium 14.

[0031] FIG. 6 shows an internal block diagram of the vehicle sensor unit 40. The vehicle sensor unit 40 is equipped with sensors 41 to 47. Sensors 41, 42, and 43 are an accelerator pedal sensor, a brake pedal sensor, and a steering wheel sensor, respectively. The host vehicle V1 is provided with operational components that receive driving operations from the driver, and the operational components include an accelerator pedal, a brake pedal, and a steering wheel. Sensors 44, 45, 46, and 47 are a vehicle speed sensor, a steering angle sensor, a G sensor, and a GPS sensor, respectively.

[0032] The accelerator pedal sensor 41 detects the operation of the accelerator pedal of the host vehicle V1 by the driver of the host vehicle V1, and generates and outputs accelerator pedal operation information indicating the operation of the accelerator pedal. The brake pedal sensor 42 detects the operation of the brake pedal of the host vehicle V1 by the driver of the host vehicle V1, and generates and outputs brake pedal operation information indicating the operation of the brake pedal. The steering wheel sensor 43 detects the operation of the steering wheel of the host vehicle V1 by the driver of the host vehicle V1, and generates and outputs steering wheel operation information indicating the operation of the steering wheel.

[0033] The vehicle speed sensor 44 detects the speed of the host vehicle V1 and generates and outputs vehicle speed information (vehicle speed pulses) representing the detected speed. The steering angle sensor 45 detects the steering angle (steering angle) of the host vehicle V1 and generates and outputs steering angle information representing the detected steering angle. The G sensor 46 detects acceleration in a predetermined axial direction applied to the host vehicle V1 and generates and outputs the acceleration detection result as acceleration information. The G sensor 46 may detect acceleration in two mutually orthogonal axial directions or may detect acceleration in three mutually orthogonal axial directions. The GPS sensor 47 receives signals from multiple GPS satellites that form a GPS (Global Positioning System) and generates and outputs vehicle position information based on the received signals. The vehicle position information generated by the GPS sensor 47 represents the current location (current position) of the host vehicle V1 using longitude and latitude, or represents the current location of the host vehicle V1 using longitude, latitude, and altitude.

[0034] The vehicle sensor information generated by the vehicle sensor unit 40 is information corresponding to the traveling state of the host vehicle V1 and includes output information from each sensor within the vehicle sensor unit 40. Therefore, the vehicle sensor information includes accelerator pedal operation information, brake pedal operation information, steering wheel operation information, vehicle speed information, steering angle information, acceleration information, and vehicle position information. However, any of this information may not be included in the vehicle sensor information. Each sensor within the vehicle sensor unit 40 periodically updates the information it should generate, and the latest vehicle sensor information is sequentially output from the vehicle sensor unit 40. Sensors other than sensors 41 to 47 (for example, distance measurement sensors, rain sensors, illuminance sensors, shift lever sensors, and door lock sensors) may also be provided in the vehicle sensor unit 40.

[0035] FIG. 7 shows an example of the input image IN. The input image IN is a two-dimensional still image obtained by one capture by the camera 51. Image data of the input image IN is supplied from the camera 51 to the in-vehicle device 10, whereby the controller 11 acquires the input image IN. The measurement target area of ​​the thermal sensor 71 encompasses the entire capture area of ​​the camera 51. However, the measurement target area may encompass only a portion of the capture area of ​​the camera 51. The controller 11 recognizes the relationship between the capture area of ​​the camera 51 and the measurement target area of ​​the thermal sensor 71, and converts the thermal sense information from the thermal sensor 71 into a temperature map based on this relationship. The temperature map is two-dimensional temperature information indicating the temperature at each position within the input image IN. FIG. 7 schematically shows a temperature map corresponding to the input image IN of FIG. 7. Hereinafter, vehicles other than the host vehicle V1 will be referred to as other vehicles. In the example of FIG. 7, the input image IN includes images of two other vehicles, and it is assumed that the temperature is uniform throughout the entire body of each other vehicle and that the temperatures of vehicles other than the other vehicles are uniform at other temperatures.

[0036] Here, it is assumed that the camera 51 and the thermal sensor 71 are provided separately, but the thermal sensor 71 may be built into the camera 51. In this case, the camera 51 incorporating the thermal sensor 71 generates and outputs image data of the input image IN and thermal sense information indicating the measurement results of the temperature at each position within the input image IN. The controller 11 can generate and acquire a temperature map from the thermal sense information from the camera 51 incorporating the thermal sensor 71.

[0037] In this embodiment, the other vehicle is a vehicle parked in a space where parking is possible. The space where parking is possible may be a parking lot, or may be a space that does not qualify as a parking lot. However, in the following, any other vehicle is assumed to be parked (stopped) in a parking lot. In other words, any other vehicle is a parked vehicle in a parking lot. When the host vehicle V1 is traveling in the parking lot, the in-vehicle device 10 realizes a characteristic function of assisting in avoiding contact between the host vehicle V1 and an occupant of the other vehicle.

[0038] Below, specific configuration examples, operation examples, application techniques, modified techniques, etc. related to the in-vehicle system 1 and the in-vehicle device 10 (particularly the above-mentioned characteristic functions) will be described in multiple embodiments. The matters described above in this embodiment are applied to each of the following embodiments unless otherwise stated and unless there is a contradiction. If there are any matters in each embodiment that contradict the matters described above, the description in each embodiment may take precedence. Furthermore, unless there is a contradiction, the matters described in any of the multiple embodiments shown below can also be applied to any of the other embodiments (i.e., any two or more of the multiple embodiments can be combined).

[0039] <<First Example>> A first embodiment will be described. FIG. 8 is a functional block diagram of the controller 11. The controller 11 includes functional blocks F1 to F6. The controller 11 is a program execution device (computer) capable of executing any program. All or part of the functions of the functional blocks F1 to F6 may be realized by the controller 11 (arithmetic processing unit 11a) executing a program recorded in the memory 12 or any other recording medium. Input images IN generated by sequential photography by the camera 51 are sequentially input to the controller 11. The thermal sensor 71 repeatedly executes a measurement process for measuring the temperature at each position within the measurement target area at a predetermined cycle, and sequentially supplies the obtained latest thermal sense information to the controller 11. The functional blocks F1 to F6 may be capable of referencing any information handled by the controller 11. Note that, with respect to any image, the input, output, recording, saving, generation, and acquisition of an image are synonymous with the input, output, recording, saving, generation, and acquisition of image data of the image. Similar expressions are also interpreted in the same manner. Furthermore, any image-based process or operation is specifically a process or operation based on the image data of that image.

[0040] ---Object detection unit F1--- The functional block F1 is an object detection unit. The object detection unit F1 performs object detection processing on each input image IN. The object detection processing is sometimes simply referred to as object detection. The object detection unit F1 can be configured using a known object detector.

[0041] In the object detection process, the object detection unit F1 detects whether a detection object exists within a detection target area in the input image IN based on the image data of the input image IN. The presence of a detection object in an input image or a detection target area within an input image specifically refers to the presence of an image of the detection object (in other words, image data of the detection object) within the input image or detection target area. The detection target area may be the entire image area of ​​the input image IN, or a partial area of ​​the entire image area of ​​the input image IN. When the object detection process detects the presence of a detection object within the detection target area, the position and shape of the detection object in the input image IN are detected, as well as the type of the detection object. In the object detection unit F1, the detection object includes a vehicle and further includes exterior parts of the vehicle. Exterior parts as detection objects include all or part of the hood, trunk lid, tires, doors, front window, door windows, and rear window. Hereinafter, the hood, trunk lid, tires, doors, front window, door windows, and rear window are all considered to be included in the detection object.

[0042] The object detection unit F1 generates and outputs object detection information as a result of object detection processing on the input image IN. The object detection unit F1 supplies the object detection information to the function block F3. For each detected object in the input image IN, the object detection information includes object position information that specifies the position and shape of the detected object in the input image IN, and class information that indicates the type of the detected object.

[0043] FIG. 9 shows an input image IN including an image of one other vehicle V2. In the object detection process for the input image IN in FIG. 9, a rectangular area in the input image IN where the image of the other vehicle V2 exists is identified as a bounding box (hereinafter referred to as BBOX) 610. The object detection unit F1 identifies the area in the input image IN where each exterior part of the other vehicle V2 exists by instant segmentation or semantic segmentation of the input image IN. As a result, object position information and class information for each exterior part of the other vehicle V2 are generated.

[0044] For example, for the input image IN in FIG. 9, an area 611 in which the hood of the other vehicle V2 exists is identified within the input image IN, and object position information indicating the identification result and class information indicating that the object in area 611 is the hood are generated. Also, an area 612 in which the front windshield of the other vehicle V2 exists is identified within the input image IN, and object position information indicating the identification result and class information indicating that the object in area 612 is the front windshield are generated. With reference to FIG. 9, attention is focused only on the hood and the front windshield as exterior parts of the other vehicle V2, but object position information and class information are generated similarly for other exterior parts as well. However, depending on the positional relationship between the host vehicle V1 and the other vehicle V2, some exterior parts of the other vehicle V2 (such as the trunk lid in the example of FIG. 9) may not be detectable in the object detection process.

[0045] ---Temperature map generation section F2--- The function block F2 is a temperature map generator. The temperature map generator F2 executes a temperature map generation process for each input image IN. In the temperature map generation process, the temperature map generator F2 generates a temperature map from the thermal sense information using the method described above. The temperature map generator F2 supplies the generated temperature map to the function block F3. The temperature map indicates the temperature at each position in the input image IN.

[0046] ---Other vehicle temperature identification unit F3--- The functional block F3 is an other-vehicle temperature identification unit. The other-vehicle temperature identification unit F3 performs a temperature identification process to identify the temperature of target components included in the exterior components of the other vehicle V2 based on the object detection information and the temperature map. The other-vehicle temperature identification unit F3 can set one or more exterior components of the other vehicle V2 as one or more target components. However, the target components are assumed to be exterior components whose presence areas in the input image IN are identified by the object detection unit F1. The other-vehicle temperature identification unit F3 generates other-vehicle temperature information indicating the results of the identification process and supplies it to the functional block F5. In the example of Figure 9, the temperature of the hood of the other vehicle V2 is identified by extracting the temperature at a position in the temperature map corresponding to the position of the hood presence area 611. The same applies to exterior components other than the hood. The thermal sensor 71 has the function of measuring the temperature of each exterior component of the other vehicle V2 (and therefore the temperature of the target components of the other vehicle V2).

[0047] Here, it is assumed that the bonnet temperature T2_B and window temperature T2_W are identified by the other vehicle temperature identification unit F3. Therefore, information indicating the bonnet temperature T2_B and window temperature T2_W is included in the other vehicle temperature information. The bonnet temperature T2_B is the temperature of the bonnet of the other vehicle V2 identified by the temperature identification process. The bonnet of the other vehicle V2 corresponds to the above-mentioned target part. The window temperature T2_W is the temperature of the window of the other vehicle V2 identified by the temperature identification process. The window of the other vehicle V2 is the front window, door window, or rear window of the other vehicle V2. The front window, door window, or rear window of the other vehicle V2 corresponds to the above-mentioned target part. The window temperature T2_W is the temperature of a window included in the exterior parts of the other vehicle V2.

[0048] ---Auxiliary information acquisition section F4--- The function block F4 is an auxiliary information acquisition unit. The auxiliary information acquisition unit F4 acquires auxiliary information and supplies it to the subsequent function block F5. The auxiliary information will be described in detail later.

[0049] ---Prediction section F5--- The functional block F5 is a prediction unit. The prediction unit F5 performs prediction processing based on the temperature information of other vehicles. In the prediction processing, the prediction unit F5 predicts the possibility of a person jumping out. The possibility of a person jumping out refers to the possibility that a person will appear (move) from inside the other vehicle V2 to the outside of the other vehicle V2. The prediction of the possibility of a person jumping out may be a prediction of whether or not there is a high possibility that a person will appear (move) from inside the other vehicle V2 to the outside of the other vehicle V2. Prediction result information indicating the result of the prediction by the prediction unit F5 is supplied to the functional block F6.

[0050] A person appearing (moving) from inside the other vehicle V2 toward the outside of the other vehicle V2 specifically refers to an occupant of the other vehicle V2 leaving the cabin of the other vehicle V2 through a door of the other vehicle V2 to the outside of the other vehicle V2 (external space). The prediction in the prediction process can also be said to be equivalent to predicting whether a person will jump out from inside the other vehicle V2 toward the outside of the other vehicle V2. Therefore, hereinafter, the prediction and prediction process by the prediction unit F5 may be referred to as jumping-out prediction and jumping-out prediction process. A person who jumps out from inside the other vehicle V2 to the outside of the other vehicle V2 (external space) is a person located in the vicinity of the host vehicle V1 and may potentially step onto the path of the host vehicle V1. Therefore, jumping-out may be interpreted as equivalent to a person who has left the inside of the other vehicle V2 to the outside of the other vehicle V2 (external space) stepping out onto the path of the host vehicle V1. Jumping-out prediction is a prediction of the behavior related to the jumping-out of the person. After the door of the other vehicle V2 transitions from a closed state to an open state, the occupant of the other vehicle V2 can exit from the inside of the other vehicle V2 to the outside of the other vehicle V2 (external space) through the door of the other vehicle V2. Therefore, it can be said that the prediction in the prediction process corresponds to a prediction of whether the door of the other vehicle V2 will be opened.

[0051] The prediction unit F5 may perform prediction processing based only on the other vehicle temperature information. However, in this embodiment, in order to improve the accuracy of prediction, it is mainly assumed that the prediction unit F5 performs prediction processing based on the other vehicle temperature information and auxiliary information.

[0052] ---Safety Support Department F6--- The functional block F6 is a safety support unit. The safety support unit F6 executes safety support processing to support the safe driving of the host vehicle V1 or the safety of pedestrians, etc., based on the prediction result information. The safety support processing may include notification to the user U1. The optional notification may be a notification by displaying a video on the display device 61 (a notification that affects the user U1's vision) or a notification by outputting a sound from the speaker 62 (a notification that affects the user U1's hearing), or a combination thereof. If the HMI 60 includes the vibration device, the optional notification may include a notification by generating a vibration from the vibration device (a notification that affects the user U1's tactile sense).

[0053] Consider a case where the prediction unit F5 predicts that there is a high possibility of a person jumping out. In this case, the safety support unit F6 issues a warning notification to inform the user U1 of the prediction result. In this case, the safety support unit F6 may cause the driving control device 20 to perform driving control of the host vehicle V1 (for example, control to reduce the speed of the host vehicle V1) to avoid contact between the host vehicle V1 and a person who may appear from inside the other vehicle V2.

[0054] ---Outline of the pop-out prediction method--- An outline of the method for predicting a sudden jump by the prediction unit F5 will be described with reference to FIG. 10. FIG. 10 is a plan view of a scene to be predicted, in which the host vehicle V1 and multiple other vehicles V2 parked in a parking lot are observed from above. The scene to be predicted is a scene in which the host vehicle V1, having traveled outside the parking lot, enters the parking lot and then travels forward within the parking lot, and in which an image of the other vehicle V2 is included in the input image IN. A scene in which the host vehicle V1 enters the parking lot from outside the parking lot and in which an image of the other vehicle V2 is included in the input image IN also falls under the prediction target scene. In a scene in which an image of the other vehicle V2 is included in the input image IN, the position and shape of the other vehicle V2 in the input image IN are detected by an object detection process, and object detection information related to the other vehicle V2 is generated. Note that the two dashed lines in FIG. 10 represent the outer edge of the image capture area of ​​the camera 51 and the outer edge of the measurement target area of ​​the thermal sensor 71 (the same applies to FIG. 15 and FIGS. 20(a) and (b) described below). 10 includes other vehicles V2[1] to V2[4], and the bonnet temperatures of the other vehicles V2[1] to V2[4] are identified by the on-board device 10 of the host vehicle V1. In the example of FIG. 10, the bonnet temperatures of the other vehicles V2[1] to V2[4] are 18°C, 41°C, 19°C, and 17°C, respectively.

[0055] For any other vehicle V2, when the other vehicle V2 is traveling, the heat generated by the engine of the other vehicle V2 increases the hood temperature of the other vehicle V2. Therefore, the hood temperature of the other vehicle V2 differs between immediately after the other vehicle V2 stops in a parking lot and some time after the other vehicle V2 stops. Therefore, based on the hood temperature of each other vehicle V2, it is possible to determine whether the other vehicle V2 has just stopped. Furthermore, immediately after the other vehicle V2 stops in a parking lot, there is a high possibility that a person will emerge from inside the other vehicle V2 to get out of the other vehicle V2. Taking these factors into consideration, the prediction unit F5 performs a running-out prediction for each other vehicle V2 based on the hood temperature of the other vehicle V2. In the example of FIG. 10, among the other vehicles V2[1] to V2[4], it can be predicted that there is a high possibility that a person will emerge from inside the other vehicle V2 only for the other vehicle V2[2]. However, the hood temperature of each other vehicle can vary depending on various factors (external temperature, presence of direct sunlight, etc.), so auxiliary information is referenced to take these various factors into consideration when predicting whether a vehicle will suddenly appear.

[0056] Furthermore, on days with high outside temperatures, the hood temperature of each other vehicle V2 is likely to be consistently high. On the other hand, when the engine of the other vehicle V2 is running on a day with high outside temperatures, the air conditioner installed in the other vehicle V2 cools the interior of the other vehicle V2, resulting in a lower window temperature of the other vehicle V2 than when the air conditioner is not operating. In other words, when comparing the window temperature of the other vehicle V2 immediately after the other vehicle V2 has parked in a parking lot with the window temperature some time after the other vehicle V2 has parked in the parking lot on a day with high outside temperatures, the former is likely to be lower. Therefore, the prediction unit F5 also predicts whether each other vehicle V2 will jump out based on the window temperature of the other vehicle V2. However, the window temperature of each other vehicle can vary depending on various factors (such as the outside temperature and the presence or absence of direct sunlight). Therefore, the prediction of a person jumping out is performed by referring to auxiliary information to take various factors into account. In addition, "immediately after the other vehicle V2 has stopped" specifically refers to the timing immediately after the other vehicle V2's speed changes from a non-zero state to a zero state (for example, within 30 seconds of that timing). "Immediately after the other vehicle V2 has parked" refers to the timing immediately after the other vehicle V2 has stopped in the parking lot and the ignition is switched from on to off. Switching the ignition of the other vehicle V2 from on to off causes the engine of the other vehicle V2 to transition from an operating state (a state in which the engine is running) to a non-operating state (a state in which the engine is not running). Therefore, when the other vehicle V2 is parked in the parking lot, the engine of the other vehicle V2 is considered to be in a non-operating state.

[0057] When the input image IN includes images of multiple other vehicles V2 (thus, when multiple other vehicles V2 are located within the measurement target area of ​​the thermal sensor 71), the prediction unit F5 executes the jump-out prediction process for each other vehicle V2. In the first embodiment, for the sake of concreteness and clarity of the explanation, attention will be focused on one other vehicle V2, and the jump-out prediction process for the focused other vehicle V2 will be explained below.

[0058] ---Supplementary Information (Figure 11)--- 11 is a diagram showing the configuration of auxiliary information. The auxiliary information acquired by the auxiliary information acquisition unit F4 includes first type reference information, second type reference information, and third type reference information.

[0059] The first type reference information is information related to the hood, and includes information indicating a hood temperature T1_B (first host vehicle temperature information), information indicating a hood color C2_B (other vehicle color information), and information indicating a hood color C1_B (host vehicle color information).

[0060] The hood temperature T1_B is the temperature of the hood of the host vehicle V1. In the scene to be predicted, the temperature of the hood of the host vehicle V1 is higher due to the engine being running compared to when the engine is not running. If the other vehicle V2 has just stopped, it is estimated that the temperature of the hood of the other vehicle V2 will be similar to the temperature of the hood of the host vehicle V1. In order to reflect this estimation in the prediction of a sudden departure, the temperature of the hood of the host vehicle V1 (hood temperature T1_B) is included in the auxiliary information. As described above, the temperature sensor 73 (see Figure 3) measures the temperature of specific exterior parts of the host vehicle V1. The specific exterior parts include the hood of the host vehicle V1. The temperature sensor 73 measures the hood temperature T1_B, and the auxiliary information acquisition unit F4 acquires information indicating the hood temperature T1_B from the temperature sensor 73.

[0061] The hood color C2_B is the color of the hood of the other vehicle V2. The temperature of the hood of the other vehicle V2 is affected by the color of the hood. To reflect this effect in the prediction of an object jumping out, the color of the hood of the other vehicle V2 (hood color C2_B) is included in the auxiliary information. The auxiliary information acquisition unit F4 can detect the hood color C2_B based on the image data of the input image IN while referring to the object detection information output from the object detection unit F1. In other words, the auxiliary information acquisition unit F4 can detect the hood color C2_B based on the image data of the area in the input image IN where the hood of the other vehicle V2 is present. Note that the input image IN generated using the camera 51 is a color image.

[0062] The hood color C1_B is the color of the hood of the host vehicle V1. The hood temperature of the host vehicle V1 is affected by the color of the hood. That is, the hood temperature T1_B of the host vehicle V1, which is referenced in the prediction of a vehicle jumping out, is affected by the color of the hood of the host vehicle V1. In order to reflect this influence in the prediction of a vehicle jumping out, the color of the hood of the host vehicle V1 (hood color C1_B) is included in the auxiliary information. Information indicating the hood color C1_B is stored in advance in the memory 12 (see Figure 5) when the in-vehicle device 10 is installed in the host vehicle V1. The auxiliary information acquisition unit F4 reads and acquires the information indicating the hood color C1_B from the memory 12.

[0063] The second type reference information is information related to the vehicle's windows and includes information indicating the window temperature T1_W (second host vehicle temperature information).

[0064] The window temperature T1_W is the temperature of the window of the host vehicle V1. The window of the host vehicle V1 refers to the front window, door window, or rear window of the host vehicle V1. The window temperatures of the host vehicle V1 and the other vehicle V2 fluctuate due to the operation of the air conditioners installed in those vehicles. However, in the prediction target scene of the host vehicle V1, if the other vehicle V2 has just stopped, it is estimated that the window temperature of the other vehicle V2 will be similar to the window temperature of the host vehicle V1. To reflect this estimation in the prediction of a sudden outward movement, the window temperature of the host vehicle V1 (window temperature T1_W) is included in the auxiliary information. As described above, the temperature sensor 73 (see FIG. 3) measures the temperature of specific exterior components of the host vehicle V1. The specific exterior components include the windows of the host vehicle V1. The window temperature T1_W is measured by the temperature sensor 73, and the auxiliary information acquisition unit F4 acquires information indicating the window temperature T1_W from the temperature sensor 73.

[0065] The timing at which prediction unit F5 makes a prediction of a vehicle jumping out is referred to as the prediction timing. The hood temperature T2_B and window temperature T2_W (see FIG. 8) are the temperatures of the hood and windows of the other vehicle V2 at the prediction timing. Similarly, the hood temperature T1_B and window temperature T1_W (see FIG. 11) are the temperatures of the hood and windows of the host vehicle V1 at the prediction timing.

[0066] The third type of reference information is environmental information, and includes outside temperature information, date and time information, weather information, and direct sunlight information.

[0067] The outside temperature information indicates the outside temperature at the prediction timing. The outside temperature refers to the temperature outside the host vehicle V1 at the location where the host vehicle V1 is located. The outside temperature affects the temperature of each exterior part of the host vehicle V1 and the other vehicle V2. In order to reflect this effect in the prediction of a sudden departure, the outside temperature information is included in the auxiliary information. The auxiliary information acquisition unit F4 can acquire the outside temperature information from the outside temperature sensor 72 (see Figure 3). Alternatively, the auxiliary information acquisition unit F4 may acquire the outside temperature information via wireless communication from an outside temperature sensor (not shown) installed on a building or the like outside the host vehicle V1. In this case, the outside temperature information may be acquired by the auxiliary information acquisition unit F4 via a server device connected to an external vehicle network.

[0068] The date and time information indicates the date to which the predicted timing belongs and the time at which the predicted timing belongs. The vehicle described in this embodiment is assumed to be located in Japan, and therefore the temperature and sunlight in the area where the vehicle is located fluctuate depending on the date. Furthermore, even if the date is the same, the temperature and sunlight in the area where the vehicle is located will fluctuate if the time is different. In other words, the date and time to which the predicted timing belongs affect the temperature of each exterior component of the host vehicle V1 and the other vehicle V2. To reflect this effect in the prediction of a vehicle jumping out, the date and time information is included in the auxiliary information. The auxiliary information acquisition unit F4 has a clock function that identifies the current date and the current time, and acquires the date and time information using the clock function. The auxiliary information acquisition unit F4 may acquire the date and time information from a server device connected to an external network.

[0069] The weather information indicates the weather at the predicted timing in the area where the host vehicle V1 and the other vehicle V2 are located. The weather indicated by the weather information may be, for example, sunny, cloudy, or rainy. The weather in the area where the host vehicle V1 and the other vehicle V2 are located affects the temperature of each exterior part of the host vehicle V1 and the other vehicle V2. In order to reflect this effect in the prediction of a sudden departure, the weather information is included in the auxiliary information. The auxiliary information acquisition unit F4 may acquire weather information from a server device connected to an external network. The server device from which the weather information is acquired provides weather information for the area where the host vehicle V1 and the other vehicle V2 are located to the in-vehicle device 10 in real time or via a weather forecast.

[0070] The direct sunlight information indicates whether or not the other vehicle V2 is exposed to direct sunlight. That is, the direct sunlight information indicates whether or not the other vehicle V2 is exposed to direct sunlight at the prediction timing. Whether or not direct sunlight is irradiating the other vehicle V2 affects the temperature of each exterior component of the other vehicle V2. To reflect this effect in the prediction of a vehicle jumping out, the direct sunlight information is included in the auxiliary information. The auxiliary information acquisition unit F4 can acquire the direct sunlight information by image analysis based on the image data of the input image IN. The auxiliary information acquisition unit F4 may determine whether or not the other vehicle V2 is exposed to direct sunlight from brightness information of an area in the input image IN where the image of the other vehicle V2 is located. The auxiliary information acquisition unit F4 may also extract the road surface temperature near the other vehicle V2 from a temperature map and estimate whether or not the other vehicle V2 is exposed to direct sunlight based on the road surface temperature.

[0071] The prediction unit F5 performs the above-mentioned prediction process (ramming prediction process) based on other vehicle temperature information (see FIG. 8) including the hood temperature T2_B and window temperature T2_W, and auxiliary information including the first to third types of reference information.

[0072] ---Internal structure of the prediction unit (Fig. 12)--- FIG. 12 shows the internal configuration of the prediction unit F5. The prediction unit F5 in FIG. 12 includes a prediction model F5a, a prediction model F5b, and an integrator F5c. In the first embodiment, the prediction model F5a is a prediction model for the hood, and the prediction model F5b is a prediction model for the window. Each of the prediction models F5a and F5b is a learning model that has undergone machine learning in advance. The learning model is configured using a support vector machine or a deep neural network.

[0073] The prediction model F5a receives first prediction information including information indicating a hood temperature T2_B of the other vehicle V2, as well as first and third type reference information. The prediction model F5a performs a jump-out prediction for the other vehicle V2 based on the first prediction information, and outputs prediction data Pa indicating the result of the jump-out prediction. The prediction model F5b receives second prediction information including information indicating a window temperature T2_W of the other vehicle V2, as well as second and third type reference information. The prediction model F5b performs a jump-out prediction for the other vehicle V2 based on the second prediction information, and outputs prediction data Pb indicating the result of the jump-out prediction.

[0074] The prediction models F5a and F5b each predict the possibility of a person jumping out. As described above, the possibility of a person jumping out refers to the possibility that a person will emerge from inside the other vehicle V2. The prediction of the possibility of a person jumping out may be a prediction of whether or not there is a high possibility that a person will emerge from inside the other vehicle V2. A high possibility of a person jumping out means that the possibility of a person jumping out is equal to or greater than a reference value, and a low possibility of a person jumping out means that the possibility of a person jumping out is lower than the reference value. The reference value is a value greater than 0% and less than 100%. The prediction models F5a and F5b may predict and derive the possibility of a person jumping out as a continuous value (i.e., predict and derive the probability that a person will emerge from inside the other vehicle V2 as a continuous value). However, in this example, each of the prediction models F5a and F5b is assumed to binarize and output the possibility of a person jumping out. A binarization unit (not shown) that binarizes the possibility of a person jumping out derived as a continuous value by each prediction model may be considered to be located subsequent to each prediction model.

[0075] The prediction model F5a outputs predicted data Pa having a value of "1" when it predicts that there is a high possibility of popping out. The prediction model F5a outputs predicted data Pa having a value of "0" when it does not predict that there is a high possibility of popping out (in other words, when it predicts that there is a low possibility of popping out). Therefore, predicted data Pa of "1" indicates that the prediction model F5a predicted that there is a high possibility of popping out. In contrast, predicted data Pa of "0" does not indicate that there is a high possibility of popping out by the prediction model F5a (in other words, it indicates that there is a low prediction possibility of popping out).

[0076] Similarly, prediction model F5b outputs predicted data Pb having a value of "1" when it predicts that there is a high possibility of popping out. When prediction model F5b does not predict that there is a high possibility of popping out (in other words, when it predicts that there is a low possibility of popping out), it outputs predicted data Pb having a value of "0". Therefore, predicted data Pb of "1" indicates that there is a prediction that there is a high possibility of popping out by prediction model F5b. In contrast, predicted data Pb of "0" does not indicate that there is a prediction that there is a high possibility of popping out by prediction model F5b (in other words, it indicates that there is a prediction that there is a low possibility of popping out).

[0077] The integrator F5c receives input of predicted data Pa from the prediction model F5a and predicted data Pb from the prediction model F5b. The integrator F5c generates and outputs prediction result information based on the predicted data Pa and Pb. Here, the prediction result information is assumed to be information having a value of "1" or "0." When at least one of the predicted data Pa and Pb has a value of "1," the integrator F5c sets the value of the prediction result information to "1." When both the predicted data Pa and Pb have a value of "0," the integrator F5c sets the value of the prediction result information to "0." Prediction result information of "1" indicates a high possibility of popping out. In contrast, prediction result information of "0" does not indicate a high possibility of popping out. In other words, prediction result information of "0" indicates a low possibility of popping out.

[0078] ---Operational flowchart for pop-out prediction (Fig. 13)--- FIG. 13 shows a flowchart of the operation of the controller 11 related to the jump-out prediction. Each process of steps S11 to S18 shown in FIG. 13 is executed by the controller 11 (arithmetic processing unit 11a). The in-vehicle device 10 also starts up in conjunction with the start of the engine of the host vehicle V1, and the operation of the controller 11 related to the jump-out prediction starts from the process of step S11 when the in-vehicle device 10 starts up. In step S11, a scene determination unit (not shown) included in the controller 11 performs scene determination. In the scene determination, it may be determined to which of a plurality of candidate scenes the driving scene of the host vehicle V1 corresponds, but here we will only focus on the above-mentioned prediction target scene. That is, in step S11, it is determined whether the driving scene of the host vehicle V1 corresponds to the prediction target scene.

[0079] For example, the scene determination unit determines that the driving scene of the host vehicle V1 corresponds to the prediction target scene only when both a first condition and a second condition are satisfied. The scene determination unit determines whether the host vehicle V1 is driving in a parking lot or whether the host vehicle V1 has entered the parking lot from outside, based on the vehicle speed information, vehicle position information, and map information of the host vehicle V1. The map information indicates the location and type of each facility (including parking lots) in the area where the vehicle V1 is located, and is stored in advance in the memory 12 or provided to the in-vehicle device 10 from the server device. The first condition is satisfied when it is determined that the host vehicle V1 is driving in a parking lot or has entered the parking lot from outside. Furthermore, the second condition is satisfied when the object detection unit F1 detects the position and shape of another vehicle V2 in the input image IN and generates object detection information related to the other vehicle V2. Note that the object detection process may be executed constantly and continuously after the in-vehicle device 10 is started.

[0080] In step S12 following step S11, the result of the scene determination is confirmed. If it is determined that the driving scene of the host vehicle V1 corresponds to the scene to be predicted (Y in step S12), the process proceeds from step S12 to step S13. If it is not determined that the driving scene of the host vehicle V1 corresponds to the scene to be predicted (N in step S12), the process returns to step S11.

[0081] In step S13, the object detection information generated by the object detection process of the object detection unit F1 and the temperature map generated by the temperature map generation process of the temperature map generation unit F2 are input to the other vehicle temperature identification unit F3 (see also FIG. 8). As a result, in step S13, other vehicle temperature information is generated by the other vehicle temperature identification unit F3. The other vehicle temperature information generated here includes information indicating the hood temperature T2_B and information indicating the window temperature T2_W. Meanwhile, in step S13, auxiliary information is acquired by the auxiliary information acquisition unit F4. After step S13, the process proceeds to step S14.

[0082] In step S14, of the other vehicle temperature information and auxiliary information generated or acquired in step S13, the first prediction information is input to the prediction model F5a and the second prediction information is input to the prediction model F5b (see FIG. 12). As a result, prediction data Pa and Pb are generated in step S14. After step S14, the process proceeds to step S15.

[0083] In step S15, the integrator F5c checks whether at least one of the predicted data Pa and Pb has a value of "1." If at least one of the predicted data Pa and Pb has a value of "1" (Y in step S15), the process proceeds to step S16. If both the predicted data Pa and Pb have a value of "0" (N in step S15), the process proceeds to step S17.

[0084] In step S16, the integrator F5c outputs prediction result information having a value of "1". In step S17, the integrator F5c outputs prediction result information having a value of "0". After step S16, the process proceeds to step S18. After step S17, the process returns to step S11 without proceeding to step S18. In step S18, the safety support unit F6 receives the prediction result information having a value of "1" and performs the above-mentioned safety support processing. Then, the process returns to step S11. For example, the safety support unit F6 involved in step S18 notifies the user U1 using the display device 61, the speaker 62, or the like that there is a possibility that a person may appear from inside the other vehicle V2. Furthermore, for example, the safety support unit F6 involved in step S18 may cause the driving control device 20 to perform driving control of the host vehicle V1 (for example, control to reduce the speed of the host vehicle V1) to avoid contact between the host vehicle V1 and a person who may appear from inside the other vehicle V2.

[0085] In this way, the in-vehicle device 10 (arithmetic processing unit 11a) acquires other vehicle temperature information (see FIG. 8) that indicates the temperature of target parts included in the exterior parts of the other vehicle V2, and performs a jump-out prediction based on the other vehicle temperature information. This makes it possible to perform a jump-out prediction (prediction of a person's appearance from the other vehicle V2) without capturing pedestrians or the like with a camera. Utilizing the results of the jump-out prediction supports safe driving of the host vehicle V1 and ensuring the safety of people who get out of the other vehicle.

[0086] <<Second Example>> A second embodiment will now be described. Note that the second embodiment and any of the embodiments described below are based on the first embodiment, and for matters not specifically mentioned in the second embodiment and any of the embodiments described below, the description of the first embodiment also applies to the second embodiment and any of the embodiments described below unless there is a contradiction.

[0087] In the second example, a method for generating prediction models F5a and F5b through machine learning will be described. FIG. 14 shows the hardware configuration involved in machine learning. The data collection device 200, database 220, and learning device 240 are connected to a communication network including the Internet. The data collection device 200 and learning device 240 each comprise one or more computer devices. The database 220 may be built into the data collection device 200 or the learning device 240. The prediction models F5a and F5b are created through machine learning performed by the learning device 240.

[0088] Before machine learning is performed, a data collection step is performed using the data collection device 200, and a large amount of learning data is collected in the data collection step. Each learning data is stored in the database 220.

[0089] Cooperating vehicles are used to collect learning data. FIG. 15 shows cooperative vehicles V3 and V4. The cooperative vehicle V3 is a vehicle that simulates the subject vehicle V1 in the data collection process, and the cooperative vehicle V4 is a vehicle that simulates the other vehicle V2 in the data collection process. The cooperative vehicle V4 is parked in a parking lot. The cooperative vehicle V3 is equipped with an in-vehicle system 1 equivalent to the in-vehicle system 1. However, the controller 11 of the in-vehicle device 10 installed in the cooperative vehicle V3 does not have a prediction unit F5. The cooperative vehicle V3 is a vehicle that travels outside the parking lot and then enters the parking lot where the cooperative vehicle V4 is parked. The cooperative vehicle V4 is located within the imaging area of ​​the camera 51 installed in the cooperative vehicle V3, and an image of the cooperative vehicle V4 is included in the input image captured by the camera 51 installed in the cooperative vehicle V3. For convenience, the in-vehicle device 10 installed in the cooperative vehicle V3 will be referred to as the cooperative in-vehicle device 10.

[0090] A method for generating one set of learning data in the data collection step will be described. In the data collection step, the cooperative in-vehicle device 10 performs the processes of steps S11 to S13, thereby obtaining other vehicle temperature information and auxiliary information in the cooperative in-vehicle device 10. One set of learning data includes other vehicle temperature information and auxiliary information, and a correct label. In one set of learning data, the other vehicle temperature information and auxiliary information are associated with the correct label.

[0091] The correct answer label indicates whether the cooperation vehicle V4 has just stopped at the time when the other vehicle temperature information and auxiliary information are obtained in the data collection process. More specifically, after starting its engine, the cooperation vehicle V4 drives on a given road with the engine running continuously for a certain period of time (e.g., 10 minutes) or more, then enters a parking lot, stops in the parking lot, and remains parked for a sufficiently long time thereafter. The parking timing is the time when the cooperation vehicle V4's engine switches from an operating state to an inoperable state after the cooperation vehicle V4's speed changes from a state where it is greater than zero to a state where it is zero in the parking lot. The elapsed time from the parking timing is referred to as time TJ. If the time TJ at the time when the other vehicle temperature information and auxiliary information are obtained in the data collection process is within a specified period of time (e.g., 1 minute), a correct answer label with a value of "1" is associated with the other vehicle temperature information and auxiliary information. In the data collection step, if the time TJ at the time when the other vehicle temperature information and auxiliary information are obtained is longer than a specified time, a correct label having a value of “0” is associated with the other vehicle temperature information and auxiliary information. The specified time indicates a boundary value that clearly distinguishes whether or not the cooperative vehicle V4 has just stopped, and is determined in advance in the data collection device 200.

[0092] A correct label of "1" indicates that the cooperating vehicle V4 has just stopped, and a correct label of "0" indicates that the cooperating vehicle V4 has not just stopped. The correct label can also be said to indicate the possibility that a person will emerge from inside the cooperating vehicle V4, and this possibility is higher for a correct label of "1" than for a correct label of "0". In other words, when comparing the correct label of "1" with the correct label of "0", the correct label of "1" indicates that the possibility that a person will emerge from inside the cooperating vehicle V4 is relatively high, and the correct label of "0" indicates that this possibility is relatively low.

[0093] Any person who can recognize time TJ may input time TJ to data collection device 200. At this time, data collection device 200 can generate learning data by combining a correct label based on the input time TJ with other vehicle temperature information and auxiliary information obtained by collaborative learning device 10. Alternatively, for example, data collection device 200 may identify time TJ based on an image captured by a fixed camera capturing images of the inside of a parking lot. Then, data collection device 200 can generate learning data by combining a correct label based on the identification result of time TJ with other vehicle temperature information and auxiliary information obtained by collaborative learning device 10.

[0094] Alternatively, a microphone that picks up sounds emitted from the cooperative vehicle V4 may be installed in the cooperative vehicle V3 or in the parking lot. In this case, an idling determination device determines whether the cooperative vehicle V4 is idling (parked with the engine running) based on the sound picked up by the microphone. The idling determination device sets the correct answer label to a value of "1" when it determines that the cooperative vehicle V4 is idling. When it determines that the cooperative vehicle V4 is not idling, the idling determination device compares the time TJ, which is the elapsed time from the parking timing, with a specified time (e.g., one minute). If the time TJ is equal to or shorter than the specified time, the correct answer label is set to a value of "1." If the time TJ is longer than the specified time, the correct answer label is set to a value of "0." The idling determination device may be built into the cooperative learning device 10 or the data collection device 200, or may be installed in the parking lot. Learning data can be generated by combining the correct label, the value of which is set using the idling determination device, with the temperature information of other vehicles and auxiliary information obtained by the collaborative learning device 10.

[0095] In the data collection step, the data collection device 200 collects a large amount of learning data (for example, tens of thousands to hundreds of thousands of sets of learning data are collected). The cooperative vehicle V3 used to obtain some learning data and the cooperative vehicle V3 used to obtain other learning data may be the same vehicle or different vehicles. The same applies to the cooperative vehicle V4. For example, a large amount of learning data can be collected by using many taxis managed by a taxi company as the cooperative vehicles V3 and V4.

[0096] After the data collection step, a machine learning step is performed in the learning device 240. The learning device 240 has learning models 241 and 242. The learning models 241 and 242 are formed by a calculation processing unit included in the learning device 240. Each learning model is an artificial intelligence based on a support vector machine or a deep neural network.

[0097] In the machine learning process, the learning device 240 acquires each piece of learning data from the database 220. The learning device 240 extracts first and second prediction information from each piece of learning data (see FIG. 12 ), inputs the extracted first prediction information to a learning model 241, and inputs the extracted second prediction information to a learning model 242.

[0098] In the machine learning process, the learning model 241 derives a first output value based on the input first prediction information. The learning device 240 derives the error between the first output value and the correct label for each piece of learning data, and performs machine learning to update the parameters of the learning model 241 so as to reduce the error. The machine learning is repeated until the error becomes sufficiently small. A model having a configuration equivalent to the learning model 241 after machine learning is incorporated into the in-vehicle device 10 as prediction model F5a. Similarly, in the machine learning process, the learning model 242 derives a second output value based on the input second prediction information. The learning device 240 derives the error between the second output value and the correct label for each piece of learning data, and performs machine learning to update the parameters of the learning model 242 so as to reduce the error. The machine learning is repeated until the error becomes sufficiently small. A model having a configuration equivalent to the learning model 242 after machine learning is incorporated into the in-vehicle device 10 as prediction model F5b.

[0099] <<Third Example>> A third embodiment will be described. For any vehicle, some of the exterior parts of the vehicle are specific exterior parts that experience a temperature rise when the vehicle is moving. In other words, for any vehicle, the temperature of the specific exterior parts rises when the vehicle is moving compared to when the vehicle is stopped (stationary). The hood corresponds to the specific exterior part. In the first embodiment, attention is focused on the hood, and the predicted data Pa is derived based on the temperature of the hood. However, the predicted data Pa may also be derived based on the temperature of specific exterior parts other than the hood.

[0100] For example, tires also fall under the category of specific exterior parts. In any vehicle, the rubber portion of the tire forms the outer circumferential surface of the tire, which generates heat when the vehicle is running due to contact with the road surface. In other words, in any vehicle, the temperature of the tire, like the temperature of the hood, rises when the vehicle is running compared to when the vehicle is stopped (stationary). Here, the temperature of the tire refers to the temperature of the rubber portion of the tire, and is preferably the temperature of the portion as close as possible to the outer circumferential surface of the tire.

[0101] In the third embodiment, a modified configuration is adopted in which predicted data Pa is derived using tire temperature instead of hood temperature. When this modified configuration is adopted, it is sufficient to modify the first embodiment as follows. That is, in the third embodiment (this modified configuration), the first embodiment is used as the basis, and "hood" in the description of the first embodiment is replaced with "tire." This replacement also includes replacing the description "hood temperature" with the description "tire temperature," and replacing the description "hood temperature" with the description "tire temperature."

[0102] For this reason, in the third embodiment, as shown in FIG. 16, the other vehicle temperature information includes information indicating tire temperature T2_T instead of information indicating bonnet temperature T2_B (see also FIG. 8 as appropriate). Tire temperature T2_T is the temperature of a tire of other vehicle V2 identified by the temperature identification process. Tire temperature T2_T may be the temperature of any tire of other vehicle V2. FIG. 16 is a functional block diagram of a controller 11 according to the third embodiment. Similarly, in the third embodiment, as shown in FIG. 17, the first type reference information includes information indicating tire temperature T1_T instead of information indicating bonnet temperature T1_B (see also FIG. 11 as appropriate). FIG. 17 is a configuration diagram of auxiliary information according to the third embodiment. Tire temperature T1_T is the temperature of a tire of host vehicle V1. At this time, tire temperature T1_T is measured by temperature sensor 73, and auxiliary information acquisition unit F4 acquires information indicating tire temperature T1_T from temperature sensor 73. The tire temperature T1_T may be the temperature of any tire on the host vehicle V1.

[0103] The first type reference information in the third embodiment does not include information on the hood colors C2_B and C1_B. Because tire color is generally fixed to black or a color close to black, the first type reference information in the third embodiment does not need to include information indicating the tire color of the other vehicle V2 or the tire color of the host vehicle V1. As shown in FIG. 18, the prediction model F5a in the third embodiment generates and outputs prediction data Pa based on first prediction information including information indicating the tire temperature T2_T of the other vehicle V2, the first type reference information, and the third type reference information. FIG. 18 is a diagram showing the internal configuration of the prediction unit F5 in the third embodiment. Like the prediction model F5a in the first embodiment, the prediction model F5a in the third embodiment can also be generated through the machine learning shown in the second embodiment.

[0104] <<Fourth Example>> A fourth embodiment will be described. To maintain high accuracy in pop-out prediction under various circumstances, it is preferable to install both prediction models F5a and F5b in the prediction unit F5. However, it is also possible to omit one of the prediction models F5a and F5b in the prediction unit F5 of FIG. 12 or FIG. 18. In this case, the integrator F5c is not required in the prediction unit F5. When the prediction unit F5 is equipped with the prediction model F5a but not with the prediction model F5b, the prediction data Pa from the prediction model F5a itself is output from the prediction unit F5 as prediction result information. When the prediction unit F5 is equipped with the prediction model F5b but not with the prediction model F5a, the prediction data Pb from the prediction model F5b itself is output from the prediction unit F5 as prediction result information.

[0105] <<Fifth Example>> A fifth embodiment will be described. It is believed that the accuracy of the prediction of sudden departure can be improved by referring to various information in addition to the temperature information of other vehicles. However, in the first embodiment, it is also possible to omit any part of the information shown in FIG. 11 from the auxiliary information.

[0106] For example, in the first embodiment, information indicating bonnet colors C2_B and C1_B may be omitted from the first-type reference information (and therefore omitted from the auxiliary information). Also, for example, in the first embodiment, information indicating bonnet temperature T1_B may be omitted from the first-type reference information (and therefore omitted from the auxiliary information). Also, for example, in the first embodiment, any one or more pieces of information selected from outside temperature information, date and time information, weather information, and direct sunlight information may be omitted from the third-type reference information (and therefore omitted from the auxiliary information).

[0107] However, immediately after the other vehicle V2 has stopped, the bonnet temperature T2_B of the other vehicle V2 is considered to be relatively close to the bonnet temperature T1_B of the host vehicle V1 while it is moving. Conversely, if a sufficiently long time has passed since the other vehicle V2 began parking, the bonnet temperature T2_B of the other vehicle V2 is likely to be considerably lower than the bonnet temperature T1_B of the host vehicle V1 while it is moving. In other words, by referring to the bonnet temperature T1_B of the host vehicle V1, it is considered possible to accurately predict the sudden jumping out based on the bonnet temperature. For this reason, it is preferable to include information indicating the bonnet temperature T1_B in the auxiliary information.

[0108] Similarly, immediately after the other vehicle V2 has stopped, the bonnet temperature T2_B of the other vehicle V2 is likely to be higher than the outside air temperature. Conversely, if a sufficient amount of time has passed since the other vehicle V2 began parking, the bonnet temperature T2_B of the other vehicle V2 is likely to be close to the outside air temperature. In other words, by referring to the outside air temperature information, it is thought that a person running out into the road can be predicted accurately based on the bonnet temperature. For this reason, it is preferable to include the outside air temperature information in the auxiliary information.

[0109] In summary, in the first embodiment, it is preferable that the first prediction information for the prediction model F5a includes, in addition to information indicating the bonnet temperature T2_B, at least one of information indicating the bonnet temperature T1_B of the host vehicle V1 and outside air temperature information. The same applies to other embodiments in which a vehicle jump-out prediction is performed based on the bonnet temperature. This improves the accuracy of the vehicle jump-out prediction.

[0110] From a similar perspective, in the first embodiment, it is preferable that the second prediction information for the prediction model F5b includes, in addition to information indicating the window temperature T2_W, at least one of information indicating the window temperature T1_W of the host vehicle V1 and outside air temperature information. This is also true for other embodiments in which a vehicle jump-out prediction is performed based on the window temperature. This improves the accuracy of the vehicle jump-out prediction.

[0111] If the auxiliary information includes the information shown in FIG. 11 as in the first embodiment, the accuracy of pop-out prediction can be further improved.

[0112] The same can be said for the third embodiment, which uses tire temperature instead of bonnet temperature. That is, in the third embodiment, it is preferable that the first prediction information for the prediction model F5a includes, in addition to information indicating tire temperature T2_T, at least one of information indicating tire temperature T1_T of the host vehicle V1 and outside air temperature information. The same is true for other embodiments in which a jump-out prediction is performed based on tire temperature. This improves the accuracy of the jump-out prediction.

[0113] As shown in the third embodiment, if the auxiliary information includes the information shown in FIG. 17, the accuracy of pop-out prediction can be further improved.

[0114] <<Sixth Example>> A sixth embodiment will be described. Instead of providing separate prediction models F5a and F5b in the prediction unit F5, a single model (hereinafter referred to as the integrated model) that integrates the prediction models F5a and F5b may be provided. In this case, integrated prediction information that includes all of the first and second prediction information is input to the integrated model, and the integrated model directly generates and outputs prediction result information based on the integrated prediction information (therefore, the integrator F5c in FIG. 12 or FIG. 18 is unnecessary). The integrated model can be formed using machine learning similar to that shown in the second embodiment.

[0115] <<Seventh Example>> A seventh embodiment will be described. Instead of forming the prediction models F5a and F5b using machine learning, the prediction models F5a and F5b may be configured by a determiner that performs pop-out prediction using a threshold value. This will be described in more detail below.

[0116] The determiner F5_B shown in Fig. 19(a) may be used as the prediction model F5a in Fig. 12. The determiner F5_B predicts the possibility of the vehicle jumping out based on the bonnet temperature T2_B.

[0117] Specifically, for example, the determiner F5_B compares the bonnet temperature T2_B with the bonnet temperature T1_B, and determines the temperature difference (T1_B-T2_B) as a positive threshold temperature difference TH B1 (for example, 5°C). At this time, the temperature difference (T1_B-T2_B) is compared with the threshold temperature difference TH B1 If the temperature difference (T1_B-T2_B) is equal to or less than the threshold temperature difference TH, the decision unit F5_B predicts that there is a high possibility of the temperature jumping out and outputs predicted data Pa having a value of "1." B1 If it is greater than this, the decision unit F5_B predicts that the possibility of the vehicle V2 suddenly jumping out is low, and outputs prediction data Pa having a value of "0." This is because the bonnet temperature T2_B of the other vehicle V2 immediately after it has stopped is considered to be relatively close to the bonnet temperature T1_B of the host vehicle V1 that has entered the parking lot and is now driving. The decision unit F5_B determines the threshold temperature difference TH according to at least one of the bonnet color C2_B and the bonnet color C1_B. B1 The determiner F5_B may set and change the threshold temperature difference TH in accordance with at least one of the outside temperature information, the date and time information, the weather information, and the direct sunlight information. B1 can be set and changed (see Figure 11).

[0118] Alternatively, for example, the determiner F5_B may set the bonnet temperature T2_B to a positive threshold temperature TH B2 (for example, 50°C). At this time, the bonnet temperature T2_B is compared with the threshold temperature TH B2 If the temperature T2_B is equal to or higher than the threshold temperature TH, the decision unit F5_B predicts that there is a high possibility of the vehicle suddenly jumping out, and outputs prediction data Pa having a value of "1." B2If the temperature is less than the threshold temperature T1_B, the decision unit F5_B predicts that the possibility of the vehicle V2 jumping out is low and outputs prediction data Pa having a value of "0." This is because the bonnet temperature T2_B of the other vehicle V2 is relatively high immediately after the vehicle has stopped, and is likely to decrease as time passes after the vehicle has stopped. The decision unit F5_B determines the threshold temperature TH in accordance with at least one of the bonnet temperature T1_B, the bonnet color C2_B, and the bonnet color C1_B. B2 The determiner F5_B may set and change the threshold temperature TH in accordance with at least one of the outside temperature information, the date and time information, the weather information, and the direct sunlight information. B2 can be set and changed (see Figure 11).

[0119] A determiner F5_T shown in Fig. 19(b) may be used as the prediction model F5a in Fig. 18. The determiner F5_T predicts the possibility of the tire breaking out based on the tire temperature T2_T.

[0120] Specifically, for example, the determiner F5_T compares the tire temperature T2_T with the tire temperature T1_T and determines the temperature difference (T1_T-T2_T) as a positive threshold temperature difference TH T1 (for example, 5°C). At this time, the temperature difference (T1_T-T2_T) is compared with the threshold temperature difference TH T1 If the temperature difference (T1_T-T2_T) is equal to or less than the threshold temperature difference TH, the decision unit F5_T predicts that there is a high possibility of jumping out and outputs predicted data Pa having a value of "1." T1 If it is greater than 0, the decision unit F5_T predicts that the possibility of the vehicle V2 suddenly jumping out is low and outputs prediction data Pa having a value of "0". This is because the tire temperature T2_T of the other vehicle V2 immediately after stopping is considered to be relatively close to the tire temperature T1_T of the host vehicle V1 that has entered the parking lot and is running. The decision unit F5_T determines the threshold temperature difference TH T1 can be set and changed (see Figure 17).

[0121] Alternatively, for example, the determiner F5_T may set the tire temperature T2_T to a positive threshold temperature TH T2 (for example, 50°C). At this time, the tire temperature T2_T may be compared with the threshold temperature THT2 If the tire temperature T2_T is equal to or higher than the threshold temperature TH, the determiner F5_T predicts that the possibility of the tire breaking out is high and outputs the prediction data Pa having a value of "1." T2 If the tire temperature T1_T is less than the threshold temperature T1_T, the determiner F5_T predicts that the possibility of the other vehicle V2 jumping out is low and outputs prediction data Pa having a value of "0". This is because the tire temperature T2_T of the other vehicle V2 is relatively high immediately after the vehicle has stopped, and is likely to decrease as time passes after the vehicle has stopped. The determiner F5_T determines the threshold temperature T T2 The determiner F5_T may set and change the threshold temperature TH in accordance with at least one of the following information: outside temperature information, date and time information, weather information, and direct sunlight information. T2 can be set and changed (see Figure 17).

[0122] The determiner F5_W shown in Fig. 19(c) may be used as the prediction model F5b in Fig. 12. The determiner F5_W predicts the possibility of the vehicle jumping out based on the window temperature T2_W.

[0123] Specifically, for example, the decision unit F5_W compares the window temperature T2_W with the window temperature T1_W, and determines the absolute value of the temperature difference (T1_W-T2_W) as a positive threshold temperature difference TH W1 (for example, 5°C). At this time, the absolute value of the temperature difference (T1_W-T2_W) is the threshold temperature difference TH W1 If the absolute value of the temperature difference (T1_W-T2_W) is equal to or less than the threshold temperature difference TH, the decision unit F5_W predicts that there is a high possibility of jumping out and outputs prediction data Pb having a value of "1." W1 If it is greater than this, the decision unit F5_W predicts that the possibility of the vehicle V2 suddenly jumping out is low and outputs prediction data Pb with a value of "0". This is because the window temperature T2_W of the other vehicle V2 immediately after it has stopped is considered to be relatively close to the window temperature T1_W of the host vehicle V1 that has entered the parking lot and is driving. The decision unit F5_W determines the threshold temperature difference TH in accordance with any one or more of the following information: outside temperature information, date and time information, weather information, and direct sunlight information. W1 can be set and changed (see Figure 11).

[0124] Alternatively, for example, the determiner F5_W may set the window temperature T2_W to a positive threshold temperature THW2 Compared with the window temperature T2_W and threshold temperature TH W2 In this case, the determiner F5_W determines the threshold temperature TH in accordance with one or more of information indicating the window temperature T1_W, outside temperature information, date and time information, weather information, and direct sunlight information. W2 may be set and changed (see Figure 11). For example, consider a midsummer daytime case where the prediction timing is 2:00 PM on August 1st. In the midsummer daytime case, immediately after the other vehicle V2 has stopped, the window temperature T2_W is expected to be relatively low due to the air conditioner that is estimated to have been operating up until that point. On the other hand, there is a high possibility that the air conditioner will stop operating at the same time as the other vehicle V2 stops in the parking lot. Therefore, in the midsummer daytime case, the window temperature T2_W is expected to rise as time passes after the other vehicle V2 has stopped. For this reason, the determiner F5_W for the midsummer daytime case determines whether "T2_W≦TH W2 When the condition "T2_W>TH" is met, it predicts that there is a high possibility of the vehicle jumping out and outputs prediction data Pb having a value of "1." W2 When the condition "is met," the possibility of the jumping out is predicted to be low, and prediction data Pb having a value of "0" is output.

[0125] <<Eighth Example>> An eighth embodiment will now be described.

[0126] The prediction unit F5 may be provided with prediction models F5a and F5b (see FIG. 12) according to the first embodiment and decision devices F5_B and F5_W according to the seventh embodiment. In this case, prediction result information may be generated based on the prediction results of the prediction models F5a and F5b according to the first embodiment and the prediction results of the decision devices F5_B and F5_W according to the seventh embodiment. Alternatively, prediction result information may be generated by switching between a block consisting of prediction models F5a and F5b and a block consisting of decision devices F5_B and F5_W depending on various conditions. The latter block is considered to require less time and computational complexity for prediction than the former block. Therefore, it is possible to perform pop-out prediction using the former block as a general rule, and then use the latter block to perform pop-out prediction when the required prediction accuracy is obtained.

[0127] The prediction unit F5 may be provided with prediction models F5a and F5b according to the third embodiment (see FIG. 18 ) and decision devices F5_T and F5_W according to the seventh embodiment. In this case, prediction result information may be generated based on the prediction results of the prediction models F5a and F5b according to the third embodiment and the prediction results of the decision devices F5_T and F5_W according to the seventh embodiment. Alternatively, prediction result information may be generated by switching between a block consisting of prediction models F5a and F5b and a block consisting of decision devices F5_T and F5_W depending on various conditions. The latter block is considered to require less time and computational complexity for prediction than the former block. Therefore, it is possible to perform pop-out prediction using the former block as a general rule, and then use the latter block to perform pop-out prediction when the latter block provides the required prediction accuracy.

[0128] <<Ninth Example>> A ninth embodiment will be described. As described above, the other vehicle temperature identification unit F3 identifies the temperature of target components included in the exterior components of the other vehicle V2 based on the object detection information and the temperature map (see FIG. 8). The other vehicle temperature identification unit F3 according to the ninth embodiment identifies the temperatures of the first and second target components of the other vehicle V2. Of these, the first target components are the specific exterior components (exterior components whose temperature rises while the vehicle is moving) described in the third embodiment, such as the hood or tires. In contrast, the second target components are exterior components that do not fall under the category of specific exterior components, such as the doors, trunk lid, front window, side windows, or rear window. As the other vehicle V2 moves, the temperature of the first target components rises more than the temperature of the second target components.

[0129] Figures 20(a) and (b) show prediction target scenes assumed in the ninth embodiment. In each of the examples in Figures 20(a) and (b), the first and second target parts are the hood and door, respectively. In the example in Figure 20(a), the temperatures of the hood and door of the other vehicle V2 are 42°C and 18°C, respectively. In the example in Figure 20(b), the temperatures of the hood and door of the other vehicle V2 are 19°C and 18°C, respectively.

[0130] In the ninth embodiment, the other vehicle temperature identification unit F3 identifies the temperatures of the first and second target components of the other vehicle V2 and generates and outputs information indicating the temperatures of the first and second target components of the other vehicle V2 as other vehicle temperature information. In the ninth embodiment, the temperature identified by the other vehicle temperature identification unit F3 and that is the temperature of the first target component of the other vehicle V2 will be referred to as "T2_α." The temperature identified by the other vehicle temperature identification unit F3 and that is the temperature of the second target component of the other vehicle V2 will be referred to as "T2_β." The prediction unit F5 according to the ninth embodiment predicts the possibility of a vehicle jumping out based on the difference between the temperature T2_α of the first target component and the temperature T2_β of the second target component, i.e., based on the temperature difference (T2_α-T2_β).

[0131] The temperature of the first target component rises more than the temperature of the second target component due to the movement of the other vehicle V2. Therefore, the temperature difference (T2_α-T2_β) is relatively large for the other vehicle V2 immediately after parking, and is considered to approach zero as time passes from the start of parking. Therefore, the prediction unit F5 according to the ninth embodiment calculates the temperature difference (T2_α-T2_β) by a positive threshold temperature difference TH DIF The temperature difference (T2_α-T2_β) is compared with the threshold temperature difference TH DIF If the temperature difference (T2_α-T2_β) is greater than or equal to the threshold temperature difference TH DIF If the temperature difference is less than the threshold temperature difference TH, the prediction unit F5 predicts that the possibility of the vehicle jumping out is low and outputs the prediction result information of "0". DIF For example, if T2_α-T2_β≧TH DIF ", but in the example of Figure 20(b), "T2_α-T2_β <TH DIF "

[0132] The method shown in this embodiment also makes it possible to perform good prediction of a person jumping out (prediction of a person appearing from another vehicle V2).

[0133] Threshold temperature difference TH DIF may be a preset fixed value. The prediction unit F5 calculates the threshold temperature difference TH based on the color of the first target part of the other vehicle V2 or the color of the second target part of the other vehicle V2. DIF The prediction unit F5 may set and change the threshold temperature difference TH based on any one or more of the following information: outside temperature information, date and time information, weather information, and direct sunlight information. DIF may be set and changed.

[0134] <<Tenth Example>> A tenth embodiment will now be described.

[0135] The host vehicle V1 may be an autonomous vehicle without a driver. When the host vehicle V1 is an autonomous vehicle, the host vehicle V1 travels according to the travel control of the travel control device 20 without the need for a driver to operate the vehicle. In this case, the travel control device 20 controls the travel of the host vehicle V1 based on information about the surrounding environment of the host vehicle V1, including image data of images captured by the camera 51.

[0136] A program that causes a computer device to execute any of the methods described in the embodiments of the present invention, and a non-volatile recording medium on which the program is recorded, are included within the scope of the embodiments of the present invention. The program that causes a computer device to execute any of the methods described in the embodiments of the present invention may be a subprogram incorporated into any main program or called by any main program. The in-vehicle device 10 or the controller 11 is a type of computer device. Any processing in the embodiments of the present invention may be realized by hardware such as a semiconductor integrated circuit, software equivalent to the program, or a combination of hardware and software.

[0137] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above-described embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each constituent element are not limited to those described in the above-described embodiments. The specific numerical values ​​shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. [Explanation of symbols]

[0138] VV vehicle BN bonnet DR Door TL trunk lid TR Tire FW Front window DW Door Window RW rear window V1 Vehicle V2 Other vehicles 1. In-vehicle systems U1 user ST1 seat 10 Onboard equipment 11 Controller 11a Processing unit 12 Memory 13 Communications Department 14 Recording media 20 Driving control device 30 Actuator section 40 Vehicle sensor unit 50 Camera Department 51 Camera 60 HMI 61 Display device 62 Speaker 63 Operation input section 70 Temperature detection unit 71 Thermal Sensor 72 Outside air temperature sensor 73 Temperature Sensor IN Input image F1 Object detection unit F2 Temperature map generator F3 Other vehicle temperature identification unit F4 Auxiliary information acquisition section F5 Prediction Department F6 Safety Support Department F5a, F5b prediction models F5c integrator 200 Data Collection Device 220 databases 240 Learning Device 241, 242 Learning Model V3, V4 cooperation vehicles F5_B, F5_T, F5_W determiner

Claims

1. An in-vehicle device mounted on a vehicle and equipped with a controller, The controller acquires other vehicle temperature information indicating the temperature of a target part included in an exterior part of the other vehicle while the other vehicle is stopped, and predicts the possibility that a person will appear from inside the other vehicle toward the outside of the other vehicle based on the other vehicle temperature information. , in-vehicle equipment.

2. the target parts include, among the exterior parts of the other vehicle, specific exterior parts whose temperature increases when the other vehicle is running, the controller makes the prediction based on the other vehicle temperature information and auxiliary information; The auxiliary information is Vehicle temperature information indicating the temperature of an exterior part corresponding to the specific exterior part among the exterior parts of the vehicle; and outside temperature information at the prediction timing when the prediction is made. The in-vehicle device according to claim 1 .

3. the specific exterior part is a hood covering an engine compartment of the other vehicle, The subject vehicle temperature information indicates a temperature of a hood covering an engine compartment of the subject vehicle. The in-vehicle device according to claim 2 .

4. The auxiliary information is the vehicle temperature information; Other vehicle color information indicating the color of a hood of the other vehicle; Vehicle color information indicating the color of a hood of the vehicle; The outside temperature information; Date and time information indicating a date to which the predicted timing belongs and a time at the predicted timing; Weather information at the predicted timing in an area where the host vehicle and the other vehicle are located; and direct sunlight information indicating whether or not the other vehicle is being irradiated with direct sunlight. The in-vehicle device according to claim 3 .

5. the specific exterior part is a tire of the other vehicle, The subject vehicle temperature information indicates a temperature of a tire of the subject vehicle. The in-vehicle device according to claim 2 .

6. The auxiliary information is the vehicle temperature information; The outside temperature information; Date and time information indicating a date to which the predicted timing belongs and a time at the predicted timing; Weather information at the predicted timing in an area where the host vehicle and the other vehicle are located; and direct sunlight information indicating whether or not the other vehicle is being irradiated with direct sunlight. The in-vehicle device according to claim 5 .

7. the target part includes a window included in an exterior part of the other vehicle, The auxiliary information is Vehicle temperature information indicating the temperature of a window included in an exterior part of the vehicle; and outside temperature information at the prediction timing when the prediction is made. The in-vehicle device according to claim 1 .

8. The auxiliary information is the vehicle temperature information; The outside temperature information; Date and time information indicating a date to which the predicted timing belongs and a time at the predicted timing; Weather information at the predicted timing in an area where the host vehicle and the other vehicle are located; and direct sunlight information indicating whether or not the other vehicle is being irradiated with direct sunlight. The in-vehicle device according to claim 7 .

9. the target parts include a hood covering an engine compartment of the other vehicle and a window included in an exterior part of the other vehicle, the other vehicle temperature information includes first other vehicle temperature information indicating a temperature of a hood of the other vehicle and second other vehicle temperature information indicating a temperature of a window of the other vehicle, the controller makes the prediction based on the first other vehicle temperature information, the second other vehicle temperature information, and auxiliary information; The auxiliary information is First host vehicle temperature information indicating a temperature of a hood covering an engine compartment of the host vehicle; Other vehicle color information indicating the color of a hood of the other vehicle; Vehicle color information indicating the color of a hood of the vehicle; Second vehicle temperature information indicating the temperature of a window included in an exterior part of the vehicle; Outside temperature information at the prediction timing when the prediction is made; Date and time information indicating a date to which the predicted timing belongs and a time at the predicted timing; Weather information at the predicted timing in an area where the host vehicle and the other vehicle are located; and direct sunlight information indicating whether or not the other vehicle is being irradiated with direct sunlight. The in-vehicle device according to claim 1 .

10. the target parts include, among exterior parts of the other vehicle, a first target part whose temperature increases when the other vehicle is traveling, and a second target part different from the first target part; the other vehicle temperature information indicates a temperature of the first target component and a temperature of the second target component, The controller makes the prediction based on a difference between the temperature of the first target component and the temperature of the second target component. The in-vehicle device according to claim 1 .

11. An in-vehicle device according to any one of claims 1 to 10; a thermal sensor for measuring the temperature of the target component.

12. An in-vehicle device according to any one of claims 1 to 10; a thermal sensor for measuring the temperature of the target component; and a vehicle equipped with the thermal sensor.

13. A program executed by a computer device associated with a vehicle, and causing the computer device to perform a step of acquiring other vehicle temperature information indicating the temperature of a target part included in an exterior part of the other vehicle, and predicting the possibility of a person appearing from inside the other vehicle based on the other vehicle temperature information. , vehicle programs.

Images