Information processing device, information processing method, and program

Abstract

An information processing device comprising: a recognition unit that recognizes an object included in captured image data of the surroundings of a moving body; a prediction unit that predicts a future trajectory of the object; and a notification control unit that, on the basis of the future trajectory of the object, causes a notification device to notify the occupants of the moving body of the presence of the object. The notification control unit causes the notification device to notify in different modes when the object is predicted to approach the moving body from behind and when the object is predicted to pass by the side of the moving object.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] In the past, there have been known techniques for notifying a vehicle occupant of a vehicle of the presence of another vehicle traveling nearby. For example, Patent Document 1 discloses a technique for instructing a warning device installed in the vehicle to issue a warning when a relative speed, which is a difference between the speed of a vehicle behind and the speed of the vehicle, is equal to or greater than a first threshold value or when a change in the relative speed is equal to or greater than a second threshold value.

[0003] Japanese Patent Application Laid-Open No. 2020-129178

[0004] However, in the conventional technology, it was sometimes not possible to notify the occupants of the moving body in an optimal manner according to the predicted future trajectory of the surrounding object.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objects is to provide an information processing device, an information processing method, and a program that can notify occupants of a moving vehicle in an optimal manner according to the predicted future trajectory of surrounding objects.

[0006] The information processing device, information processing method, and program according to the present invention employ the following configuration: (1): An information processing device according to one aspect of the present invention includes a recognition unit that recognizes an object included in image data of an image of the periphery of a moving body, a prediction unit that predicts a future trajectory of the object, and a notification control unit that causes a notification device to notify a passenger of the moving body of the presence of the object based on the future trajectory of the object, and the notification control unit changes the mode of notification by the notification device depending on whether the object is predicted to approach from behind the moving body or whether the object is predicted to pass by the side of the moving body.

[0007] (2): In the above aspect (1), the notification device is an audio output device, and the notification control unit causes the audio output device to output a warning sound when it is predicted that the object will approach from behind the moving body and the moving body enters a first predetermined area.

[0008] (3): In the above aspect (1), the notification device is configured such that, when the object is predicted to pass on the side of the moving body and the moving body enters a second specified area, the notification control unit outputs a warning sound indicating either the left or right side.

[0009] (4): In the above aspect (2), when the object is predicted to pass beside the moving body and the moving body enters a second predetermined area, the notification control unit outputs a warning sound indicating either the left or right side, the first predetermined area being an area located within a first distance from the moving body, and the second predetermined area being an area located at least the first distance from the moving body and within a second distance greater than the first distance.

[0010] (5): In the aspect (1) above, the notification device is a display device, and the information processing device further includes a reception unit that receives instruction information indicating whether the moving body is a four-wheeled vehicle or a two-wheeled vehicle, and the notification control unit, when the reception unit receives instruction information indicating that the moving body is a two-wheeled vehicle, predicts that the object will approach from behind the moving body and when the moving body enters a first predetermined area, causes the display device to display a predetermined figure in the center of the display device.

[0011] (6): In the aspect (1) above, the notification device is a display device, and the information processing device further includes a reception unit that receives instruction information indicating whether the moving body is a four-wheeled vehicle or a two-wheeled vehicle, and when the reception unit receives instruction information indicating that the moving body is a four-wheeled vehicle, the notification control unit dynamically displays a predetermined figure diagonally upward on the display device from the lower left corner or lower right corner of the display device, which corresponds to the direction of entry of the object, when the object is predicted to pass beside the moving body and the moving body enters a second predetermined area.

[0012] (7) In the aspect (6) above, the notification control unit increases the speed at which the predetermined figure is dynamically displayed as the relative speed of the object with respect to the moving body increases.

[0013] (8) In the aspect of (6) above, the notification control unit increases the size of the predetermined figure as the object approaches the moving body.

[0014] (9): Another aspect of the information processing method of the present invention is a method in which a computer recognizes an object contained in image data captured around a moving body, predicts the future trajectory of the object, and causes a notification device to notify the occupants of the moving body of the presence of the object based on the future trajectory of the object, and changes the notification mode by the notification device depending on whether the object is predicted to approach from behind the moving body or whether the object is predicted to pass by the side of the moving body.

[0015] (10): Another aspect of the present invention is a program that causes a computer to recognize objects contained in image data captured around a moving body, predict the future trajectory of the object, and, based on the future trajectory of the object, cause a notification device to notify the occupants of the moving body of the presence of the object, and change the notification mode by the notification device depending on whether the object is predicted to approach from behind the moving body or whether the object is predicted to pass by the side of the moving body.

[0016] (11): Another aspect of the present invention is a program that causes a computer to recognize objects contained in image data captured around a motorcycle, predict the future trajectory of the object, and, based on the future trajectory of the object, cause a notification device to notify the rider of the motorcycle of the presence of the object. The program changes the notification mode of the notification device depending on whether the object is predicted to approach from behind the motorcycle or whether the object is predicted to pass by the side of the motorcycle. When the object is predicted to approach from behind the motorcycle and the moving body enters a first predetermined area, the program causes the notification device to display a predetermined graphic in the center of the notification device.

[0017] (12): A program according to another aspect of the present invention causes a computer to recognize an object contained in image data of the area around a four-wheeled vehicle, to predict the future trajectory of the object, and to cause a notification device to notify a passenger of the four-wheeled vehicle of the presence of the object based on the future trajectory of the object. The program changes the notification mode by the notification device depending on whether the object is predicted to approach from behind the four-wheeled vehicle or whether the object is predicted to pass by the side of the four-wheeled vehicle. When the object is predicted to pass by the side of the four-wheeled vehicle and the moving body enters a second predetermined area, the program causes the notification device to dynamically display a predetermined graphic diagonally upward from the lower left or lower right corner of the notification device, which corresponds to the direction of entry of the object.

[0018] According to the aspects (1) to (12), it is possible to notify the occupants of the moving body in an optimal manner according to the predicted future trajectories of the surrounding vehicles.

[0019] 1 is a diagram showing an example of the configuration of the on-board camera 10 and the terminal device 100 mounted on the host vehicle M. FIG. 1 is a diagram for explaining a method by which the prediction unit 150 predicts the future trajectory of an object. FIG. 2 is a diagram showing a schematic diagram of a density distribution. FIG. 3 is a diagram showing the height of the density value D at line 4-4 in FIG. 3. FIG. 4 is another diagram for explaining a method by which the prediction unit 150 predicts the future trajectory of an object. FIG. 6 is a diagram showing an example of a case in which an object is predicted to approach from behind the host vehicle M and a case in which an object is predicted to pass by the side of the host vehicle M. FIG. 7 is a diagram showing an example of a predetermined area behind the host vehicle M and a predetermined area to the side of the host vehicle M used by the control unit 160 to determine whether or not a notification is necessary. FIG. 8 is a diagram showing another example of a predetermined area behind the host vehicle M and a predetermined area to the side of the host vehicle M used by the control unit 160 to determine whether or not a notification is necessary. FIG. 9 is a diagram for explaining the operation of the notification device when an object enters the central area CA. FIG. 10 is a diagram for explaining the operation of the notification device when an object enters the left area LA. FIG. 11 is a diagram for explaining the operation of the notification device when an object enters the right area RA. FIG. 1 is a diagram showing an example of a reception screen IM1 for instruction information received by the reception unit 170. FIG. 2 is a diagram showing an example of a screen displayed by the display unit 120 when the host vehicle M is a two-wheeled vehicle. FIG. 3 is a diagram showing an example of a screen displayed by the display unit 120 when the host vehicle M is a four-wheeled vehicle. FIG. 4 is a flowchart showing an example of the flow of processing executed by the terminal device 100. FIG. 5 is a flowchart showing another example of the flow of processing executed by the terminal device 100.

[0020] Hereinafter, an information processing device, an information processing method, and a program according to an embodiment of the present invention will be described with reference to the drawings.

[0021] 1 is a diagram showing an example of the configuration of an on-board camera 10 and a terminal device 100 mounted on a host vehicle M. The host vehicle M is, for example, a two-wheeled vehicle or a four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

[0022] The vehicle-mounted camera 10 is a camera attached to the vehicle M. The vehicle-mounted camera 10 includes a camera attached at a position capable of capturing images of at least the rear of the vehicle M, and captures images of the scenery outside the vehicle through the rear windshield, or is provided with a lens exposed outside the vehicle to directly capture images of the surroundings outside the vehicle. The vehicle-mounted camera 10 transmits captured images IM of the surroundings outside the vehicle to the terminal device 100 via wireless communication using a method such as Bluetooth (registered trademark) or Wi-Fi.

[0023] The terminal device 100 is, for example, a portable terminal device such as a smartphone. If the host vehicle M is a four-wheeled vehicle, the terminal device 100 is used by being set in a holder provided, for example, on the passenger compartment side of the front windshield of the vehicle. If the host vehicle M is a two-wheeled vehicle, the terminal device 100 is used by being set in a holder provided, for example, near the handlebars of the vehicle. That is, in either case, the terminal device 100 is set in a position where the display unit 120 of the terminal device 100 can be seen by a passenger of the host vehicle M.

[0024] The communication unit 110 is a wireless communication device that uses the above-mentioned Bluetooth (registered trademark) or Wi-Fi, etc. The communication unit 110 communicates with the vehicle-mounted camera 10 and receives a captured image IM of the surroundings outside the vehicle.

[0025] The display unit 120 is, for example, a display device such as a touch panel, a liquid crystal display, etc. The display unit 120 displays information about objects around the host vehicle M in accordance with the control of the control unit 160, which will be described later.

[0026] The audio output unit 130 is, for example, a speaker device. The audio output unit 130 outputs information about objects in the vicinity of the vehicle M in audio form under the control of the control unit 160. The display unit 120 and the audio output unit 130 are an example of a "notification device."

[0027] The terminal device 100 further includes a recognition unit 140, a prediction unit 150, a control unit 160, and a reception unit 170. The recognition unit 140, the prediction unit 150, the control unit 160, and the reception unit 170 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device. The recognition unit 140, the prediction unit 150, the control unit 160, and the reception unit 170 may hereinafter be referred to as a "driving assistance app." Furthermore, the terminal device 100 equipped with the driving assistance app is an example of an "information processing device."

[0028] The recognition unit 140 recognizes objects captured in images captured by the on-board camera 10. More specifically, for example, the recognition unit 140 recognizes objects using a trained model that has been trained to output information such as the presence, position, and type of an object when an image captured by the on-board camera 10 is input. The recognition unit 140 uses this trained model to identify the area occupied by the vehicle in the captured image IM while distinguishing between types such as two-wheeled vehicles and four-wheeled vehicles.

[0029] The prediction unit 150 predicts the future trajectory of the object recognized by the recognition unit 140. Fig. 2 is a diagram for explaining a method by which the prediction unit 150 predicts the future trajectory of the object. When multiple vehicles are captured in the captured image IM, the prediction unit 150 performs the process described below for each of the multiple vehicles. Fig. 2 shows a case where the on-board camera 10 captures an image of the area behind the host vehicle M.

[0030] The prediction unit 150 first identifies the position of the vehicle on the image plane of the captured image, for example, a position near the center of the bottom edge of the area identified by the recognition unit 140. In Figure 2, A1 to A4 indicate the area occupied by each of the identified vehicles M1 to M4, and P1 to P4 indicate the respective positions of the identified vehicles M1 to M4. In the figure, M1 to M3 are four-wheeled vehicles, and M4 is a two-wheeled vehicle.

[0031] Furthermore, the prediction unit 150 converts the position of the vehicle on the image plane into the position of the vehicle on a virtual plane S. The virtual plane S is a virtual plane seen from above, and approximately coincides with the road plane. The prediction unit 150 identifies the position of the representative point of the vehicle on the virtual plane S, for example, based on a conversion rule for converting coordinates on the image plane into coordinates on the virtual plane. P#1 to P#4 in FIG. 2 indicate the positions of vehicles M1 to M4 on the virtual plane S.

[0032] The prediction unit 150 sets an index value I having a distribution according to the vehicle position on the virtual plane S, using each of the acquired positions of the multiple vehicles as a reference. The index value I is set, for example, so that the contour lines form a circle with the vehicle position as the center. If the index value I is set as height, the distribution of the indexes on the virtual plane S has a dome-like shape with the vehicle position as the ceiling. The prediction unit 150 may or may not change the degree of density spread depending on the vehicle model.

[0033] The prediction unit 150 then generates density distribution information by overlaying the index values ​​I found for multiple vehicles (adding them up for each coordinate on the virtual plane S). Hereinafter, the value obtained by adding up the index values ​​for multiple vehicles will be referred to as the density value D. The index value I and the density value D may be set with a specified value (for example, 1) as their upper limit. In this case, if the result of adding up the index values ​​I found for multiple vehicles exceeds the specified value, the prediction unit 150 sets the specified value at that point as the density value D. FIG. 3 is a diagram schematically showing a density distribution. FIG. 4 is a diagram showing the height of the density value D at line 4-4 in FIG. 3.

[0034] FIG. 5 is another diagram illustrating a method by which the prediction unit 150 predicts a future trajectory of an object. As shown in the figure, the prediction unit 150 estimates that the target vehicle M4 will travel along a trajectory Tj that is a series of points with low density values ​​D (valleys in the density distribution). The prediction unit 150 also predicts a small acceleration (including negative acceleration, i.e., deceleration) of the target vehicle M4 in a section along the trajectory Tj where the density value D increases between points (the section up to point P#4-1 in the figure), and a large acceleration of the target vehicle M4 in a section where the density value D decreases between points (the section after point P#4-1 in the figure). The prediction unit 150 generates a future speed profile of the target vehicle M4 based on the predicted acceleration. In this case, the prediction unit 150 may predict a small acceleration when the density value D increases rapidly between points (a large positive gradient) compared to when the increasing speed is small, and may predict a large acceleration when the density value D decreases rapidly between points (a large negative gradient) compared to when the decreasing speed is small. In a scene where the field of view of the target vehicle suddenly opens up, the target vehicle is expected to accelerate significantly, and therefore the prediction unit 150 can more accurately predict the behavior of the target vehicle in such a scene.

[0035] The control unit 160 causes the display unit 120 or the audio output unit 130 to notify the presence of the object based on the future trajectory of the object predicted by the prediction unit 150 (hereinafter, the display unit 120 and the audio output unit 130 may be collectively referred to as the "notification device"). More specifically, the control unit 160 changes the mode of notification by the notification device depending on whether the object is predicted to approach from behind the vehicle M or whether the object is predicted to pass by the side of the vehicle M.

[0036] 6 is a diagram showing an example of a case where an object is predicted to approach from behind the host vehicle M and a case where an object is predicted to pass by the side of the host vehicle M. In Fig. 6, Tj1 represents the future trajectory of the target vehicle M1, and Tj2 represents the future trajectory of the target vehicle M2. The control unit 160 sets, for example, a center determination area CDA at the center rear of the host vehicle M for determining whether an object is approaching, and sets a left determination area LDA and a right determination area RDA at the sides of the host vehicle M for determining whether an object is passing by.

[0037] When the predicted future trajectory Tj of the object enters the center judgment area CDA, the control unit 160 predicts that the object will approach the host vehicle M. On the other hand, when the predicted future trajectory Tj of the object enters the left judgment area LDA or the right judgment area RDA, the control unit 160 predicts that the object will pass beside the host vehicle M. In the case of FIG. 6 , the future trajectory Tj1 of the target vehicle M1 enters the left judgment area LDA, and the future trajectory Tj2 of the target vehicle M2 enters the center judgment area CDA. Therefore, the control unit 160 predicts that the target vehicle M1 will pass beside the host vehicle M, and predicts that the target vehicle M2 will approach the host vehicle M. Note that in FIG. 6 , the approach prediction and the passage prediction are performed based on whether the predicted trajectory enters a predetermined judgment area. However, the present invention is not limited to such a configuration. For example, the passage prediction may be performed based on whether an extension of the front end of the host vehicle M intersects with the future trajectory.

[0038] The control unit 160 determines whether an object predicted to pass beside the host vehicle M and an object predicted to approach the host vehicle M have actually entered a predetermined area, and if it is determined that the object has entered the predetermined area, causes the notification device to issue a notification. FIG. 7 is a diagram showing an example of a predetermined area behind the host vehicle M and a predetermined area to the side of the host vehicle M that the control unit 160 uses to determine whether or not a notification is necessary. In FIG. 7, the symbol CA represents a predetermined area behind the center of the host vehicle M, the symbol LA represents a predetermined area to the side (left) of the host vehicle M, and the symbol RA represents a predetermined area to the side (right) of the host vehicle M. The predetermined area behind the host vehicle M is an example of a "first predetermined area," and the predetermined area to the side of the host vehicle M is an example of a "second predetermined area."

[0039] 7 , the central area CA is defined as, for example, an area extending a distance D1 from the rear end of the host vehicle M in the opposite direction to the traveling direction of the host vehicle M. The left side area LA is defined as, for example, an area to the left of the host vehicle M in the opposite direction to the traveling direction of the host vehicle M, the area being at least the distance D1 from the rear end of the host vehicle M and within a distance D2 greater than the distance D1. The right side area RA is defined as, for example, an area to the right of the host vehicle M in the opposite direction to the traveling direction of the host vehicle M, the area being at least the distance D1 from the rear end of the host vehicle M and within a distance D2 greater than the distance D1. When the control unit 160 determines that an object predicted to approach or pass by the side of the host vehicle M has entered any one of the central area CA, the left side area LA, or the right side area RA, the control unit 160 causes the notification device to notify the occupant of the host vehicle M of the presence of the object in a manner corresponding to the determined area.

[0040] 7, the side areas LA or RA on the sides of the host vehicle M are set at a longer distance from the host vehicle M than the central area CA on the center of the host vehicle M. This is because, in general, to check the side areas LA or RA on the sides of the host vehicle M, the occupants of the host vehicle M need to use the side mirrors of the host vehicle M, which tends to take longer to check than the central area CA, which can be checked using the rearview mirror of the host vehicle M. By setting the distance to the side areas LA or RA longer, the occupants of the host vehicle M can check the areas on the sides of the host vehicle M with ample time to spare.

[0041] FIG. 8 is a diagram showing another example of a predetermined area behind the host vehicle M and a predetermined area to the side of the host vehicle M, which the control unit 160 uses to determine whether or not a notification is necessary. In FIG. 8 , the predetermined area CA may be defined as an area within a distance D1 in the direction opposite to the traveling direction of the host vehicle M. In other words, the control unit 160 may determine that an object predicted to approach the host vehicle M has entered the predetermined area CA when the distance between the host vehicle M and the object is within D1. The same applies to the left area LA and the right area RA. The control unit 160 may determine that an object predicted to pass by the side of the host vehicle M has entered the left area LA or the right area RA when the distance between the host vehicle M and the object is within D2. More generally, the center area CA, the left area LA, and the right area RA may be defined as finite areas as in FIG. 7 or as infinite areas extending infinitely in any direction as in FIG. 8.

[0042] 9 is a diagram illustrating the operation of the notification device when an object enters the central area CA. Fig. 9 shows a scene in which a two-wheeled vehicle M1 has entered the central area CA. In this case, the control unit 160 causes the audio output unit 130 to output a warning sound (e.g., "Center!") indicating that the two-wheeled vehicle M1 has entered the central area CA and is approaching the vehicle M.

[0043] 9, the symbol B1 represents a bounding box surrounding the two-wheeled vehicle M1, and the display unit 120 displays the object recognized by the recognition unit 140 together with the bounding box B1. The symbol BV represents a bird's-eye view showing the surroundings of the host vehicle M, and the display unit 120 displays the position of the object on the virtual plane S derived by the prediction unit 150. The display of the bounding box and bird's-eye view is merely an example, and other information suitable for driving assistance may also be displayed.

[0044] 10 is a diagram illustrating the operation of the notification device when an object enters the left area LA. FIG. 10 shows a scene in which a two-wheeled vehicle M1 has entered the left area LA. In this case, the control unit 160 causes the audio output unit 130 to output a warning sound (e.g., "Left!") indicating that the two-wheeled vehicle M1 has entered the left area LA and will pass to the left of the host vehicle M.

[0045] 11 is a diagram illustrating the operation of the notification device when an object enters the right area RA. Fig. 11 shows a scene in which a two-wheeled vehicle M1 has entered the right area RA. In this case, the control unit 160 causes the audio output unit 130 to output a warning sound (e.g., "Right!") indicating that the two-wheeled vehicle M1 has entered the right area RA and will pass to the right of the vehicle M.

[0046] 11 , the control unit 160 causes the audio output unit 130 to output a warning sound (e.g., "Center!") indicating that the motorcycle M1 is predicted to enter the center area CA. However, the present invention is not limited to such a configuration. For example, the control unit 160 may cause the audio output unit 130 to output any electronic sound that does not have a specific meaning. This is because, if a warning sound for a side area has meaning, even if a warning sound for a center area has no meaning, the occupant of the host vehicle M can identify the warning sound as being for the center area.

[0047] In this way, by predicting the future trajectory of an object located behind the vehicle M and changing the notification format depending on whether the predicted future trajectory is predicted to approach the vehicle M from behind or to pass by the side of the vehicle M, it is possible to notify the occupants of the moving vehicle in the most optimal manner depending on the predicted future trajectory of the surrounding object.

[0048] The reception unit 170 receives instruction information indicating whether the host vehicle M is a four-wheeled vehicle or a two-wheeled vehicle. The control unit 160 changes the mode of notification by the notification device in accordance with the instruction information received by the reception unit 170, as will be described later.

[0049] 12 is a diagram showing an example of a reception screen IM1 for instruction information received by the reception unit 170. The reception screen IM1 is a screen that is displayed, for example, when a user of the terminal device 100 downloads and installs a driving assistance app via a network (not shown) and launches the driving assistance app for the first time. Furthermore, for example, the screen may also be displayed in response to a user selection even after the driving assistance app has been launched for the first time.

[0050] 12, symbol B1 represents a button (software switch) that the user of the terminal device 100 selects when the host vehicle M is a four-wheeled vehicle, and symbol B2 represents a button that the user of the terminal device 100 selects when the host vehicle M is a two-wheeled vehicle. As will be described below, the reception unit 170 receives the user's selection of four-wheeled vehicle mode B1 or two-wheeled vehicle mode B2, and the control unit 160 controls the display by the display unit 120 according to the received mode.

[0051] Fig. 13 is a diagram showing an example of a screen displayed by the display unit 120 when the host vehicle M is a two-wheeled vehicle. Fig. 13 shows a screen that the control unit 160 causes the display unit 120 to display when an object predicted to approach the host vehicle M from behind enters the central area CA. As shown in Fig. 13, when an object predicted to approach the host vehicle M from behind enters the central area CA, the control unit 160 causes the display unit 120 to display a predetermined graphic near the center of the screen of the terminal device 100. This visually notifies the occupant of the host vehicle M that an object is approaching the host vehicle M from the rear center.

[0052] 13, for example, the display unit 120 dynamically displays a plurality of dot-like objects in a clockwise circular fashion in the center of the screen. On the other hand, when an object predicted to pass beside the host vehicle M enters the side area LA or RA, the control unit 160 causes the display unit 120 to not display a predetermined graphic on the screen of the terminal device 100. This is because, when the host vehicle M is a two-wheeled vehicle, the occupant of the host vehicle M is generally expected to pay more attention to an object approaching from the center rear of the host vehicle M (i.e., an object that poses a risk of rear-end collision) than to an object approaching from diagonally behind the host vehicle M.

[0053] FIG. 14 is a diagram showing an example of a screen displayed by the display unit 120 when the host vehicle M is a four-wheeled vehicle. FIG. 14 shows a screen displayed by the control unit 160 on the display unit 120 when an object predicted to pass to the left of the host vehicle M enters the left area LA. As shown in FIG. 14 , when an object predicted to pass to the left of the host vehicle M enters the left area LA, the control unit 160 causes the display unit 120 to display a predetermined graphic diagonally upward from the lower left corner of the display unit 120 corresponding to the left area LA. Similarly, when an object predicted to pass to the right of the host vehicle M enters the right area RA, the control unit 160 causes the display unit 120 to display a predetermined graphic diagonally upward from the lower right corner of the display unit 120 corresponding to the right area RA. This visually notifies the occupants of the host vehicle M that an object is about to pass beside the host vehicle M.

[0054] For example, in the case of the screen shown in FIG. 14 , the display unit 120 displays multiple linear objects flowing diagonally upward from the lower left corner of the display unit 120 corresponding to the left area LA. Similarly, in this embodiment, when an object predicted to pass to the right of the host vehicle M enters the right area RA, the display unit 120 displays multiple linear objects flowing diagonally upward from the lower right corner of the display unit 120 corresponding to the right area RA. On the other hand, when an object predicted to approach from behind the host vehicle M enters the central area CA, the control unit 160 causes the display unit 120 to not display a predetermined graphic on the screen of the terminal device 100. This is because, when the host vehicle M is a four-wheeled vehicle, it is generally considered that an occupant of the host vehicle M should pay more attention to an object approaching from diagonally behind the host vehicle M (i.e., an object that poses a risk of being hit by the host vehicle M) than to an object approaching from the center rear of the host vehicle M.

[0055] Furthermore, when an object enters a predetermined area, the control unit 160 may change the display mode of the display unit 120 depending on the relationship between the object and the host vehicle M. For example, the control unit 160 may calculate an index value (e.g., relative speed or TTC (time to collision)) representing the degree of proximity of the object to the host vehicle M based on the magnification rate of the object captured in time series by the on-board camera 10, and may increase the speed at which the predetermined figure is dynamically displayed as the calculated index value increases. For example, in the case of the screen shown in FIG. 13 , the display unit 120 may increase the speed at which multiple point-like objects display a circle in a clockwise direction as the calculated index value representing the degree of proximity increases, or in the case of the screen shown in FIG. 14 , the display unit 120 may increase the speed at which multiple line-like objects flow. Also, for example, the control unit 160 may calculate the distance between the object and the vehicle M, and the smaller the calculated distance becomes, the more the display unit 120 may display the object thicker (larger) to alert the occupants of the vehicle M.

[0056] Next, the flow of processing executed by the terminal device 100 will be described with reference to Figures 15 and 16. Figure 15 is a flowchart showing an example of the flow of processing executed by the terminal device 100. The processing of the flowchart in Figure 15 is repeatedly executed in a predetermined control cycle while the host vehicle M is traveling.

[0057] First, the recognition unit 140 recognizes objects around the vehicle M that are captured in images captured by the on-board camera 10 (step S100). Next, the prediction unit 150 predicts the future trajectories of the objects around the vehicle M that have been recognized by the recognition unit 140 (step S102).

[0058] Next, the control unit 160 determines whether the recognized peripheral object is approaching the vehicle M from behind, more specifically, whether the future trajectory predicted by the prediction unit 150 has entered the central determination area CDA and the peripheral object has entered the central area CA (step S104). If it is determined that the peripheral object is approaching the vehicle M from behind, the control unit 160 causes the audio output unit 130 to output a warning sound indicating that the peripheral object is approaching the vehicle M from the rear center (step S106).

[0059] On the other hand, if it is not determined that the peripheral object is approaching the vehicle M from behind, the control unit 160 determines whether the peripheral object is approaching the vehicle M from the side, more specifically, whether the future trajectory predicted by the prediction unit 150 has entered the side determination area (left side determination area LDA or right side determination area RDA) and whether the peripheral object has entered the lateral area (left side area LA or right side area RA) (step S108). If it is determined that the peripheral object is approaching the vehicle M from the side, the control unit 160 causes the audio output unit 130 to output a warning sound from either the left or right side corresponding to the approaching direction (step S110). This ends the processing of this flowchart.

[0060] Fig. 15 is a flowchart showing another example of the flow of processing executed by the terminal device 100. The processing of the flowchart in Fig. 15 is executed when the terminal device 100 starts up a driving assistance application.

[0061] First, the recognition unit 140 recognizes objects around the vehicle M that are captured in images captured by the on-board camera 10 (step S200). Next, the prediction unit 150 predicts the future trajectories of the objects around the vehicle M that have been recognized by the recognition unit 140 (step S202).

[0062] Next, the control unit 160 determines whether the host vehicle M is a motorcycle, more specifically, whether the mode accepted by the accepting unit 170 is the motorcycle mode (step S204). If it is determined that the host vehicle M is a motorcycle, the control unit 160 determines whether the recognized peripheral object is approaching the host vehicle M from behind (step S206), similar to step S104. If it is determined that the recognized peripheral object is approaching the host vehicle M from behind, the control unit 160 causes the display unit 120 to display multiple dot-like objects in a clockwise circle (step S208). On the other hand, if it is not determined that the recognized peripheral object is approaching the host vehicle M from behind, the control unit 160 returns the process to step S200.

[0063] If it is determined that the host vehicle M is a four-wheeled vehicle, the control unit 160 determines whether the recognized peripheral object is approaching the host vehicle M from the side (step S210), similar to step S108. If it is determined that the recognized peripheral object is not approaching the host vehicle M from the side, the control unit 160 returns the process to step S200. On the other hand, if it is determined that the recognized peripheral object is approaching the host vehicle M from the side, the control unit 160 causes the display unit 120 to display multiple linear objects diagonally upward from the corner corresponding to the approaching direction (step S212). This ends the process of this flowchart.

[0064] In the above description, the driving assistance app includes a reception unit 170, and the reception unit 170 changes the notification mode of the notification device depending on whether the host vehicle M is a four-wheeled vehicle or a two-wheeled vehicle. However, the present invention is not limited to such a configuration, and the driving assistance app may be provided separately as a four-wheeled vehicle-specific app installed in the four-wheeled vehicle and a two-wheeled vehicle-specific app installed in the two-wheeled vehicle. In this case, the four-wheeled vehicle-specific app executes the process shown in the flowchart of FIG. 15 and, when it predicts that a nearby object is approaching from the side, causes the display unit 120 to display multiple linear objects diagonally upward from a corner corresponding to the approaching direction. On the other hand, the two-wheeled vehicle-specific app executes the process shown in the flowchart of FIG. 15 and, when it predicts that a nearby object is approaching from behind the host vehicle M, causes the display unit 120 to display multiple dot-like objects in a clockwise circular pattern.

[0065] According to the present embodiment described above, the future trajectory of a surrounding object recognized by the recognition unit is predicted, and the notification mode by the notification device is changed depending on the predetermined area that the predicted future trajectory will enter. Furthermore, the notification mode by the notification device is changed depending on whether the host vehicle M is a two-wheeled vehicle or a four-wheeled vehicle. This makes it possible to notify the occupant of the moving body in an optimal manner depending on the predicted future trajectory of the surrounding vehicle.

[0066] The above-described embodiment can be expressed as follows: An information processing device comprising: a storage medium storing computer-readable instructions; and a processor connected to the storage medium, wherein the processor executes the computer-readable instructions to: recognize an object included in image data captured around a moving body; predict a future trajectory of the object; cause a notification device to notify a passenger of the moving body of the presence of the object based on the future trajectory of the object; and change the mode of notification by the notification device depending on whether the object is predicted to approach from behind the moving body or to pass by the side of the moving body.

[0067] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0068] REFERENCE SIGNS LIST 10 In-vehicle camera 100 Terminal device 110 Communication unit 120 Display unit 130 Audio output unit 140 Recognition unit 150 Prediction unit 160 Control unit 170 Reception unit

Claims

1. A recognition unit that recognizes objects included in image data captured around a moving object; A prediction unit for predicting a future trajectory of the object; a notification control unit that causes a notification device to notify a passenger of the moving body of the presence of the object based on a future trajectory of the object; the notification control unit changes a mode of notification by the notification device depending on whether the object is predicted to approach from behind the moving body or whether the object is predicted to pass by the side of the moving body. Information processing device.

2. the notification device is an audio output device; the notification control unit causes the audio output device to output a warning sound when it is predicted that the object will approach from behind the moving body and the object has entered a first predetermined area; The information processing device according to claim 1 .

3. the notification device is an audio output device; the notification control unit, when it is predicted that the object will pass on the side of the moving body and the object has entered a second predetermined area, outputs a warning sound indicating either the left or right side. The information processing device according to claim 1 .

4. the notification control unit, when it is predicted that the object will pass on the side of the moving body and the object has entered a second predetermined area, outputs a warning sound indicating either the left or right side; The first predetermined area is an area that exists within a first distance from the moving body, and the second predetermined area is an area that is away from the moving body by at least the first distance and within a second distance that is greater than the first distance. The information processing device according to claim 2 .

5. the notification device is a display device, The information processing device further includes a reception unit that receives instruction information indicating whether the moving object is a four-wheeled vehicle or a two-wheeled vehicle, When the receiving unit receives instruction information indicating that the moving body is a two-wheeled vehicle, the notification control unit causes the display device to display a predetermined figure in a center portion of the display device when the object is predicted to approach from behind the moving body and the object enters a first predetermined area. The information processing device according to claim 1 .

6. the notification device is a display device, The information processing device further includes a reception unit that receives instruction information indicating whether the moving object is a four-wheeled vehicle or a two-wheeled vehicle, When the receiving unit receives instruction information indicating that the moving body is a four-wheeled vehicle, when the object is predicted to pass beside the moving body and the object enters a second predetermined area, the notification control unit dynamically displays a predetermined figure diagonally upward on the display device from a lower left corner or a lower right corner of the display device corresponding to a direction in which the object enters. The information processing device according to claim 1 .

7. the notification control unit increases a speed at which the predetermined figure is dynamically displayed as the relative speed of the object with respect to the moving body increases; The information processing device according to claim 6.

8. The notification control unit increases the size of the predetermined figure as the object approaches the moving body. The information processing device according to claim 6.

9. The computer Recognizing objects contained in image data captured around the moving object, predicting a future trajectory of the object; causing a notification device to notify a passenger of the moving body of the presence of the object based on a future trajectory of the object; changing a mode of notification by the notification device depending on whether the object is predicted to approach from behind the moving body or whether the object is predicted to pass by the side of the moving body; Information processing methods.

10. On the computer, Recognizing objects contained in image data captured around the moving object, predicting a future trajectory of said object; causing a notification device to notify a passenger of the moving body of the presence of the object based on a future trajectory of the object; changing a mode of notification by the notification device depending on whether the object is predicted to approach from behind the moving body or whether the object is predicted to pass by the side of the moving body; program.

11. On the computer, Recognizing objects contained in image data captured around the motorcycle, predicting a future trajectory of said object; causing a notification device to notify a rider of the two-wheeled vehicle of the presence of the object based on a future trajectory of the object; changing a mode of notification by the notification device depending on whether the object is predicted to approach from behind the motorcycle or whether the object is predicted to pass beside the motorcycle; when it is predicted that the object will approach from behind the two-wheeled vehicle and the two-wheeled vehicle has entered a first predetermined area, causing the notification device to display a predetermined figure in a center portion of the notification device. program.

12. On the computer, Recognizing objects contained in image data captured around the four-wheeled vehicle, predicting a future trajectory of said object; causing a notification device to notify a passenger of the four-wheeled vehicle of the presence of the object based on a future trajectory of the object; changing a mode of notification by the notification device depending on whether the object is predicted to approach from behind the four-wheeled vehicle or whether the object is predicted to pass beside the four-wheeled vehicle; when it is predicted that the object will pass beside the four-wheeled vehicle and the four-wheeled vehicle has entered a second predetermined area, causing the notification device to dynamically display a predetermined figure diagonally upward from a lower left corner or a lower right corner of the notification device corresponding to the direction in which the object will enter; program.