Information processing device, information processing method, and program

Abstract

An information processing device is provided with: a recognition unit that recognizes an object included in image data obtained by capturing an image of the periphery of a moving body; a prediction unit that predicts a future trajectory of the object; and a notification control unit that causes a notification device to notify an occupant of the moving body of the presence of the object on the basis of the future trajectory of the object, the notification control unit causing the notification device to notify the occupant of the moving body of the presence of the object when it is predicted that the object approaches from the rear of the moving body and when it is predicted that the object passes the side of the moving body. The form of the notification performed by the notification device is changed.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method, and a program. Background Art

[0002] Conventionally, there is a known technology for notifying passengers of the presence of other vehicles traveling around the vehicle. For example, Patent Document 1 discloses a technology for instructing an alarm device provided in the vehicle to sound an alarm when a relative speed, which is the difference between the speed of a rear vehicle and the speed of the vehicle itself, is greater than a first threshold or when a change in the relative speed is greater than a second threshold.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-129178 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, in the conventional technology, there are cases where it is not possible to notify the occupants of the moving object in an optimal form based on the predicted future trajectory of the surrounding objects.

[0008] The present invention has been made in consideration of such circumstances, and one of its objects is to provide an information processing device, an information processing method, and a program that can notify the occupants of a moving object in an optimal form based on the predicted future orbits of surrounding objects.

[0009] Solutions to Solve Problems

[0010] The information processing device, information processing method, and program of the present invention employ the following structures.

[0011] (1): An information processing device according to one embodiment of the present invention comprises: a recognition unit that recognizes an object included in image data obtained by photographing the periphery of a moving body; a prediction unit that predicts the future trajectory of the object; and a notification control unit that causes a notification device to notify an occupant of the moving body of the existence of the object based on the future trajectory of the object, wherein the notification control unit changes the form of notification by the notification device when the object is predicted to approach from behind the moving body and when the object is predicted to pass by the side of the moving body.

[0012] (2): Based on the solution of (1) above, the notification device is a sound output device,

[0013] The notification control unit causes the sound output device to output a warning sound when it is predicted that the object approaches from behind the moving body and the moving body enters a first predetermined area.

[0014] (3): In the above-mentioned aspect (1), the notification device is configured such that, when it is predicted that the object passes by a side of the moving body and the moving body enters a second predetermined area, the notification control unit outputs a warning sound indicating either left or right of the side.

[0015] (4): Based on the scheme of (2) above, the notification control unit outputs a warning sound indicating either left or right of the side when it is predicted that the object passes by the side of the moving body and the moving body enters a second specified area, wherein the first specified area is an area within a first distance from the moving body, and the second specified area is an area that is separated from the moving body by more than the first distance and within a second distance from the moving body, and the second distance is greater than the first distance.

[0016] (5): Based on the scheme of (1) above, the notification device is a display device, and the information processing device further includes a receiving unit for receiving indication information indicating whether the moving body is a four-wheeled vehicle or a two-wheeled vehicle, and when the receiving unit receives indication information indicating that the moving body is a two-wheeled vehicle, the notification control unit causes the display device to display a specified graphic in the center of the display device when it is predicted that the object is approaching from behind the moving body and the moving body enters a first specified area.

[0017] (6): Based on the scheme of (1) above, the notification device is a display device, and the information processing device further includes a receiving unit for receiving indication information indicating whether the moving body is a four-wheeled vehicle or a two-wheeled vehicle, and when the receiving unit receives indication information indicating that the moving body is a four-wheeled vehicle, the notification control unit causes the display device to dynamically display a specified graphic from the lower left corner or the lower right corner of the display device corresponding to the entry direction of the object to the upper side when it is predicted that the object passes by the side of the moving body and the moving body enters a second specified area.

[0018] (7): In the above-mentioned aspect (6), the notification control unit increases the speed at which the predetermined graphic is dynamically displayed as the relative speed of the object relative to the moving body increases.

[0019] (8) In the above configuration of (6), the notification control unit increases the size of the predetermined graphic as the object approaches the moving body.

[0020] (9): The information processing method of another scheme of the present invention causes a computer to perform the following processing: identifying an object contained in image data obtained by photographing the periphery of a moving body; predicting the future trajectory of the object; based on the future trajectory of the object, causing a notification device to notify the occupants of the moving body of the existence of the object; and changing the form of notification performed by the notification device in a case where the object is predicted to approach from the rear of the moving body and a case where the object is predicted to pass by the side of the moving body.

[0021] (10): A program in another embodiment of the present invention causes a computer to perform the following processing: identifying an object contained in image data obtained by photographing the periphery of a moving body; predicting the future trajectory of the object; causing a notification device to notify occupants of the moving body of the existence of the object based on the future trajectory of the object; and changing the form of notification performed by the notification device in a case where the object is predicted to approach from behind the moving body and a case where the object is predicted to pass by the side of the moving body.

[0022] (11): A program according to another embodiment of the present invention causes a computer to perform the following processing: identifying an object contained in image data obtained by photographing the periphery of a two-wheeled vehicle; predicting the future trajectory of the object; causing a notification device to notify an occupant of the two-wheeled vehicle of the existence of the object based on the future trajectory of the object; changing the form of notification by the notification device in a case where the object is predicted to approach from the rear of the two-wheeled vehicle and a case where the object is predicted to pass by the side of the two-wheeled vehicle; and causing the notification device to display a specified graphic in the center of the notification device in a case where the object is predicted to approach from the rear of the two-wheeled vehicle and the moving body enters a first specified area.

[0023] (12): A program in another embodiment of the present invention causes a computer to perform the following processing: identifying an object contained in image data obtained by photographing the periphery of a four-wheeled vehicle; predicting the future trajectory of the object; causing a notification device to notify the occupants of the four-wheeled vehicle of the existence of the object based on the future trajectory of the object; changing the form of notification performed by the notification device in a case where the object is predicted to approach from the rear of the four-wheeled vehicle and in a case where the object is predicted to pass by the side of the four-wheeled vehicle; and causing the notification device to dynamically display a specified graphic from the lower left corner or the lower right corner of the notification device corresponding to the entry direction of the object to the diagonally upward direction in a case where the object is predicted to pass by the side of the four-wheeled vehicle and the mobile body enters a second specified area.

[0024] Effects of the Invention

[0025] According to the aspects (1) to (12), it is possible to notify the occupants of the moving object in an optimal form based on the predicted future trajectories of the surrounding vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a diagram showing an example of the configuration of an in-vehicle camera 10 and a terminal device 100 mounted on the vehicle M.

[0027] Figure 2 This is a diagram for explaining a method in which the prediction unit 150 predicts the future trajectory of an object.

[0028] Figure 3 is a diagram schematically showing density distribution.

[0029] Figure 4 It is shown Figure 3 Graph of the height of the density value D at line 4-4 in .

[0030] Figure 5 This is another diagram for explaining the method by which the prediction unit 150 predicts the future trajectory of an object.

[0031] Figure 6 1 and 10 are diagrams showing an example of a case where an object is predicted to approach the host vehicle M from behind and a case where an object is predicted to pass by the side of the host vehicle M.

[0032] Figure 7 1 is a diagram showing an example of a predetermined area behind the host vehicle M and a predetermined area to the side of the host vehicle M used by the control unit 160 to determine whether notification is necessary.

[0033] Figure 8 1 is a diagram showing another example of the predetermined area behind the host vehicle M and the predetermined area to the side of the host vehicle M used by the control unit 160 to determine whether notification is necessary.

[0034] Fig. 9 This is a diagram for explaining the operation of the notification device when an object enters the central area CA.

[0035] Fig.10 This is a diagram for explaining the operation of the notification device when an object enters the left area LA.

[0036] Fig.11 This is a diagram for explaining the operation of the notification device when an object enters the right area RA.

[0037] Fig.12 1 is a diagram showing an example of an acceptance screen IM1 of instruction information accepted by the acceptance unit 170 .

[0038] Fig.131 is a diagram showing an example of a screen displayed on the display unit 120 when the host vehicle M is a two-wheeled vehicle.

[0039] Fig.14 1 is a diagram showing an example of a screen displayed on the display unit 120 when the host vehicle M is a four-wheeled vehicle.

[0040] Fig.15 This is a flowchart showing an example of the flow of processing executed by the terminal device 100 .

[0041] Fig.16 This is a flowchart showing another example of the flow of processing executed by the terminal device 100 . DETAILED DESCRIPTION

[0042] Hereinafter, embodiments of an information processing device, an information processing method, and a program according to the present invention will be described with reference to the drawings.

[0043] [structure]

[0044] Figure 1 1 is a diagram showing an example of the structure of a vehicle-mounted camera 10 and a terminal device 100 mounted on a vehicle M. The vehicle M is, for example, a two-wheeled vehicle or a four-wheeled vehicle, and its driving source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a secondary battery or a fuel cell.

[0045] The vehicle-mounted camera 10 is a camera installed in the vehicle M. The vehicle-mounted camera 10 includes at least a camera installed at a position where it can shoot the rear of the vehicle M, and shoots the scenery outside the vehicle through the rear windshield, or is set in a manner that the lens is exposed to the outside of the vehicle and directly shoots the surroundings outside the vehicle. The vehicle-mounted camera 10 sends the captured image IM obtained by shooting the surroundings outside the vehicle to the terminal device 100 via wireless communication using Bluetooth (registered trademark), Wi-Fi, etc.

[0046] The terminal device 100 is, for example, a mobile terminal device such as a smart phone. When the vehicle M is a four-wheeled vehicle, the terminal device 100 is placed on a holder provided on the cabin side of the front windshield of the vehicle, for example, for use. When the vehicle M is a two-wheeled vehicle, the terminal device 100 is placed on a holder provided near the handle of the vehicle, for example, for use. That is, in any case, the terminal device 100 is placed at a position where the occupants of the vehicle M can visually confirm the display unit 120 of the terminal device 100.

[0047] The communication unit 110 is a wireless communication device using the above-mentioned Bluetooth (registered trademark), Wi-Fi, etc. The communication unit 110 communicates with the vehicle-mounted camera 10 and receives a captured image IM obtained by capturing the surroundings outside the vehicle.

[0048] The display unit 120 is a display device such as a touch panel or a liquid crystal display. The display unit 120 displays information on objects around the host vehicle M in accordance with control by a control unit 160 described later.

[0049] The sound output unit 130 is, for example, a speaker device. The sound output unit 130 outputs information about objects around the host vehicle M by sound according to the control of the control unit 160. The display unit 120 and the sound output unit 130 are examples of “notification devices”.

[0050] The terminal device 100 also includes a recognition unit 140, a prediction unit 150, a control unit 160, and a receiving unit 170. The recognition unit 140, the prediction unit 150, the control unit 160, and the receiving unit 170 are implemented by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can be implemented by hardware (including circuitry) such as LSI (LargeScale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc., or can be implemented by the cooperation of software and hardware. The program can be pre-stored in a storage device such as an HDD (Hard Disk Drive) or a flash memory (a storage device having a non-temporary storage medium), or can be stored in a removable storage medium such as a DVD or CD-ROM (a non-temporary storage medium), and installed by assembling the storage medium in a drive device. There is a case where the recognition unit 140, the prediction unit 150, the control unit 160, and the receiving unit 170 are hereinafter referred to as "driving support applications". In addition, the terminal device 100 equipped with the driving support application is an example of an “information processing device”.

[0051] The recognition unit 140 recognizes an object shown in the image captured by the vehicle-mounted camera 10. More specifically, for example, the recognition unit 140 recognizes the object using a learned model that has been learned in such a way as to output information such as the existence, position, and category of the object when the image captured by the vehicle-mounted camera 10 is input. The recognition unit 140 uses the learned model to distinguish between categories such as two-wheeled vehicles and four-wheeled vehicles, and to determine the area occupied by the vehicle in the captured image IM.

[0052] The prediction unit 150 predicts the future trajectory of the object recognized by the recognition unit 140 . Figure 2 1 is a diagram for explaining a method of predicting the future trajectory of an object by the prediction unit 150. When a plurality of vehicles are shown in the captured image IM, the prediction unit 150 performs the processing described below for each of the plurality of vehicles. Figure 2 The situation where the vehicle-mounted camera 10 captures the rear of the vehicle M is shown.

[0053] The prediction unit 150 first determines a position near the center of, for example, the lower end of the region determined by the recognition unit 140 as the position of the vehicle on the image plane of the captured image. Figure 2 In FIG. 1 , A1 to A4 represent the areas occupied by the determined vehicles M1 to M4, and P1 to P4 represent the positions of the determined vehicles M1 to M4. In the figure, M1 to M3 are four-wheeled vehicles, and M4 is a two-wheeled vehicle.

[0054] Furthermore, the prediction unit 150 transforms the position of the vehicle on the image plane into the position of the vehicle on the virtual plane S. The virtual plane S is a virtual plane observed from above and is substantially consistent with the road plane. The prediction unit 150 determines the position of the representative point of the vehicle on the virtual plane S based on, for example, a transformation rule for transforming coordinates on the image plane into coordinates on the virtual plane. Figure 2 P#1 to P#4 in represent the positions of the vehicles M1 to M4 on the imaginary plane S.

[0055] The prediction unit 150 sets the index value I having a distribution corresponding to the position of the vehicle on the virtual plane S based on the positions of the obtained multiple vehicles. The index value I is set, for example, in such a way that the contour line is in the shape of a circle centered on the position of the vehicle. When the index value I is set as the height, the distribution of the index on the virtual plane S has a dome-shaped shape with the position of the vehicle as the top. The prediction unit 150 may change the degree of spread of the density according to the type of vehicle, or may not change it.

[0056] Furthermore, the prediction unit 150 generates density distribution information by superimposing the index values ​​I obtained for a plurality of vehicles (adding them for each coordinate on the imaginary plane S). Hereinafter, the value obtained by adding up the index values ​​for a plurality of vehicles is referred to as the density value D. The index value I and the density value D may also be set to a predetermined value (e.g., 1) as an upper limit. In this case, when the result obtained by adding up the index values ​​I obtained for a plurality of vehicles exceeds the predetermined value, the prediction unit 150 sets the predetermined value of the location as the density value D. Figure 3 is a diagram schematically showing density distribution. Figure 4 It is shown Figure 3 Graph of the height of the density value D at line 4-4 in .

[0057] Figure 5 1 is another diagram for explaining the method of predicting the future track of an object by the prediction unit 150. As shown in the figure, the prediction unit 150 estimates that the target vehicle M4 travels along the track Tj formed by connecting the points (valleys of density distribution) with small density values ​​D. In addition, the prediction unit 150 predicts that the acceleration of the target vehicle M4 is small (including negative acceleration, i.e., deceleration) in the interval where the density value D increases between the points along the track Tj (in the figure, the interval to the point P#4-1), and predicts that the acceleration of the target vehicle M4 is large in the interval where the density value D decreases between the points (in the figure, the interval after the point P#4-1), and generates the future speed curve of the target vehicle M4 based on these. At this time, the prediction unit 150 may also predict that the acceleration is small when the rising speed between the points of the density value D is large (the gradient in the positive direction is large) compared with the case where the rising speed is small, and predict that the acceleration is large when the decreasing speed between the points of the density value D is large (the gradient in the negative direction is large) compared with the case where the decreasing speed is small. Since it is assumed that the target vehicle greatly accelerates in a scenario where the field of vision of the target vehicle is suddenly opened, the prediction unit 150 can more accurately predict the behavior of the target vehicle in this scenario.

[0058] The control unit 160 causes the display unit 120 or the sound output unit 130 to notify the existence of the object based on the future trajectory of the object predicted by the prediction unit 150 (hereinafter, the display unit 120 and the sound output unit 130 may be collectively referred to as a "notification device"). More specifically, the control unit 160 changes the form of notification by the notification device when the object is predicted to approach from the rear of the vehicle M and when the object is predicted to pass by the side of the vehicle M.

[0059] Figure 6 1 is a diagram showing an example of a case where an object is predicted to approach from behind the host vehicle M and a case where an object is predicted to pass by the side of the host vehicle M. Figure 6In FIG. 1 , Tj1 represents the future trajectory of the target vehicle M1, and Tj2 represents the future trajectory of the target vehicle M2. For example, the control unit 160 sets a central determination area CDA for determining the approach of an object at the center rear of the host vehicle M, and sets a left determination area LDA and a right determination area RDA at the side of the host vehicle M for determining the passage of an object.

[0060] When the predicted future trajectory Tj of the object enters the central determination area CDA, the control unit 160 predicts that the object is approaching the host vehicle M. On the other hand, when the predicted future trajectory Tj of the object enters the left determination area LDA or the right determination area RDA, the control unit 160 predicts that the object is passing the side of the host vehicle M. Figure 6 In the case of , the future trajectory Tj1 of the target vehicle M1 enters the left determination area LDA, and the future trajectory Tj2 of the target vehicle M2 enters the center determination area CDA. Therefore, the control unit 160 predicts that the target vehicle M1 passes the side of the host vehicle M, and predicts that the target vehicle M2 approaches the host vehicle M. It should be noted that in Figure 6 In the method, approach prediction and passing prediction are performed based on whether the predicted track enters a predetermined judgment area, but the present invention is not limited to such a structure. For example, passing prediction can also be performed based on whether the extension line of the front end of the vehicle M intersects with the future track.

[0061] The control unit 160 determines whether an object predicted to pass by the side of the host vehicle M or an object predicted to approach the host vehicle M has actually entered a predetermined area, and causes the notification device to make a notification if it is determined that the object has entered the predetermined area. Figure 7 1 is a diagram showing an example of a predetermined area behind the host vehicle M and a predetermined area to the side of the host vehicle M used for the control unit 160 to determine whether notification is necessary. Figure 7 , reference numeral CA indicates a prescribed area behind the center of the host vehicle M, reference numeral LA indicates a prescribed area behind the side (left) of the host vehicle M, and reference numeral RA indicates a prescribed area behind the side (right) of the host vehicle M. The prescribed area behind the host vehicle M is an example of a “first prescribed area”, and the prescribed area on the side of the host vehicle M is an example of a “second prescribed area”.

[0062] like Figure 7As shown, the central area CA is defined as, for example, an area obtained by extending a distance D1 from the rear end of the vehicle M in the opposite direction of the travel direction of the vehicle M. The left area LA is defined as, for example, an area on the left side of the vehicle M in the opposite direction of the travel direction of the vehicle M, which is within a distance D2 greater than the distance D1 from the rear end of the vehicle M. The right area RA is defined as, for example, an area on the right side of the vehicle M in the opposite direction of the travel direction of the vehicle M, which is within a distance D2 greater than the distance D1 from the rear end of the vehicle M. When the control unit 160 determines that an object predicted to approach the vehicle M or pass the side of the vehicle M enters any of the central area CA, the left area LA, and the right area RA, the notification device notifies the occupants of the vehicle M of the existence of the object in a form corresponding to the determined area.

[0063] It should be noted that in Figure 7 In the embodiment, the side area LA or RA related to the side of the host vehicle M is set to be longer in distance from the host vehicle M than the central area CA related to the center of the host vehicle M. This is because, in general, in order to confirm the side area LA or RA related to the side of the host vehicle M, the occupant of the host vehicle M needs to use the side rearview mirror of the host vehicle M, and there is a tendency to take longer time to confirm than the central area CA confirmed by using the interior rearview mirror of the host vehicle M. By setting the distance to the side area LA or RA longer, the occupant of the host vehicle M can confirm the area on the side of the host vehicle M with some room.

[0064] Figure 8 FIG. 2 is a diagram showing another example of a predetermined area behind the host vehicle M and a predetermined area to the side of the host vehicle M used by the control unit 160 to determine whether notification is necessary. Figure 8 In the figure, the specified area CA can also be defined as an area within a distance D1 in the opposite direction of the traveling direction of the vehicle M. In other words, the control unit 160 can also determine that the object has entered the specified area CA when the distance between the vehicle M and an object predicted to be close to the vehicle M is within D1. The same is true for the left area LA and the right area RA. The control unit 160 can also determine that the object has entered the left area LA or the right area RA when the distance between the vehicle M and an object predicted to pass by the side of the vehicle M is within D2. More generally, the central area CA, the left area LA, and the right area RA can be as follows. Figure 7 As a finite region, it can also be defined as Figure 8 That is defined as an infinite region extending infinitely in any direction.

[0065] Fig. 9This is a diagram for explaining the operation of the notification device when an object enters the central area CA. Fig. 9 This is a scene showing that the two-wheeled vehicle M1 has entered the central area CA. In this case, the control unit 160 causes the sound output unit 130 to output a warning sound (for example, “Center!”) indicating that the two-wheeled vehicle M1 has entered the central area CA and is approaching the host vehicle M.

[0066] And, in Fig. 9 In the figure, reference numeral B1 denotes a bounding box surrounding the two-wheeled vehicle M1, and the display unit 120 displays the object recognized by the recognition unit 140 together with the bounding box B1. In addition, reference numeral BV denotes a bird's-eye view showing the surroundings of the host vehicle M, and the display unit 120 displays the position of the object in the virtual plane S derived by the prediction unit 150. The display of these bounding boxes and the bird's-eye view is just an example, and other information suitable for driving support may also be displayed.

[0067] Fig.10 This is a diagram for explaining the operation of the notification device when an object enters the left area LA. Fig.10 The scene in which the two-wheeled vehicle M1 enters the left area LA is shown. In this case, the control unit 160 causes the sound output unit 130 to output a warning sound indicating that the two-wheeled vehicle M1 enters the left area LA and passes to the left of the host vehicle M (for example, “Left!”).

[0068] Fig.11 This is a diagram for explaining the operation of the notification device when an object enters the right area RA. Fig.11 In this case, the control unit 160 causes the sound output unit 130 to output a warning sound indicating that the two-wheeled vehicle M1 has entered the right area RA and passed the right side of the host vehicle M (for example, "right!").

[0069] It should be noted that in the above description, Fig.11 In the scenario of , the control unit 160 causes the sound output unit 130 to output a warning sound (e.g., "Center!") indicating that the two-wheeled vehicle M1 is predicted to enter the central area CA. However, the present invention is not limited to such a configuration, and for example, the control unit 160 may cause the sound output unit 130 to output an arbitrary electronic sound that does not have a specific meaning. This is because, if the warning sound for the side area has a meaning, even if the warning sound for the central area has no meaning, the occupants of the vehicle M can recognize that the warning sound is a warning sound for the central area.

[0070] In this way, the future trajectory of an object existing behind the vehicle M is predicted, and the form of notification is changed when the predicted future trajectory is predicted to approach from the rear of the vehicle M or predicted to pass the side of the vehicle M, thereby enabling notification to the occupants of the moving body in the best form based on the predicted future trajectory of the surrounding objects.

[0071] The receiving unit 170 receives instruction information indicating whether the host vehicle M is a four-wheel vehicle or a two-wheel vehicle. The control unit 160 changes the form of notification by the notification device based on the instruction information received by the receiving unit 170 as described later.

[0072] Fig.12 1 is a diagram showing an example of an acceptance screen IM1 of instruction information accepted by the acceptance unit 170. The acceptance screen IM1 is, for example, a screen displayed when the user of the terminal device 100 downloads and installs the driving support application via a network (not shown) and starts the driving support application for the first time. In addition, for example, the screen may also be displayed according to the user's selection after the driving support application is started for the first time.

[0073] exist Fig.12 In the figure, reference numeral B1 indicates a button (software switch) selected by the user of the terminal device 100 when the vehicle M is a four-wheel vehicle, and reference numeral B2 indicates a button selected by the user of the terminal device 100 when the vehicle M is a two-wheel vehicle. As described below, the receiving unit 170 receives the selection of the four-wheel vehicle mode B1 or the two-wheel vehicle mode B2 by the user, and the control unit 160 controls the display by the display unit 120 according to the received mode.

[0074] Fig.13 1 is a diagram showing an example of a screen displayed on the display unit 120 when the host vehicle M is a two-wheeled vehicle. Fig.13 1 shows a screen that the control unit 160 causes the display unit 120 to display when an object approaching from the rear of the host vehicle M is predicted to enter the central area CA. Fig.13 As shown, when an object approaching from the rear of the host vehicle M is predicted to enter the central area CA, the control unit 160 causes the display unit 120 to display a predetermined graphic near the center of the screen of the terminal device 100. This is to visually convey to the occupants of the host vehicle M the fact that the object is approaching the host vehicle M from the center and rear.

[0075] For example, in Fig.13In the case of the screen shown in FIG. 1 , the display unit 120 dynamically displays a plurality of dot-shaped objects in the center of the screen by drawing a circle in a clockwise direction. On the other hand, when an object passing by the side of the vehicle M is predicted to enter the side area LA or RA, the control unit 160 causes the display unit 120 not to display a predetermined graphic on the screen of the terminal device 100. This is because, when the vehicle M is a two-wheeled vehicle, for the occupants of the vehicle M, an object approaching from the center rear of the vehicle M (i.e., an object that poses a risk of rear-end collision) is generally considered to be an object that requires more attention than an object approaching from the oblique rear of the vehicle M.

[0076] Fig.14 1 is a diagram showing an example of a screen displayed on the display unit 120 when the host vehicle M is a four-wheeled vehicle. Fig.14 1 shows a screen that the control unit 160 causes the display unit 120 to display when an object passing to the left of the host vehicle M is predicted to enter the left area LA. Fig.14 As shown in FIG. 1 , when an object passing the left side of the host vehicle M is predicted to enter the left area LA, the control unit 160 causes the display unit 120 to display a predetermined graphic from the lower left corner of the display unit 120 corresponding to the left area LA to the upper side. Similarly, when an object passing the right side of the host vehicle M is predicted to enter the right area RA, the control unit 160 causes the display unit 120 to display a predetermined graphic from the lower right corner of the display unit 120 corresponding to the right area RA to the upper side. This is to visually convey to the occupants of the host vehicle M the fact that the object is about to pass by the side of the host vehicle M.

[0077] For example, in Fig.14 In the case of the screen shown in FIG. 1 , the display unit 120 displays a plurality of linear objects in a flowing manner from the lower left corner of the display unit 120 corresponding to the left area LA to diagonally upward. Similarly, in the present embodiment, when an object passing through the right side of the vehicle M is predicted to enter the right area RA, the display unit 120 displays a plurality of linear objects in a flowing manner from the lower right corner of the display unit 120 corresponding to the right area RA to diagonally upward. On the other hand, when an object approaching from the rear of the vehicle M is predicted to enter the central area CA, the control unit 160 causes the display unit 120 not to display a prescribed graphic on the screen of the terminal device 100. This is because, in the case where the vehicle M is a four-wheel vehicle, for the occupants of the vehicle M, an object approaching from the oblique rear of the vehicle (i.e., an object that is dangerous to be involved by the vehicle M) is generally considered to be an object that should be given more attention than an object approaching from the center rear of the vehicle M.

[0078] Furthermore, when an object enters a predetermined area, the control unit 160 may change the display mode of the display unit 120 according to the relationship between the object and the vehicle M. For example, the control unit 160 may calculate an index value (e.g., relative speed, TTC (time to collison)) indicating the degree of proximity of the object to the vehicle M based on the magnification of the object captured by the vehicle-mounted camera 10 in a time series, and the larger the calculated index value, the faster the predetermined graphic is displayed dynamically. For example, in Fig.13 In the case of the screen shown in FIG. 1 , the display unit 120 may increase the speed at which the plurality of dot-shaped objects are displayed in a circle clockwise as the calculated index value indicating the degree of proximity increases. Fig.14 In the case of the screen shown, the display unit 120 may also increase the speed of the flow of multiple linear objects. In addition, for example, the control unit 160 may calculate the distance between the object and the vehicle M. The smaller the calculated distance is, the thicker (larger) the display unit 120 displays the object, thereby calling the attention of the occupants of the vehicle M.

[0079] Next, refer to Fig.15 as well as Fig.16 The flow of processing executed by the terminal device 100 will be described. Fig.15 This is a flowchart showing an example of the flow of processing executed by the terminal device 100 . Fig.15 The processing in the flowchart is repeatedly executed in a predetermined control cycle while the vehicle M is traveling.

[0080] First, the recognition unit 140 recognizes the surrounding objects of the host vehicle M that appear in the image captured by the vehicle-mounted camera 10 (step S100 ). Next, the prediction unit 150 predicts the future trajectory of the surrounding objects recognized by the recognition unit 140 (step S102 ).

[0081] Next, the control unit 160 determines whether the recognized surrounding object is approaching from the rear of the vehicle M, more specifically, whether the future track predicted by the prediction unit 150 enters the central determination area CDA and the surrounding object enters the central area CA (step S104). If it is determined that the surrounding object is approaching from the rear of the vehicle M, the control unit 160 causes the sound output unit 130 to output a warning sound indicating that the surrounding object is approaching from the center rear (step S106).

[0082] On the other hand, if it is not determined that the surrounding object is approaching from the rear of the vehicle M, the control unit 160 determines whether the surrounding object is approaching from the side of the vehicle M, more specifically, whether the future track predicted by the prediction unit 150 enters the side determination area (left determination area LDA or right determination area RDA) and the surrounding object enters the side area (left area LA or right area RA) (step S108). If it is determined that the surrounding object is approaching from the side of the vehicle M, the control unit 160 causes the sound output unit 130 to output a warning sound in either the left or right direction corresponding to the approaching direction (step S110). Thus, the processing of this flowchart ends.

[0083] Fig.15 This is a flowchart showing another example of the flow of processing executed by the terminal device 100 . Fig.15 The process of the flowchart is executed when the terminal device 100 starts the driving support application.

[0084] First, the recognition unit 140 recognizes the surrounding objects of the host vehicle M that appear in the image captured by the vehicle-mounted camera 10 (step S200 ). Next, the prediction unit 150 predicts the future trajectory of the surrounding objects recognized by the recognition unit 140 (step S202 ).

[0085] Next, the control unit 160 determines whether the vehicle M is a two-wheeled vehicle, more specifically, whether the mode received by the receiving unit 170 is a two-wheeled vehicle mode (step S204). When it is determined that the vehicle M is a two-wheeled vehicle, the control unit 160 determines whether the recognized surrounding objects are approaching from the rear of the vehicle M in the same manner as step S104 (step S206). When it is determined that the recognized surrounding objects are approaching from the rear of the vehicle M, the control unit 160 causes the display unit 120 to display a plurality of dot-shaped objects in a clockwise circle (step S208). On the other hand, when it is not determined that the recognized surrounding objects are approaching from the rear of the vehicle M, the control unit 160 returns the process to step S200.

[0086] When it is determined that the vehicle M is a four-wheel vehicle, the control unit 160 determines whether the recognized surrounding objects are approaching from the side of the vehicle M in the same manner as step S108 (step S210). When it is determined that the recognized surrounding objects are not approaching from the side of the vehicle M, the control unit 160 returns the process to step S200. On the other hand, when it is determined that the recognized surrounding objects are approaching from the side of the vehicle M, the control unit 160 causes the display unit 120 to display a plurality of linear objects obliquely upward from the corner corresponding to the approaching direction (step S212). Thus, the process of this flowchart ends.

[0087] It should be noted that in the above description, the driving support application includes a receiving unit 170, and the receiving unit 170 changes the form of the notification by the notification device according to whether the vehicle M is a four-wheeled vehicle or a two-wheeled vehicle. However, the present invention is not limited to such a structure, and the driving support application can also be separately provided as a four-wheeled vehicle-specific application installed on a four-wheeled vehicle and a two-wheeled vehicle-specific application installed on a two-wheeled vehicle. In this case, the four-wheeled vehicle-specific application executes Fig.15 , and when it is predicted that a surrounding object is approaching from the side, the display unit 120 displays a plurality of linear objects diagonally upward from the corner corresponding to the approaching direction. Fig.15 The process shown in the flowchart is performed, and when it is predicted that a surrounding object is approaching from the rear of the host vehicle M, the display unit 120 is caused to display a plurality of dot-shaped objects in a manner of drawing a circle in a clockwise direction.

[0088] According to the present embodiment described above, the future track of the surrounding object identified by the identification unit is predicted, and the form of the notification by the notification device is changed according to the predetermined area that the predicted future track enters. In addition, the form of the notification by the notification device is changed according to whether the host vehicle M is a two-wheeled vehicle or a four-wheeled vehicle. Thus, it is possible to notify the occupants of the mobile body in the best form according to the predicted future track of the surrounding vehicle.

[0089] The above-described embodiment can be expressed as follows.

[0090] An information processing device, wherein:

[0091] The information processing device comprises:

[0092] a storage medium that stores computer-readable instructions; and

[0093] a processor connected to the storage medium,

[0094] The processor executes the computer-readable instructions to perform the following processing:

[0095] Identifying objects included in image data obtained by photographing the periphery of a moving object;

[0096] predicting the future orbit of the object;

[0097] causing a notification device to notify an occupant of the moving body of the existence of the object based on the future trajectory of the object; and

[0098] The form of notification by the notification device is changed between a case where the object is predicted to approach from behind the moving body and a case where the object is predicted to pass by the side of the moving body.

[0099] Although specific embodiments of the present invention have been described above using the embodiments, the present invention is not limited to such embodiments at all, and various modifications and substitutions can be made without departing from the gist of the present invention.

[0100] Description of Reference Numerals

[0101] 10 Car Camera

[0102] 100 Terminal Devices

[0103] 110 Ministry of Communications

[0104] 120 Display unit

[0105] 130 Sound output unit

[0106] 140 Identification Department

[0107] 150 Forecasting Department

[0108] 160 Control Department

[0109] 170 Receiving Department.

Claims

1. An information processing device, wherein: The information processing device comprises: a recognition unit that recognizes an object included in image data obtained by photographing the periphery of the moving object; a prediction unit that predicts the future trajectory of the object; as well as a notification control unit that causes a notification device to notify an occupant of the moving body of the existence of the object based on the future trajectory of the object, The notification control unit changes the form of notification by the notification device when the object is predicted to approach from behind the moving body and when the object is predicted to pass by the side of the moving body.

2. The information processing device according to claim 1, wherein: The notification device is a sound output device, The notification control unit causes the sound output device to output a warning sound when it is predicted that the object approaches from behind the moving body and the moving body enters a first predetermined area.

3. The information processing device according to claim 1, wherein: The notification device is a sound output device, The notification control unit outputs a warning sound indicating either left or right of the side, when it is predicted that the object passes by the side of the moving body and the moving body enters a second predetermined area.

4. The information processing device according to claim 2, wherein: The notification control unit outputs a warning sound indicating either left or right of the side when it is predicted that the object passes by the side of the moving body and the moving body enters a second predetermined area. The first prescribed area is an area located within a first distance from the moving body, and the second prescribed area is an area located at a position separated from the moving body by more than the first distance and within a second distance from the moving body, wherein the second distance is greater than the first distance.

5. The information processing device according to claim 1, wherein: The notification device is a display device, The information processing device further includes a receiving unit for receiving instruction information indicating whether the moving body is a four-wheeled vehicle or a two-wheeled vehicle. When the receiving unit receives indication information indicating that the moving body is a two-wheeled vehicle, the notification control unit causes the display device to display a specified graphic in the center of the display device when it is predicted that the object is approaching from behind the moving body and the moving body enters a first specified area.

6. The information processing device according to claim 1, wherein: The notification device is a display device, The information processing device further includes a receiving unit for receiving instruction information indicating whether the moving body is a four-wheeled vehicle or a two-wheeled vehicle. When the receiving unit receives indication information indicating that the moving body is a four-wheeled vehicle, the notification control unit causes the display device to dynamically display a specified graphic diagonally upward from the lower left corner or the lower right corner of the display device corresponding to the entry direction of the object when it is predicted that the object passes by the side of the moving body and the moving body enters a second specified area.

7. The information processing device according to claim 6, wherein: The notification control unit increases the speed at which the predetermined graphic is dynamically displayed as the relative speed of the object relative to the moving body increases.

8. The information processing device according to claim 6, wherein: The notification control unit increases the size of the predetermined graphic as the object approaches the moving body.

9. An information processing method, wherein: The information processing method enables the computer to perform the following processing: Identifying objects included in image data obtained by photographing the periphery of a moving object; predicting the future orbit of the object; causing a notification device to notify an occupant of the moving body of the presence of the object based on the future trajectory of the object; as well as The form of notification by the notification device is changed between a case where the object is predicted to approach from behind the moving body and a case where the object is predicted to pass by the side of the moving body.

10. A program, wherein: The program causes the computer to perform the following processing: Identifying objects included in image data obtained by photographing the periphery of a moving object; predicting the future orbit of the object; causing a notification device to notify an occupant of the moving body of the presence of the object based on the future trajectory of the object; as well as The form of notification by the notification device is changed between a case where the object is predicted to approach from behind the moving body and a case where the object is predicted to pass by the side of the moving body.

11. A program, wherein: The program causes the computer to perform the following processing: Recognizing an object included in image data obtained by photographing the periphery of the two-wheeled vehicle; predicting the future orbit of the object; causing a notification device to notify an occupant of the two-wheeled vehicle of the presence of the object based on the future trajectory of the object; changing the form of notification by the notification device in a case where the object is predicted to approach from behind the two-wheeled vehicle and a case where the object is predicted to pass by the side of the two-wheeled vehicle; as well as When it is predicted that the object approaches from behind the two-wheeled vehicle and the two-wheeled vehicle enters a first predetermined area, the notification device is caused to display a predetermined graphic in the center portion of the notification device.

12. A program, wherein: The program causes the computer to perform the following processing: Recognizing an object included in image data obtained by photographing the surroundings of the four-wheeled vehicle; predicting the future orbit of the object; causing a notification device to notify an occupant of the four-wheeled vehicle of the presence of the object based on the future trajectory of the object; changing the form of notification by the notification device in a case where the object is predicted to approach from behind the four-wheeled vehicle and a case where the object is predicted to pass by the side of the four-wheeled vehicle; as well as When it is predicted that the object passes by the side of the four-wheeled vehicle and the four-wheeled vehicle enters the second specified area, the notification device dynamically displays a specified graphic diagonally upward from the lower left corner or lower right corner of the notification device corresponding to the entry direction of the object.

Images