Information providing method, storage medium, and terminal device

Abstract

The present application provides an information providing method, a storage medium, and a terminal device capable of more accurately and favorably estimating an intention of a moving body, and capable of providing useful information to a user by displaying an image of a report related to the presence of a traffic participant based on the estimated intention of the moving body. The information providing method causes a computer of a terminal device mounted on a moving body and equipped with a camera to execute the following processing: acquiring images captured at different times; estimating whether or not the moving body has an intention to make a right turn or a left turn based on the images captured at the different times; and causing a display portion provided on a housing in which the camera is provided to display an image of a report related to the presence of a traffic participant based on the result of the estimation.

Description

Technical Field

[0001] This invention relates to information provision methods, storage media, and terminal devices. Background Technology

[0002] Previously, a device for determining the rapid deceleration of a vehicle based on captured images was disclosed (Japanese Patent Application Publication No. 2002-79896). Summary of the Invention

[0003] However, in the aforementioned techniques, it is sometimes impossible to presume the intention of the moving body's actions.

[0004] The present invention was made in consideration of such circumstances, and one of its objectives is to provide an information providing method, storage medium, and terminal device that can more accurately and effectively estimate the intention of a moving body to perform its actions.

[0005] Moreover, one of the objectives is to provide an information delivery method, storage medium, and terminal device that can provide useful information to users by displaying images of reports related to the presence of traffic participants based on the presumed intent of the action.

[0006] Solution for solving the problem

[0007] The information providing method, storage medium, and terminal device of the present invention adopt the following structure.

[0008] (1): The information providing method causes a computer of a terminal device equipped with a camera and installed on a mobile body to perform the following processing: acquiring images taken at different times; based on the images taken at the different times, inferring whether the mobile body intends to turn right or left; and based on the inference result, causing the display unit of the terminal device provided on the housing on which the camera is provided to display images of a report related to the presence of a traffic participant.

[0009] (2): Based on the above (1) scheme, the computer also performs the following processing: presumes whether the moving body is in a straight state or the moving body is turning right or left.

[0010] (3): Based on the above scheme (1) or (2), the computer further performs the following processing: rotates the image captured at the first moment to the left or right by a predetermined angle from the reference position; compares the image captured at the second moment, which is different from the first moment, with the image obtained by rotating the predetermined angle; and based on the comparison result, infers whether the moving body intends to turn right or left.

[0011] (4): Based on the above (3) scheme, the computer further performs the following processing: comparing the first image of the central part of the image captured at the first moment with the second image of the central part of the image obtained by rotating the predetermined angle.

[0012] (5): Based on the above (4) scheme, the computer further performs the following processing: obtaining a first image of the central portion of each of the multiple images obtained by rotating the image captured at the first moment by a predetermined angle to the left and right from the reference position; comparing the obtained first image of the central portion with the second image of the central portion of the image captured at the second moment; if the rotation angle of the first image with the smallest difference between the first image and the second image is above a threshold, it is presumed that the moving body has the intention to turn right or left.

[0013] (6): Based on the above (5) scheme, the computer further performs the following processing: the size of the cropped image obtained by cropping a specific area from the image captured at the first moment is close to the size of the image captured at the second moment, the cropped image is rotated by the specified angle, and the image captured at the second moment is compared with the image obtained by rotating by the specified angle.

[0014] (7): Based on any of the above schemes (1) to (6), the computer further performs the following processing: if it is presumed that the moving body intends to turn right or left, and it is presumed that the traffic participant is present or near the path or path through which the moving body made the right or left turn, the display unit displays an image of a report related to the proximity of the traffic participant.

[0015] (8): An application program stored in the storage medium of one aspect of the present invention causes the computer of the terminal device to perform the following processing: acquiring images taken at different times; based on the images taken at said different times, inferring whether the moving body intends to turn right or left; and based on the inference result, causing the display unit to display images of a report related to the presence of traffic participants.

[0016] (9): A terminal device according to one aspect of the present invention includes: a camera disposed on a housing; a display unit disposed on the housing; an estimation unit that estimates whether a moving body intends to turn right or left based on images captured by the camera at different times; and a display control unit that, based on the estimation result, causes the display unit disposed on the housing where the camera is disposed to display images of reports related to the presence of traffic participants.

[0017] Invention Effects

[0018] According to (1)-(9), the information providing method, the application stored in the storage medium, or the terminal device performs processing based on images captured at different times to presume whether a moving vehicle intends to turn right or left, thereby enabling a more accurate and reliable estimation of the moving vehicle's intention. The information providing method also performs processing based on the estimation result to display images of a report related to the presence of a traffic participant on the display unit, thereby providing useful information to the user.

[0019] According to (7), the information providing method displays an image of a report related to the proximity of a traffic participant when the mobile vehicle turns right or left, thereby enabling a report to the operator of the mobile vehicle. Attached Figure Description

[0020] Figure 1 It is a diagram used to illustrate the overview of the support system.

[0021] Figure 2 This is a diagram illustrating an example of a scenario related to support.

[0022] Figure 3 This is another example of a scenario related to support.

[0023] Figure 4 This is another example of a scenario related to support.

[0024] Figure 5 This is another example of a scenario related to support.

[0025] Figure 6 This is a diagram illustrating an example of the functional structure of a support system.

[0026] Figure 7 This diagram shows an example of a terminal device installed on this vehicle.

[0027] Figure 8 This is a diagram illustrating an example of the structure of a terminal device, including its functional components.

[0028] Figure 9 This is a flowchart illustrating an example of a process executed by an application (terminal device).

[0029] Figure 10 This is an example of an image displayed on the display section of a terminal device.

[0030] Figure 11 This is a diagram used to illustrate the processing related to the estimation of the amount of movement.

[0031] Figure 12 This diagram illustrates the processing related to posture determination.

[0032] Figure 13 This is a flowchart illustrating an example of a processing flow performed by the processing department.

[0033] Figure 14 This diagram illustrates the process of detecting traffic participants.

[0034] Figure 15 This is a diagram used to illustrate the process of predicting the actions of detected traffic participants.

[0035] Figure 16 This is a diagram illustrating an example of the definition of risk when traveling straight.

[0036] Figure 17 This diagram illustrates an example of how to handle the risk of a vehicle traveling straight.

[0037] Figure 18 This is a diagram illustrating an example of the definition of risk when turning left.

[0038] Figure 19 This diagram illustrates an example of how to handle the risk of a vehicle making a left turn.

[0039] Figure 20 This is a diagram illustrating an example of the definition of the risk when turning right.

[0040] Figure 21 This diagram illustrates an example of how to handle the risk of this vehicle turning right.

[0041] Figure 22 This is a diagram illustrating an example of the treatment of presumed potential risks.

[0042] Figure 23 This is a flowchart illustrating an example of a processing flow performed by the processing department.

[0043] Figure 24 This is a flowchart illustrating an example of a process related to alerts / attention.

[0044] Figure 25 It is a diagram used to illustrate the scenarios and notifications that generate forward-looking warning risks.

[0045] Figure 26 It is a diagram used to illustrate the scenarios and notifications that generate left-side or right-side alarm risks.

[0046] Figure 27 It is a diagram used to illustrate scenarios and notifications that generate risk awareness.

[0047] Figure 28 It is a diagram used to illustrate scenarios and notifications that may generate potential risks. Detailed Implementation

[0048] The following description, with reference to the accompanying drawings, illustrates the information provision method, application program, and terminal device involved in the embodiments of the present invention.

[0049] [Summary 1]

[0050] Figure 1 This diagram illustrates the general outline of support system 1. In this embodiment, as an example, a two-wheeled vehicle (hereinafter, sometimes referred to as "this vehicle") is described, but it can also be applied to various other mobile bodies such as light vehicles and three-wheeled vehicles. Hereinafter, the forward / backward direction of this vehicle is sometimes referred to as the X direction, the direction orthogonal to the forward / backward direction is referred to as the Y direction, and the direction orthogonal to both the X and Y directions is referred to as the Z direction.

[0051] Support system 1 includes a passing indicator light PL, a terminal device (e.g., a smartphone) SP, and an application app (AP) installed on the terminal device SP. The passing indicator light PL is installed, for example, around the headlights, in a manner that illuminates in the positive X direction (forward direction) (passing indication). The passing indicator light PL-R within the passing indicator light PL is installed in the positive Y direction of the vehicle, and the passing indicator light PL-L within the passing indicator light PL is installed in the negative Y direction of the vehicle.

[0052] The terminal device SP has a camera and a light mounted on its first side, and a display unit on its second side, opposite to the first side. The terminal device SP is installed in the vehicle. It is positioned such that the camera can capture images of the vehicle in the X-direction, and the light illuminates in the X-direction (overtaking indicator). The terminal device SP is positioned so that the driver (operator) can visually recognize the display unit.

[0053] The support system 1 uses one or both of the hazard lights and the overtaking indicator light PL of the terminal device SP to instruct traffic participants to overtake. The support system 1 further causes the display unit of the terminal device SP to display an alarm. Hereinafter, the two aspects of instructing overtaking and displaying an alarm will sometimes be referred to as "support".

[0054] For example, two-wheeled vehicles tend to have long braking distances, and therefore tend to have a low ability to avoid approaching other road users. In order to suppress approaching other road users, support system 1 reports the approach to the driver in advance, conveys the possibility of approach to other road users through automatic overtaking instructions via overtaking indicator lights, and entrusts cooperation in suppressing the approach (deceleration, stopping, avoidance, etc.).

[0055] Support System 1, such as Figure 2 The system provides support to prevent head-on collisions. For example, support system 1 provides support to avoid the vehicle from approaching pedestrians, two-wheeled vehicles, or four-wheeled vehicles.

[0056] Support System 1, such as Figure 3 The system shown provides support to suppress approaching traffic participants at intersections. Support system 1 is as follows: Figure 4 The indicated method is to provide support for suppressing the approach of a traffic participant when the vehicle is squeezed laterally past a four-wheeled vehicle in a four-wheeled vehicle congestion, or to provide support for suppressing the approach of a traffic participant who has stopped in front of it.

[0057] Support System 1, such as Figure 5 The system provides support to suppress approaching vehicles based on potential risks. For example, support system 1 suppresses the risk of the vehicle turning right in a four-wheeled vehicle congestion, or the risk of a traffic participant suddenly running from behind a parked vehicle. For example, in this embodiment, as... Figure 5 As shown in the right figure, if there is a line of objects (four-wheeled vehicles) located near the vehicle and connected in the same direction as the vehicle's travel direction, it is determined that there is a potential risk. In this embodiment, as... Figure 5 As shown in the left diagram, if a vehicle traveling on the first road enters the second road connecting to the first road (by turning right from the first road), and there is an object (a four-wheeled vehicle) in front of or behind the junction (intersection) of the first road and the second road in the direction of extension of the first road, it is determined that there is a potential risk. The details are explained below.

[0058] [Functional Structure of the Support System]

[0059] Figure 6 This is a diagram illustrating an example of the functional structure of support system 1. Support system 1, for example, is as follows: Figure 1 As explained, it includes a terminal device SP and a passing indicator PL. For example, the passing indicator PL and the terminal device SP are connected to the vehicle M's battery BT via a power line and operate by receiving power from the battery BT. The terminal device SP can also operate by receiving power from the battery BT via the passing indicator PL, or it can operate by using power from the battery mounted on the terminal device SP.

[0060] The terminal device SP and the overtaking indicator PL are connected by a communication line and can communicate with each other. For example, the terminal device SP sends an execution signal to activate the overtaking indicator PL, and the overtaking indicator PL executes the overtaking instruction according to the execution signal.

[0061] Terminal device SP such as Figure 7 The device is mounted on the vehicle by a retainer. For example, the camera Cam mounted as the terminal device SP takes pictures in the positive X direction, the lamp Lig of the terminal device SP emits light in the positive X direction, and the display unit Dis faces the negative X direction.

[0062] Figure 8 This diagram illustrates an example of the functional structure of a terminal device SP. The terminal device SP includes, for example, an acquisition unit 10, a processing unit 20, a notification unit 30, and a storage unit 50. The acquisition unit 10, processing unit 20, and notification unit 30 are implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing an application program (AP) stored in the storage unit 50. The application program AP is provided, for example, by a server device that provides the application program AP. The application program AP may also be provided, for example, by an operator that manufactures or sells this vehicle. Furthermore, the application program AP includes, for example, information that functions as a DNN (Deep Neural Network) 52; details about this DNN will be described later.

[0063] Some or all of the components of the aforementioned functional units can be implemented using hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), including the circuitry unit, or through the coordinated use of software and hardware. The application program (AP) can also be pre-stored in a storage device such as a flash memory (a storage device with a non-transitory storage medium) in the storage unit 50, or it can be stored in a removable storage medium such as an SD card, and installed in the storage unit of the terminal device SP by mounting the storage medium (a non-transitory storage medium) onto the terminal device SP.

[0064] The acquisition unit 10 acquires images captured by the camera Cam. The processing unit 20 analyzes the images and performs various processing steps based on the analysis results. The notification unit 30 issues warnings such as alerts and cautions to the driver based on the processing results of the processing unit 20. Details regarding these processing steps will be described later.

[0065] [flow chart]

[0066] Figure 9 This is a flowchart illustrating an example of a process executed by an application (terminal device SP). First, the terminal device SP estimates the amount of movement (step S100) and estimates the posture (step S110). The amount of movement refers to the amount of movement of a traffic participant or the movement of the vehicle itself. The posture refers to the posture of the vehicle itself.

[0067] Next, the terminal device SP detects traffic participants (step S120) and predicts the actions of the detected traffic participants (step S130). Next, the terminal device SP detects the risk to its own vehicle when going straight (step S140), the risk when turning left (step S150), and the risk when turning right (step S160). Next, the terminal device SP detects potential risks (step S170). Next, the terminal device SP performs a decision-making process (step S180).

[0068] In the aforementioned determination process, the terminal device SP displays information based on the determination result. Figure 10 This refers to an alarm level report or attention level report as shown. Alarm level reports or attention level reports can be reports related to the front, left or left-front, right or right-front, etc. This concludes the processing of the first routine in this flowchart. The details of these processes are explained below.

[0069] [Estimation of movement]

[0070] Figure 11 This diagram illustrates the processing related to the estimation of movement. The acquisition unit 10 of the terminal device SP acquires images of a time series (k…kn) captured by the camera Cam ("images captured at different times") (step S101). Next, the processing unit 20 of the terminal device SP derives the difference between the image captured at the current time and the image captured i times ago (step S102). Then, the processing unit 20 binarizes the derived difference (step S104) and calculates the sum of all pixel values ​​(step S106). The sum of all pixel values ​​refers to the sum of pixels where a difference is determined to exist. For example, the processing unit 20 derives the difference in brightness values ​​for each pixel and binarizes the difference based on whether the derived difference is above or below a threshold. For example, pixels with a difference above the threshold are represented as black, and pixels with a difference below the threshold are represented as white. Differences in RGB components can also be used. In this case, binarization can also be performed for each pixel by comparing the value obtained by statistically processing the differences of each component with a threshold.

[0071] For example, when the vehicle is stationary (parked), the difference between the images is small (due to high similarity), so the whiteness increases when the difference is binarized. Conversely, when the vehicle is moving, the difference is large (due to low similarity), so the whiteness decreases when the difference is binarized.

[0072] Next, the processing unit 20 determines whether the sum of all pixel values ​​is below a threshold (step S108). If the sum is below the threshold (high whiteness), the processing unit 20 determines that the vehicle has stopped (F_RUN = 0) (step S110). If the sum is not below the threshold (low whiteness), the processing unit 20 determines that the vehicle is in motion or that the movement of surrounding traffic participants is high (F_RUN = 1) (step S110).

[0073] As described above, the terminal device SP can easily determine the status of the vehicle and the status of traffic participants based on images.

[0074] [Presumption of Posture]

[0075] Figure 12 This diagram illustrates the processing related to posture determination. The processing unit 20 extracts a rectangular region AR1 from the image at time k in the time series image. This rectangular region AR1 is centered on the center of the image and is centered on a reference position (e.g., center of gravity) of the image. The processing unit 20 extracts a region from the image at time ki in the time series image that is the same as or approximately the region captured in the image at time k, and enlarges the image of the extracted region in a manner close to that of the image at time k. This processing is an example of a process that "makes the size of the extracted image obtained by extracting a specific region from the image captured at the first time close to the size of the image captured at the second time."

[0076] Regarding the image magnification described above, the magnification amount can be determined as follows. For example, when the vehicle's speed is obtained by a positioning device that measures the position of the terminal device SP, the processing unit 20 can magnify the image by the amount of movement at sampling time i, based on the vehicle's speed. For example, the processing unit 20 can magnify the image by referring to information that establishes a correspondence between speed and magnification. For example, the processing unit 20 can also magnify the image by a magnification corresponding to the speed when the speed is obtained as described above, and magnify the image by a predetermined value when the speed is not obtained.

[0077] Alternatively, instead of the magnification process described above, image processing can be used to adjust the image range at time k to match the image range at time k-1. For example, the processing unit 20 can also obtain an area including a specified landscape from the images at time k and k-1, and reduce the size of the image obtained from the image at time k that includes the aforementioned area to a size close to that of the image obtained from the image at time k-1 that includes the aforementioned area.

[0078] Next, the processing unit 20 acquires multiple images obtained by rotating the magnified image to the left and right by a predetermined angle from a reference position. For example, it acquires images obtained by rotating the image counterclockwise and clockwise around the reference position. The images are assigned parameters ID according to the degree of rotation. For example, it acquires images obtained by rotating to the left with parameters ID +1…+n and images obtained by rotating to the right with parameters ID -1…--n.

[0079] The processing unit 20 sets a rectangular region AR2 centered on the center of the image to which the parameter ID is assigned. Rectangular region AR2 is, for example, the region corresponding to rectangular region AR1. Rectangular region AR2 is a region whose reference position is set centered on the center of the image, regardless of image rotation.

[0080] Next, the processing unit 20 compares the image of the rectangular region AR1 at time k (an example of the "second image") with the image of the rectangular region AR1 assigned parameter ID (an example of the "first image") to derive the difference. Then, the processing unit 20 binarizes the derived differences and sums the total pixel values ​​for each image. Furthermore, the processing unit 20 selects the image with the smallest total pixel sum (the sum of pixels determined to have a difference) as its parameter ID.

[0081] At times k and ki, if the vehicle is traveling straight, choose... Figure 12 The parameter ID is "0". At times ki and k, if the vehicle is about to turn left, for example, by selecting... Figure 12 The parameter ID "+n" is used when the vehicle is about to turn right at times ki and k, for example, when selecting... Figure 12 The parameter ID is "-n".

[0082] Figure 13 This is a flowchart illustrating an example of the processing flow performed by the processing unit 20. First, the processing unit 20 determines whether the parameter ID is greater than the threshold + a (step S110). If the parameter ID is greater than the threshold + a, the image is rotating to the left, therefore, the processing unit 20 determines that the vehicle intends to turn left (step S112). The determination is "F_LTURN = 1" or "F_RTURN = 0".

[0083] If parameter ID is not greater than threshold + a, the processing unit 20 determines whether parameter ID is less than threshold - a (step S114). If parameter ID is less than threshold - a, the image is rotating to the right, therefore, the processing unit 20 determines that the vehicle intends to turn right (step S116). The determination is "F_LTURN = 0" or "F_RTURN = 1".

[0084] If parameter ID is not less than the threshold -a, the processing unit 20 determines that the vehicle intends to proceed straight (step S118). The determination is "F_LTURN=0" and "F_RTURN=0". Thus, step 1 of this flowchart ends.

[0085] As described above, the terminal device SP can accurately determine whether the vehicle intends to turn right, left, or go straight based on images captured at different times.

[0086] Alternatively, the processing unit 20 may replace the above-described processing (or further) determine the vehicle's intention to turn left, right, or go straight based on the detection results of an acceleration sensor mounted on the terminal device SP that can detect the tilt of the terminal device SP. For example, the processing unit 20 may also determine that the vehicle intends to turn in the direction in which the terminal device SP is tilted.

[0087] In the above example, we illustrated the case of preparing multiple images obtained by rotating each by a predetermined angle. However, we can also compare an image obtained by rotating to the left and an image obtained by rotating to the right with the image taken at time k, and determine that the vehicle is about to travel in the direction corresponding to the image with the smallest difference. In the above example, we illustrated the case of comparing the central portions of the images with each other, but we can also compare other areas instead of the central portions. Furthermore, the processing of adjusting the size of the image at time ki or time k in the time series images when the interval between the captured images is less than a predetermined value (e.g., the processing of enlarging the image at time k-1) can be omitted. If we can determine the vehicle's intention to turn right, turn left, or go straight with good accuracy, we can also omit the above processing.

[0088] [Inspecting traffic participants]

[0089] Figure 14 This diagram illustrates the process of detecting traffic participants. The processing unit 20 uses a DNN52 to determine the category of traffic participants in the image. The DNN52 is a learned model that, when given an input image, outputs information representing the category of traffic participants in the image and information used to determine the location of those traffic participants. The categories of traffic participants include, for example, four-wheeled vehicles, two-wheeled vehicles, bicycles, and pedestrians. As information for determining location, for example, a correspondence is established between the traffic participant and a detection box, which is a rectangular region. This detection box is, for example, a region defined in a way that includes the traffic participant.

[0090] [Predicting and handling the actions of detected traffic participants]

[0091] Figure 15This diagram illustrates the process of predicting the actions of detected traffic participants. The processing unit 20 infers the direction of movement of traffic participants based on changes in their positions determined from images taken at different times, and predicts their actions based on the inference results.

[0092] When the processing unit 20 enlarges the detection box that corresponds to the traffic participant obtained from the image at time k compared to the detection box that corresponds to the traffic participant obtained from the image at time k-1, it determines that there is a tendency for the vehicle to approach the traffic participant. In this case, for example, the traffic participant is approaching the vehicle, or the vehicle is approaching the traffic participant.

[0093] When the processing unit 20 determines that the vehicle is moving away from the traffic participant if the detection box corresponding to the traffic participant obtained from the image at time k is reduced in size compared to the detection box corresponding to the traffic participant obtained from the image at time k-1, this indicates a tendency for the vehicle to move away from the traffic participant. In this case, for example, the traffic participant is moving away from the vehicle, or the vehicle is moving away from the traffic participant. The processing unit 20 estimates the amount of movement of the traffic participant based on the scaling or shrinking of the detection box.

[0094] If the position of the detection frame at time k moves to the right relative to the position of the detection frame at time k-1, the processing unit 20 determines that the traffic participant has moved to the right; if the position of the detection frame at time k moves to the left, the processing unit 20 determines that the traffic participant has moved to the left. The processing unit 20 estimates the amount of movement of the traffic participant based on the amount of movement of the detection frame.

[0095] As described above, the processing unit 20 estimates the direction and amount of movement of traffic participants and predicts their actions. Based on the detection results of the positions of traffic participants at time k-1 and time k, the processing unit 20 predicts the future positions of traffic participants.

[0096] [Handling of detected risks]

[0097] (Risks related to driving straight)

[0098] Figure 16This diagram illustrates an example of the definition of risk when traveling straight. The risk of traveling straight (straight-ahead risk) is set within a defined range in front of the vehicle. The straight-ahead risk is set within a first angular range, based on a line segment (d1) in the vehicle's direction of travel or the road's extension direction (reference direction). The second angular range, smaller than the first angle, based on the line segment in the direction of travel, is the forward risk range. Within the range between the first and second angular ranges, the range to the left is the left-hand risk range, and the range to the right is the right-hand risk range. The range within the aforementioned risk range at a first distance from the vehicle is the warning level range, and the range at a second distance from the vehicle is the caution level range.

[0099] Hereinafter, the region encompassing the first angle range and the first distance, with the reference direction (direction of travel) as the reference, is sometimes referred to as "region a"; the region encompassing the range between the first angle and the second angle (which may also be an angle different from the second angle mentioned above, as described below) in the left direction, and the range encompassing the first distance, is referred to as "region b"; and the region encompassing the range between the first angle and the second angle in the right direction, and the range encompassing the first distance, is referred to as "region c" (see below). Figure 17 , Figure 19 , Figure 21 ).

[0100] Sometimes, the region between the first and second angles, and between the first and second distances, with the reference direction (direction of travel) as the reference, is called "region d"; the region between the first and second angles, and between the first and second distances, in the left direction is called "region e"; and the region between the first and second angles, and between the first and second distances, in the right direction is called "region f" (see below). Figure 17 , Figure 19 , Figure 21 .

[0101] Figure 17 This diagram illustrates an example of the processing for determining the risk of the vehicle proceeding straight. Based on the results of the aforementioned processing, the processing unit 20 determines whether the vehicle should proceed straight (step S300). If the vehicle is not proceeding straight, the process described later continues. Figure 19 The processing of step S400.

[0102] When the vehicle is traveling straight, the processing unit 20 determines which area among areas a-e the risk to the vehicle exists in. For example, in the above-described processing, the processing unit 20 determines which area among areas a-e the location (object location) or future location (object location) of the target traffic participant (e.g., the traffic participant located closest to the vehicle) is located in. In the figure, "★" represents the object location.

[0103] For example, if the object is located in region a, assign the flag "F_FR=1", if the object is located in region b, assign the flag "F_LR=1", and if the object is located in region c, assign the flag "F_RR=1".

[0104] For example, if the object is located in region d, assign the flag "F_FC = 1"; if the object is located in region d, assign the flag "F_LC = 1"; if the object is located in region e, assign the flag "F_RC = 1". For all other cases, assign the flag 0.

[0105] (Regarding the risks of turning left)

[0106] Figure 18 This diagram illustrates an example of the definition of risk when turning left. The risk of turning left (left turn risk) centers on the front left of the vehicle. While the risk of going straight is defined with the vehicle's direction of travel as the reference direction, the risk of turning left is defined using a line segment (dL) obtained by rotating the line segment in the vehicle's direction of travel to the left by a specified angle as the reference direction. The specified angle is, for example, an angle within the range of 30 to 60 degrees, determined according to the shape of the road.

[0107] Figure 19 This diagram illustrates an example of processing to determine the risk of the vehicle making a left turn. Based on the results of the above processing, the processing unit 20 determines whether the vehicle is making a left turn or has the intention to do so (step S400). If the vehicle is neither making a left turn nor has the intention to do so, it proceeds to... Figure 21 The processing step S500. If the vehicle is making a left turn or has the intention to make a left turn, the processing unit 20 determines which area of ​​area a-area e the risk of the vehicle exists in.

[0108] like Figure 17 As explained, in the relationship between object location and region, assign the flags "F_FR=1", "F_LR=1", "F_RR=1", "F_FC=1", "F_LC=1", "F_RC=1" or other flags = 0.

[0109] (Risks associated with turning right)

[0110] Figure 20This diagram illustrates an example of the definition of risk when turning right. The risk of turning right (right turn risk) centers on the right front of the vehicle. While the risk of going straight is defined with the vehicle's direction of travel as the reference direction, the risk of turning right is defined using a line segment (dR) obtained by rotating the line segment in the vehicle's direction of travel to the right by a specified angle as the reference direction. The specified angle is, for example, an angle within the range of 30 to 60 degrees, depending on the shape of the road.

[0111] Figure 21 This diagram illustrates an example of processing to determine the risk of the vehicle making a right turn. Based on the results of the above processing, the processing unit 20 determines whether the vehicle is making a right turn or intends to do so (step S500). If the vehicle is neither making a right turn nor intends to turn right, the process proceeds to... Figure 23 The processing step S600. If the vehicle is making a right turn or has the intention to make a right turn, the processing unit 20 determines which area of ​​area a-area e the risk of the vehicle exists in.

[0112] like Figure 17 As explained, in the relationship between object location and region, assign the flags "F_FR=1", "F_LR=1", "F_RR=1", "F_FC=1", "F_LC=1", "F_RC=1" or other flags = 0.

[0113] As described above, the processing unit 20 sets a risk (warning zone) in front of the vehicle when the vehicle is traveling straight, sets a risk in front of the vehicle and to the left when the vehicle is turning left, and sets a risk in front of the vehicle and to the right when the vehicle is turning right, thereby setting an appropriate risk corresponding to the direction of travel of the vehicle.

[0114] [Estimation of potential risks]

[0115] Figure 22 This diagram illustrates an example of the process for presuming potential risks. The processing unit 20 detects the average inter-vehicle distance Lvv_ave and the lateral distance Lev based on the positions of other vehicles in the image. The average inter-vehicle distance Lvv_ave is, for example, the detected average inter-vehicle distance from the current vehicle to a specified number of vehicles (e.g., all vehicles) located on the side of other vehicles. Other vehicles refer to vehicles located at positions closest to the current vehicle. The inter-vehicle distance is, for example, the distance between the reference positions of vehicles forming a column in the direction of travel. When there is only one vehicle, the inter-vehicle distance is set to a value equal to a pre-set distance (e.g., 30m).

[0116] Lateral distance Lev is the lateral distance between this vehicle and other vehicles. Other vehicles, for example, are those located at the closest possible distance to this vehicle. The average inter-vehicle distance Lvv_ave and lateral distance Lev mentioned above are used in the following treatments related to the assumption of potential risks.

[0117] Figure 23 This is a flowchart illustrating an example of the processing flow performed by the processing unit 20. The processing unit 20 determines whether the average inter-vehicle distance Lvv_ave is equal to the threshold Lvv_LR (step S600). The average inter-vehicle distance Lvv_ave is, for example, the longitudinal distance between the first object about to enter the intersection (the connection between the first road and the second road) and the second object behind that first object, or the average inter-vehicle distance based on the inter-vehicle distances from the first object to the nth object located behind the second object. In step S600, it may also be determined whether other vehicles are stopped and whether the average inter-vehicle distance Lvv_ave is equal to the threshold Lvv_LR. To reduce the computational load in this process, the determination process for whether other vehicles are stopped is omitted; if the average inter-vehicle distance Lvv_ave is equal to the threshold Lvv_LR, it is determined instead that other vehicles are stopped. If the average inter-vehicle distance Lvv_ave is not equal to the threshold Lvv_LR, proceed to... Figure 24 The processing of step S700.

[0118] When the average inter-vehicle distance Lvv_ave is equal to the threshold Lvv_LR, the processing unit 20 determines whether the vehicle is turning right or intends to turn right (step S602). If the vehicle is turning right or intends to turn right, the processing unit 20 determines that there is a potential risk in the left direction (step S604). If there is a potential risk in the left direction, the flag "F_LLR = 1" is set.

[0119] If the vehicle is not turning right or does not intend to turn right, the processing unit 20 determines whether the vehicle is turning left or intends to turn left (step S606). If the vehicle is turning left or intends to turn left, the processing unit 20 determines that there is a potential risk in the right direction (step S608). If there is a potential risk in the right direction, the flag "F_RLR=1" is set.

[0120] When this vehicle is making a left turn, there is a possibility that a traffic participant may approach the vehicle from the left. However, since the driver can identify potential approaching traffic participants by observing the gaps between vehicles before making the left turn, this focus is on traffic participants approaching from the right, which are difficult to identify. While the driver is looking to the left when making a left turn, the possibility that the driver's driving may become unstable due to shifting their face to that side cannot be ruled out. Therefore, to suppress this possibility, the driver focuses on the right.

[0121] If the vehicle is not making a left turn or does not intend to make a left turn, the processing unit 20 determines whether the lateral distance |Lev| is less than the threshold Lev_LR (step S610). If the lateral distance |Lev| is less than the threshold Lev_LR, the processing unit 20 determines that there is a potential risk in front of the vehicle (step S612). If there is a potential risk in front, the flag "F_FLR = 1" is set.

[0122] If the lateral distance |Lev| is not less than the threshold Lev_LR, the processing unit 20 determines that there is no potential risk ahead (step S614). In this case, the flags “F_LLR=0”, “F_RLR=0”, and “F_FLR=0” are set.

[0123] [Handling related to alarms / attention]

[0124] Figure 24 This is a flowchart illustrating an example of a process related to alarm / attention handling (determination processing). The processing unit 20 determines whether the vehicle is in motion (step S700). If the vehicle is not in motion, step 1 of this flowchart ends.

[0125] (Handling of alert-related risks)

[0126] When the vehicle is in motion, the processing unit 20 determines whether there is a forward warning risk (step S702). If there is no forward warning risk, the processing unit 20 determines whether there is a left turn warning risk (step S704). If there is no left turn warning risk, the processing unit 20 determines whether there is a right turn warning risk (step S706).

[0127] In the event of a forward warning risk, a left-hand warning risk, or a right-hand warning risk, the notification unit 30 displays a warning corresponding to the risk level (step S708), and further provides an overtaking instruction in the first manner (step S710). As described above... Figure 17 , Figure 19 , Figure 21If the flags "F_FR=1 (signal indicating a forward warning risk)," "F_LR=1 (signal indicating a left-direction warning risk)," or "F_RR=1 (signal indicating a right-direction warning risk)" are set as described above, then steps S708 and S710 are executed. For details regarding these settings, please refer to the description below. Figure 25 , Figure 26 To explain.

[0128] (Handling related to risk awareness)

[0129] If there is no right-direction warning risk, the processing unit 20 determines whether there is a forward warning risk (step S712). If there is no forward warning risk, the processing unit 20 determines whether there is a left-direction warning risk (step S714). If there is no left-direction warning risk, the processing unit 20 determines whether there is a right-direction warning risk (step S716).

[0130] In cases where there is a risk of danger ahead, a risk of danger to the left, or a risk of danger to the right, the notification unit 30 displays the corresponding risk warning (step S718), and further provides a second-mode overtaking instruction (step S720). As described above... Figure 17 , Figure 19 , Figure 21 When the flags "F_FC = 1 (signal indicating forward danger)," "F_LC = 1 (signal indicating left-hand danger)," or "F_RC = 1 (signal indicating right-hand danger)" are set as described above, the processes in steps S718 and S720 are executed. For details regarding these, please refer to the description below. Figure 27 To explain.

[0131] (Handling of potential risks)

[0132] If there is no risk of danger to the right, the processing unit 20 determines whether there is a potential risk ahead (step S722). If there is no potential risk ahead, the processing unit 20 determines whether there is a potential risk to the left (step S724). If there is no potential risk to the left, the processing unit 20 determines whether there is a potential risk to the right (step S726).

[0133] If there is a potential risk ahead, a potential risk to the left, or a potential risk to the right, the notification unit 30 displays the information corresponding to the potential risk (step S728). As described above... Figure 17 , Figure 19 , Figure 21If the flags "F_FLR = 1 (signal indicating potential risk ahead)," "F_LLR = 1 (signal indicating potential risk to the left)," or "F_RLR = 1 (signal indicating potential risk to the right)" are set as described above, then the process in step S728 is executed. For details regarding these flags, please refer to the description below. Figure 28 This concludes the explanation of the first routine in this flowchart.

[0134] [Notifications corresponding to the alert risk (Level 1)]

[0135] Figure 25 This diagram illustrates a scenario and notification that generates a forward warning risk. When a forward warning risk exists, the notification unit 30 causes the display unit to show warning information indicating a forward warning risk, further executing the overtaking instruction in the first manner. This warning display is a pre-defined display showing a forward-direction arrow (e.g., a red arrow). The first method involves illuminating the overtaking indicator light PL at a high cycle.

[0136] Figure 26 This diagram illustrates the scenario and notification of a left-hand or right-hand traffic warning risk. When a left-hand traffic warning risk exists, the notification unit 30 causes the display unit to show warning information indicating a left-hand traffic warning risk, further executing the overtaking instruction of the first method. This warning display is a prescribed display, such as a left-hand arrow (e.g., a red arrow), as shown in the diagram.

[0137] If there is a warning risk in the right direction, the notification unit 30 causes the display unit to display a warning message indicating a warning risk in the right direction, and further executes the overtaking instruction of the first method. This warning display is a prescribed display, such as showing a right-direction arrow (e.g., a red arrow), as illustrated.

[0138] [Notification corresponding to the risk awareness (Level 2 alert)]

[0139] Figure 27 This diagram illustrates a scenario where a risk of danger arises and provides notification. When a risk of danger exists in the forward direction, the notification unit 30 causes the display unit to show a warning message indicating the presence of such a risk, further executing the overtaking instruction in a second manner. This warning display is a pre-defined display, such as showing a forward-direction arrow (e.g., a yellow arrow). The second method involves illuminating the overtaking indicator light PL at a low cycle.

[0140] If there is a risk of danger to the left, the notification unit 30 causes the display unit to display a warning message indicating a risk of danger to the left, and further executes the overtaking instruction in the second manner. This warning display is a pre-defined display, such as a left-hand arrow (e.g., a yellow arrow), as shown in the illustration.

[0141] If a risk of danger exists to the right, the notification unit 30 causes the display unit to show a warning message indicating a risk of danger to the right, and further executes the overtaking instruction in the second manner. This warning display is a pre-defined display showing a right-direction arrow (e.g., a yellow arrow), as illustrated. As described above, the color of the image displayed on the display unit when a risk of danger exists (an arrow or other warning image) is different from the color of the image displayed on the display unit when a warning risk exists.

[0142] [Notification corresponding to potential risks (Level 3 Alert)]

[0143] Figure 28 This diagram illustrates a scenario where a potential risk arises and provides notification. When a potential risk exists in the forward direction, the notification unit 30 causes the display unit to show potential risk information indicating the presence of a potential risk in the forward direction. This potential risk display is a predefined display, such as showing an arrow pointing forward (e.g., a yellow arrow), as illustrated.

[0144] If there is a potential risk to the left, the notification unit 30 causes the display unit to show a warning message indicating a potential risk to the left. This potential risk display is a pre-defined display, such as a forward-direction arrow (e.g., a yellow arrow), as shown in the illustration. Even when a potential risk exists, sometimes there are no approaching traffic participants, and therefore the overtaking instruction is not executed.

[0145] The above example illustrates a scenario where the camera and display are located on the terminal device SP. However, it is also possible for one or both of the camera and display to be located on other devices or mobile bodies instead of the terminal device SP. In this case, the terminal device SP communicates with other devices or mobile bodies to acquire images or display information.

[0146] In the above embodiments, it was explained that the system determines whether it is a first alert level or a second alert level, and provides different information for the first and second alert levels. However, alternatively (or based on this), the application may not determine whether it is a first alert level or a second alert level, and, in the case of a situation requiring alert (where the alert level is above a predetermined level), display information related to the alert on the display unit. Furthermore, in this case, the application may also execute an overtaking instruction. A situation requiring alert refers to a situation of first alert level, a situation of second alert level, or a situation where there is a potential risk.

[0147] According to the above-described implementation method, the application causes the computer of the terminal device equipped with a camera and installed on the mobile body to perform the following processing: acquiring an image; determining, based on the surrounding conditions of the mobile body obtained from the image, whether the level of alertness for the operator of the mobile body is a first level of alertness or a second level of alertness that is lower than the first level of alertness; and, based on the determination result, causing the display unit provided on the housing on which the camera is installed on the terminal device to display information corresponding to the level of alertness, thereby improving the convenience for the user.

[0148] According to the implementation described above, the application causes the computer of a terminal device equipped with a camera and installed on a mobile body to perform the following processing: acquiring images taken at different times; based on the images taken at different times, inferring whether the mobile body intends to turn right or left; and based on the inference result, causing the display unit of the terminal device provided on the housing on which the camera is provided to display an image of a report related to the presence of a traffic participant, thereby enabling a more accurate and better estimation of the mobile body's intention to move.

[0149] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. An information providing method, wherein the information providing method causes a computer of a terminal device mounted with a camera and installed on a moving body to execute processing of: acquiring first images of central portions of a plurality of images each obtained by rotating an image captured at a first time by a predetermined angle in each of a left direction and a right direction from a reference position; comparing the acquired first images of the central portions with second images of central portions of images captured at a second time different from the first time, respectively; in a case where a rotation angle of a first image in which a difference between the first image and the second image becomes the smallest is equal to or more than a threshold value, inferring that there is an intention of a right turn or a left turn of the moving body; and based on a result of the inference, causing a display portion provided on a housing in which the camera is provided to display an image of a report related to presence of a traffic participant.

2. The information providing method according to claim 1, wherein the information providing method causes the computer to further execute processing of: inferring whether a state in which the moving body is straight or a state in which the moving body is in a right turn or a left turn.

3. The information providing method according to claim 1, wherein the information providing method causes the computer to further execute processing of: causing a size of a cropped image obtained by cropping a specific region from the image captured at the first time to be close to a size of the image captured at the second time, rotating the cropped image by the predetermined angle, and comparing the image captured at the second time with the image obtained by rotating by the predetermined angle.

4. The information providing method according to any one of claims 1 to 3, wherein the information providing method causes the computer to further execute processing of: in a case where it is inferred that there is an intention of a right turn or a left turn of the moving body and it is inferred that a traffic participant is present in or near a path or a path vicinity through which the moving body has made a right turn or a left turn, causing the display portion to display an image of a report related to approach of the traffic participant.

5. A storage medium storing an application program, wherein the application program causes a computer of a terminal device to execute processing of: acquiring first images of central portions of a plurality of images each obtained by rotating an image captured at a first time by a predetermined angle in each of a left direction and a right direction from a reference position; comparing the acquired first images of the central portions with second images of central portions of images captured at a second time different from the first time, respectively; in a case where a rotation angle of a first image in which a difference between the first image and the second image becomes the smallest is equal to or more than a threshold value, inferring that there is an intention of a right turn or a left turn of the moving body; and based on a result of the inference, causing a display portion to display an image of a report related to presence of a traffic participant.

6. A terminal device, wherein the terminal device includes: a camera provided to a housing; a display portion provided to the housing; a presumption unit that, in a case where a rotation angle of a first image in which a central portion of an image captured at a first time is rotated by a predetermined angle in each of a left direction and a right direction from a reference position is equal to or greater than a threshold value, and a second image is a central portion of an image captured at a second time different from the first time, presumes that the mobile body has an intention to right turn or left turn, if a difference between the first image and the second image is the smallest; and a display control unit that causes a display unit provided on the housing in which the camera is provided to display an image of a report related to the presence of a traffic participant, based on a result of the presumption.

Images