Prediction device, prediction system, prediction method for prediction device, and program

Abstract

To provide a prediction device, a prediction system, a prediction method for the prediction device, and a program for predicting positions of target objects to be detected that enter a blind spot area.SOLUTION: A prediction device includes a detection target detection unit, a blind spot area detection unit, a prediction unit, and an output information generation unit. The detection target detection unit identifies a moving speed of a detection target detected from a video captured in front of the moving object. The blind spot area detection unit detects a blind spot area from the video captured in front of the moving object. The prediction unit predicts a plurality of predicted positions of the detection target in the blind spot area using the moving speed of the detection target that entered the blind spot and sets the probability of the detection target being present at each predicted position. The output information generation unit generates the output information according to the predicted result of the prediction unit and outputs the generated output information.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present disclosure relates to a prediction device, a prediction system, a prediction method, and a program. [Background technology]

[0002] In the fields of driving assistance and autonomous driving, it is necessary to correctly detect objects such as pedestrians. The surroundings monitoring device described in Patent Document 1 detects objects by analyzing images captured by an on-board camera using a classifier. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2023 / 286540 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology disclosed in Patent Document 1, when a detection target such as a pedestrian enters a blind spot area, the detection target cannot be detected.

[0005] An object of the present disclosure is to provide a prediction device, a prediction system, a prediction method, and a program for predicting the position of a detected object that has entered a blind spot area. [Means for solving the problem]

[0006] A prediction device of one aspect of the present disclosure includes a detection object detection means for identifying the movement speed of a detection object detected from an image captured in front of a moving body, a blind spot area detection means for detecting blind spot areas from the image, a prediction means for predicting multiple predicted positions of the detection object in the blind spot area using the movement speed of the detection object that has entered the blind spot area and setting the presence probability of the detection object being present at each predicted position, and an output information generation means for generating output information according to the prediction result of the prediction means and outputting the generated output information.

[0007] In a screen control method according to one aspect of the present disclosure, the movement speed of a detected object is identified from an image captured in front of a moving body, a blind spot area is detected from the image, and multiple predicted positions of the detected object in the blind spot area are predicted using the movement speed of the detected object that has entered the blind spot area, a presence probability that the detected object will be present at each predicted position is set, output information according to the predicted position and presence probability is generated, and the generated output information is output.

[0008] A program of one embodiment of the present disclosure causes a computer to perform the following processes: determining the movement speed of a detected object from an image captured in front of a moving body; detecting a blind spot area from the image; predicting multiple predicted positions of the detected object in the blind spot area using the movement speed of the detected object that has entered the blind spot area and setting the presence probability of the detected object being present at each predicted position; generating output information according to the predicted position and presence probability; and outputting the generated output information. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a prediction device, a prediction system, a prediction method, and a program for predicting the position of a detected object that has entered a blind spot area. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a conceptual diagram for explaining a prediction device according to the present disclosure. [Diagram 2] FIG. 1 is a conceptual diagram for explaining a prediction device according to the present disclosure. [Diagram 3] FIG. 1 is a conceptual diagram for explaining a prediction device according to the present disclosure. [Figure 4] FIG. 1 is a block diagram showing an example of a configuration of a prediction system according to the present disclosure. [Diagram 5] 10 is a conceptual diagram for explaining a procedure for predicting a predicted position of a detection object in a blind spot area. FIG. [Figure 6]FIG. 13 is a conceptual diagram for explaining a process when a predicted position goes outside a blind spot area. [Figure 7] FIG. 11 is a conceptual diagram for explaining a procedure for re-predicting a predicted position. [Figure 8] FIG. 11 is a diagram illustrating an example of output information. [Figure 9] FIG. 11 is a diagram illustrating an example of output information. [Figure 10] 10 is a flowchart illustrating an example of an operation of the prediction device according to the present disclosure. [Figure 11] 11 is a flowchart illustrating an example of an operation of a prediction process of a prediction device according to the present disclosure. [Figure 12] 1 is a block diagram showing an example of a configuration of a prediction system according to the present disclosure. [Figure 13] 10 is a flowchart illustrating an example of an operation of the prediction device according to the present disclosure. [Figure 14] 11 is a flowchart illustrating an example of an operation of a prediction process of a prediction device according to the present disclosure. [Figure 15] 1 is a block diagram showing an example of a configuration of a prediction device according to the present disclosure. [Figure 16] 10 is a flowchart illustrating an example of an operation of the prediction device according to the present disclosure. [Figure 17] 1 is a block diagram showing an example of a configuration of a prediction device according to the present disclosure. [Figure 18] 10 is a flowchart illustrating an example of an operation of the prediction device according to the present disclosure. [Figure 19] FIG. 2 is a block diagram showing an example of a hardware configuration for executing control and processing according to each embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. The embodiments described below are limited in a manner that is technically preferable for implementing the present disclosure, but the scope of the invention is not limited to the following. In all the drawings used to explain the following embodiments, the same reference numerals are used for similar parts unless otherwise specified. In the following embodiments, repeated explanations of similar configurations and operations may be omitted.

[0012] [First embodiment]

[0013] (overview) The prediction device 10 of this embodiment predicts the position of a detection object that has entered a blind spot area, and outputs output information using the predicted position to a display device provided in a moving body MB. Figures 1 to 3 are conceptual diagrams for explaining a prediction system 1 according to the present disclosure. First, an overview of the prediction system 1 will be described with reference to Figures 1 to 3.

[0014] For example, the prediction device 10 is mounted on a moving body MB. In this embodiment, the moving body MB is a vehicle such as an automobile, a bus, a motorcycle, or a construction machine. The moving body MB is equipped with a camera 60. The camera 60 captures an area including a direction V1, which is the traveling direction of the moving body MB. The prediction device 10 detects a detection target T from the captured image P by analyzing the captured image P by the camera 60 using a known method. The detection target T is an object that may be located on the path of the moving body MB. For example, the detection target T is an object such as a pedestrian, an animal, or a vehicle other than the moving body MB. In the example shown in FIG. 1, if the traveling direction of the moving body MB is the direction V1, the area E1 behind the obstacle O is a blind area for the moving body MB. When the detection target T travels in the direction V2, it enters the area E1, which is the blind area of ​​the moving body MB.

[0015] Fig. 2 is a conceptual diagram showing a situation t seconds after the situation in Fig. 1. In the example shown in Fig. 2, the detection object T enters an area E1 that is a blind spot area of ​​the moving object MB. Therefore, the detection object T is not detected from the captured image P captured by the camera 60.

[0016] FIG. 3 is a conceptual diagram showing a situation in which the moving object MB advances in the direction V1 for t' seconds in the situation of FIG. 2. If the driver of the moving object MB does not notice the detection object T, the detection object T suddenly appears from the region E1, which is the blind spot region of the moving object MB. In such a situation, there is a possibility that the moving object MB and the detection object T collide. In addition, there is a possibility that the moving object MB must brake suddenly to avoid the collision. In order to prevent these from happening, the prediction device 10 of this embodiment predicts the position of the detection object that has entered the blind spot region. Furthermore, the prediction device 10 presents output information using the predicted position to the driver. For example, the prediction device 10 of this embodiment displays the output information on a display device (not shown in FIGS. 1 to 3) mounted on the vehicle.

[0017] (composition) Fig. 4 is a block diagram showing an example of a configuration of a prediction system according to the present disclosure. As shown in Fig. 4, the prediction system 1 includes a prediction device 10, a camera 60, a distance measurement sensor 70, and a display device 80. The prediction device 10 is connected to each of the camera 60, the distance measurement sensor 70, and the display device 80 so as to be able to communicate with each other.

[0018] The camera 60 is an on-board camera mounted on the moving body MB. The camera 60 captures an image of the area ahead of the moving body MB. The imaging range of the camera 60 includes the area diagonally ahead of the moving body MB. The camera 60 may be a monocular camera or a stereo camera. The image captured by the camera 60 is a moving image composed of images captured at a predetermined frame rate. The image captured by the camera 60 is transmitted to the prediction device 10.

[0019] The distance measurement sensor 70 is a sensor for measuring the distance between the moving body MB and a detection target. The distance measurement sensor 70 acquires sensor data used to calculate the distance to an object located in front of the moving body MB. For example, the distance measurement sensor 70 is realized by an ultrasonic sensor, an infrared sensor, a LiDAR (Light Detection And Ranging), a millimeter wave radar, or the like. When the distance can be measured from an image captured by the camera 60, the prediction system 1 does not need to include the distance measurement sensor 70.

[0020] The display device 80 has a display unit that displays the output information generated by the prediction device 10. For example, the display device 80 is realized by a display device such as a display. The display device 80 may be realized by a display unit of a car navigation device mounted on the mobile object MB. When the display device 80 is realized by a display unit of the car navigation device, the prediction device 10 may be realized by an application installed in the car navigation device.

[0021] [Prediction device] Next, the prediction device 10 will be described with reference to FIG. 4. For example, the prediction device 10 is an in-vehicle device. The prediction device 10 may be realized by an application installed in an in-vehicle device such as a car navigation device mounted on the mobile body MB. Alternatively, the prediction device 10 may be realized by an application installed in a mobile terminal carried by the driver of the mobile body MB. The prediction device 10 includes an acquisition unit 11, a detection object detection unit 12, a blind spot area detection unit 13, a prediction unit 14, and an output information generation unit 15.

[0022] The acquisition unit 11 acquires an image and sensor data captured in front of the moving object MB. The acquisition unit 11 acquires an image captured by the camera 60. The acquisition unit 11 acquires sensor data obtained by the distance measurement sensor .

[0023] The detection target detection unit 12 detects the detection target T from the video acquired by the acquisition unit 11. The detection target detection unit 12 identifies the moving speed and moving direction of the detected detection target T. As described above, the detection target T is an object that may be located on the path of the moving object MB. For example, the detection target T is a pedestrian, an animal, a vehicle other than the moving object MB, etc.

[0024] The detection target detection unit 12 detects the detection target T from a frame image included in the video. For example, the detection target detection unit 12 may detect the detection target T by performing pattern matching using a discriminator for identifying the detection target T for the frame images included in the video. In another example, the detection target detection unit 12 may create a background image using a plurality of consecutive frame images, and detect the difference between the background image and the current frame image as the detection target T. The detection target detection unit 12 tracks the detection target T detected in one frame image constituting the video in other frame images. For example, the detection target detection unit 12 tracks the detection target T using a known object tracking technique. That is, the detection target detection unit 12 detects the detection target T for each of a plurality of consecutive frame images. The method of detecting the detection target T by the detection target detection unit 12 is not limited to the example given here.

[0025] The detection target detection unit 12 further specifies the moving speed and moving direction of the detected detection target T. For example, the detection target detection unit 12 specifies the moving speed and moving direction of the detection target T using an inter-frame difference of the detected detection target T. In another example, the detection target detection unit 12 specifies the moving speed and moving direction of the detection target T using sensor data acquired by the acquisition unit 11. The method of specifying the moving speed and moving direction of the detection target T by the detection target detection unit 12 is not limited to the example given here.

[0026] Hereinafter, an example of a procedure in which the detection object detection unit 12 identifies the moving speed and moving direction using the sensor data acquired by the acquisition unit 11 will be described. First, the detection object detection unit 12 measures the distance from the moving body MB to the detected detection object T using the sensor data acquired by the acquisition unit 11. Next, the detection object detection unit 12 compares the current frame with the previous frame. It calculates the moving distance of the detection object T in one frame. The moving speed of the detection object T is calculated by dividing the moving distance of the detection object T by the number of seconds for one frame. In addition, the moving direction of the detection object T is calculated by, for example, optical flow.

[0027] The blind spot area detection unit 13 detects a blind spot area from the video acquired by the acquisition unit 11. For example, the blind spot area detection unit 13 detects a blind spot area in the following procedure. First, the blind spot area detection unit 13 identifies a frame in which the tracking of the detection target T detected by the detection target detection unit 12 is lost. Next, the blind spot area detection unit 13 refers to the frame image one frame before the frame in which the tracking is lost and the sensor data corresponding to the frame image. The blind spot area detection unit 13 identifies the area of ​​an object that is an obstacle to the detection target T. Specifically, the blind spot area detection unit 13 identifies the area of ​​an object located in front of the detection target T using the sensor data. The blind spot area detection unit 13 detects the area of ​​the identified object as a blind spot area. The method of detecting a blind spot area by the blind spot area detection unit 13 is not limited to the example given here.

[0028] The prediction unit 14 predicts a plurality of predicted positions of the detection object T in the blind spot area. The prediction unit 14 further sets the presence probability of the detection object T at each predicted position. The prediction unit 14 predicts the predicted position of the detection object T in the blind spot area in the following procedure. First, the prediction unit 14 acquires the movement speed of the detection object T when it enters the blind spot area (hereinafter, also referred to as the entry speed). The movement speed of the detection object T when it enters the blind spot area is, in other words, the movement speed of the detection object T immediately before the tracking of the detection object T is lost. Next, the prediction unit 14 sets a plurality of predicted moving speeds of the detection object T in the blind spot area based on the moving speed when it enters. The prediction unit 14 calculates a plurality of predicted positions according to the predicted moving speeds. The prediction unit 14 predicts a plurality of predicted positions of the detection object T using the elapsed time since the detection object entered the blind spot, the predicted moving speed, and the size of the blind spot area.

[0029] With reference to FIG. 5, a specific example in which the prediction unit 14 predicts the position of the detection object T in the blind spot area will be described. In the example of FIG. 5, the prediction unit 14 sets the predicted travel speed in the blind spot area to four values, namely, −40%, −20%, ±0%, and +20%, with the entry speed of the detection object T as a reference (100%). The prediction unit 14 calculates four predicted positions (predicted positions PT1 to PT4) of the detection object T according to the predicted travel speed. Specifically, the prediction unit 14 predicts a plurality of predicted positions of the detection object T using the elapsed time since the detection object entered the blind spot, the predicted travel speed, and the size of the blind spot area. Note that the above-mentioned value of the predicted travel speed is an example, and the predicted travel speed may be set to another value. In addition, the number of predicted travel speeds and predicted positions PT is not limited to four, and may be other values.

[0030] The prediction unit 14 further calculates the existence probability that the detection target T exists at each of the predicted positions PT1 to PT4. The existence probability is set to be higher as the absolute value of the difference between the entry speed and the predicted traveling speed is smaller. The prediction unit 14 sets the existence probability of the predicted position assumed to be traveling at the same speed as the entry speed to the highest. The prediction unit 14 sets the existence probability to be lower as the absolute value of the difference between the entry speed and the predicted traveling speed is larger. In the example of FIG. 5, the prediction unit 14 sets the existence probability of the predicted position PT3 to 50%, the existence probabilities of the predicted positions PT2 and PT4 to 20%, and the existence probability of the predicted position PT1 to 10%. Note that the above-mentioned existence probability values ​​are merely examples, and the existence probability values ​​that are set may be other values.

[0031] Next, referring to FIG. 6, a process when the predicted position goes outside the blind spot area will be described. FIG. 6 is a diagram showing a predicted position of the detection object T in an image n frames later (n is a natural number) from the frame image shown in FIG. 5. The prediction unit 14 updates the predicted positions PT1 to PT4 every time a new frame image is acquired. As shown in FIG. 6, the predicted position goes outside the blind spot area n frames later (n is a natural number) after the start of updating the predicted position. If the detection object T is not detected in the image n frames later, the prediction unit 14 changes the existence probability of the predicted position that has gone outside the blind spot area to 0%. In the example shown in FIG. 6, the predicted position PT4 goes outside the blind spot area. Furthermore, in the example shown in FIG. 6, since the detection object T is not detected, the prediction unit 14 changes the existence probability of the predicted position PT4 to 0%. The prediction unit 14 does not update the prediction process for the predicted positions with an existence probability of 0%. The prediction unit 14 continues the update process for the predicted positions PT1 to PT4 until the existence probabilities of all predicted positions become 0%. When the detection target T is detected by the detection target detection section 12, the prediction section 14 ends the prediction process.

[0032] Next, a procedure for re-predicting a predicted position will be described with reference to Fig. 7. If the detection target T is not detected by the detection target detection unit 12 even when the existence probabilities of all predicted positions become 0%, the prediction unit 14 re-predicts the predicted position. For example, as shown in Fig. 7, the prediction unit 14 sets a slower predicted progress speed. The prediction unit 14 predicts predicted positions PT5 to PT8 corresponding to the predicted progress speed. The prediction unit 14 further resets the existence probabilities of the predicted positions PT5 to PT8.

[0033] The output information generating unit 15 generates output information according to the prediction result of the predicting unit 14. The output information generating unit 15 outputs the generated output information.

[0034] FIG. 8 is an example of output information generated by the output information generating unit 15. The output information shown in FIG. 8 includes at least an image P1 obtained by cutting out an area including a blind spot area, and an icon IC1 displayed by superimposing it on the image P1. The icon IC1 is an icon displayed at a predicted position of the detection object T. In the example shown in FIG. 8, the icon IC1 is a human-shaped icon, but other icons may be used. The icon IC1 may be changed according to the type of the detection object T. For example, if the detection object T is an animal, an animal icon may be used. The icon IC1 may be displayed in a display method according to the high probability of existence. For example, as shown in FIG. 8, the icon IC1 may be displayed in a gradation according to the high probability of existence. In the example of FIG. 8, the darker the display color, the higher the probability of existence. For example, the icon IC1 may be color-coded according to the probability of existence. The output information shown in FIG. 8 may also include display information W1 indicating the probability of existence at each predicted position. The output information shown in FIG. 8 may also include display information W2 including content that calls attention to the driver of the moving object MB. For example, in addition to the display information W1, a sound including a message calling the driver of the moving object MB to be careful may be output. For example, a sound calling the driver's attention to the detection object T may be output.

[0035] Fig. 9 is another example of output information generated by the output information generating unit 15. The example shown in Fig. 9 assumes output information in the case where the display device 80 is realized by a display unit of a car navigation device. The output information generated by the output information generating unit 15 may be displayed superimposed on a map M of route guidance provided by the car navigation device, as shown in Fig. 9. For example, when there is a possibility that a detection target T will appear from a blind spot area, the output information generating unit 15 displays an icon IC3 at the point where the possibility has arisen. The output information shown in Fig. 9 may also include display information W3 that calls the attention of the driver of the moving body MB.

[0036] (operation) Next, an example of the operation of the prediction device 10 in this embodiment will be described with reference to Fig. 10 to Fig. 11. For example, when the prediction device 10 detects that the moving object MB has started to move, the prediction device 10 starts the process shown in Fig. 10.

[0037] First, the acquisition unit 11 acquires an image of the area in front of the moving object MB from the camera 60. Furthermore, the acquisition unit 11 acquires sensor data of the area in front of the moving object MB (step S11). The area in front of the moving object MB includes the area diagonally in front of the moving object MB.

[0038] Next, the detection target detection unit 12 detects the detection target T from the video acquired by the acquisition unit 11 (step S12). The detection target detection unit 12 detects the detection target T from a frame image included in the video. The detection target detection unit 12 tracks the detected detection target T.

[0039] Next, the detected object detection unit 12 identifies the moving speed and moving direction of the detected detected object (step S13). If the prediction system 1 does not include the distance measurement sensor 70, the detected object detection unit 12 performs the processes of steps S12 and S13 using the inter-frame difference of the acquired video. If the prediction system 1 includes the distance measurement sensor 70, the detected object detection unit 12 performs the processes of steps S12 and S13 using the inter-frame difference of the acquired video and the acquired sensor data.

[0040] Next, the blind spot detection unit 13 detects a blind spot from the video acquired by the acquisition unit 11 (step S14). The blind spot detection unit 13 identifies an object in front of the detection target T by using the frame image immediately before when tracking of the detection target T is lost. The blind spot detection unit 13 detects an area that becomes a blind spot due to the identified object as a blind spot.

[0041] Next, the prediction unit 14 performs a prediction process (step S15). Fig. 11 is a diagram showing an example of a series of operations in the prediction process. The operations in the prediction process will be described with reference to Fig. 11.

[0042] First, the prediction unit 14 acquires frame images from the video acquired by the acquisition unit 11 (step S151).

[0043] Next, the prediction unit 14 predicts multiple positions of the detection object T using the entry speed of the detection object T into the blind spot area (step S152). The entry speed of the detection object T is the moving speed of the detection object T just before the tracking of the detection object T is lost. In other words, the entry speed of the detection object T is specified in step S13. The prediction unit 14 sets multiple predicted moving speeds of the detection object T in the blind spot area based on the moving speed when the detection object T enters the blind spot area. The prediction unit 14 calculates multiple predicted positions according to the predicted moving speeds.

[0044] Next, the prediction unit 14 sets a presence probability that the detection object T exists at each predicted position (step S153). The presence probability is set to be higher as the absolute value of the difference between the entry speed and the predicted travel speed is smaller.

[0045] Next, the prediction unit 14 determines whether or not there is a predicted position located outside the blind spot area among the multiple predicted predicted positions (step S154). If there is no predicted position located outside the blind spot area (No in step S154), the prediction unit 14 proceeds to the process of step S156. If there is a predicted position located outside the blind spot area (Yes in step S154), the prediction unit 14 sets the existence probability of a predicted position outside the blind spot area to 0% (step S155). In addition, the prediction unit 14 determines whether or not there is a predicted position located within the blind spot area (step S156). If there is a predicted position located within the blind spot area (Yes in step S156), the series of prediction processes ends. If there is no predicted position located within the blind spot area (No in step S156), the prediction unit 14 returns to the process of step S151 and resets the predicted position.

[0046] When the series of prediction processes described above is completed, the prediction device 10 proceeds to the process of step S16 in Fig. 10. The output information generation unit 15 generates output information according to the prediction result of step S15. The output information generation unit 15 outputs the generated output information to the display device 80 (step S16). For example, the output information generation unit 15 generates output information as shown in Fig. 8 or Fig. 9.

[0047] The prediction device 10 repeats the processes of steps S15 to S16 until the detection target detection unit 12 detects that the detection target T has appeared from the blind spot area (No in step S17).

[0048] When the detection target T appears from the blind spot area (Yes in step S17), the prediction device 10 judges whether or not a predetermined end condition is satisfied (step S18). When the predetermined end condition is not satisfied (No in step S18), the processing of steps S11 to S17 is repeated. For example, the predetermined end condition includes the power supply of the prediction device 10 being turned off, a command to stop processing being issued from the driver of the moving body MB, etc. When the prediction device 10 satisfies the predetermined end condition (Yes in step S18), it ends the series of processes described above.

[0049] As described above, the prediction device of this embodiment includes an acquisition unit, a detection target detection unit, a blind spot area detection unit, a prediction unit, and an output information generation unit. The acquisition unit acquires an image and sensor data captured in front of the moving object. The detection target detection unit identifies the moving speed and moving direction of the detection target T detected from the image acquired by the acquisition unit. The blind spot area detection unit detects a blind spot area from the image acquired by the acquisition unit. The prediction unit predicts a plurality of positions of the detection target in the blind spot area using the moving speed of the detection target that has entered the blind spot area. The prediction unit sets the presence probability of the detection target at each predicted position. The output information generation unit generates output information according to the prediction result of the prediction unit 14, and outputs the generated output information. With the above configuration, the prediction device of this embodiment can obtain the predicted position of the detection target even when the detection target enters the blind spot area and the detection target is not detected from the acquired image. In other words, the prediction device of this embodiment can predict the position of the detection target that has entered the blind spot area.

[0050] The prediction unit included in the prediction device of this embodiment sets multiple predicted moving speeds of the detection object in the blind spot area based on the moving speed when the detection object enters the blind spot area. The prediction unit predicts multiple predicted positions of the detection object in the blind spot area according to the predicted moving speeds. Furthermore, the prediction unit repeats the above-mentioned prediction process until the detection object emerges from the blind spot area. According to the prediction device of this embodiment, the driver of the moving body can predict the position of the detection object even if the detection object decelerates or accelerates in the blind spot area.

[0051] The prediction unit included in the prediction device of this embodiment sets the presence probability of the detection object at each predicted position. The presence probability is set so that it is higher as the absolute value of the difference between the entry speed and the predicted travel speed is smaller. When the output information generated by the output information generation unit includes information indicating the presence probability, the driver of the moving body can grasp the probability of the detection object being present at the predicted position. This allows the driver to change the control of the moving body according to the presence probability. For example, when passing near a blind spot area, if there is no predicted position with a high presence probability near the exit of the blind spot area, the driver only decelerates the moving body. On the other hand, when passing near a blind spot area, if there is a predicted position with a high presence probability near the exit of the blind spot area, the driver stops the moving body temporarily. In this way, by including the presence probability of the detection object in the output information, the driver can change the control of the moving body according to the presence probability.

[0052] The output information generating unit included in the prediction device of this embodiment generates output information according to the prediction result of the prediction unit, and outputs it to a display device included in the moving body. As an example, the output information includes an image of an area including a blind spot area, and an icon displayed at the predicted position superimposed on the cut-out image. The icon is displayed in a display method according to the degree of the existence probability. For example, the icon is displayed in a gradation according to the degree of the existence probability. In another example, the output information includes a map of route guidance and an icon displayed superimposed on the map. The icon is displayed at a point where the detected object may appear from the blind spot area. The output information may also include display information that calls the driver of the moving body to attention. By outputting these output information to the display device, the driver can intuitively grasp the point and timing at which he or she should be careful of the jumping out of the moving object.

[0053] The prediction unit included in the prediction device of this embodiment sets the presence probability corresponding to a predicted position located outside a blind spot area to 0. The prediction unit does not update the prediction process for a predicted position whose presence probability is 0. This allows the prediction device of this embodiment to reduce calculation costs because it does not need to update the prediction process for a predicted position located outside a blind spot area.

[0054] [Second embodiment]

[0055] (composition) Next, the prediction system 2 and the prediction device 20 according to this embodiment will be described with reference to the drawings. In the following description, the same parts as those in the first embodiment will be omitted as appropriate. The prediction system 1 of the first embodiment outputs output information according to the prediction result to a display device equipped with a moving body MB. The prediction system 2 of this embodiment differs from the prediction device 10 of the first embodiment in that it generates a control signal according to the prediction result for the moving body MB that is automatically driven, and outputs it to a control unit that controls the moving body MB. In other words, the prediction system 1 is intended to alert the driver of the moving body MB, and the prediction system 2 is intended to safely control the moving body MB that is automatically driven, and thus the two are different. The moving body MB in this embodiment is an automatically driven moving body. Specifically, the moving body MB may be a moving body such as a robot or a drone, in addition to a vehicle such as an automatically driven car, bus, motorcycle, or construction machine.

[0056] 12 is a block diagram showing an example of a configuration of a prediction system according to the present disclosure. The prediction system 2 includes a prediction device 20, a control unit 50, a camera 60, and a distance measurement sensor 70. The camera 60 and the distance measurement sensor 70 are the same as the camera 60 and the distance measurement sensor 70 in the first embodiment, and therefore a description thereof will be omitted.

[0057] The control unit 50 is a control unit (controller) that controls the moving object MB. The control unit 50 controls the moving object MB by a control signal. For example, the control unit 50 controls the running speed, moving direction, and moving route of the moving object MB.

[0058] [Prediction device] Next, the prediction device 20 will be described with reference to Fig. 12. The prediction device 20 includes an acquisition unit 21, a detection object detection unit 22, a blind spot area detection unit 23, a prediction unit 24, and an output information generation unit 25. The acquisition unit 21, the detection object detection unit 22, and the blind spot area detection unit 23 are the same as the acquisition unit 11, the detection object detection unit 12, and the blind spot area detection unit 13 in the first embodiment, respectively, and therefore description thereof will be omitted.

[0059] The prediction unit 24 predicts a plurality of appearance times at which the detection object T will appear at a position outside the blind spot area. The prediction unit 24 predicts a plurality of appearance times at which the detection object T will appear at a position outside the blind spot area using the time at which the detection object enters the blind spot area, the predicted travel speed, and the size of the blind spot area. The prediction unit 24 sets an appearance probability, which is the probability that the detection object will appear, for each appearance time.

[0060] The prediction unit 24 changes the occurrence probability of a predicted time past the current time to 0%. For example, if the prediction unit 24 predicts that the predicted time is 10:10:10, and the detection object does not appear even at 10:10:11, the prediction unit 24 changes the occurrence probability of the predicted time to 0%. Furthermore, if the detection object T is not detected by the detection object detection unit 12 even if the occurrence probability of all predicted times becomes 0%, the prediction unit 24 re-predicts the predicted time. When the detection object T is detected by the detection object detection unit 22, the prediction unit 24 ends the prediction process.

[0061] The output information generating unit 25 generates a control signal according to the prediction result. The output information generating unit 25 outputs the generated control signal to the control unit 50. The output information generating unit 25 uses the predicted time predicted by the prediction unit 24 to generate a control signal that prevents the moving body MB from colliding or braking suddenly. Specifically, the output information generating unit 25 controls the running speed and the stop timing using the predicted time. The output information generating unit 25 may change the content of the control signal according to the appearance probability. For example, if the appearance probability from the blind spot area is low at the time when the moving body MB passes near the blind spot area, the output information generating unit 25 generates a control signal to decelerate the moving body MB. On the other hand, if the appearance probability from the blind spot area is high at the time when the moving body MB passes near the blind spot area, the output information generating unit 25 generates a control signal to gradually decelerate the moving body MB and then temporarily stop it.

[0062] (operation) Next, an example of the operation of the prediction device 20 in this embodiment will be described with reference to Fig. 13 to Fig. 14. For example, when the prediction device 20 detects that the moving object MB has started to move, it starts the process shown in Fig. 13.

[0063] First, the acquisition unit 21 acquires an image of the area in front of the moving object MB from the camera 60. Furthermore, the acquisition unit 21 acquires sensor data of the area in front of the moving object MB (step S21). The area in front of the moving object MB includes an area diagonally in front of the moving object MB.

[0064] Next, the detection target detection unit 22 detects the detection target T from the video acquired by the acquisition unit 21 (step S22). The detection target detection unit 22 detects the detection target T from frame images included in the video. The detection target detection unit 22 tracks the detected detection target T. Next, the detection target detection unit 12 identifies the moving speed and moving direction of the detected detection target (step S23).

[0065] Next, the blind spot area detection unit 23 detects a blind spot area from the video acquired by the acquisition unit 21 (step S24). When the detection target object detection unit 22 loses tracking of the detection target object T, the blind spot area detection unit 23 uses the previous frame image to identify an object in front of the detection target object T. The blind spot area detection unit 23 detects an area that becomes a blind spot due to the identified object as a blind spot area.

[0066] Next, the prediction unit 24 performs a prediction process (step S25). Fig. 14 is a diagram showing an example of a series of operations in the prediction process. The operations in the prediction process will be described with reference to Fig. 14.

[0067] First, the prediction unit 24 acquires a frame image from the video acquired by the acquisition unit 21 (step S251).

[0068] Next, the prediction unit 24 predicts a plurality of predicted travel speeds of the detection object T using the entry speed of the detection object T into the blind spot area (step S252). The entry speed of the detection object T is the moving speed of the detection object T immediately before the tracking of the detection object T is lost. That is, the entry speed of the detection object T is specified in step S23.

[0069] Next, the prediction unit 24 predicts the appearance time when the detection object T appears at a position outside the blind spot area by using the predicted travel speed (step S253). Furthermore, the prediction unit 24 sets the appearance probability that the detection object will appear at each appearance time (step S254).

[0070] Next, the prediction unit 24 judges whether or not there is an appearance time after the current time among the multiple predicted appearance times (step S255). If there is no appearance time after the current time (No in step S255), the prediction unit 24 proceeds to the process of step S257. If there is an appearance time after the current time (Yes in step S255), the prediction unit 24 sets the appearance probability of the appearance time after the current time to 0% (step S256). In addition, the prediction unit 24 judges whether or not there is an appearance time before the current time (step S257). If there is an appearance time before the current time (Yes in step S257), the series of prediction processes ends. If there is no appearance time before the current time (No in step S257), the prediction unit 14 returns to the process of step S251 and resets the predicted time.

[0071] When the series of prediction processes described above is completed, the prediction device 20 proceeds to the process of step S26 in Fig. 13. The output information generation unit 25 generates a control signal for the moving object MB according to the prediction result of step S25. The output information generation unit 25 outputs the generated control signal to the control unit 50 of the moving object MB (step S26).

[0072] The prediction device 20 repeats the processes of steps S25 to S26 until the detection target detection unit 22 detects that the detection target T has appeared in a position outside the blind spot area (No in step S27).

[0073] If the detection target T appears at a position outside the blind spot area (Yes in step S27), the prediction device 20 judges whether or not a predetermined end condition is satisfied (step S28). If the predetermined end condition is not satisfied (No in step S28), the processing of steps S21 to S27 is repeated. For example, the predetermined end condition includes the power supply of the prediction device 20 or the moving object MB being turned off. When the prediction device 20 satisfies the predetermined end condition (Yes in step S28), the prediction device 20 ends the series of processes described above.

[0074] As described above, the prediction device of this embodiment includes an acquisition unit, a detection object detection unit, a blind spot area detection unit, a prediction unit, and an output information generation unit. The prediction unit predicts a plurality of appearance times at which the detection object T appears at a position outside the blind spot area. In other words, the prediction device of this embodiment predicts the position of the detection object that has entered the blind spot area. The prediction unit further sets an appearance probability, which is the probability that the detection object will appear, for each appearance time. The output information generation unit generates a control signal according to the prediction result and outputs the generated control signal to the control unit. As a result, the prediction device of this embodiment can generate a control signal that prevents the moving object from colliding with the detection object or from suddenly braking the moving object. In other words, the prediction device of this embodiment makes it possible to safely control an autonomously driven moving object.

[0075] [Third embodiment]

[0076] (composition) Next, a prediction device 30 according to the present disclosure will be described with reference to Fig. 15 and Fig. 16. The prediction device 30 has a simplified configuration of the prediction device 10 according to the first embodiment.

[0077] As shown in FIG. 15, the prediction device 30 includes a detection target object detection unit 32, a blind spot area detection unit 33, a prediction unit , and an output information generation unit .

[0078] The detection target detection unit 32 is an example of a detection target detection means that identifies the moving speed of the detection target T detected from an image captured in front of the moving object MB.

[0079] The blind spot detection unit 33 is an example of a blind spot detection means that detects a blind spot from an image captured in front of the moving object MB.

[0080] The prediction unit 34 is an example of a prediction means that predicts a plurality of positions of the detection target T in the blind spot area using the moving speed of the detection target T that has entered the blind spot area and sets the presence probability of the detection target T at each position. The moving speed of the detection target T is identified by the detection target detection unit 31.

[0081] The output information generating unit 35 is an example of an output information generating means that generates output information according to the prediction result and outputs the generated output information.

[0082] (operation) Next, an example of the operation of the prediction device 30 according to the present disclosure will be described with reference to FIG.

[0083] First, the detection target detection unit 32 identifies the moving speed and moving direction of the detection target T detected from an image capturing an area ahead of the moving object MB (step S31).

[0084] Next, the blind spot detection unit 33 detects a blind spot from an image captured in front of the moving object MB (step S32).

[0085] Next, the prediction unit 34 predicts multiple positions of the detection object T in the blind spot area using the movement speed of the detection object T that entered the blind spot area, and sets the presence probability of the detection object T being present at each position (step S33).

[0086] Next, the output information generating unit 35 generates output information according to the prediction result, and outputs the generated output information (step S34).

[0087] According to the prediction device of this embodiment, it is possible to obtain the same advantageous effects as those of the prediction device of the first embodiment.

[0088] [Fourth embodiment]

[0089] (composition) Next, a prediction device 40 according to the present disclosure will be described with reference to Fig. 17 and Fig. 18. The prediction device 40 has a simplified configuration of the prediction device 20 according to the second embodiment.

[0090] 17, the prediction device 30 includes a detection object detection unit 42, a blind spot area detection unit 43, a prediction unit 44, and an output information generation unit 45. The detection object detection unit 42, the blind spot area detection unit 43, and the output information generation unit 45 are the same as the detection object detection unit 32, the blind spot area detection unit 33, and the output information generation unit 35 of the third embodiment.

[0091] The prediction unit 44 predicts a plurality of appearance times at which the detection target T will appear at a position outside the blind spot area. The prediction unit 44 sets an appearance probability at which the detection target T will appear at each appearance time.

[0092] (operation) Next, an example of the operation of the prediction device 40 according to the present disclosure will be described with reference to FIG.

[0093] First, the detection target detection unit 42 identifies the moving speed and moving direction of the detection target T detected from an image captured in front of the moving object MB (step S41).

[0094] Next, the blind spot detection unit 43 detects a blind spot from an image captured in front of the moving object MB (step S42).

[0095] Next, the prediction unit 44 predicts multiple appearance times at which the detection object T will appear at a position outside the blind spot area using the movement speed of the detection object T that has entered the blind spot area, and sets the appearance probability of the detection object T at each appearance time (step S43).

[0096] Next, the output information generating unit 35 generates output information according to the prediction result, and outputs the generated output information (step S44).

[0097] According to the prediction device of this embodiment, it is possible to obtain the same advantageous effects as those of the prediction device of the second embodiment.

[0098] [Hardware configuration] The functions of each of the components in each of the embodiments of the present disclosure described above can be realized not only by hardware but also by a computer device and firmware based on program control.

[0099] Fig. 19 is a diagram showing an example of a hardware configuration in which a schedule adjustment device in the present disclosure is realized by a computer device 90 including a processor. The screen sharing control device of each embodiment is realized by the computer device 90. As shown in Fig. 19, the computer device 90 includes a CPU (Central Processing Unit) 91, a memory 92, a storage device 93 such as a hard disk for storing programs, an input / output interface 94 for connecting input devices and output devices, and a communication interface 95 for connecting to a network.

[0100] The CPU 91 operates an operating system to control the prediction device of the present disclosure. For example, the CPU 91 reads out programs and data from a storage medium attached to a drive device or the like into a memory 92. The CPU 91 also functions as a part of the acquisition unit 11, the detection target detection unit 12, the blind spot area detection unit 13, the prediction unit 14, and the output information generation unit 15 in the prediction device 10 of the present disclosure, and executes processes or commands based on the programs.

[0101] The storage device 93 is, for example, an optical disk, a flexible disk, a magneto-optical disk, an external hard disk, or a semiconductor memory. A part of the storage device is a storage medium that is a non-volatile storage device, and the program is recorded therein. The program may also be downloaded from an external computer (not shown) connected to a communication network.

[0102] The input device connected to the input / output interface 94 is realized by, for example, a mouse, a keyboard, etc., and is used for input operations. Similarly, the output device connected to the input / output interface 94 is realized by, for example, a display, etc., and is used for displaying and checking output results.

[0103] Although the present disclosure has been described above with reference to each embodiment, the present disclosure is not limited to the above embodiment. Various changes that a person skilled in the art can understand within the scope of the present disclosure can be made to the configuration and details of the present disclosure. For example, although multiple operations are described in order in the form of a flowchart, the order of description does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations can be changed to the extent that the content is not affected.

[0104] Some or all of the above embodiments can be described as follows:

[0105] (Appendix 1) a detection object detection means for identifying a moving speed of a detection object detected from an image captured in front of the moving body; a blind spot detection means for detecting a blind spot from the image; a prediction means for predicting a plurality of predicted positions of the object to be detected in the blind spot area using the moving speed of the object to be detected that has entered the blind spot area, and setting a probability that the object to be detected is present in each predicted position; an output information generating means for generating output information according to a prediction result of the prediction means and outputting the generated output information; A prediction device comprising:

[0106] (Appendix 2) the output information includes an image obtained by cutting out an area including the blind spot area, and an icon superimposed on the image and displayed at each of the predicted positions; The prediction device according to claim 1, wherein the icon is displayed in a manner that corresponds to the degree of the existence probability.

[0107] (Appendix 3) 3. The prediction device according to claim 2, wherein the output information further includes display information indicating the existence probability.

[0108] (Appendix 4) 2. The prediction device according to claim 1, wherein the output information includes a map for route guidance and an icon superimposed on the map and displayed at a location where the detected object is likely to appear from the blind spot area.

[0109] (Appendix 5) The prediction means includes: setting a plurality of predicted travel speeds of the detection object within the blind spot area based on the moving speed when the detection object enters the blind spot area; 2. The prediction device of claim 1, further comprising: a predictor configured to predict the predicted position in accordance with the predicted travel speed.

[0110] (Appendix 6) The prediction means includes: predicting a plurality of appearance times at which the detection object will appear at a position outside the blind spot area; setting an appearance probability of the detection object at each of the appearance times; The output information generating means 2. The prediction device according to claim 1, further comprising: a control signal for controlling a moving object as the output information.

[0111] (Appendix 7) The prediction device according to claim 1, wherein the prediction means sets the presence probability corresponding to the predicted position located outside the blind spot area to 0.

[0112] (Appendix 8) A prediction device according to any one of appendix 1 to 7; A camera for photographing the front of the moving object; A prediction system comprising:

[0113] (Appendix 9) The computer Identifying the moving speed of the detected object from the image captured in front of the moving object, Detecting blind spots from the image; predicting a plurality of predicted positions of the detection object in the blind spot area using the moving speed of the detection object that has entered the blind spot area; A probability that the detection object is present at each of the predicted positions is set; generating output information according to the predicted position and the presence probability; The generated output information is output.

[0114] (Appendix 10) the output information includes an image obtained by cutting out an area including the blind spot area, and an icon superimposed on the image and displayed at each of the predicted positions; The prediction method according to claim 9, wherein the icon is displayed in a manner corresponding to the degree of the presence probability.

[0115] (Appendix 11) The prediction method of claim 10, wherein the output information further includes display information indicating the existence probability.

[0116] (Appendix 12) 10. The prediction method of claim 9, wherein the output information includes a map for route guidance and an icon superimposed on the map and displayed at a location where the detected object is likely to appear from the blind spot area.

[0117] (Appendix 13) In the prediction, The computer setting a plurality of predicted travel speeds of the detection object within the blind spot area based on the moving speed when the detection object enters the blind spot area; 10. The prediction method of claim 9, further comprising predicting the predicted position depending on the predicted travel speed.

[0118] (Appendix 14) In the prediction, The computer predicting a plurality of appearance times at which the detection object will appear at a position outside the blind spot area; setting an appearance probability of the detection object at each of the appearance times; A prediction method as described in Appendix 9, wherein the generation method generates a control signal for controlling a moving object as the output information.

[0119] (Appendix 15) 10. The prediction method of claim 9, wherein in the prediction, the computer sets the presence probability corresponding to the predicted position located outside the blind spot area to 0.

[0120] (Appendix 16) On the computer, A process of identifying a moving speed of a detection target object detected from an image captured in front of the moving body; A process of detecting a blind spot area from the image; A process of predicting a plurality of predicted positions of the detection object in the blind spot area using the moving speed of the detection object that has entered the blind spot area, and setting a presence probability of the detection object being present at each predicted position; generating output information according to the predicted position and the presence probability; and outputting the generated output information.

[0121] (Appendix 17) the output information includes an image obtained by cutting out an area including the blind spot area, and an icon superimposed on the image and displayed at each of the predicted positions; The program according to claim 16, wherein the icon is displayed in a manner that corresponds to the degree of the presence probability.

[0122] (Appendix 18) 18. The program of claim 17, wherein the output information further includes display information indicating the presence probability.

[0123] (Appendix 19) 17. The program of claim 16, wherein the output information includes a map for route guidance and an icon superimposed on the map and displayed at a location where the detected object may appear from the blind spot area.

[0124] (Appendix 20) In the prediction process, A process of setting a plurality of predicted travel speeds of the detection object within the blind spot area based on the moving speed when the detection object enters the blind spot area; 17. The program according to claim 16, which causes a computer to execute a process of predicting the predicted position according to the predicted travel speed.

[0125] (Appendix 21) In the prediction process, A process of predicting a plurality of appearance times at which the detection object will appear at a position outside the blind spot area; A process of setting an appearance probability that the detection object will appear at each of the appearance times; 17. The program according to claim 16, which causes a computer to execute, in the generating process, a process of generating a control signal for controlling a moving object as the output information.

[0126] (Appendix 22) The program according to claim 16, which causes a computer to execute a process of setting the presence probability corresponding to the predicted position located outside the blind spot area to 0 in the prediction process. [Explanation of symbols]

[0127] 1, 2, 3, 4 Prediction System 10, 20, 30, 40 Predictor 11, 21 Acquisition Department 12, 22, 32, 42 Object detection section 13, 23, 33, 43 Blind spot detection unit 14, 24, 34, 44 Prediction Section 15, 25, 35, 45 Output information generation section 50 Control section 60 Camera 70 Distance Sensor 80 Display device 90 Computer Equipment 91 CPU(Central Processing Unit) 92 Memory 93 Storage device 94 Input / Output Interface 95 Communication Interface

Claims

1. a detection object detection means for identifying a moving speed of a detection object detected from an image captured in front of the moving body; a blind spot detection means for detecting a blind spot from the image; a prediction means for predicting a plurality of predicted positions of the detection object in the blind spot area using the moving speed of the detection object that has entered the blind spot area, and setting a probability that the detection object is present at each predicted position; an output information generating means for generating output information according to a prediction result of the prediction means and outputting the generated output information; A prediction device comprising:

2. the output information includes an image obtained by cutting out an area including the blind spot area, and an icon superimposed on the image and displayed at each of the predicted positions; The prediction device according to claim 1 , wherein the icon is displayed in a manner that corresponds to the degree of the existence probability.

3. The prediction device according to claim 2 , wherein the output information further includes display information indicating the existence probability.

4. The prediction device according to claim 1 , wherein the output information includes a map for route guidance and an icon superimposed on the map and displayed at a point where the detected object is likely to appear from the blind spot area.

5. The prediction means includes: setting a plurality of predicted travel speeds of the detection object within the blind spot area based on the moving speed when the detection object enters the blind spot area; The prediction device according to claim 1 , wherein the predicted position is predicted in accordance with the predicted travel speed.

6. The prediction means includes: predicting a plurality of appearance times at which the detection object will appear at a position outside the blind spot area; setting an appearance probability of the detection object at each of the appearance times; The output information generating means The prediction device according to claim 1 , further comprising a control signal for controlling a moving object, the control signal being generated as the output information.

7. The prediction device according to claim 1 , wherein the prediction means sets the presence probability corresponding to the predicted position located outside the blind spot area to 0.

8. A prediction device according to any one of claims 1 to 7, A camera for photographing the front of the moving object; A prediction system comprising:

9. The computer Identifying the moving speed of the detected object from the image captured in front of the moving object, Detecting blind spots from the image; predicting a plurality of predicted positions of the detection object in the blind spot area using the moving speed of the detection object that has entered the blind spot area; A probability that the detection object is present at each of the predicted positions is set; generating output information according to the predicted position and the presence probability; The generated output information is output.

10. On the computer, A process of identifying a moving speed of a detection target object detected from an image captured in front of the moving body; A process of detecting a blind spot area from the image; A process of predicting a plurality of predicted positions of the detection object in the blind spot area using the moving speed of the detection object that has entered the blind spot area; A process of setting a probability that the detection object is present at each of the predicted positions; generating output information according to the predicted position and the presence probability; and outputting the generated output information.

Images