Behavior prediction device and behavior prediction method

Abstract

A behavior prediction device according to one aspect of the present invention comprises: a detection unit that analyzes an image obtained by imaging a region including a second area adjacent to a first area and detects a plurality of passersby appearing in the image; a prediction unit that obtains an entry index indicating the possibility that each of the passersby will enter the first area from the second area; and an aggregation unit that aggregates the entry indices of the passersby and determines a synthesis index indicating the possibility that at least one of the plurality of passersby will enter the first area from the second area.

Description

Behavior prediction device and behavior prediction method

[0001] This application claims priority to Japanese Patent Application No. 2023-201098, filed November 28, 2023, and incorporates by reference all of the contents of said Japanese application.

[0002] Patent Document 1 describes a system that analyzes the line of sight of people in an image captured by a camera that captures the waiting area of ​​a crosswalk, detects pedestrians waiting to cross the crosswalk, and, when a waiting pedestrian is detected, controls pedestrian and vehicle traffic lights to allow the pedestrian to cross the crosswalk.

[0003] Japanese Patent Application Laid-Open No. 2017-208141

[0004] A behavior prediction device according to one aspect includes a detection unit that analyzes an image of an area surrounding a first area, including a second area, to detect multiple passersby appearing in the image; a prediction unit that calculates an entry index indicating the likelihood that each passerby will enter the first area; and an aggregation unit that aggregates the entry indexes of each passerby to calculate a composite index indicating the likelihood that at least one of the multiple passersby will enter the first area.

[0005] FIG. 1 is a schematic diagram of a behavior prediction system according to an embodiment. FIG. 2 is a diagram showing an imaging area. FIG. 3 is a block diagram showing the functional configuration of a behavior prediction device. FIG. 4(a) shows an example of a frame included in a moving image. FIG. 4(b) shows an example of a frame including position information of a passerby. FIG. 5 shows an example of a movement trajectory of a passerby. FIG. 6 is a diagram for explaining an example of a method for determining the entry probability of each passerby. FIG. 7 is a diagram for explaining another example of a method for determining the entry probability of each passerby. FIG. 8 is a block diagram showing an example of the hardware configuration of a behavior prediction device. FIG. 9 is a flowchart showing a behavior prediction method according to an embodiment.

[0006] [Problem to be Solved by the Present Disclosure] For efficient traffic control, it is important to predict the behavior of pedestrians. For example, as described in the above-mentioned Patent Document 1, if it is possible to predict that pedestrians will move into an area that includes a crosswalk, it is possible to control traffic lights to allow pedestrians to cross safely. However, in actual traffic environments, there are often many pedestrians. Even in such situations, it is desirable to be able to predict the entry of pedestrians into a specific area.

[0007] Effect of the Present Disclosure According to the present disclosure, in a scene where a plurality of passersby are present, it is possible to predict the entry of passersby into a specific area.

[0008] [Description of Embodiments of the Present Disclosure] First, the contents of the embodiments of the present disclosure will be listed and described.

[0009] [1] A behavior prediction device according to one aspect of the present disclosure includes a detection unit that analyzes an image of an area surrounding a first area, including a second area, to detect multiple passersby appearing in the image; a prediction unit that calculates an entry index indicating the likelihood that each passerby will enter the first area; and an aggregation unit that aggregates the entry indexes of each passerby to calculate a composite index indicating the likelihood that at least one of the multiple passersby will enter the first area.

[0010] This behavior prediction device aggregates the indicators of each passerby entering the first area and determines the indicator of at least one of multiple passersby entering the first area, so that in a scene where multiple passersby are present, it can predict whether at least one of the multiple passersby will enter the first area.

[0011] [2] In the behavior prediction device of [1] above, the first area may be an area including a crosswalk with a traffic light installed, and the device may further include a control unit that controls the traffic light to allow multiple pedestrians to cross the crosswalk when the composite index is higher than a reference value. In this case, the traffic light is controlled when there is a pedestrian attempting to cross the crosswalk. Therefore, the pedestrian can cross safely.

[0012] [3] In the behavior prediction device described in [1] or [2] above, the detection unit may identify a movement trajectory of each passerby based on the position information of the passersby, and the prediction unit may calculate an entry indicator for each passerby based on the movement trajectory. By using the movement trajectory of the passerby, it is possible to predict with high accuracy whether the passerby will enter the first area.

[0013] [4] In the behavior prediction device according to any one of [1] to [3] above, the prediction unit may input a movement trajectory of each identified passerby and calculate an entry index for each passerby using a prediction model that has been machine-learned to output an entry index for each passerby. By using the prediction model, it is possible to predict with high accuracy whether a passerby will enter the first area.

[0014] [5] In the behavior prediction device described in any one of [1] to [3] above, the detection unit may identify the positions of multiple passersby, and the prediction unit may calculate an entry indicator for each passerby based on the distance between each passerby and the first area. The closer a passerby is to the first area, the more likely the passerby is to enter the first area. Therefore, by calculating the entry indicator for each passerby based on the distance between each passerby and the first area, it is possible to predict whether or not the passerby will enter the first area.

[0015] [6] In the behavior prediction device described in [5] above, the prediction unit may further identify attributes of multiple passersby, and the prediction unit may calculate an entry index for each passerby based on the distance and the attributes of each passerby. Since the behavior of passersby may have characteristic patterns depending on the attributes, calculating the entry index using the attributes of each passerby can improve the prediction accuracy of the entry index for each passerby.

[0016] [7] In the behavior prediction device described in any one of [1] to [3] above, the detection unit may identify the positions of multiple passersby, and the prediction unit may identify multiple candidate routes for each passerby to travel from the identified positions to the first area, and calculate an entry index for each passerby based on the likelihood that each candidate route will take each of the multiple candidate routes. In this case, it is possible to appropriately predict whether or not a passerby will enter the first area.

[0017] [8] In the behavior prediction device according to any one of [1] to [7] above, the aggregation unit may perform statistical processing on the entry indexes of each passerby to obtain a composite index. In this case, it is possible to appropriately predict whether at least one passerby will enter the first area.

[0018] [9] In the behavior prediction device described in any one of [1] to [8] above, the aggregation unit may calculate a weighted average using the entry index of each passerby as a weight. By calculating the weighted average using the entry index of each passerby as a weight, it is possible to predict with high accuracy whether at least one of the multiple passersby will enter the first area.

[0019]

[10] In the behavior prediction device described in any one of [1] to [8] above, the aggregation unit may calculate an index indicating that all passersby will not enter the first area based on the entry index of each passerby, and may obtain a composite index by subtracting the index indicating that all passersby will not enter the first area from the overall index. In this case, it is possible to predict with high accuracy whether at least one of the multiple passersby will enter the first area.

[0020]

[11] A behavior prediction method according to one aspect of the present disclosure includes the steps of: analyzing an image of an area surrounding a first area, including a second area, to detect multiple passersby appearing in the image; calculating an entry index indicating a likelihood that each passerby will enter the first area; and aggregating the entry indexes of each passerby to calculate a composite index indicating a likelihood that at least one of the multiple passersby will enter the first area. This behavior prediction method can predict whether at least one of the multiple passersby will enter the first area in a scene where multiple passersby are present.

[0021] [Details of the embodiments of the present disclosure] Specific examples of the embodiments of the present disclosure will be described below with reference to the drawings. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims. In the description of the drawings, the same elements are given the same reference numerals, and duplicate explanations will be omitted.

[0022] 1 is a schematic diagram of a behavior prediction system 1 according to an embodiment. The behavior prediction system 1 predicts the behavior of multiple passersby M, in particular, whether or not at least one of the multiple passersby M will cross a pedestrian crossing.

[0023] In the following description, the behavior prediction system 1 is applied to a situation where a sidewalk 2 and a roadway 3 are provided, and an example will be described in which the system predicts whether at least one of multiple pedestrians M traveling on the sidewalk 2 will cross a crosswalk 4 installed on the roadway 3. Passersby M are typically pedestrians, but the pedestrians M also include people riding on vehicles such as bicycles, kick scooters, and wheelchairs. As shown in FIG. 1 , pedestrian traffic lights S1 are installed on both sides of the crosswalk 4 in the direction of travel. A vehicle traffic light S2 is installed on the roadway 3 for vehicles passing through the crosswalk 4. The sidewalk 2 may be a shoulder or a shoulder strip. The roadway 3 is a road where pedestrians M may cross. In the following description, pedestrians M traveling on the sidewalk 2 who intend to cross the crosswalk 4 may be referred to as a "crosser."

[0024] As shown in FIG. 1 , the behavior prediction system 1 includes a camera 10, a roadside device 20, and a behavior prediction device 30. The camera 10 is an imaging device mounted, for example, on top of a support pole 5 installed on a sidewalk 2 and captures an image of an imaging area C. In the following description, the term "image" includes both moving and still images. Although not limited thereto, the camera 10 may be installed at a height of 3 m or more from the ground surface. In one embodiment, as shown in FIG. 2 , the imaging area C is an area including a waiting area (second area) R2 around the roadway 3. The waiting area R2 is located in front of the crosswalk 4 and is an area on the sidewalk 2 where a pedestrian M intending to cross the crosswalk 4 temporarily waits. As shown in FIG. 2 , the imaging area C may include a part or all of a crossing area (first area) R1 including the crosswalk 4.

[0025] The camera 10 may be disposed at an angle as long as it can capture an image of the image capture area C. For example, the central axis (optical axis) of the camera 10 may be perpendicular to the ground surface or may be inclined relative to the ground surface. The camera 10 may be installed on the opposite side of the crosswalk 4 from the image capture area C.

[0026] The camera 10 captures an image of the imaging area C at a predetermined frame rate to generate a moving image. The captured moving image is transmitted to the roadside unit 20 via wired or wireless communication. The camera 10 may periodically transmit images (still images) captured in the imaging area C to the roadside unit 20.

[0027] The roadside device 20 is a computer such as a PLC (Programmable Logic Controller) equipped with a processor, a storage device, a communication device, etc., and is installed adjacent to the roadway 3. For example, the roadside device 20 is provided on a support pole 5. The roadside device 20, for example, loads a program stored in the storage device and executes the loaded program on the processor, thereby realizing various functions described below.

[0028] The roadside device 20 is communicably connected to the camera 10, the behavior prediction device 30, the pedestrian traffic light S1, and the vehicular traffic light S2. The roadside device 20 transmits moving images captured by the camera 10 to the behavior prediction device 30. The roadside device 20 also receives control signals from the behavior prediction device 30 and controls the pedestrian traffic light S1 and the vehicular traffic light S2.

[0029] The behavior prediction device 30 is a stationary or portable computer or workstation equipped with a processor, a storage device, a communication device, etc., and is typically located at a location remote from the roadside device 20 .

[0030] The behavior prediction device 30 can communicate with the roadside device 20 via a network N. The network N is a communication network for wireless or wired communication. As will be described later, the behavior prediction device 30 calculates an index indicating the possibility that one of multiple passersby M captured in a video image captured by the camera 10 will enter the crossing area R1. The index may be, for example, a probability, likelihood, or reliability. Note that "entering the crossing area R1" is a concept that includes the passerby M passing through the boundary line between the crossing area R1 and the waiting area R2. The behavior prediction device 30, for example, loads a program stored in a storage device and executes the loaded program on a processor to realize various functions, which will be described later.

[0031] 3 , the behavior prediction device 30 has, as functional components, an acquisition unit 31, a detection unit 32, a prediction unit 33, a collection unit 34, and a control unit 35. The acquisition unit 31 acquires moving images of an imaging area C captured by the camera 10 via the roadside device 20. The acquisition unit 31 acquires moving images that capture at least one passerby M from the moving images captured by the camera 10. Note that the acquisition unit 31 may periodically acquire images (still images) that capture multiple passersby M.

[0032] The detection unit 32 analyzes the moving image acquired by the acquisition unit 31 and detects multiple passersby M appearing in the moving image. The detection unit 32 identifies multiple passersby M appearing in each frame (image) of the moving image, for example, by image recognition processing, and outputs position information indicating the positions of the identified multiple passersby M. Fig. 4(a) shows an example of one frame included in the moving image input to the detection unit 32. Fig. 4(b) shows an example of a frame including position information of passersby M. In Figs. 4(a) and 4(b), passersby M1, passersby M2, and passersby M3 are shown as multiple passersby M.

[0033] The position information of passersby M can be identified using known image recognition algorithms such as a convolutional neural network (CNN), histogram of oriented gradients (HOG), and YOLO (you only look once). Note that the detection unit 32 may further identify attribute information indicating the attributes of each passerby M in addition to the position information of each passerby M. The attribute information of passersby M is information indicating the attributes of the passerby M, such as the age, gender, and type (pedestrian, cyclist, wheelchair user, kickboard user, white cane user, etc.) of the passerby M. The detection unit 32 identifies the position information and attributes of each passerby M for all frames included in the moving image.

[0034] Furthermore, the detection unit 32 may identify the movement trajectory of the passerby M based on the position information of the passerby M in multiple frames. For example, the detection unit 32 connects the positions of each passerby M between multiple frames arranged in time series to generate the movement trajectory of each passerby M. The movement trajectory of the passerby M can be generated using a known tracking algorithm such as a Kalman filter, a particle filter, Optical Flow, or Deep SORT.

[0035] 5 shows examples of movement trajectories of each passerby M identified by the detection unit 32. In Fig. 5, movement trajectory 50A represents the movement trajectory of passerby M1, movement trajectory 50B represents the movement trajectory of passerby M2, and movement trajectory 50C represents the movement trajectory of passerby M3. The detection unit 32 associates information indicating the movement trajectory of each passerby M with an identifier (ID) that identifies each passerby M, and outputs the information to the prediction unit 33.

[0036] The prediction unit 33 calculates an entry index for each passerby M, which indicates the possibility that each passerby M will enter the crossing area R1. The entry index is the probability, likelihood, or reliability that each passerby M will enter the crossing area R1. In the following explanation, an example will be described in which an entry probability, which indicates the probability that each passerby M will enter the crossing area R1, is calculated as the entry index. The entry probability can also be said to be the probability that each passerby M will cross the crosswalk 4. The entry probability for each passerby M can be calculated using various methods.

[0037] A first method for calculating the entry probability of each passerby M will be described. In the first method, the entry probability of each passerby M is calculated using a prediction model that has been machine-learned to input the movement trajectory of each passerby M and output the entry probability of each passerby M. The prediction model is a classifier constructed by machine learning a data set of training data including the movement trajectories of passersby M captured in previously captured video images and label information indicating the presence or absence of a pedestrian crossing. The prediction model is generated by optimizing learning parameters using known machine learning algorithms such as a convolutional neural network and a recurrent neural network. The prediction unit 33 reads the prediction model stored in the storage device, inputs the movement trajectories of each passerby M identified by the detection unit 32 into the prediction model, and obtains the probability that each passerby M will enter the crossing area R1.

[0038] The training data used to generate the prediction model may include attribute information indicating the attributes of passersby M. The behavior of passersby M may have characteristic patterns depending on their attributes. For example, adult pedestrians tend to move quickly and in a straight line. In contrast, the movement speeds of children, pedestrians with strollers, and people using white canes tend to be slower than that of adult pedestrians. Passersby M riding bicycles or kick scooters tend to move quickly and change their movement direction less than pedestrians. When the training data includes attribute information of passersby M, a prediction model can be generated that takes into account the movement patterns of passersby M described above.

[0039] As described above, when a prediction model is generated using training data including attribute information of passersby M, the prediction unit 33 inputs the attribute information of passersby M identified by the detection unit 32 into the prediction model in addition to the movement trajectory of each passerby M. This makes it possible to calculate the entry probability of each passerby M with high accuracy.

[0040] Next, a second method for calculating the entry probability of each passerby M will be described. In the second method, the entry probability of each passerby M is calculated based on the distance between each passerby M and the crossing area R1. In general, the closer the position of a passerby M is to the crossing area R1, the more likely the passerby M is to enter the crossing area R1. Therefore, in the second method, the prediction unit 33 calculates the distance between each passerby M and the crossing area R1 based on the position information of each passerby M identified by the detection unit 32, and sets the entry probability of the passerby M to be higher the shorter the distance.

[0041] In addition, the prediction unit 33 may identify the movement direction of each passerby M from the movement trajectory of each passerby M, and set the entry probability of the passerby M to be high when the passerby M is moving toward the crossing area R1, and set the entry probability of the passerby M to be low when the passerby M is moving in a direction away from the crossing area R1.

[0042] Furthermore, even if the distance between the passerby M and the crossing area R1 is small, if the movement speed of the passerby M is fast, the possibility that the passerby M will enter the crossing area R1 increases. Therefore, the prediction unit 33 may set the entry probability of the passerby M to be high if the movement speed of the passerby M is fast, and may set the entry probability of the passerby M to be low if the movement speed of the passerby M is slow. The movement speed of the passerby M is determined based on, for example, the attributes of the passerby M.

[0043] An example of a method for determining the entry probability of each passerby M using the second technique will be described with reference to FIG. 6 . In FIG. 6 , passerby M1, passerby M2, passerby M3, and passerby M4 are illustrated as multiple passerby M. Passerby M1, M2, and M4 are pedestrians, and passerby M3 is a person riding a bicycle. In the scene illustrated in FIG. 6 , passerby M1 is located closer to the crossing area R1 than passerby M2, M3, and M4. Therefore, the prediction unit 33 sets the entry probability of passerby M1 higher than the entry probabilities of passerby M2, M3, and M4. Passerby M3 is riding a bicycle and moves faster than passerby M1, M2, and M3. Therefore, the prediction unit 33 sets, for example, the entry probability of passerby M3 higher than the entry probability of passerby M2.

[0044] Next, a third method for determining the entry probability of each passerby M will be described. In the third method, multiple candidate routes for traveling from the position of each passerby M to the crossing area R1 are identified, and the entry probability of each passerby M is determined based on the probability that each passerby M will take each of the multiple candidate routes. An example of a method for determining the entry probability of each passerby M using the third method will be described with reference to FIG. 7 . FIG. 7 illustrates a passerby M located in the image capture area C. In the third method, the prediction unit 33 first sets multiple candidate routes 51A, 51B, and 51C for traveling from the position of the passerby M to the crossing area R1. Note that the number of candidate routes set by the prediction unit 33 is arbitrary.

[0045] Next, the prediction unit 33 determines the probabilities of each of the multiple route candidates 51A, 51B, and 51C being selected as the route along which the passerby M will travel. The probabilities of the passerby M selecting the route candidates 51A, 51B, and 51C can be determined, for example, based on the past movement trajectories of the passerby M. In the following description, the probability that the passerby M will select the route candidate 51A is represented as P1(A), the probability that the passerby M will select the route candidate 51B is represented as P1(B), and the probability that the passerby M will select the route candidate 51C is represented as P1(C).

[0046] Next, the prediction unit 33 determines the probability that the passerby M will travel along the route candidates 51A, 51B, and 51C. The probability that the passerby M will travel along the route candidates 51A, 51B, and 51C indicates the probability that the passerby M will travel along each of the route candidates 51A, 51B, and 51C from the start point to the end point without turning back along the route candidates 51A, 51B, and 51C. In the following description, the probability that the passerby M will travel along the route candidate 51A is represented as P2(A), the probability that the passerby M will travel along the route candidate 51B is represented as P2(B), and the probability that the passerby M will travel along the route candidate 51C is represented as P2(C). The probabilities P2(A) to P2(C) can be determined based on, for example, the past movement trajectories of the passerby M.

[0047] Next, the prediction unit 33 calculates the probabilities that passerby M will enter the traversal area R1 via the route candidates 51A, 51B, and 51C. In the following description, the probability that passerby M will enter the traversal area R1 via the route candidate 51A will be represented as P3(A), the probability that passerby M will enter the traversal area R1 via the route candidate 51B will be represented as P3(B), and the probability that passerby M will enter the traversal area R1 via the route candidate 51C will be represented as P3(C). In this case, the probabilities P3(A), P3(B), and P3(C) can be calculated using the following formulas: P3(A) = P1(A) x P2(A) P3(B) = P1(B) x P2(B) P3(C) = P1(C) x P2(C)

[0048] Next, the prediction unit 33 calculates the probability P(M) of the passerby M entering the crossing area R1 using the following formula: P(M)=P3(A)+P3(B)+P3(C).

[0049] The prediction unit 33 outputs the entry probability of each passerby M calculated by the various methods described above to the aggregation unit 34 .

[0050] The aggregation unit 34 aggregates the entry probabilities of each passerby M to determine a composite index indicating the likelihood that at least one of the multiple passersby M will enter the crossing area R1. The composite index is the probability, likelihood, or reliability that at least one of the multiple passersby M will enter the crossing area R1. In the following explanation, an example will be described in which a composite probability indicating the probability that at least one of the multiple passersby M will enter the crossing area R1 is calculated as the composite index. The composite probability can also be said to be the probability that at least one of the multiple passersby M will cross the crosswalk 4. For example, the aggregation unit 34 calculates the composite probability by performing statistical processing on the entry probabilities of each passerby M calculated by the prediction unit 33.

[0051] The composite probability can be calculated using various methods. First, a first method for calculating the composite probability will be described. In the first method, the aggregation unit 34 calculates the composite probability by averaging or weighted averaging, with the entry probability of each passerby M as a weight. For example, if the entry probability of multiple passersby Mn (n = 1, 2, 3, ..., N) into the crossing area R1 is represented as P(Mn), the aggregation unit 34 can calculate the composite probability Pc using the following formula: Pc = (P(M1) + P(M2) + P(M3) + ... + P(MN)) / N

[0052] As described above, the combined probability Pc is calculated by dividing the sum of the entry probabilities of multiple passersby M by the number of passersby M. For example, if there are 10 passersby M in the video and the entry probabilities of the 10 passersby are 0.9, 0.8, 0.7, 0.9, 0.6, 0.5, 0.8, 0.9, 0.4, and 0.7, respectively, the combined probability Pc is 0.72. This result indicates that the probability that at least one of the 10 passersby will cross the crosswalk 4 is 72%.

[0053] Next, a second method for calculating the composite probability will be described. In the second method, the aggregation unit 34 uses a complement to subtract the probability that none of the passersby M enter the crossing area R1 from the overall probability to calculate the composite probability Pc. For example, if the probability of multiple passersby Mn (n = 1, 2, 3, ..., N) entering the crossing area R1 is represented as P(Mn), the aggregation unit 34 calculates the probability Pf that none of the multiple passersby Mn enter the crossing area R1 using the following formula: Pf = (1 - P(M1)) * (1 - P(M2)) * (1 - P(M3)) ... (1 - P(MN))

[0054] Then, the aggregation unit 34 calculates the combined probability Pc using the following formula: Pc=1−Pf

[0055] As described above, the second method calculates the combined probability Pc by subtracting the probability Pf that none of the passersby M will enter the crossing area R1 from the overall probability. For example, if the number of passersby M captured in the video is four and the entry probabilities of the four passersby are 0.7, 0.3, 0.6, and 0.4, respectively, the combined probability Pc is 0.9496. This result indicates that the probability that at least one of the four passersby will cross the crosswalk 4 is approximately 95%.

[0056] The aggregating unit 34 can also calculate the composite probability by performing any statistical processing on the entry probability of each passerby M using a method other than the above. For example, the aggregating unit 34 may calculate the composite probability by performing a likelihood evaluation, or may calculate the composite probability using a probability mass function. The aggregating unit 34 outputs the calculated composite probability to the control unit 35.

[0057] The control unit 35 controls the pedestrian traffic light S1 and the vehicular traffic light S2 based on the combined probability output from the aggregation unit 34. For example, if the combined probability is higher than a reference value, the control unit 35 controls the pedestrian traffic light S1 and the vehicular traffic light S2 via the roadside device 20 so that the pedestrian traffic light S1 changes its signal display to green and the vehicular traffic light S2 changes its signal display to red. If the pedestrian traffic light S1 is a push-button traffic light that changes its signal display by pressing a button, the control unit 27 may output a control signal to physically or electrically press the button. This allows pedestrian M to cross the crosswalk 4 safely. The reference value is a setting value that is preset by a designer.

[0058] Next, the hardware configuration of the behavior prediction device 30 will be described. Fig. 8 is a block diagram showing an example of the hardware configuration of the behavior prediction device 30. The behavior prediction device 30 is configured by one or more computers. The computer includes a CPU (processor) 101, a main memory unit 102, an auxiliary memory unit 103, a communication control unit 104, an input device 105, and an output device 106. Each of the behavior prediction devices 30 is configured by one or more computers configured by this hardware and software such as a program.

[0059] When the behavior prediction device 30 is configured by multiple computers, the multiple computers may be connected locally or via a communication network such as the Internet or an intranet. This connection logically constructs one behavior prediction device 30.

[0060] The CPU 101 is a processor that executes an operating system, application programs, etc. The main memory unit 102 is composed of a ROM (Read Only Memory) and a RAM (Random Access Memory). The auxiliary memory unit 103 is a storage medium composed of a hard disk, flash memory, etc. The auxiliary memory unit 103 generally stores a larger amount of data than the main memory unit 102. The communication control unit 104 is composed of a network card or a wireless communication module. At least part of the communication function between the camera 10, the roadside unit 20, and the behavior prediction device 30 is realized by the communication control unit 104. The input device 105 is composed of a keyboard, a mouse, a touch panel, a microphone for voice input, etc. The output device 106 is composed of a display, a printer, etc.

[0061] The auxiliary storage unit 103 stores programs and data necessary for processing. The programs cause the computer to execute each functional element of the behavior prediction device 30. The programs realize the functions of the behavior prediction device 30 in the computer. For example, the programs are read by the CPU 101 or the main storage unit 102, and cause at least one of the CPU 101, the main storage unit 102, the auxiliary storage unit 103, the communication control unit 104, the input device 105, and the output device 106 to operate. For example, the programs read and write data from and to the main storage unit 102 and the auxiliary storage unit 103.

[0062] The program may be provided in the form of a tangible storage medium such as a CD-ROM, a DVD-ROM, a semiconductor memory, etc. The program may be provided as a data signal via a communication network.

[0063] The behavior prediction device 30 does not necessarily have to be configured by a computer that operates by a program, and some or all of the functions of the behavior prediction device 30 may be implemented in an ASIC (Application Specific Integrated Circuit) that integrates logic circuits.

[0064] Next, a behavior prediction method according to an embodiment will be described. Fig. 9 is a flowchart showing the behavior prediction method according to an embodiment. This behavior prediction method is executed by the behavior prediction device 30 described above.

[0065] 9 , first, the acquisition unit 31 of the behavior prediction device 30 acquires an image of the imaging area C captured by the camera 10 (step ST1). The image acquired by the acquisition unit 31 is an image that includes at least one passerby M among the images captured by the camera 10. The acquisition unit 31 periodically acquires images from the camera 10, for example.

[0066] Next, the detection unit 32 detects multiple passersby M appearing in the image (step ST2). At this time, the detection unit 32 acquires position information and attribute information of each passerby M. Next, the detection unit 32 tracks each passerby M based on the time-series position information of each passerby M to identify their movement trajectories (step ST3).

[0067] Next, the prediction unit 33 calculates the entry probability of each passerby M (step ST4). For example, the prediction unit 33 inputs the movement trajectory of each passerby M into the above-mentioned prediction model, and calculates the probability that multiple passersby M will enter the crossing area R1 for each passerby M.

[0068] Next, the prediction unit 33 aggregates the entry probabilities of each passerby M to calculate a composite probability indicating the probability that at least one of the multiple passersby M will enter the crossing area R1 (step ST5). For example, the prediction unit 33 performs statistical processing on the entry probabilities of each passerby M, such as a weighted average using the entry probabilities as weights, to calculate the composite probability.

[0069] Next, the control unit 35 determines whether the combined probability of the multiple passersby M output from the aggregation unit 34 is higher than a reference value (step ST6). If the combined probability is higher than the reference value, the control unit 35 sends a control signal to the roadside device 20 to control the pedestrian traffic light S1 and the vehicular traffic light S2 so that the passerby M can cross the crosswalk 4 (step ST7). On the other hand, if the crossing probability is equal to or lower than the reference value, the control unit 35 ends the series of processes without controlling the pedestrian traffic light S1 and the vehicular traffic light S2.

[0070] As explained above, the behavior prediction device 30 aggregates the probability that each passerby M will enter the crossing area R1 and determines the probability that any one of the multiple passersby M will enter the crossing area R1. Therefore, in a situation where multiple passersby M are present, it is possible to predict whether at least one of the multiple passersby M will enter the first area.

[0071] The behavior prediction device 30 and behavior prediction method according to various embodiments have been described above, but the present invention is not limited to the above-described embodiments and various modifications can be made without departing from the spirit of the invention.

[0072] For example, in the above embodiment, the behavior prediction device 30 acquires images from the camera 10 via the roadside device 20 and controls the pedestrian traffic light S1 and the vehicular traffic light S2 via the roadside device 20. However, the behavior prediction device 30 may acquire images directly from the camera 10 and directly control the pedestrian traffic light S1 and the vehicular traffic light S2. The behavior prediction device 30 may be integrated with the camera 10. That is, the behavior prediction device 30 may be provided inside the housing of the camera 10. The behavior prediction device 30 may also be integrated with the roadside device 20. That is, the behavior prediction device 30 may be provided inside the housing of the roadside device 20.

[0073] In the above embodiment, whether or not each passerby M will enter the crossing area R1 is predicted based on the movement trajectory of the passerby M, but it is possible to determine whether or not the passerby M will enter the crossing area R1 using any method. For example, the prediction unit 33 may calculate the entry probability of each passerby M using only the position information of each passerby M without using the movement trajectory of each passerby M.

[0074] Furthermore, in the above embodiment, the behavior prediction device 30 predicts whether at least one of a plurality of passersby M will cross the crosswalk 4, but the behavior prediction device 30 can also be applied to roads where a crosswalk 4 is not installed. For example, the behavior prediction device 30 can predict whether at least one of a plurality of passersby M shown in an image will enter a specific area on the road. Furthermore, the pedestrian traffic light S1 and the vehicular traffic light S2 do not necessarily have to be installed, and the behavior prediction device 30 only needs to predict the entry of a passerby M into a specific area, and does not necessarily have to control the pedestrian traffic light S1 and the vehicular traffic light S2.

[0075] The various embodiments described above can be combined to the extent that no contradiction occurs.

[0076] DESCRIPTION OF SYMBOLS 1... Behavior prediction system 2... Sidewalk 3... Roadway 4... Crosswalk 5... Support pole 10... Camera 20... Roadside device 30... Behavior prediction device 31... Acquisition unit 32... Detection unit 33... Prediction unit 34... Aggregation unit 35... Control unit 50A, 50B, 50C... Movement trajectory 51A, 51B, 51C... Route candidates 101... CPU (processor) 102... Main memory unit 103... Auxiliary memory unit 104... Communication control unit 105... Input device 106... Output device C... Imaging area M... Passerby N... Network R1... Crossing area (first area) R2... Waiting area (second area) S1... Pedestrian traffic light S2... Vehicle traffic light

Claims

1. A behavior prediction device comprising: a detection unit that analyzes an image captured of an area surrounding a first area including a second area to detect multiple passersby appearing in the image; a prediction unit that calculates an entry indicator indicating the possibility of each passerby entering the first area; and an aggregation unit that aggregates the entry indicators of each passerby to calculate a composite indicator indicating the possibility of at least one of the multiple passersby entering the first area.

2. The behavior prediction device of claim 1, wherein the first area is an area including a crosswalk equipped with a traffic light, and the behavior prediction device further comprises a control unit for controlling the traffic light so that the plurality of pedestrians can cross the crosswalk when the composite index is higher than a reference value.

3. The behavior prediction device according to claim 1 or claim 2, wherein the detection unit identifies a movement trajectory of each of the multiple passersby based on position information of the passersby, and the prediction unit determines the entry indicator of each passerby based on the movement trajectory.

4. The behavior prediction device of claim 3, wherein the prediction unit inputs the movement trajectory of each identified passerby and determines the entry index of each passerby using a prediction model that has been machine-learned to output the entry index of each passerby.

5. The behavior prediction device according to claim 1 or claim 2, wherein the detection unit identifies the positions of the plurality of passersby, and the prediction unit determines the entry indicator of each passerby based on the distance between each passerby and the first area.

6. The behavior prediction device according to claim 5, wherein the prediction unit further identifies attributes of the plurality of passersby, and the prediction unit determines the entry index of each passerby based on the distance and the attributes of each passerby.

7. The behavior prediction device described in claim 1 or claim 2, wherein the detection unit identifies the positions of the multiple passersby, and the prediction unit identifies multiple candidate routes for each passerby to travel from the identified positions to the first area, and calculates the entry indicator of each passerby based on the possibility that each candidate route will take each of the multiple candidate routes.

8. The behavior prediction device according to claim 1, wherein the aggregation unit performs statistical processing on the entry index of each passerby to obtain the composite index.

9. The behavior prediction device according to claim 1, wherein the aggregation unit determines the composite index by a weighted average in which the entry index of each passerby is used as a weighting factor.

10. A behavior prediction device as described in any one of claims 1 to 8, wherein the aggregation unit determines an index indicating that all passersby do not enter the first area based on the entry index of each passerby, and subtracts the index indicating that all passersby do not enter the first area from the overall index to obtain the composite index.

11. A behavior prediction method comprising the steps of: analyzing an image captured of an area surrounding a first area including a second area to detect a plurality of passersby appearing in the image; calculating an entry index indicating the possibility of each passerby entering the first area; and aggregating the entry indexes of each passerby to calculate a composite index indicating the possibility of at least one of the plurality of passersby entering the first area.

Images