Monitoring control device, monitoring method, and computer program

Abstract

To prevent monitoring accuracy of behavior of an object existing in a predetermined area from being affected by an environment in the area.SOLUTION: The monitoring control apparatus 30 is configured to monitor a behavior of an object in a predetermined area based on a captured image in which the area is captured, divide a predetermined range of the captured image into a plurality of blocks to detect a motion vector of each block, and determine, based on a direction and a magnitude of the motion vector of each block, whether an object moving in a doorway direction of the area is present in the captured image without detecting the object captured in the captured image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a monitoring control device, a monitoring method, and a computer program. [Background technology]

[0002] Patent Document 1 discloses a technology for determining the disembarking behavior of a passenger by identifying and tracking the head of the passenger from video data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2018-523234 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, the conventional technology detects people (identifies human heads) based on video data. However, for example, when the vehicle is crowded, multiple passengers may appear overlapping in the video data. In such cases, the conventional technology may not be able to detect the heads of passengers at the rear. Furthermore, when the illumination inside the vehicle is low (the vehicle is dark), for example, at night, the heads may not be clearly visible in the video data, and therefore the heads of passengers may not be detected.

[0005] As described above, when detecting people from video data, the ease of detection varies depending on the environment within the area being monitored, such as the level of congestion and illuminance, etc. Therefore, with the conventional technology described above, the accuracy of determining passenger disembarking behavior may be affected by the in-vehicle environment.

[0006] The present invention has been made in response to such problems, and aims to reduce the influence of the environment within a specified area on the accuracy of monitoring the behavior of objects present within the specified area. [Means for solving the problem]

[0007] In order to solve the above problem, a monitoring control device according to one aspect of the present invention is a monitoring control device configured to monitor the behavior of objects within a specified area based on an captured image of the area, and is configured to divide a specified range of the captured image into multiple blocks, detect the motion vector of each block, and determine whether there is an object in the captured image heading toward the entrance / exit of the area based on the direction and magnitude of the motion vector of each block, without detecting the object in the captured image.

[0008] Furthermore, a monitoring method according to one aspect of the present invention is a monitoring method for monitoring the behavior of objects within a specified area based on an image captured of the area, in which a specified range of the captured image is divided into a plurality of blocks, a motion vector of each block is detected, and based on the direction and magnitude of the motion vector of each block, it is determined whether an object heading toward an entrance / exit of the area is present in the captured image without detecting the object in the captured image.

[0009] Furthermore, a computer program according to one aspect of the present invention is a computer program for a monitoring control device configured to monitor the behavior of objects within a specified area based on an image captured of the area, and causes the monitoring control device to perform a process of dividing a specified range of the captured image into multiple blocks, detecting the motion vector of each block, and determining whether there is an object in the captured image heading toward the entrance / exit of the area, without detecting the object in the captured image, based on the direction and magnitude of the motion vector of each block. [Effects of the Invention]

[0010] According to these aspects of the present invention, the behavior of an object in a captured image of a specified area is inferred based on the direction and magnitude of the motion vector of each block in the captured image without detecting the object in the captured image, thereby reducing the dependence of the monitoring accuracy of the object's behavior on the environment within the area. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of a monitoring device according to an embodiment of the present invention; [Figure 2] FIG. 10 is a diagram showing a captured image of the passenger area inside a bus. [Figure 3] FIG. 10 is a diagram showing a captured image of the passenger area inside a bus. [Figure 4] FIG. 10 is a diagram showing a captured image of a bus with its doors open, showing the passenger area inside the bus and the boarding and disembarking area outside the bus. [Figure 5] 3 is a flowchart illustrating monitoring control according to an embodiment of the present invention. [Figure 6] 3 is a flowchart illustrating monitoring control according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, like components are designated by like reference numerals.

[0013] FIG. 1 is a schematic diagram of a monitoring device 1 according to an embodiment of the present invention.

[0014] The monitoring device 1 includes an imaging device 10, a notification device 20, and a monitoring control device 30.

[0015] The imaging device 10 generates captured images of a predetermined area to be monitored (hereinafter referred to as the "monitored area"). In this embodiment, the imaging device 10 includes one or more cameras 11 for capturing images of the passenger area inside the autonomously driven shuttle bus. The cameras 11 capture images of at least the passenger area inside the bus at a predetermined frame rate (for example, 10 Hz to 40 Hz) and generate captured images that capture the passenger area inside the bus. Each time the cameras 11 generate a captured image, they transmit the generated captured image to the monitoring control device 30.

[0016] In this embodiment, the monitoring area is described using an example in which the passenger area on a bus is used as the monitoring target area, but of course the monitoring target area is not limited to the passenger area on a bus. For example, the monitoring target area can be part or all of a moving body such as a bus, train, or airplane, a room, or a venue. Furthermore, the monitoring target area can be the entire captured image, or just a part of it.

[0017] The notification device 20 is a device for making notifications to people inside and outside the bus. In this embodiment, the notification device 20 includes a display 21 for making visual notifications and a speaker 22 for making auditory notifications. The notification device 20 displays information (e.g., text information or image information) corresponding to a display signal received from the monitoring control device 30 on the display 21, and outputs a sound from the speaker 22 corresponding to a sound signal received from the monitoring control device 30.

[0018] The monitoring and controlling device 30 is an ECU (Electronic Control Unit) including a communication unit 31, a storage unit 32, and a processing unit 33.

[0019] The communication unit 31 includes an interface circuit for connecting the monitoring control device 30 to the internal network 2. The communication unit 31 supplies the captured images received from the camera 11 to the processing unit 33. The communication unit 31 also transmits the display signal and audio signal output from the processing unit 33 to the alarm device 20.

[0020] The storage unit 32 has a storage medium such as a hard disk drive (HDD), a solid disk drive (SSD), or a semiconductor memory, and stores various computer programs and data used in the processing by the processing unit 33.

[0021] The processing unit 33 has one or more central processing units (CPUs) and their peripheral circuits, and executes various computer programs stored in the storage unit 32. The processing unit 33 is, for example, a processor. The processing unit 33 may further include other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit. The processing unit 33 executes processing in accordance with the computer programs, thereby functioning as a motion vector calculation unit 41, an alighting determination unit 42, an entry determination unit 43, and a notification unit 44, and operates as functional units (modules) that realize predetermined functions. In the following description, when processing is described using each of the functional units 41 to 44 as the subject, it indicates that the processing unit 33 is executing a program that realizes each of the functional units 41 to 44.

[0022] Below, the specific processing contents performed in the monitoring control device 30, i.e., the contents of each functional unit 41 to 44 realized by the processing unit 33 executing processing according to a computer program, will be explained with reference to Figures 2 to 4 as necessary.

[0023] Figures 2 and 3 are images showing the passenger area inside the bus, with the bus doors closed. Figure 4 is an image showing the passenger area inside the bus (unhatched area) and the boarding and disembarking area outside the bus (hatched area) with the bus doors open.

[0024] The motion vector calculation unit 41 detects a motion vector indicating the direction and size of movement of an object between frames (still images) taken at different times, based on the captured image received from the camera 11. As shown in FIG. 2, in this embodiment, the motion vector calculation unit 41 divides a preset monitoring range in the captured image into blocks of a predetermined size and detects a motion vector for each divided block. Note that to avoid cluttering the drawing, FIG. 2 shows only the direction and size (corresponding to the length of the arrow) of the motion vector for some blocks in the upper left corner, but in reality, motion vectors are detected for all blocks. In this embodiment, a block includes multiple pixels, and as an example, the size of the block (vertical × horizontal) is set to 5 pixels × 5 pixels.

[0025] The monitoring range can be set to any range that allows for understanding the behavior of passengers in the passenger area, but it is desirable that the range does not include the window area, as shown in Figure 2. This is because if the window area is included in the monitoring range, a motion vector indicating the movement direction and size of an object outside the window will be detected, making it difficult to predict the behavior of passengers inside the vehicle.

[0026] The alighting determination unit 42 determines whether there are any passengers getting off the bus based on the direction and magnitude of the motion vector of each block.

[0027] The alighting determination unit 42, for example, extracts from each block blocks those blocks that have a motion vector pointing in the direction toward the bus entrance / exit (boarding / exit door), calculates an integrated value (hereinafter referred to as the "entrance / exit direction motion vector integrated value") by accumulating the magnitude (scalar value) of the motion vector of each extracted block, and if the calculated entrance / exit direction motion vector integrated value is equal to or greater than a predetermined alighting determination threshold, it can determine that there is a passenger attempting to alight from the bus.

[0028] As a variant example, the disembarkation determination unit 42 can also use the integrated value obtained by accumulating the magnitude values ​​(scalar values) of the directional components of the motion vector of each block toward the bus entrance / exit (boarding / exit) as the motion vector integrated value in the entrance / exit direction.

[0029] Whether the direction of the motion vector of each block is heading toward the bus entrance / exit can be determined, for example, as follows, by defining a 360-degree range, with a motion vector direction directly upward being an angle of 0 degrees and a motion vector direction directly downward being an angle of 180 degrees. That is, as shown in Fig. 3, whether the direction of the motion vector of a certain block 3 is heading toward the bus entrance / exit is determined by determining whether the direction of the motion vector of that block 3 falls within the angular range of the apex angle of a triangle whose apex is the center of that block 3 and whose base is an imaginary entrance / exit line drawn at a part of the captured image that will be the bus entrance / exit (for example, the part of the boarding / alighting door), and if the direction of the motion vector of that block 3 falls within the angular range of the apex angle of the triangle, it can be determined that the direction of the motion vector of that block 3 is heading toward the bus entrance / exit.

[0030] As described above, according to this embodiment, when it is determined that there are many movements toward the bus entrance / exit in the captured image based on the direction and magnitude of the motion vectors of each block in the captured image, it is determined that there are passengers about to get off the bus. That is, according to this embodiment, without detecting passengers (people) in the captured image, it is determined whether there are passengers about to get off the bus by inferring the behavior of passengers in the captured image based on the direction and magnitude of the motion vectors of each block in the captured image. Therefore, it is possible to prevent this determination from being influenced by the environment inside the bus (for example, the degree of congestion, illuminance, etc.).

[0031] Based on the captured images received from camera 11, boarding determination unit 43 determines whether there are passengers about to board the bus in the boarding and alighting area outside the bus, as shown in Fig. 4. By sequentially inputting the captured images to a classifier, boarding determination unit 43 detects people appearing in the boarding and alighting area in the captured images and tracks the detected people in chronological order to determine whether there are passengers about to board the bus in the boarding and alighting area outside the bus. The classifier can be, for example, a convolutional neural network (CNN) having multiple convolution layers connected in series from the input side to the output side.

[0032] The determination of whether or not there are passengers in the boarding and disembarking area outside the bus may be made based on, for example, an image (exterior image) captured by a separate dedicated camera that is attached to the outside of the bus and captures the boarding and disembarking area of ​​the bus.

[0033] If the alighting determination unit 42 determines that there are passengers who are about to alight from the bus, the notification unit 44 issues a notification that there are passengers alighting. Also, if the boarding determination unit 43 determines that there are passengers who are about to board the bus, the notification unit 44 issues a notification that it is possible to board the bus.

[0034] FIG. 5 is a flowchart illustrating the monitoring control according to this embodiment, and in particular, a flowchart illustrating a process for notifying passengers that there are passengers getting off the bus when they get on or off the bus.

[0035] In step S1, based on the captured image received from the camera 11, the monitoring control device 30 detects the motion vector of each block in the monitoring range within the captured image.

[0036] In step S2, the monitoring control device 30 extracts from each block blocks those blocks that have a motion vector in a direction toward the bus entrance / exit (boarding / alighting door), and calculates an integrated value by accumulating the magnitude of the motion vector of each extracted block, i.e., an integrated value of the motion vector in the entrance / exit direction. As described above, the integrated value of the motion vector in the entrance / exit direction may be obtained by accumulating the magnitude of the directional component of the motion vector of each block that is toward the bus entrance / exit (boarding / alighting door).

[0037] In step S3, the monitoring control device 30 determines whether the integrated value of the motion vector in the entrance / exit direction is equal to or greater than a predetermined alighting determination threshold. If the integrated value of the motion vector in the entrance / exit direction is equal to or greater than the alighting determination threshold, the monitoring control device 30 determines that there is a passenger getting off the bus, and proceeds to processing in step S4. On the other hand, if the integrated value of the motion vector in the entrance / exit direction is less than the alighting determination threshold, the monitoring control device 30 determines that there is no passenger getting off the bus, and proceeds to processing in step S5.

[0038] In step S4, the monitoring control device 30 notifies the passengers that there are passengers disembarking using at least one of the display 21 and the speaker 22. At this time, the monitoring control device 30 may also notify the passengers that they are requested to board the bus after they have disembarked. While the notification is being made, the time that the boarding / alighting doors of the bus are open is automatically extended in the case of an automatically driven bus, for example, so that the boarding / alighting doors do not close.

[0039] In step S5, if the monitoring control device 30 has notified the user that there is a passenger disembarking, it ends the notification.

[0040] FIG. 6 is a flowchart illustrating the monitoring control according to this embodiment, and in particular, a flowchart illustrating the process for notifying passengers that they can board the bus when they get on or off the bus.

[0041] In step S11, the monitoring control device 30 determines whether any passengers are getting off the bus. If there are no passengers getting off the bus, i.e., if the disembarking notification has ended, the monitoring control device 30 proceeds to the processing of step S12. On the other hand, if there are passengers getting off the bus, i.e., if the disembarking notification is currently being implemented, the monitoring control device 30 ends the current processing.

[0042] In step S12, based on the captured image received from camera 11, monitoring control device 30 detects people in the captured image in the boarding and disembarking area outside the bus and tracks the detected people in chronological order to determine whether there are passengers about to board the bus in the boarding and disembarking area outside the bus. If there are passengers about to board the bus in the boarding and disembarking area outside the bus, monitoring control device 30 proceeds to processing in step S13. On the other hand, if there are no passengers about to board the bus in the boarding and disembarking area outside the bus, monitoring control device 30 proceeds to processing in step S14.

[0043] In step S13, the monitoring control device 30 notifies the passengers that it is possible to board the bus using at least one of the display 21 and the speaker 22. While the notification is being made, the time that the boarding / alighting doors of the bus are open is automatically extended in the case of an automatically operated bus, for example, so that the boarding / alighting doors do not close.

[0044] In step S14, if the monitoring control device 30 has notified the passenger that the bus is available for boarding, the monitoring control device 30 ends the notification. After the notification ends, for example, in the case of an automatically operated bus, when the departure time arrives, the boarding doors are closed and the bus departs.

[0045] The monitoring control device 30 according to the present embodiment described above is configured to monitor the behavior of objects within a predetermined monitoring area based on captured images of the area. Specifically, the monitoring control device 30 divides a predetermined range of the captured image into multiple blocks, detects a motion vector for each block, and determines whether an object heading toward an entrance or exit of the monitoring area is present in the captured image based on the direction and magnitude of the motion vector for each block, without detecting the object itself in the captured image.

[0046] More specifically, the monitoring control device 30 is configured to extract from each block a block whose motion vector direction is oriented toward the entrance / exit direction of the monitored area, calculate the integrated value of the magnitude of the motion vector of the extracted block as the integrated motion vector value in the entrance / exit direction, and determine that an object heading toward the entrance / exit direction of the monitored area is present if the integrated motion vector value in the entrance / exit direction is equal to or greater than the alighting determination threshold (predetermined threshold).Alternatively, the monitoring control device 30 is configured to calculate the integrated value of the magnitude of the component of the motion vector of each block in the entrance / exit direction of the monitored area as the integrated motion vector value in the entrance / exit direction, and determine that an object heading toward the entrance / exit direction of the monitored area is present if the integrated motion vector value in the entrance / exit direction is equal to or greater than the alighting determination threshold (predetermined threshold).

[0047] According to this embodiment, the behavior of an object in a captured image of a specific area can be estimated based on the direction and magnitude of the motion vector of each block in the captured image, without detecting the object in the captured image. Therefore, when determining whether an object is present in the captured image heading toward the entrance / exit of the area, the determination can be prevented from being influenced by the environment within the area, such as the degree of congestion or illuminance of the area.

[0048] In addition, the monitoring control device 30 of this embodiment is configured to monitor the behavior of passengers on the bus based on captured images of the passenger area inside the bus, and when there are passengers on the bus heading towards the bus entrance / exit, it is configured to notify the presence of passengers disembarking via the alarm device 20 installed on the bus.

[0049] This allows passengers who are about to board the bus, for example, outside the bus, to be notified of the presence of passengers heading toward the bus entrance / exit via the alarm device 20. This allows passengers who are about to board the bus to be urged to wait outside the bus until passengers inside the bus have disembarked, preventing confusion between passengers getting off the bus and passengers getting on the bus. As a result, safety when boarding and disembarking can be improved and the time it takes to board and disembark can be reduced.

[0050] In this embodiment, the monitoring range (predetermined range) of the captured image is a range within the bus in which passengers can move, but does not include the windows of the bus.

[0051] This eliminates the need to detect motion vectors that indicate the direction and size of movement of objects outside the window, making it possible to accurately estimate the behavior of passengers inside the vehicle.

[0052] In addition, when there are no passengers inside a bus with its boarding / alighting doors open and an outside passenger is detected in front of the bus's boarding / alighting doors based on an image taken or an exterior image of the outside of the bus, the monitoring control device 30 of this embodiment is configured to notify via the alarm device 20 that it is possible to board the bus.

[0053] This allows, for example, passengers who are outside the bus and are about to board the bus to be notified of the appropriate time to board the bus.

[0054] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments.

[0055] Also, for example, in the above embodiment, the computer program executed in the monitoring control device 30 may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. [Explanation of symbols]

[0056] 20. Alarm device 30 Monitoring and control device

Claims

1. A monitoring control device configured to monitor the behavior of an object within a predetermined area based on a captured image of the area, Dividing a predetermined range of the captured image into a plurality of blocks and detecting a motion vector for each block; and determining whether an object moving toward an entrance / exit direction of the area is present in the captured image without detecting the object in the captured image, based on the direction and magnitude of the motion vector of each block. Monitoring and control equipment.

2. extracting, from each of the blocks, the block whose direction of the motion vector is directed toward the entrance / exit direction of the area; calculating an integrated value of the magnitude of the motion vector of the extracted block; If the integrated value is equal to or greater than a predetermined threshold, it is determined that the object heading in the direction of the entrance / exit of the area is present. The monitoring and control device according to claim 1 .

3. calculating an integrated value of the magnitude of the component of the motion vector of each block in the entrance / exit direction of the area; If the integrated value is equal to or greater than a predetermined threshold, it is determined that the object heading in the direction of the entrance / exit of the area is present. The monitoring and control device according to claim 1 .

4. the area is the interior of a bus and the object is a passenger; When there is a passenger on the bus heading toward the bus entrance / exit, the presence of the passenger getting off is notified via an alarm device provided on the bus. The monitoring and control device according to any one of claims 1 to 3.

5. the predetermined range of the captured image is a range within which the passenger can move inside the bus, and does not include windows of the bus; The monitoring and control device according to claim 4.

6. When there is no passenger heading toward the entrance / exit direction inside the bus with the boarding / alighting door open, and a passenger outside the bus is detected in front of the boarding / alighting door based on the captured image or an exterior image of the outside of the bus, the system is configured to notify via the notification device that the passenger is able to board the bus. The monitoring and control device according to claim 4.

7. A monitoring method for monitoring the behavior of an object in a predetermined area based on a captured image of the area, comprising: Dividing a predetermined range of the captured image into a plurality of blocks and detecting a motion vector for each block; determining whether an object heading toward an entrance / exit direction of the area is present in the captured image without detecting the object in the captured image based on the direction and magnitude of the motion vector of each block; Monitoring method.

8. A computer program for a monitoring control device configured to monitor behavior of an object within a predetermined area based on a captured image of the area, Dividing a predetermined range of the captured image into a plurality of blocks and detecting a motion vector for each block; causing the monitoring control device to execute a process of determining whether an object heading toward an entrance / exit direction of the area is present in the captured image without detecting the object in the captured image, based on the direction and magnitude of the motion vector of each of the blocks; Computer program.

Images