Monitoring system, monitoring device, monitoring method, and program

Abstract

To provide a technology capable of notifying a unique behavior of a specific kind of person on a station platform.SOLUTION: According to an embodiment, a monitoring system 100 includes an imaging unit 110 installed on a platform PF of a station ST to acquire an image of the platform PF, a behavior recognition unit 150 for recognizing a behavior of a specific kind of person on the basis of the image of the platform PF, and a notification unit 160 for notifying a recognition result by the behavior recognition unit 150. For example, the monitoring system 100 includes a person recognition unit 120 for recognizing persons included in the image of the platform PF, and a person classification unit 130 for classifying the persons recognized by the person recognition unit 120 on the basis of the image of the platform PF, and the behavior recognition unit 150 recognizes a behavior of a person classified to the specific kind by the person classification unit 130 on the basis of the image of the platform PF.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a monitoring system and the like.

Background Art

[0002] For example, a technology for detecting a specific type of person in a station building and notifying station staff or the like is known (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, although a specific type of person can be detected, it is not possible to detect and notify the specific actions unique to that type of person.

[0005] Therefore, in view of the above problems, an object of the present disclosure is to provide a technology capable of detecting and notifying specific actions unique to a specific type of person on a station platform.

Means for Solving the Problems

[0006] To achieve the above object, in one embodiment of the present disclosure, an imaging unit installed on a station platform for acquiring an image of the platform, an action recognition unit for recognizing the actions of a specific type of person based on the image, and a notification unit for notifying the recognition result by the action recognition unit are provided. A monitoring system is provided.

[0007] Also, in another embodiment of the present disclosure, An action recognition unit that recognizes the actions of a specific type of person based on an image of the platform obtained by an imaging unit installed on the platform of the station, A notification unit that notifies the recognition result by the action recognition unit, and A monitoring device is provided.

[0008] In still another embodiment of the present disclosure, An information processing device includes an action recognition step of recognizing the actions of a specific type of person based on an image of the platform obtained by an imaging unit installed on the platform of the station, An information processing device includes a notification step of notifying the recognition result in the action recognition step, and A monitoring method is provided.

[0009] In still another embodiment of the present disclosure, In an information processing device, An action recognition step of recognizing the actions of a specific type of person based on an image of the platform obtained by an imaging unit installed on the platform of the station, A notification step of notifying the recognition result in the action recognition step, are executed, A program is provided.

Advantages of the Invention

[0010] According to the above-described embodiment, it is possible to detect and notify actions specific to a specific type of person on the platform of the station.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described with reference to the drawings.

[0013] [First Example of Monitoring System] With reference to FIGS. 1 to 6, a first example of the monitoring system 100 according to the present embodiment will be described.

[0014] [Configuration] FIG. 1 is a diagram showing a first example of the monitoring system 100. FIG. 2 is a diagram showing an example of an imaging image (image IM2) acquired by the imaging unit 110. FIG. 3 is a diagram for explaining an example of the processing by the person recognition unit 120. FIG. 4 is a diagram for explaining an example of the processing by the person classification unit 130. FIG. 5 is a diagram showing another example of an imaging image (image IM5) acquired by the imaging unit 110.

[0015] The monitoring system 100 is installed at the station ST and monitors the situation of the platform PF within the station ST and the railway vehicle 200 that enters the platform PF and stops.

[0016] For example, the monitoring system 100 monitors the presence or absence of specific types of people's unique behaviors on the platform PF and notifies the monitoring results. Specific types of people are, for example, station staff. In this case, the unique behavior is, for example, signals made by station staff using hand flags or signal lights. Station staff may, for example, assist and guide passengers using wheelchairs or white canes to board the railway vehicle 200 and in such cases, may notify the crew such as the driver of the guiding situation using hand flags, signal lights, etc.

[0017] The monitoring system 100 includes an imaging unit 110, a person recognition unit 120, a person classification unit 130, a posture estimation unit 140, an action recognition unit 150, and a notification unit 160.

[0018] The imaging unit 110 is provided on the platform PF and images an imaging range including the platform PF and the railway vehicle 200 that is entering or stopped on the platform PF. The imaging range includes the location adjacent to the railway vehicle 200 that is entering or stopped on the platform PF. Thereby, the imaging unit 110 can acquire an imaging image including station staff who perform various operations on the railway vehicle 200 that is entering or stopped on the platform PF.

[0019] For example, the imaging unit 110 is a monocular surveillance camera permanently installed on the platform PF and is attached to the ceiling of the platform PF.

[0020] For example, as shown in FIG. 1, one imaging unit 110 is provided, and its imaging range is set to cover the entire monitoring range. Alternatively, a plurality of imaging units 110 may be provided, and the imaging ranges of each of them may be set by the plurality of imaging units to cover the entire monitoring range.

[0021] For example, during the business hours of station ST, the imaging unit 110 acquires an image every 1 / 30 second and outputs its data. The imaging unit 110 is communicably connected to the person recognition unit 120 through a one-to-one communication line, a local area network (LAN), or the like, and the output of the imaging unit 110 (specifically, the data of the image) is taken in from the imaging unit 110 by the person recognition unit 120. Also, the imaging unit 110 may be communicably connected to the person classification unit 130 through a one-to-one communication line, a local area network, or the like, and the output of the imaging unit 110 (the data of the image) may be taken in from the imaging unit 110 by the person classification unit 130.

[0022] Based on the image acquired by the imaging unit 110, the person recognition unit 120 performs image analysis to recognize the person included in (i.e., shown in) the image.

[0023] At each predetermined processing cycle, based on the latest image data input from the imaging unit 110, the person recognition unit 120 performs a person recognition process on the image and outputs information regarding the result of the person recognition process, including the result of the person recognition process.

[0024] The result of the human recognition process includes, for example, information indicating the presence or absence of human recognition. Further, the result of the human recognition process may include information representing a region (e.g., a rectangular region) in the entire image where a person is included when the person is recognized. The region in the entire image where a person is included is, for example, a region including the entire body of the person. The region in the entire image where a person is included is usually a partial region in the entire image, but depending on how the person appears in the image, etc., it may be extracted as the entire region of the image. Further, the result of the human recognition process may include data of an image (hereinafter, "partial image") of a region in the entire image input from the imaging unit 110 and including the person recognized by the human recognition unit 120. Further, instead of the data of the partial image, the human recognition unit 120 may output the original data (i.e., the latest image data) acquired by the imaging unit 110 together with information representing the region in the entire image input from the imaging unit 110 and including a person.

[0025] The function of the human recognition unit 120 may be realized by any hardware, or any combination of hardware and software, etc. For example, the human recognition unit 120 is mainly configured around a computer (information processing device) including a CPU (Central Processing Unit), a memory device, an auxiliary storage device, an interface device, etc. The memory device is, for example, SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), etc. The auxiliary storage device is, for example, HDD (Hard Disc Drive), SSD (Solid State Drive), flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), etc. The interface device includes, for example, a communication interface for communicating with other components of the monitoring system 100 such as the imaging unit 110 and the human classification unit 130. In addition, the interface device includes, for example, an external interface for connecting to a recording medium. Thereby, the human recognition unit 120 can, for example, through the external interface, take in various programs from the recording medium and install them in the auxiliary storage device. Further, the information processing device may further include a high-speed arithmetic device that operates in conjunction with the CPU. The high-speed arithmetic device includes, for example, GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc.

[0026] For example, based on local feature amounts obtained from the image data or the image data itself, the human recognition unit 120 applies the learned model LM1 to recognize a person in the image and estimate the area on the image that includes that person.

[0027] The learned model LM1 is obtained, for example, by machine learning (i.e., supervised learning) of the base learning model M1 using the teacher dataset TRD1. The learned model LM1 may be generated by machine learning of the learning model M1 in the human recognition unit 120, or may be generated by machine learning of the learning model M1 in an information processing device different from the human recognition unit 120. For example, each piece of teacher data included in the teacher dataset TRD1 is a combination of an image feature amount or the image data itself obtained from the image data as input data and data specifying an area (i.e., a partial image) in which a person on the image is included as output data. Thereby, the learned model LM1 can recognize a person on the image using the image feature amount or the image data itself obtained from the image data as input, and output data of an area (e.g., a rectangular area) in which the person on the image is included. For example, the learned model LM1 is mainly configured by a deep neural network (DNN: Deep Neural Network), and the machine learning of the DNN is made more efficient by applying backpropagation (error backpropagation method) based on the teacher data. Further, the learned model LM1 may include a U-Net capable of recognizing a person from image data with the image data as input.

[0028] Further, the human recognition unit 120 may recognize a person in the image by applying a rule-based algorithm such as pattern matching based on feature points, local feature amounts, etc. of the image data.

[0029] For example, as shown in FIG. 2, in the latest image IM2 acquired by the imaging unit 110, a railway vehicle 200 that enters adjacent to the platform PF, a platform door PD installed on the platform PF, and passengers PS and station staff SA standing on the platform PF are shown. For example, as shown in FIG. 3, the human recognition unit 120 recognizes the passengers PS and the station staff SA as persons from the latest image IM2 acquired by the imaging unit 110, and outputs data of partial images of rectangular areas AR1 and rectangular areas AR2 that include the entire bodies of the passengers PS and the station staff SA, respectively.

[0030] In the present specification, the term "passenger" is used with the intention of including all persons who plan to board the railway vehicle 200, persons who are boarding the railway vehicle 200, and persons who have alighted after boarding the railway vehicle 200.

[0031] The person recognition unit 120 is communicably connected to the person classification unit 130 through a one-to-one communication line, a local network, or the like. The output of the person recognition unit 120 (specifically, information regarding the result of the person recognition process, etc.) is taken in from the person recognition unit 120 by the person classification unit 130. The person recognition unit 120 is communicably connected to the posture estimation unit 140 through a one-to-one communication line, a local network, or the like, and the output of the person recognition unit 120 (information regarding the result of the person recognition process, etc.) may be taken in from the person recognition unit 120 by the posture estimation unit 140.

[0032] The person classification unit 130 classifies the person recognized by the person recognition unit 120 based on the output of the person recognition unit 120. Specifically, the person classification unit 130 classifies the person recognized by the person recognition unit 120 into any one of a plurality of predefined types of attributes based on the data of the partial image including the person recognized by the person recognition unit 120. The data of the partial image including the person recognized by the person recognition unit 120 is input from the person recognition unit 120 to the person classification unit 130, for example, as part of the information regarding the result of the person recognition process from the person recognition unit 120. Also, the data of the partial image including the person recognized by the person recognition unit 120 may be obtained by being cut out from the latest image acquired by the imaging unit 110 based on the output of the person recognition unit 120. In the latter case, the latest image data acquired by the imaging unit 110 may be transmitted from the person recognition unit 120 to the person classification unit 130, or may be directly transmitted from the imaging unit 110 to the person classification unit 130.

[0033] Each time the output of the human recognition unit 120 is input, the human classification unit 130 performs classification processing on the humans included in the partial image, and outputs information regarding the result of the classification processing, including the result of the classification processing. The result of the classification processing includes, for example, label information representing the classified human attributes for each region (i.e., partial image) corresponding to the human recognized by the human recognition unit 120. Further, the result of the classification processing may include information representing the region including the human corresponding to the label information in the entire image acquired by the imaging unit 110. Further, the result of the classification processing may include data of the partial image of the region including the human corresponding to the label information in the entire image acquired by the imaging unit 110. Further, instead of the data of the partial image, the human classification unit 130 may output the latest image data acquired by the imaging unit 110 together with information representing the region including the human corresponding to the label information in the entire image.

[0034] The function of the human classification unit 130 may be realized by any hardware or any combination of hardware and software. For example, the human classification unit 130 is mainly configured by a computer (information processing device) including a CPU, a memory device, an auxiliary storage device, an interface device, and the like. The memory device is, for example, SRAM, DRAM, or the like. The auxiliary storage device is, for example, HDD, SSD, flash memory, EEPROM, or the like. The interface device includes, for example, a communication interface for communicating with other components of the monitoring system 100 such as the human recognition unit 120 and the posture estimation unit 140. Further, the interface device includes, for example, an external interface for connecting to a recording medium. Thereby, the human classification unit 130 can, for example, take in various programs from the recording medium through the external interface and install them in the auxiliary storage device. Further, the information processing device may further include a high-speed arithmetic device that operates in conjunction with the CPU. The high-speed arithmetic device includes, for example, GPU, FPGA, ASIC, or the like.

[0035] For example, the human classifier 130 applies the learned model LM2 based on local feature amounts obtained from data of a partial image corresponding to a region including a human in the entire image and data of the partial image, and estimates which attribute among a plurality of types of attributes the human included in the partial image corresponds to. The learned model LM2 corresponds to a classifier that classifies a human in a partial image into any one of a plurality of types of attributes, and may be a set of a plurality of classifiers that classify whether or not each corresponds to each of the plurality of types of attributes.

[0036] The learned model LM2 is obtained, for example, by machine learning (i.e., supervised learning) of the base learning model M2 using the teacher dataset TRD2. The learned model LM2 may be generated by machine learning of the learning model M2 in the human classifier 130, or may be generated by machine learning of the learning model M2 in an information processing device different from the human classifier 130. For example, each piece of teacher data included in the teacher dataset TRD2 is a combination of a local feature amount obtained from data of an image including a human as input data or the image data itself and data of a label representing an attribute of the human included in the image as output data. Thereby, the learned model LM2 can estimate the attribute of the human included in the image using the image feature amount obtained from the data of the image including the human or the image data itself as input, and output data of a label representing the attribute of the human included in the image. For example, the learned model LM2 is mainly configured by a DNN, and the machine learning of the DNN is made more efficient by applying backpropagation based on teacher data. Also, the learned model LM2 may be a support vector machine (SVM).

[0037] Further, the human classifier 130 may apply a rule-based algorithm such as pattern matching based on feature points, local feature amounts, etc. of data of a partial image including a human, and classify the attributes of the human included in the partial image.

[0038] For example, as shown in FIG. 4, the person classification unit 130 classifies a person included in a partial image into either a "station staff member" or a "person other than station staff member" using the data of the partial image as input. Persons other than station staff members include, for example, general passengers getting on or off the train, as well as workers such as cleaners. In this example, the person classification unit 130 classifies the person (passenger PS) included in the partial image of the rectangular area AR1 of the image IM2, which is output from the person recognition unit 120, as a "person other than station staff member". Also, the person classification unit 130 classifies the person (station staff member SA) included in the partial image of the rectangular area AR2, which is output from the person recognition unit 120, as a "station staff member".

[0039] Note that the person classification unit 130 may classify a person included in a partial image into one of three or more attributes including "station staff member" based on the data of the partial image including a person, which is output from the person recognition unit 120.

[0040] The person classification unit 130 is communicably connected to the posture estimation unit 140 through a one-to-one communication line, a local network, or the like, and the output of the person classification unit 130 (specifically, information regarding the result of the classification process, etc.) is taken in from the person classification unit 130 to the posture estimation unit 140. Also, the output of the person classification unit 130 may include the output of the person recognition unit 120 (information regarding the result of the person recognition process, etc.). That is, the output of the person recognition unit 120 may be taken in to the posture estimation unit 140 via the person classification unit 130.

[0041] The posture estimation unit 140 estimates the posture state of a person classified into a specific type by the person classification unit 130 based on the data of the partial image including the person classified into a specific type by the person classification unit 130. For example, the posture estimation unit 140 estimates the positions and orientations of a plurality of parts of the body in the partial image including the person classified into a specific type by the person classification unit 130. The plurality of parts of the body include, for example, the ankles, knees, waist, chest, shoulders, elbows, wrists, neck, head, etc. Also, the posture estimation unit 140 may estimate which of a plurality of types of pre-specified posture states the posture state of the person classified into a specific type by the person classification unit 130 corresponds to.

[0042] Whenever the output of the person classification unit 130 is input, the posture estimation unit 140 performs a posture estimation process on the person classified into a specific type by the person classification unit 130, and outputs information regarding the result of the person posture estimation process, including the result of the person posture estimation process. The result of the person posture estimation process includes, for example, data representing the posture state of the person for each region (i.e., partial image) corresponding to the person classified into a specific type by the person classification unit 130 within the entire image acquired by the imaging unit 110. Further, the result of the person posture estimation process may include information representing the region including the person corresponding to the data representing the posture state of the person within the entire image acquired by the imaging unit 110. Further, the result of the person posture estimation process may include data of the partial image of the region including the person corresponding to the data representing the posture state of the person within the entire image acquired by the imaging unit 110. Further, instead of the data of the partial image, the posture estimation unit 140 may output the latest image data acquired by the imaging unit 110 together with the information representing the region including the person corresponding to the data representing the posture state of the person.

[0043] The function of the posture estimation unit 140 may be realized by any hardware or any combination of hardware and software, etc. For example, the posture estimation unit 140 is configured around a computer (information processing device) including a CPU, a memory device, an auxiliary storage device, an interface device, etc. The memory device is, for example, SRAM, DRAM, etc. The auxiliary storage device is, for example, HDD, SSD, flash memory, EEPROM, etc. The interface device includes, for example, a communication interface for communicating with other components of the monitoring system 100 such as the person classification unit 130 and the behavior recognition unit 150. Further, the interface device includes, for example, an external interface for connecting to a recording medium. Thereby, the posture estimation unit 140 can, for example, take in various programs from the recording medium through the external interface and install them in the auxiliary storage device. Further, the information processing device may further include a high-speed arithmetic device that operates in conjunction with the CPU. The high-speed arithmetic device includes, for example, GPU, FPGA, ASIC, etc.

[0044] For example, the posture estimation unit 140 applies the learned model LM3 based on local feature amounts obtained from data of a partial image including a person classified into a specific type by the person classification unit 130 and the data of the partial image, and estimates the posture state of the person included in the data of the partial image.

[0045] The learned model LM3 is obtained, for example, by machine learning (i.e., supervised learning) of the base learning model M3 using the teacher dataset TRD3. The learned model LM3 may be generated by machine learning of the learning model M3 in the posture estimation unit 140, or may be generated by machine learning of the learning model M3 in an information processing device different from the posture estimation unit 140. For example, each piece of teacher data included in the teacher dataset TRD3 is a combination of a local feature amount obtained from data of an image including a person as input data or the data of the image itself, and data representing the posture state of the person included in the image as output data. Thereby, the learned model LM3 can estimate the posture state of the person included in the image using, as input, the image feature amount obtained from the data of the image including the person or the data of the image itself, and output data representing the posture state of the person. For example, the learned model LM3 is mainly configured by a DNN, and the machine learning of the DNN is made more efficient by applying backpropagation based on the teacher data. Further, when estimating which of a plurality of types of posture states in which the posture state of the person included in the image is predefined the learned model LM3 may be an SVM as a classifier.

[0046] Further, the posture estimation unit 140 may apply a rule-based algorithm based on feature points, local feature amounts, etc. of the data of the partial image including a person, and estimate which of a plurality of types of posture states in which the posture state of the person included in the partial image is predefined the posture state corresponds to.

[0047] The posture estimation unit 140 is communicably connected to the action recognition unit 150 through a one-to-one communication line, a local network, or the like, and the output of the posture estimation unit 140 (specifically, information regarding the result of the human posture estimation process, etc.) is taken into the action recognition unit 150. Further, the output of the posture estimation unit 140 may include the output of the person recognition unit 120 (information regarding the result of the human recognition process, etc.) and the output of the person classification unit 130 (information regarding the result of the classification process, etc.). That is, the output of the person recognition unit 120 and the output of the person classification unit 130 may be taken into the action recognition unit 150 via the posture estimation unit 140.

[0048] Based on the output of the posture estimation unit 140 (such as the result of the posture estimation process), the action recognition unit 150 recognizes a specific action of a person classified into a specific type by the person classification unit 130. For example, based on the data representing the human posture state output from the posture estimation unit 140, the action recognition unit 150 determines whether the posture state of a person classified into a specific type by the person classification unit 130 and its temporal change represent a specific action, thereby recognizing the specific action of that person.

[0049] Every time the output of the posture estimation unit 140 is input, the action recognition unit 150 performs an action recognition process for a person classified into a specific type by the person classification unit 130, and outputs information regarding the result of the action recognition process, including the result of the action recognition process. The result of the action recognition process includes, for example, information indicating the presence or absence of a specific action for each person classified into a specific type by the person classification unit 130.

[0050] The functions of the action recognition unit 150 may be realized by any hardware, any combination of hardware and software, or the like. For example, the action recognition unit 150 is mainly configured around a computer (information processing device) including a CPU, a memory device, an auxiliary storage device, an interface device, and the like. The memory device is, for example, SRAM, DRAM, or the like. The auxiliary storage device is, for example, HDD, SSD, flash memory, EEPROM, or the like. The interface device includes, for example, a communication interface for communicating with other components of the monitoring system 100 such as the posture estimation unit 140 and the notification unit 160. Further, the interface device includes, for example, an external interface for connecting to a recording medium. Thereby, the action recognition unit 150 can, for example, take in various programs from the recording medium through the external interface and install them in the auxiliary storage device. Further, the information processing device may further include a high-speed arithmetic device that operates in conjunction with the CPU. The high-speed arithmetic device includes, for example, GPU, FPGA, ASIC, or the like.

[0051] For example, the action recognition unit 150 determines the presence or absence of a specific action by a person classified into a specific type by the person classification unit 130 based on a comparison between data representing the posture state of the person classified into a specific type by the person classification unit 130 and reference data representing the posture state corresponding to a specific action. Further, the action recognition unit 150 may determine the presence or absence of a specific action of the person based on a comparison between data representing the temporal change of the posture state of the person classified into a specific type by the person classification unit 130 and reference data representing the temporal change of the posture state corresponding to a specific action.

[0052] For example, the action recognition unit 150 recognizes the action of signaling to the crew of the railway vehicle 200 of a person classified as a "station staff member" by the person classification unit 130. The crew of the railway vehicle 200 includes, for example, a driver, a conductor, and the like. The action of signaling to the crew of the railway vehicle 200 is, for example, an action of waving a flag or a signal lamp at a position higher than the shoulder or the head. Thus, as shown in FIG. 5, when the image IM5 is acquired by the imaging unit 110, the action recognition unit 150 can recognize the action of the station staff member SA waving the flag HF based on the results of the processing of the person recognition unit 120, the person classification unit 130, and the posture estimation unit 140.

[0053] The action recognition unit 150 is communicably connected through a one-to-one communication line, a local network, or the like to the notification unit 160, and the output of the action recognition unit 150 (information regarding the result of the action recognition process, etc.) is taken in from the action recognition unit 150 by the notification unit 160.

[0054] The notification unit 160 notifies the person related to the operation of the railway vehicle 200 of the recognition result by the action recognition unit 150, that is, the result of the recognition process of a specific action of a person classified into a specific type by the person classification unit 130. For example, the notification unit 160 may notify to that effect only when a specific action of a person classified into a specific type is recognized by the action recognition unit 150, or may notify the result of the recognition process of the action recognition unit 150 regardless of the presence or absence of the recognition of a specific action.

[0055] Those related to the operation of the railway vehicle 200 include, for example, the crew of the railway vehicle 200, the station staff in the platform PF, the monitors in the command room, etc. When notifying the crew of the railway vehicle 200, the notification unit 160 transmits, for example, a signal including the notification content to the railway vehicle 200 through a predetermined communication line (for example, a wireless communication line such as WiFi or Bluetooth (registered trademark)), and causes a display device installed on the driver's cab or the like of the railway vehicle 200 to display the notification content. Further, the notification unit 160 may transmit a signal including the notification content to a portable terminal device (that is, a mobile terminal) or the like brought into the railway vehicle 200 by the crew of the railway vehicle 200, and cause the portable terminal device to display the notification content. Thereby, the crew of the railway vehicle 200 can surely confirm the notification content, for example, while being inside the railway vehicle 200. When notifying the station staff in the platform PF or the like, the notification unit 160 causes, for example, a display unit installed in the platform PF to display the notification content. When notifying the monitors in the command room or the like, a signal including the notification content is transmitted through a predetermined communication line, and the notification content is displayed on the display unit in the command room.

[0056] The function of the notification unit 160 may be realized by any hardware or any combination of hardware and software. For example, the notification unit 160 is mainly configured around a computer (information processing device) including a CPU, a memory device, an auxiliary storage device, an interface device, etc. The memory device is, for example, SRAM, DRAM, or the like. The auxiliary storage device is, for example, HDD, SSD, flash memory, EEPROM, or the like. The interface device includes, for example, a communication interface for communicating with other components of the monitoring system 100 such as the behavior recognition unit 150. Further, the interface device includes, for example, an external interface for connecting to a recording medium. Thereby, the notification unit 160 can, for example, take in various programs from the recording medium through the external interface and install them in the auxiliary storage device. Further, the information processing device may further include a high-speed arithmetic device that operates in conjunction with the CPU. The high-speed arithmetic device includes, for example, GPU, FPGA, ASIC, or the like.

[0057] For example, the notification unit 160 notifies the crew of the railway vehicle 200 of the recognition result of the presence or absence of a specific action of a person classified into a specific type by the action recognition unit 150. Thereby, the crew of the railway vehicle 200 can appropriately grasp the presence or absence of a specific action of a specific type of person on the platform PF where the railway vehicle 200 is stopped. As a result, the flow from the stop to the departure of the railway vehicle 200 can proceed smoothly.

[0058] For example, as shown in FIG. 5, when the station staff SA is sending a signal by waving a hand flag HF, the notification unit 160 can notify the crew of the railway vehicle 200, such as the driver, that a person (station staff SA) classified as a "station staff" is performing a specific action (sending a signal by waving a hand flag HF). Therefore, by checking the signal from the station staff SA, the driver or the like can surely grasp the start and end of guiding passengers who need assistance, such as those using wheelchairs or white canes, on the railway vehicle 200.

[0059] <Processing> FIG. 6 is a flowchart schematically showing a first example of the processing of the monitoring system 100.

[0060] This flowchart is started, for example, when the railway vehicle 200 arrives at and stops at the platform PF.

[0061] As shown in FIG. 6, in step S102, the person recognition unit 120 acquires the latest captured image from the imaging unit 110 (monitoring camera).

[0062] When the processing in step S102 is completed, the person recognition unit 120 proceeds to step S104.

[0063] In step S104, the person recognition unit 120 executes a process of recognizing a person (person recognition process) reflected in the captured image acquired in step S102.

[0064] When the processing in step S104 is completed, the person recognition unit 120 outputs information regarding the result of the person recognition process to the person classification unit 130.

[0065] When the information regarding the result of the person recognition process is input from the person recognition unit 120 to the person classification unit 130, the person classification unit 130 executes the process of step S106.

[0066] In step S106, the person classification unit 130 determines whether a person has been recognized (i.e., detected) based on the information regarding the result of the recognition process of the person recognition unit 120. If a person has been recognized by the person recognition unit 120, the person classification unit 130 proceeds to step S108. On the other hand, if a person has not been recognized by the person recognition unit 120, the person classification unit 130 does not execute the process of step S108 and outputs a notification signal indicating that the process is aborted to the person recognition unit 120.

[0067] In step S108, for each person recognized by the person recognition unit 120, the person classification unit 130 executes a classification process (classification processing) to classify the target person into either "station staff" or "person other than station staff" based on the data of the partial image including the person.

[0068] When the process of step S108 is completed, the person classification unit 130 outputs the information regarding the result of the classification process to the posture estimation unit 140.

[0069] When the information regarding the result of the classification process from the person classification unit 130 is input to the posture estimation unit 140, the posture estimation unit 140 executes the process of step S110.

[0070] In step S110, the posture estimation unit 140 determines whether there is a person classified as "station staff" among the persons recognized by the person recognition unit 120 based on the information regarding the result of the classification process of the person classification unit 130. If there is a person classified as "station staff", the posture estimation unit 140 proceeds to step S112. On the other hand, if there is no person classified as "station staff", the posture estimation unit 140 does not execute the process of step S112 and outputs a notification signal indicating that the process is aborted to the person recognition unit 120.

[0071] In step S112, the posture estimation unit 140 performs the posture estimation process of the person based on the data of the partial image including the person classified as "station staff" by the person classification unit 130.

[0072] When the process of step S112 is completed, the posture estimation unit 140 outputs information regarding the result of the human posture estimation process to the action recognition unit 150.

[0073] When information regarding the result of the posture estimation process is input from the posture estimation unit 140, the action recognition unit 150 executes the process of step S114.

[0074] In step S114, the action recognition unit 150 executes an action recognition process for a person classified as a "railway employee" by the person classification unit 130 based on the information regarding the result of the posture estimation process of the posture estimation unit 140.

[0075] When the process of step S114 is completed, the action recognition unit 150 outputs information regarding the result of the action recognition process to the notification unit 160.

[0076] When information regarding the result of the action recognition process is input from the action recognition unit 150, the notification unit 160 executes the process of step S116.

[0077] In step S116, the notification unit 160 determines whether or not an action of signaling to the driver by a person classified as a "railway employee" (for example, signaling with flags or a signal lamp at a position higher than the shoulder or head) has been recognized by the action recognition unit 150 based on the result of the action recognition process by the action recognition unit 150. In this example, signaling to the driver means the end of the guidance work for boarding the railway vehicle 200 for a person with a disability using a wheelchair or a white cane. If the notification unit 160 recognizes an action of signaling to the driver by a person classified as a "railway employee" by the action recognition unit 150, it proceeds to step S118. On the other hand, if the notification unit 160 does not recognize an action of signaling to the driver by a person classified as a "railway employee" by the action recognition unit 150, it does not execute the process of step S118 and outputs a notification signal indicating the termination of the process to the person recognition unit 120.

[0078] In step S118, the notification unit 160 notifies the driver of the railway vehicle 200 that there has been a signal from the railway employee.

[0079] When the process of step S118 is completed, the notification unit 160 ends the process, and the end of this process causes the process of this flowchart to end.

[0080] On the other hand, when a notification signal to abort the process is input from the person classification unit 130, the posture estimation unit 140, or the notification unit 160, the person recognition unit 120 executes the process of step S120.

[0081] In step S120, the person recognition unit 120 determines whether or not the railway vehicle 200 has departed. The person recognition unit 120 may determine whether or not the railway vehicle 200 has departed based on whether or not a departure signal received from the driver's cab or the railway vehicle 200 is received, or may determine whether or not the railway vehicle 200 has departed by recognizing the presence or absence of movement of the railway vehicle 200 or the opening / closing state of the platform door, etc. from the latest captured image of the imaging unit 110. When the railway vehicle 200 has not departed, the person recognition unit 120 returns to step S102 and executes the process of step S102 again. On the other hand, when the railway vehicle 200 has departed, the person recognition unit 120 ends the process, and the end of this process causes the process of this flowchart to end.

[0082] As described above, in this example, the monitoring system 100 monitors whether or not a signal is given from the station staff on the platform PF to the driver based on the image acquired by the imaging unit 110 between the stop and departure of the railway vehicle 200, and when a signal is recognized to be given to the driver, the driver can be notified to that effect. Therefore, the driver can surely grasp that the guidance of the station staff on the platform PF to the railway vehicle 200 for a person with a disability using a wheelchair, a white cane, etc. has been completed based on the notification.

[0083] In particular, in the case of one-person operation in which only the driver rides, it may be difficult for the driver of the railway vehicle 200 to surely grasp the situation of the platform PF. In contrast, in this example, the monitoring system 100 can notify that a signal has been given from the station staff to the driver, support the confirmation work of the signal from the station staff to the driver, and reduce the burden on the driver's confirmation work.

[0084] [Second Example of Surveillance System] Next, with reference to FIGS. 7 to 9, a second example of the surveillance system 100 according to the present embodiment will be described.

[0085] Hereinafter, in this example, the same or corresponding configurations as those in the above-described first example are denoted by the same reference numerals, and the description will be centered on the parts different from the above-described first example, and the same or corresponding descriptions as those in the above-described first example may be omitted.

[0086] [Configuration] FIG. 7 is a diagram showing a second example of the surveillance system 100. FIG. 8 is a diagram for explaining an example of processing by the region extraction unit 154.

[0087] In this example, the surveillance system 100 is different from the above-described first example in that the person classification unit 130 is omitted and the configuration of the action recognition unit 150 is changed.

[0088] The surveillance system 100 includes an imaging unit 110, a person recognition unit 120, a posture estimation unit 140, an action recognition unit 150, and a notification unit 160.

[0089] The person recognition unit 120 is communicably connected to the posture estimation unit 140 through a one-to-one communication line, a local network, or the like, and the output of the person recognition unit 120 (such as the result of the recognition process) is taken in from the person recognition unit 120 to the posture estimation unit 140.

[0090] Based on the data of the partial image including the person recognized by the person recognition unit 120, the posture estimation unit 140 estimates the posture state of the person recognized by the person recognition unit 120.

[0091] Each time the output of the person recognition unit 120 is input, the posture estimation unit 140 performs a posture estimation process on the person recognized by the person recognition unit 120 and outputs information regarding the result of the posture estimation process, including the result of the posture estimation process.

[0092] Based on the output of the posture estimation unit 140, the action recognition unit 150 recognizes a specific action unique to a specific type of person performed by the person recognized by the person recognition unit 120. The action recognition unit 150 includes a candidate recognition unit 152, a region extraction unit 154, and a determination unit 156.

[0093] The candidate recognition unit 152, the region extraction unit 154, and the determination unit 156 may be functional units realized by one information processing device corresponding to the action recognition unit 150, or may be functional units each realized by three information processing devices, with each being realized by a separate information processing device. Also, the candidate recognition unit 152, the region extraction unit 154, and the determination unit 156 may be functional units realized by two information processing devices or four or more information processing devices. Hereinafter, the description will proceed on the premise that the candidate recognition unit 152, the region extraction unit 154, and the determination unit 156 are realized by one information processing device corresponding to the action recognition unit 150.

[0094] Based on the output of the posture estimation unit 140, the candidate recognition unit 152 recognizes candidates for specific actions unique to a specific type of person performed by the person recognized by the person recognition unit 120. For example, based on data representing the posture state of the person recognized by the person recognition unit 120, the candidate recognition unit 152 recognizes an action similar to an action unique to a specific type of person performed by the person recognized by the person recognition unit 120 (hereinafter, "similar action").

[0095] An action unique to a specific type of person is, for example, an action in which a station staff member signals to a driver by waving a hand flag or a signal lamp at a position higher than the shoulder or head. In this case, a similar action is, for example, an action of raising at least one hand above the shoulder or head (hereinafter, "hand-raising action").

[0096] Based on the output of the posture estimation unit 140 (such as the result of the posture estimation process), the candidate recognition unit 152 recognizes a similar action of the person recognized by the person recognition unit 120. For example, based on data representing the posture state of a person output from the posture estimation unit 140, the action recognition unit 150 determines whether the posture state of the person recognized by the person recognition unit 120 and its temporal change represent a similar action, thereby recognizing a specific action of that person.

[0097] Each time the output of the posture estimation unit 140 is input, the candidate recognition unit 152 performs recognition processing of similar actions of the person recognized by the person recognition unit 120, and outputs information regarding the result of the recognition processing of similar actions, including the result of the recognition processing of similar actions. The result of the recognition processing of similar actions includes information indicating the presence or absence of similar actions for each person recognized by the person recognition unit 120.

[0098] Based on the data representing the posture state of the person recognized by the person recognition unit 120, the candidate recognition unit 152 may recognize similar actions of the person recognized by the person recognition unit 120 in the same manner as the action recognition unit 150 of the first example of the monitoring system 100 described above.

[0099] The region extraction unit 154 extracts a partial image of a region including a body part characteristic in the actions specific to a specific type of person of the person whose similar actions have been recognized by the candidate recognition unit 152 from the latest image data (original data) acquired by the imaging unit 110.

[0100] Each time the output of the candidate recognition unit 152 is input, the region extraction unit 154 performs a region extraction process of extracting an image of a region including a body part characteristic in the actions specific to a specific type of person of the person whose similar actions have been recognized by the candidate recognition unit 152, and outputs information regarding the result of the region extraction process, including the result of the region extraction process. The result of the region extraction process includes, for example, information representing the extracted region in the latest image acquired by the imaging unit 110. Further, instead of the information representing the extracted region in the latest image acquired by the imaging unit 110, the result of the region extraction process may include data of a partial image obtained by cutting out the region extracted from the latest image of the imaging unit 110.

[0101] For example, based on the data representing the posture state of a person whose similar behavior has been recognized by the candidate recognition unit 152, the region extraction unit 154 estimates the sizes of the torso part and arm part of the person in the image, and determines the position range and size of the region to be extracted based on the characteristic body part. Thereby, the region extraction unit 154 can appropriately extract a partial image of the region including the characteristic body part according to the size of the person shown in the image.

[0102] For example, in the case of the behavior where a station staff member signals to crew members such as a driver by waving a flag or a signal lamp at a position higher than the shoulders or head, the characteristic body part in that behavior is the hand part held at a position higher than the head and shoulders. This is because the station staff member holds a flag or a signal lamp.

[0103] For example, as shown in FIG. 8, in the latest image IM8 captured by the imaging unit 110, there are shown a railway vehicle 200 parked adjacent to the platform PF, a platform door PD installed on the platform PF, and passengers PS and a station staff member SA standing on the platform PF. In this example, the station staff member SA is signaling by raising a flag HF above the head, and the region extraction unit 154 extracts a rectangular region AR3 including the hand of the station staff member SA raised above the head and the flag HF held in that hand.

[0104] Based on the data of the partial image corresponding to the region extracted by the region extraction unit 154, the determination unit 156 determines whether the similar behavior recognized by the candidate recognition unit 152 corresponds to a specific behavior unique to a specific type of person. Thereby, the behavior recognition unit 150 can recognize a specific behavior unique to a specific type of person.

[0105] Each time the output of the region extraction unit 154 is input, the determination unit 156 performs a determination process of whether the similar behavior recognized by the candidate recognition unit 152 corresponds to a specific behavior unique to a specific type of person, and outputs information regarding the result of the determination process including the result of the determination process. The result of the determination process includes, for example, information indicating whether each similar behavior recognized by the candidate recognition unit 152 corresponds to a specific behavior unique to a specific type of person.

[0106] For example, the determination unit 156 determines whether the recognized similar behavior corresponds to a specific behavior unique to a specific type of person based on whether a specific object used in a specific behavior unique to a specific type of person is included in the partial image corresponding to the region extracted by the region extraction unit 154. Specifically, the determination unit 156 may perform recognition of a specific object in the partial image based on the data of the partial image, and if the object can be recognized, determine that the recognized similar behavior corresponds to a specific behavior unique to a specific type of person. On the other hand, when the determination unit 156 cannot recognize a specific object in the partial image, it may determine that the recognized similar behavior does not correspond to a specific behavior unique to a specific type of person.

[0107] Similar to the case of the person recognition unit 120, the determination unit 156 may perform recognition of a specific object using a learned model based on machine learning (supervised learning), or may perform recognition of a specific object using a rule-based algorithm such as pattern matching regarding color and shape. Further, the determination unit 156 may apply a known image processing technique or a learned model to remove the background from the partial image corresponding to the region extracted by the region extraction unit 154 and then perform recognition of a specific object. Thereby, the determination unit 156 can suppress a situation in which it fails to recognize a specific object due to the influence of the background or misrecognizes the background as a specific object.

[0108] For example, in the case of the behavior of a station staff member signaling to crew members such as drivers by waving a flag or a signal lamp at a position higher than the shoulders or head, the determination unit 156 performs recognition of the flag or the signal lamp in the partial image corresponding to the region extracted by the region extraction unit 154. Then, when the flag or the signal lamp can be recognized, the determination unit 156 determines that the recognized similar behavior corresponds to the behavior of the station staff member signaling to the crew members, and when the flag or the signal lamp cannot be recognized, determines that it does not correspond to the behavior of the station staff member signaling to the crew members.

[0109] For example, as shown in FIG. 8, the determination unit 156 can determine that the action of the station staff SA extracted as a similar action corresponds to an action specific to the station staff (an action of giving a signal to the crew by the station staff) by recognizing the hand flag HF included in the partial image of the extracted rectangular region AR3.

[0110] The notification unit 160 notifies the persons related to the operation of the railway vehicle 200 of the result of the action recognition process of the action recognition unit 150, that is, the result of whether or not a specific action unique to a specific type of person is recognized by the person recognized by the person recognition unit 120. Thereby, the persons related to the operation of the railway vehicle 200 can grasp that a specific action unique to a specific type of person is being performed, that is, that a specific type of person is performing a specific action unique to that person.

[0111] The result of whether or not a specific action unique to a specific type of person is recognized by the person recognized by the person recognition unit 120 corresponds to the result of the determination of whether or not it is appropriate by the determination unit 156. That is, when the result of the determination of whether or not it is appropriate by the determination unit 156 indicates that the recognized similar action corresponds to a specific type of action unique to a specific type of person, the result of the action recognition process of the action recognition unit 150 means that the person recognized by the person recognition unit 120 has recognized an action unique to a specific type of person. On the other hand, when the result of the determination of whether or not it is appropriate by the determination unit 156 indicates that the recognized similar action does not correspond to a specific type of action unique to a specific type of person, the result of the action recognition process of the action recognition unit 150 means that the person recognized by the person recognition unit 120 has not recognized an action unique to a specific type of person.

[0112] For example, as shown in FIG. 8, when the station staff SA raises the hand flag HF above the head to give a signal, the notification unit 160 can notify the crew of the railway vehicle 200 that the station staff SA recognized as a person is performing a signal action unique to the station staff, that is, that the station staff SA is performing a signal action. Therefore, the crew such as the driver can surely grasp the start and end of the guidance of the railway vehicle 200 for passengers who need assistance, such as those using a wheelchair or a white cane, by confirming the signal from the station staff SA.

[0113] <Processing> FIG. 9 is a flowchart schematically showing a second example of the processing of the monitoring system 100.

[0114] This flowchart is started, for example, when the railway vehicle 200 arrives at and stops at the home PF.

[0115] As shown in FIG. 9, since the processes of steps S202 and S204 are the same as the processes of steps S102 and S104 in FIG. 6, the description thereof is omitted.

[0116] When the process of step S204 is completed, the person recognition unit 120 outputs information regarding the result of the person recognition process to the posture estimation unit 140.

[0117] When the posture estimation unit 140 receives information regarding the result of the person recognition process from the person recognition unit 120, the posture estimation unit 140 executes the process of step S206.

[0118] In step S206, the posture estimation unit 140 determines whether a person has been recognized (i.e., detected) based on the information regarding the result of the recognition process of the person recognition unit 120. If a person is recognized by the person recognition unit 120, the posture estimation unit 140 proceeds to step S208. On the other hand, if a person is not recognized by the person recognition unit 120, the posture estimation unit 140 does not execute the process of step S208 and outputs a notification signal indicating that the process is aborted to the person recognition unit 120.

[0119] In step S208, based on the data of the partial image including the person recognized by the person recognition unit 120, the posture estimation process of that person is performed.

[0120] When the process of step S208 is completed, the posture estimation unit 140 outputs information regarding the result of the posture estimation process to the action recognition unit 150.

[0121] When the action recognition unit 150 receives information regarding the result of the posture estimation process from the posture estimation unit 140, the action recognition unit 150 executes the process of step S210.

[0122] In step S210, the candidate recognition unit 152 of the action recognition unit 150 executes a recognition process of similar actions by the person recognized by the person recognition unit 120 based on the information regarding the result of the posture estimation process of the posture estimation unit 140.

[0123] When the process of step S210 is completed, the action recognition unit 150 proceeds to step S212.

[0124] In step S212, the area extraction unit 154 of the action recognition unit 150 determines whether it was possible to recognize a raising hand action as a similar action to the action of signaling to the crew of the railway vehicle 200 by the station staff in step S210. If the area extraction unit 154 of the action recognition unit 150 can recognize the raising hand action, it proceeds to step S214. On the other hand, if the area extraction unit 154 of the action recognition unit 150 cannot recognize the raising hand action, it does not execute the process of step S214 and outputs a notification signal indicating that the process is aborted to the person recognition unit 120.

[0125] In step S214, the area extraction unit 154 of the action recognition unit 150 executes an area extraction process of extracting an area including the vicinity of the hand of the raising hand of the person recognized by the person recognition unit 120 from the latest image data of the imaging unit 110.

[0126] When the process of step S214 is completed, the action recognition unit 150 proceeds to step S216.

[0127] In step S216, the determination unit 156 of the action recognition unit 150 executes a process (appropriateness determination process) of determining whether the raising hand action recognized in step S210 corresponds to the action of signaling to the crew of the railway vehicle 200 by the station staff based on the data of the partial image corresponding to the area extracted in step S214.

[0128] When the process of step S216 is completed, the action recognition unit 150 outputs information regarding the result of the action recognition process corresponding to the result of the appropriateness determination process of the determination unit 156 to the notification unit 160.

[0129] When information regarding the result of the action recognition process is input from the action recognition unit 150 to the notification unit 160, the notification unit 160 executes the process of step S218.

[0130] In step S218, based on the information regarding the result of the action recognition process of the action recognition unit 150, the notification unit 160 determines whether or not the action recognition unit 150 has recognized the action of signaling to the crew as a specific action by the station staff. In this example, similar to the case of the first example (FIG. 6) described above, signaling to the driver means the end of the guidance work for boarding the railway vehicle 200 by the persons with disabilities using wheelchairs or white canes. If the notification unit 160 recognizes that the action recognition unit 150 has recognized the action of signaling to the crew as specific to the station staff, the process proceeds to step S220. On the other hand, if the notification unit 160 does not recognize that the action recognition unit 150 has recognized the action of signaling to the crew as specific to the station staff, the notification unit 160 does not execute the process of step S220 and outputs a notification signal indicating the termination of the process to the human recognition unit 120.

[0131] In step S220, the notification unit 160 notifies the driver of the railway vehicle 200 that there has been a signal from the station staff.

[0132] When the process of step S220 is completed, the notification unit 160 ends the process, and the end of this process causes the end of the process of this flowchart.

[0133] On the other hand, when a notification signal indicating the termination of the process is input from the posture estimation unit 140, the action recognition unit 150, or the notification unit 160 to the human recognition unit 120, the human recognition unit 120 executes the process of step S222.

[0134] Since the process of step S222 is the same as the process of step S120 in FIG. 6, the description thereof is omitted. If the railway vehicle 200 has not departed, the human recognition unit 120 returns to step S202 and executes the process of step S202 again. On the other hand, if the railway vehicle 200 has departed, the human recognition unit 120 ends the process, and the end of this process causes the end of the process of this flowchart.

[0135] Thus, in this example, similar to the above-described first example (Fig. 6), the monitoring system 100 monitors whether there is a signal from the station staff at the platform PF to the driver during the period from the stop to the departure of the railway vehicle 200, and when a signal to the driver is recognized, it can notify the driver to that effect.

[0136] [Third Example of Monitoring System] Next, with reference to Figs. 10 to 13, a third example of the monitoring system 100 according to the present embodiment will be described.

[0137] Hereinafter, in this example, the same or corresponding components as those in the above-described first example and second example are denoted by the same reference numerals, and the description will focus on the parts different from the above-described first example and second example, and the same or corresponding descriptions as those in the above-described first example and second example may be omitted.

[0138] [Configuration] Fig. 10 is a diagram showing a third example of the monitoring system 100. Fig. 11 is a diagram showing still another example (image IM11) of the captured image captured by the imaging unit 110. Fig. 12 is a diagram for explaining an example of the processing by the person recognition unit 120, the region extraction unit 154, and the determination unit 156.

[0139] In this example, the monitoring system 100 is mainly different from the above-described first example in that the content of the processing of each of the person recognition unit 120, the person classification unit 130, the posture estimation unit 140, and the behavior recognition unit 150 is changed.

[0140] Unlike the above-described first example, the person recognition unit 120 recognizes a person included in an image by recognizing the head of the person from the latest image acquired by the imaging unit 110.

[0141] The person recognition unit 120 performs a recognition process of the head of the person included in the image based on the latest image data input from the imaging unit 110 at each predetermined processing cycle, and outputs information regarding the result of the recognition process of the head of the person, including the result of the recognition process of the head of the person.

[0142] The result of the human head recognition process includes, for example, information indicating the presence or absence of human head recognition. Also, the result of the human head recognition process may include information representing the region (e.g., a rectangular region) in the entire image where the human head is included when the human head is recognized. The region in the entire image where the human head is included is, for example, a region that includes a part or all of the human head and only a part of the human body. The region in the entire image where the human head is included is typically a region that includes only the human head in the human body, or a region that includes the entire human head and a part of the shoulders in the human body. The region in the entire image where the human head is included is usually a partial region in the entire image, but depending on how the person appears in the image, etc., it may be extracted as the entire region of the image. Also, the result of the human head recognition process may include data of a partial image of the region in the entire image input from the imaging unit 110 where the human head recognized by the human recognition unit 120 is included. Further, instead of the data of the partial image, the human recognition unit 120 may output the original data (i.e., the latest image data) acquired by the imaging unit 110 together with the information representing the region in the entire image input from the imaging unit 110 where the human head is included.

[0143] For example, based on local feature amounts obtained from the image data or the image data itself, the human recognition unit 120 applies the learned model LM4 to recognize the human head in the image and estimate the region in the image where the human head is included.

[0144] The learned model LM4 is obtained, for example, by machine learning (i.e., supervised learning) of the base learning model M4 using the teacher dataset TRD4. The learned model LM4 may be generated by machine learning of the learning model M4 in the human recognition unit 120, or may be generated by machine learning of the learning model M4 in an information processing device different from the human recognition unit 120. For example, each piece of teacher data included in the teacher dataset TRD4 is a combination of an image feature amount or the image data itself obtained from the image data as input data and data specifying a region (i.e., a partial image) including the head of a person on the image as output data. Thereby, the learned model LM4 can recognize the head of a person on the image with the image feature amount or the image data itself obtained from the image data as input, and output data of a region (e.g., a rectangular region) including the head of the person on the image. For example, the learned model LM4 is mainly configured around a deep neural network (DNN), and the machine learning of the DNN is made efficient by applying backpropagation (error backpropagation method) based on the teacher data. Also, the learned model LM4 may include a U-Net capable of recognizing a person from image data with the image data as input.

[0145] Also, the human recognition unit 120 may apply a rule-based algorithm such as pattern matching based on feature points, local feature amounts, etc. of the image data to recognize the head of a person in the image.

[0146] For example, as shown in FIG. 11, in the latest image IM11 acquired by the imaging unit 110, there are shown a railway vehicle 200 entering adjacent to the platform PF, a platform door PD installed on the platform PF, and passengers PS11 to PS15 and a station staff SA1 on the platform PF. In this example, the passengers PS11 to PS15 and the station staff SA1 are densely gathered near the platform door PD. Therefore, in front of the station staff SA1 and the passenger PS14, the passengers PS12 and PS13 are positioned so as to overlap, and even if an attempt is made to recognize the entire bodies of the station staff SA1 and the passenger PS14, there is a possibility that they cannot be recognized.

[0147] In contrast, as shown in FIG. 12, the person recognition unit 120 recognizes the heads of the station staff SA1 and the passengers PS11 to PS15 from the image IM11, and outputs the data of the partial images of the rectangular regions AR10 to AR15 each containing a respective head. Thereby, the person recognition unit 120 can recognize the station staff SA1 and the passenger PS14 by recognizing the respective heads of the station staff SA1 and the passenger PS14 in the image IM11.

[0148] The output of the person recognition unit 120 (specifically, information related to the result of the recognition process of a person's head, etc.) is taken in by the person classification unit 130 from the person recognition unit 120. Also, the person recognition unit 120 is communicably connected to the action recognition unit 150 (specifically, the determination unit 153) through a one-to-one communication line, a local network, or the like, and the output of the person recognition unit 120 (information related to the result of the person recognition process, etc.) may be taken in by the posture estimation unit 140 from the person recognition unit 120.

[0149] The person classification unit 130 classifies the person corresponding to the head recognized by the person recognition unit 120 based on the output of the person recognition unit 120, specifically, information related to the result of the recognition process of a person's head. Specifically, the person classification unit 130 classifies the person corresponding to the head recognized by the person recognition unit 120 into any one of a plurality of types of attributes based on the data of the partial image including the head recognized by the person recognition unit 120. The data of the partial image including the head of the person recognized by the person recognition unit 120 is input from the person recognition unit 120 to the person classification unit 130, for example, as part of the information related to the result of the recognition process of a person's head from the person recognition unit 120. Also, the data of the partial image including the head of the person recognized by the person recognition unit 120 may be obtained by being cut out from the latest image acquired by the imaging unit 110 based on the output of the person recognition unit 120. In the latter case, the latest image data acquired by the imaging unit 110 may be transmitted from the person recognition unit 120 to the person classification unit 130, or may be directly transmitted from the imaging unit 110 to the person classification unit 130.

[0150] Whenever the output of the person recognition unit 120 is input, the person classification unit 130 performs classification processing on the person corresponding to the head included in the partial image, and outputs information regarding the result of the classification processing, including the result of the classification processing. The result of the classification processing includes, for example, label information representing the classified attributes of the person for each region (i.e., partial image) including the head of the person recognized by the person recognition unit 120. Further, the result of the classification processing may include information representing the region in the entire image acquired by the imaging unit 110 that includes the head of the person corresponding to the label information. Further, the result of the classification processing may include data of a partial image of the region in the entire image acquired by the imaging unit 110 that includes the head of the person corresponding to the label information. Further, instead of the data of the partial image, the person classification unit 130 may output the latest image data acquired by the imaging unit 110 together with information representing the region in the entire image that includes the head of the person corresponding to the label information.

[0151] For example, based on local feature amounts obtained from data of a partial image corresponding to a region including a person's head in the entire image and the data of the partial image, the person classification unit 130 applies the learned model LM5 to estimate which attribute among a plurality of types of attributes the person corresponding to the head included in the partial image corresponds to. The learned model LM5 corresponds to a classifier that classifies the person in the partial image into any one of a plurality of types of attributes, and may be a set of a plurality of classifiers that classify whether or not each corresponds to each of the plurality of types of attributes. For example, when classified into either a "railway employee" or "a person other than a railway employee", features of characteristic possessions such as the hat of the railway employee (specifically, an item worn near the head) are learned.

[0152] The learned model LM5 is obtained, for example, by machine learning (i.e., supervised learning) of the base learning model M5 using the teacher dataset TRD5. The learned model LM5 may be generated by machine learning of the learning model M5 in the human classification unit 130, or may be generated by machine learning of the learning model M5 in an information processing device different from the human classification unit 130. For example, each piece of teacher data included in the teacher dataset TRD5 is a combination of local feature amounts obtained from image data including a person's head as input data or the image data itself, and label data representing the attributes of the person corresponding to the head included in the image as output data. Thereby, the learned model LM5 can estimate the attributes of the person corresponding to the head included in the image by using the image feature amounts obtained from the image data including the person's head or the image data itself as input, and output label data representing the attributes of that person. For example, the learned model LM5 is mainly configured by a DNN, and the machine learning of the DNN is made more efficient by applying backpropagation based on the teacher data. Also, the learned model LM5 may be a support vector machine (SVM). For example, when classified into either a "railway employee" or a "person other than a railway employee", the learned model LM5 reflects the characteristics of items (specifically, items worn near the head) characteristic of the attributes of a "railway employee" such as a hat.

[0153] Also, the human classification unit 130 may classify the attributes of the person corresponding to the head included in the partial image by applying a rule-based algorithm such as pattern matching based on feature points, local feature amounts, etc. of the data of the partial image including the person's head. For example, when classified into either a "railway employee" or a "person other than a railway employee", the human classification unit 130 can apply an algorithm that utilizes the characteristics of items (specifically, items worn near the head) characteristic of the attributes of a "railway employee" such as a hat.

[0154] The human classification unit 130 is communicably connected to the posture estimation unit 140 of the action recognition unit 150 through a one-to-one communication line, a local network, or the like, and the output of the human classification unit 130 (specifically, information regarding the result of the classification process, etc.) is taken in from the human classification unit 130 by the posture estimation unit 140. Further, the output of the human recognition unit 120 (information regarding the result of the recognition process of the human head, etc.) may be taken in by the posture estimation unit 140 of the action recognition unit 150 via the human classification unit 130.

[0155] Based on the data of the partial image including the human head classified into a specific type by the human classification unit 130, the posture estimation unit 140 estimates the posture state of the human head recognized by the human recognition unit 120. For example, the posture estimation unit 140 estimates the orientation of the face corresponding to the human head recognized by the human recognition unit 120.

[0156] Every time the output of the human classification unit 130 is input, the posture estimation unit 140 performs a posture estimation process for the human head classified into a specific type by the human classification unit 130, and outputs information regarding the result of the head posture estimation process, including the result of the head posture estimation process. The result of the head posture estimation process includes, for example, data representing the posture state of a person for each region (i.e., partial image) including the head corresponding to the person classified into a specific type by the human classification unit 130 in the entire image acquired by the imaging unit 110. Further, the result of the posture estimation process may include information representing the region including the person corresponding to the data representing the posture state of the person in the entire image acquired by the imaging unit 110. Further, the result of the posture estimation process may include the data of the partial image of the region including the person corresponding to the data representing the posture state of the person in the entire image acquired by the imaging unit 110. Further, instead of the data of the partial image, the posture estimation unit 140 may output the latest image data acquired by the imaging unit 110 together with the information representing the region including the person corresponding to the data representing the posture state of the person.

[0157] For example, based on local feature quantities obtained from data of a partial image including the head of a person classified into a specific type by the person classification unit 130 and data of the partial image, the posture estimation unit 140 applies the learned model LM6 to estimate the posture state of the head included in the data of the partial image.

[0158] The learned model LM6 is obtained, for example, by machine learning (i.e., supervised learning) of the base learning model M6 using the teacher dataset TRD6. The learned model LM6 may be generated by machine learning of the learning model M6 in the posture estimation unit 140, or may be generated by machine learning of the learning model M6 in an information processing apparatus different from the posture estimation unit 140. For example, each piece of teacher data included in the teacher dataset TRD6 is a combination of a local feature quantity obtained from data of an image including a person's head as input data or the data of the image itself and data representing the posture state of the head included in the image as output data. Thereby, the learned model LM6 can estimate the posture state of the head included in the image using the image feature quantity obtained from the data of the image including the person's head or the data of the image itself as input, and output data representing the posture state. For example, the learned model LM6 is mainly configured by a DNN, and the machine learning of the DNN is made more efficient by applying backpropagation based on the teacher data. Also, when estimating which of a plurality of types of predefined posture states the posture state (e.g., face orientation) of the head included in the image corresponds to, the learned model LM6 may be an SVM as a classifier.

[0159] Also, based on feature points, local feature quantities, etc. of data of a partial image including a person, the posture estimation unit 140 may apply a rule-based algorithm to estimate which of a plurality of types of predefined posture states the posture state (e.g., face orientation) of the head included in the partial image corresponds to.

[0160] The posture estimation unit 140 is communicably connected to the action recognition unit 150 (specifically, the determination unit 153) through a one-to-one communication line, a local network, or the like. The output of the posture estimation unit 140 (specifically, information regarding the result of the head posture estimation process, etc.) is taken into the determination unit 153 of the action recognition unit 150 from the posture estimation unit 140. Further, the output of the posture estimation unit 140 may include the output of the person recognition unit 120 (information regarding the result of the person recognition process, etc.) and the output of the person classification unit 130 (information regarding the result of the classification process, etc.). That is, the output of the person recognition unit 120 and the output of the person classification unit 130 may be taken into the action recognition unit 150 via the posture estimation unit 140.

[0161] The action recognition unit 150 includes a determination unit 153, a region extraction unit 154, and a determination unit 156.

[0162] The determination unit 153, the region extraction unit 154, and the determination unit 156 may be functional units realized by one information processing device corresponding to the action recognition unit 150, or may be functional units respectively realized by three information processing devices, each by a separate information processing device. Further, the region extraction unit 154 and the determination unit 156 may be functional units realized by two or four or more information processing devices. Hereinafter, the description will proceed on the premise that the region extraction unit 154 and the determination unit 156 are realized by one information processing device corresponding to the action recognition unit 150.

[0163] The determination unit 153 determines whether a person classified into a specific type of attribute by the person classification unit 130 is likely to perform a specific action unique to the specific type of attribute based on the head posture state of the person estimated by the posture estimation unit 140. In other words, the determination unit 153 determines whether the head posture state of the person estimated by the posture estimation unit 140 may represent a specific action unique to a person classified into a specific type of attribute by the person classification unit 130.

[0164] Each time the output of the posture estimation unit 140 is input, the determination unit 153 performs a determination process on whether a person classified into a specific type of attribute by the person classification unit 130 is performing a specific action, and outputs information regarding the result of the determination process including the result of the determination process. The result of the determination process regarding the possibility includes, for example, information indicating whether a person classified into a specific type of attribute by the person classification unit 130 is performing a specific action.

[0165] For example, when the face direction estimated by the posture estimation unit 140 corresponds to a specific direction, the determination unit 153 determines that a person classified into a specific type of attribute by the person classification unit 130 is performing a specific action. For example, when the specific type of attribute is "railway employee" and the specific action is an action of signaling to the crew of the railway vehicle 200, the specific direction is the direction in which the crew of the railway vehicle 200 is present.

[0166] The output of the determination unit 153 (information such as the result of the determination process regarding the possibility) is input to the region extraction unit 154.

[0167] When it is determined by the determination unit 153 that a person classified into a specific type of attribute by the person classification unit 130 may be performing a specific action, the region extraction unit 154 performs a region extraction process and outputs information regarding the result of the region extraction process including the result of the region extraction process. Thereby, the action recognition unit 150 can perform an action recognition process of recognizing a specific action of the person only when there is a possibility that a person classified into a specific type by the person classification unit 130 is performing a specific action. Therefore, the monitoring system 100 can suppress unnecessary processing and improve the efficiency of processing.

[0168] For example, the region extraction unit 154 estimates the position range and size of a region including characteristic body parts when a specific action is performed, based on the position and size of a partial image of a region including the head of a person classified into a specific type of attribute by the person classification unit 130 from among the latest images acquired by the imaging unit 110. Thereby, the region extraction unit 154 can appropriately extract a region including characteristic body parts when a specific action unique to a specific type of person is performed, according to the size of the person shown in the image.

[0169] For example, as shown in FIG. 12, the region extraction unit 154 extracts a partial image of a rectangular region AR20, based on the position and size of a rectangular region AR10 including the head of a station staff SA1 classified as a "station staff" by the person classification unit 130. The rectangular region AR20 includes a hand part raised at a position higher than the head and shoulders, which is a characteristic body part of the action of signaling to the crew of the railway vehicle 200.

[0170] The determination unit 156 determines whether a person classified into a specific type by the person classification unit 130 is performing a specific action, based on the data of the partial image corresponding to the region extracted by the region extraction unit 154. In other words, the determination unit 156 determines whether the action of a person classified into a specific type of attribute by the person classification unit 130 corresponds to a specific action unique to a specific type of person. At this time, the determination unit 156 can determine whether the action of a person classified into a specific type of attribute by the person classification unit 130 corresponds to a specific action unique to a specific type of person, by the same method as in the second example described above. Thereby, the determination unit 156 can recognize a specific action unique to a person classified into a specific type by the person classification unit 130.

[0171] Each time the output of the region extraction unit 154 is input, the determination unit 156 performs a determination process as to whether or not the behavior of a person classified into a specific type of attribute by the person classification unit 130 corresponds to a specific behavior unique to a specific type of person, and outputs information regarding the result of the determination process, including the result of the determination process. The result of the determination includes, for example, information indicating whether or not the behavior of the target person for each person classified into a specific type of attribute by the person classification unit 130 corresponds to the behavior unique to a specific type of person.

[0172] For example, as shown in FIG. 12, the determination unit 156 recognizes the hand flag HF1 included in the rectangular region AR20 extracted by the region extraction unit 154. Thereby, the determination unit 156 can determine that the behavior of the station staff SA corresponds to a specific behavior unique to the station staff SA (that is, a signal to the crew of the railway vehicle 200 by the station staff SA).

[0173] Similar to the second example described above, the notification unit 160 notifies the person related to the operation of the railway vehicle 200 of the result of the behavior recognition process of the behavior recognition unit 150, that is, the result of whether or not a specific behavior unique to a specific type of person is recognized by the person recognized by the person recognition unit 120. Thereby, the person related to the operation of the railway vehicle 200 can grasp that a specific behavior unique to a specific type of person is being performed, that is, a specific type of person is performing a specific behavior unique to that person.

[0174] For example, as shown in FIG. 11, when the station staff SA1 raises the hand flag HF1 above the head to send a signal, the notification unit 160 can notify the crew of the railway vehicle 200 that the station staff SA1 classified as a "station staff" is performing a signal behavior unique to the station staff, that is, the station staff SA1 is performing a signal behavior. Therefore, the crew such as the driver can surely grasp the start and end of the guidance of the railway vehicle 200 for passengers who need assistance, such as those using a wheelchair or a white cane, by confirming the signal of the station staff SA.

[0175] <Process> FIG. 13 is a flowchart schematically showing a third example of the process of the monitoring system 100.

[0176] This flowchart is started, for example, when the railway vehicle 200 arrives at and stops at the platform PF.

[0177] As shown in FIG. 13, since the process of step S302 is the same as the process of step S102 in FIG. 6, the description thereof is omitted.

[0178] When the process of step S302 is completed, the human recognition unit 120 proceeds to step S304.

[0179] In step S304, the human recognition unit 120 executes the recognition process of a person's head.

[0180] When the process of step S304 is completed, the human recognition unit 120 outputs information regarding the result of the recognition process of the person's head to the human classification unit 130.

[0181] When the output of the human recognition unit 120 is input, the human classification unit 130 executes the process of step S306.

[0182] In step S306, the human classification unit 130 determines whether a person's head has been recognized (i.e., detected) by the human recognition unit 120. If the human classification unit 130 determines that a person's head has been recognized by the human recognition unit 120, it proceeds to step S308. If a person's head has not been recognized, it does not execute the process of step S308 and outputs a notification signal indicating that the process is to be aborted to the human recognition unit 120.

[0183] In step S308, for each person's head recognized by the human recognition unit 120, the human classification unit 130 executes a classification process (classification process) of classifying the person corresponding to the target head into either "station staff" or "person other than station staff" based on the data of the partial image including the person's head.

[0184] When the process of step S308 is completed, the human classification unit 130 outputs information regarding the result of the classification process to the posture estimation unit 140.

[0185] When the classification processing result from the person classification unit 130 is input, the posture estimation unit 140 executes the process of step S310.

[0186] In step S310, based on the information regarding the classification processing result of the person classification unit 130, the posture estimation unit 140 determines whether there is a person classified as a "station staff member" among the people corresponding to the head recognized by the person recognition unit 120. If the posture estimation unit 140 determines that there is a person classified as a "station staff member", it proceeds to step S312. On the other hand, if the posture estimation unit 140 determines that there is no person classified as a "station staff member", it does not execute the process of step S312 and outputs a notification signal indicating the termination of the process to the person recognition unit 120.

[0187] In step S312, based on the data of the partial image including the head of the person classified as a "station staff member" by the person classification unit 130, the posture estimation unit 140 performs the posture estimation process of the head.

[0188] When the process of step S312 is completed, the posture estimation unit 140 outputs information regarding the result of the head posture estimation process to the action recognition unit 150.

[0189] When the information regarding the result of the head posture estimation process is input from the posture estimation unit 140, the action recognition unit 150 executes the process of step S314.

[0190] In step S314, based on the information regarding the result of the head posture estimation process of the posture estimation unit 140, the determination unit 153 of the action recognition unit 150 determines whether the head of the person classified as a "station staff member" by the person classification unit 130 is facing the above-mentioned specific direction. If the determination unit 153 determines that the head of the target person is facing the specific direction, it determines that the person corresponding to the head of the target may be performing a specific action and proceeds to step S316. On the other hand, if the determination unit 153 determines that the head of the target person is not facing the specific direction, it determines that the person corresponding to the head of the target is not likely to be performing a specific action, does not execute the process of step S316, and outputs a notification signal indicating the termination of the process to the person recognition unit 120.

[0191] In step S316, the region extraction unit 154 of the action recognition unit 150 executes a process (region extraction process) of extracting a region corresponding to the vicinity of the hand of the person classified as a "station staff" by the person classification unit 130 from the data of the latest image of the imaging unit 110.

[0192] When the process of step S316 is completed, the action recognition unit 150 proceeds to step S318.

[0193] In step S318, the determination unit 156 of the action recognition unit 150 executes a process (appropriateness determination process) of determining whether the action of the person classified as a "station staff" by the person classification unit 130 corresponds to an action of signaling to the crew based on the data of the partial image corresponding to the region extracted in step S316.

[0194] When the process of step S318 is completed, the action recognition unit 150 outputs information regarding the result of the action recognition process corresponding to the result of the appropriateness determination process of the determination unit 156 to the notification unit 160.

[0195] When information regarding the result of the action recognition process is input from the action recognition unit 150, the notification unit 160 executes the process of step S320.

[0196] In step S320, the notification unit 160 determines whether an action of signaling to the crew has been recognized by the action recognition unit 150 based on the information regarding the result of the action recognition process of the action recognition unit 150. In this example, similar to the case of the above-described first example (Figure 6), signaling to the driver means the end of the guidance work for boarding the railway vehicle 200 by a person with a disability using a wheelchair or a white cane. When the notification unit 160 recognizes that an action of signaling to the crew has been recognized by the action recognition unit 150, it proceeds to step S322. On the other hand, when the notification unit 160 does not recognize an action of signaling to the crew specific to the station staff by the action recognition unit 150, it does not execute the process of step S322 and outputs a notification signal indicating that the process is aborted to the person recognition unit 120.

[0197] In step S322, the notification unit 160 notifies the driver of the railway vehicle 200 that there has been a signal from the station staff.

[0198] When the process of step S322 is completed, the notification unit 160 ends the process, and the end of this process causes the process of this flowchart to end.

[0199] On the other hand, when a notification signal indicating the end of the process is input from the posture estimation unit 140, the action recognition unit 150, or the notification unit 160, the person recognition unit 120 executes the process of step S324.

[0200] Since the process of step S324 is the same as the process of step S120 in FIG. 6, the description thereof is omitted. When the railway vehicle 200 has not departed, the person recognition unit 120 returns to step S302 and executes the process of step S302 again. On the other hand, when the railway vehicle 200 has departed, the person recognition unit 120 ends the process, and the end of this process causes the process of this flowchart to end.

[0201] Thus, in this example, similar to the above-described first example (FIG. 6), the monitoring system 100 monitors whether or not there is a signal from the station staff on the platform PF to the driver from the stop to the departure of the railway vehicle 200, and when a signal to the driver is recognized, the driver can be notified to that effect.

[0202] [Fourth Example of Monitoring System] Next, with reference to FIGS. 14 to 17, a fourth example of the monitoring system 100 according to the present embodiment will be described.

[0203] Hereinafter, in this example, the same or corresponding components as those in the above-described first to third examples are denoted by the same reference numerals, and the description will be centered on the parts different from the above-described first to third examples, and the same or corresponding descriptions as those in the above-described first to third examples may be omitted.

[0204] [Configuration] FIG. 14 is a diagram showing a fourth example of the monitoring system 100. FIG. 15 is a diagram showing still another example (image IM15) of the captured image acquired by the imaging unit 110. FIG. 16 is a diagram for explaining an example of the processes of the object recognition unit 170 and the object tracking unit 180.

[0205] In this example, the monitoring system 100 is different from the first example described above in that it includes an object recognition unit 170 and an object tracking unit 180 instead of a person recognition unit 120, a person classification unit 130, and a posture estimation unit 140, and the processing of the behavior recognition unit 150 is changed.

[0206] The imaging unit 110 is communicably connected to the object recognition unit 170 through a one-to-one communication line, a local network, or the like, and the output (image data) of the imaging unit 110 is taken in from the imaging unit 110 to the object recognition unit 170.

[0207] Based on the latest image acquired by the imaging unit 110, the object recognition unit 170 recognizes a specific object used in a specific behavior unique to a specific type of person. The object recognition unit 170 can recognize a specific object from the image of the imaging unit 110 by arbitrarily applying a machine learning-based algorithm or a rule-based algorithm in the same manner as the determination unit 156.

[0208] Each time the output of the imaging unit 110 is input, the object recognition unit 170 performs an object recognition process for recognizing a specific object used in a specific behavior unique to a specific type of person, and outputs information regarding the result of the object recognition process, including the result of the object recognition process. The result of the object recognition process includes, for example, information indicating the presence or absence of recognition of a specific object in the latest image acquired by the imaging unit 110. Further, the result of the object recognition process may include information representing the region including the specific object in the entire image acquired by the imaging unit 110. The region including the specific object in the entire image is usually a partial region in the entire image, but depending on the way the specific object appears in the image, it may be extracted as the entire region of the image. Further, the result of the object recognition process may include data of a partial image of the region including the specific object in the latest image acquired by the imaging unit 110. Further, instead of the data of the partial image, the result of the object recognition process may include the data of the image acquired by the imaging unit 110 together with information representing the region including the specific object in the entire image.

[0209] The function of the object recognition unit 170 may be realized by any hardware, or any combination of hardware and software, etc. For example, the object recognition unit 170 is mainly configured around a computer (information processing device) including a CPU, a memory device, an auxiliary storage device, an interface device, etc. The memory device is, for example, SRAM, DRAM, etc. The auxiliary storage device is, for example, HDD, SSD, flash memory, EEPROM, etc. The interface device includes, for example, a communication interface for communicating with other components of the monitoring system 100 such as the imaging unit 110 and the object tracking unit 180. Also, the interface device includes, for example, an external interface for connecting to a recording medium. Thereby, the object recognition unit 170 can, for example, through the external interface, capture various programs from the recording medium and install them in the auxiliary storage device. Further, the information processing device may further include a high-speed arithmetic device that operates in conjunction with the CPU. The high-speed arithmetic device includes, for example, GPU, FPGA, ASIC, etc.

[0210] For example, a specific object is a hand flag or a signal lamp used in the action of a station staff giving a signal to the crew of the railway vehicle 200.

[0211] For example, as shown in FIG. 15, in the latest image IM15 captured by the imaging unit 110, there are shown the railway vehicle 200 parked adjacent to the platform PF, the platform door PD installed on the platform PF, and the passengers PS21 - PS25 and the station staff SA2 on the platform PF. In this example, the station staff SA2 is giving a signal by raising the hand flag HF2 above the head. Therefore, as shown in FIG. 16, the object recognition unit 170 can recognize the hand flag HF2 and extract the area AR30 including the hand flag HF2.

[0212] The object recognition unit 170 is communicably connected to the object tracking unit 180 through a one-to-one communication line, a local network, etc., and the output of the object recognition unit 170 (information regarding the result of the object recognition process, etc.) is taken in from the object recognition unit 170 to the object tracking unit 180.

[0213] Based on the output of the object recognition unit 170 (information related to the result of the object recognition process, etc.), the object tracking unit 180 tracks the movement of a specific object used in a specific action unique to a specific type of person within the imaging range of the imaging unit 110. Specifically, the object tracking unit 180 acquires the history of the position of a specific object, that is, the trajectory, on the image of the imaging unit 110 based on the output of the object recognition unit 170 in time series during a most recent predetermined period. More specifically, the object tracking unit 180 acquires the trajectory of a specific object of interest by associating in time series the positions of the same specific object on the image for a specific object recognized in time series by the object recognition unit 170.

[0214] Each time the output of the object recognition unit 170 is input, the object tracking unit 180 determines whether a specific object is recognized by the object recognition unit 170. And when a specific object is recognized by the object recognition unit 170, the object tracking unit 180 performs object tracking processing on the specific object recognized by the object recognition unit 170, and outputs information regarding the result of the object tracking processing, including the result of the object tracking processing. The result of the object tracking processing includes, for example, information regarding the history (i.e., trajectory) of the position of the specific object recognized by the object recognition unit 170. Also, each time the output of the object recognition unit 170 is input, the object tracking unit 180 may determine whether a specific object is recognized within a specific region in the image by the object recognition unit 170. And when a specific object is recognized within a specific region in the image by the object recognition unit 170, the object tracking unit 180 may perform object tracking processing on the specific object recognized by the object recognition unit 170 and output information regarding the result of the object tracking processing. The state where a specific object is recognized within a specific region represents, for example, a state where at least a part of the specific object recognized by the object recognition unit 170 is included in the specific region. The specific region is set to include the space above the shoulders of people on the platform of the station. This is because in the action of a station staff giving a signal to the crew of the railway vehicle 200, a flag or a signal lamp is held and used at a position higher than the head. For example, when the platform PF is crowded, the image captured by the imaging unit 110 will show the bodies of multiple people overlapping, and it becomes difficult to recognize the actions of people by the parts below the head. Also, the specific region may be further limited to a range relatively close to the railway vehicle 200 on the platform of the station. This is because it is highly likely that the station staff will give a signal to the crew of the railway vehicle 200 at a location relatively close to the railway vehicle 200 on the platform PF.

[0215] For example, as shown in FIG. 16, the object tracking unit 180 determines whether the hand flag HF2 as a specific object recognized by the object recognition unit 170 is recognized in a specific area AR40 in the image IM15. The specific area AR40 corresponds to the space above the head of a person in the home PF in the image IM15 and is set in a range relatively close to the railway vehicle 200 in the home PF. In this example, the station staff SA2 gives a signal by raising the hand flag HF2 to a position higher than the head at a location relatively close to the railway vehicle 200 in the home PF, and the entire area AR30 including the hand flag HF2 is included in the specific area AR40. Therefore, the object tracking unit 180 performs object tracking processing on the hand flag HF2 and acquires the history (i.e., trajectory) of the position of the hand flag HF2.

[0216] The function of the object tracking unit 180 may be realized by any hardware or any combination of hardware and software, etc. For example, the object tracking unit 180 is centered around a computer (information processing device) including a CPU, a memory device, an auxiliary storage device, an interface device, etc. The memory device is, for example, SRAM, DRAM, etc. The auxiliary storage device is, for example, HDD, SSD, flash memory, EEPROM, etc. The interface device includes, for example, a communication interface for communicating with other components of the monitoring system 100 such as the object recognition unit 170 and the behavior recognition unit. Also, the interface device includes, for example, an external interface for connecting to a recording medium. Thereby, the object tracking unit 180 can, for example, take in various programs from the recording medium through the external interface and install them in the auxiliary storage device. Further, the information processing device may further include a high-speed arithmetic device that operates in conjunction with the CPU. The high-speed arithmetic device includes, for example, GPU, FPGA, ASIC, etc.

[0217] For example, the object tracking unit 180 applies a known technique related to MOT (Multi Object Tracking) arbitrarily, and based on the information regarding the result of the recognition process of the object recognition unit 170 in the most recent time series, acquires the trajectory of a specific object in a recent predetermined time period. Specifically, the object tracking unit 180 may acquire the trajectory of a specific object, for example, by applying time series filtering using a Kalman filter or the like. Further, the object tracking unit 180 may acquire the trajectory of a specific object by applying a machine learning-based algorithm such as SORT (Simple Online and Realtime Tracking), for example. Also, the object tracking unit 180 may acquire the trajectory of a specific object in a recent predetermined time period based on the information regarding the result of the recognition process of the object recognition unit 170 in the most recent time series by arbitrarily applying a known technique related to SOT (Single Object Tracking). Additionally, the object tracking unit 180 may acquire the history of the coordinate information representing the position of the object recognized by the object recognition unit 170 itself as the trajectory of the specific object.

[0218] The object tracking unit 180 is communicably connected through a one-to-one communication line, a local network, or the like, and the output of the object tracking unit 180 (information regarding the result of the object tracking process, etc.) is taken into the action recognition unit 150 from the object tracking unit 180.

[0219] Based on the information regarding the result of the object tracking process of the object tracking unit 180, the action recognition unit 150 determines whether a specific action unique to a specific type of person is being performed.

[0220] For example, as shown in FIG. 15, a situation may occur where people are crowded on the home PF, the station staff SA2 and the passengers PS21, PS22 overlap in the image IM15, and the entire body of the station staff SA2 cannot be recognized. Even in such a case, the action recognition unit 150 can recognize the action of signaling using the hand flag HF2 of the station staff SA2 based on the tracking result of the movement of the hand flag HF2 by the object tracking unit 180.

[0221] Based on the output of the object tracking unit 180 (information such as the result of the object tracking process), the action recognition unit 150 performs an action recognition process to recognize a specific action unique to a specific type of person, and outputs information regarding the result of the action recognition process, including the result of the action recognition process. Specifically, the action recognition unit 150 recognizes a specific action unique to a specific type of person by determining whether a specific action unique to a specific type of person is being performed based on the output of the object tracking unit 180. In other words, the action recognition unit 150 recognizes a specific action unique to a specific type of person by determining whether the movement of a specific object represented by the result of the object tracking process of the object tracking unit 180 corresponds to the movement corresponding to a specific action unique to a specific type of person.

[0222] For example, the action recognition unit 150 determines whether a specific action of the target is being performed based on a comparison between the data of the trajectory of a specific object corresponding to the output of the object tracking unit 180 and the reference data of the trajectory of a specific object corresponding to a specific action unique to a specific type of person.

[0223] <Process> FIG. 17 is a flowchart schematically showing a fourth example of the process of the monitoring system 100.

[0224] This flowchart is started, for example, when the railway vehicle 200 arrives at the platform PF and stops.

[0225] As shown in FIG. 17, in step S402, the object recognition unit 170 acquires the latest captured image from the imaging unit 110 (monitoring camera).

[0226] When the process of step S402 is completed, the object recognition unit 170 proceeds to step S204.

[0227] In step S404, the object recognition unit 170 executes a process (object recognition process) of recognizing specific objects such as hand flags and signal lights shown in the captured image acquired in step S402.

[0228] When the process of step S404 is completed, the object recognition unit 170 outputs information regarding the result of the object recognition process to the object tracking unit 180.

[0229] When information regarding the result of the object recognition process is input from the object recognition unit 170, the object tracking unit 180 executes the process of step S406.

[0230] In step S406, the object tracking unit 180 determines whether a specific object has been recognized (i.e., detected) based on the information regarding the result of the object recognition process of the object recognition unit 170. If a specific object has been recognized by the object recognition unit 170, the object tracking unit 180 proceeds to step S408. On the other hand, if a specific object has not been recognized by the object recognition unit 170, the object tracking unit 180 does not execute the process of step S408 and outputs a notification signal indicating that the process is aborted to the object recognition unit 170.

[0231] In step S408, the object tracking unit 180 determines whether the specific object recognized by the object recognition unit 170 exists in a specific region in the image acquired by the imaging unit 110 based on the information regarding the result of the object recognition process of the object recognition unit 170. If the specific object recognized by the object recognition unit 170 exists in a specific region in the image acquired by the imaging unit 110, the object tracking unit 180 proceeds to step S410. On the other hand, if the specific object recognized by the object recognition unit 170 does not exist in a specific region in the image of the imaging unit 110, the object tracking unit 180 does not execute the process of step S410 and outputs a notification signal indicating that the process is aborted to the object recognition unit 170.

[0232] In step S410, the object tracking unit 180 executes a process (object tracking process) of tracking the movement of the object recognized by the object recognition unit 170 in the most recent predetermined period.

[0233] When the process of step S410 is completed, the object tracking unit 180 outputs information regarding the result of the object tracking process to the action recognition unit 150.

[0234] When information regarding the result of the object tracking process from the object tracking unit 180 is input to the action recognition unit 150, the action recognition unit 150 executes the process of step S412.

[0235] In step S412, based on the information regarding the result of the object tracking process of the object tracking unit 180, the action recognition unit 150 executes a process (action recognition process) of recognizing an action of signaling to the crew of the railway vehicle 200 that is specific to the station staff.

[0236] When the process of step S412 is completed, the action recognition unit 150 outputs information regarding the result of the action recognition process to the notification unit 160.

[0237] When information regarding the result of the action recognition process from the action recognition unit 150 is input to the notification unit 160, the notification unit 160 executes the process of step S414.

[0238] Since the process of step S414 is the same as the process of step S218 in FIG. 9, the description thereof is omitted.

[0239] When the action recognition unit 150 recognizes an action of signaling to the crew of the railway vehicle 200 that is specific to the station staff, the notification unit 160 proceeds to step S416. On the other hand, when the action recognition unit 150 does not recognize an action of signaling to the crew of the railway vehicle 200 that is specific to the station staff, the notification unit 160 does not execute the process of step S416 and outputs a notification signal indicating that the process is aborted to the object recognition unit 170.

[0240] Since the process of step S416 is the same as the process of step S220 in FIG. 9, the description thereof is omitted.

[0241] On the other hand, when a notification signal indicating that the process is aborted is input to the object recognition unit 170 from the object tracking unit 180 or the notification unit 160, the object recognition unit 170 executes the process of step S418.

[0242] In step S418, the object recognition unit 170 determines whether the railway vehicle 200 has departed. The object recognition unit 170 may determine whether the railway vehicle 200 has departed based on whether a departure signal received from the driver's cab or the railway vehicle 200 is received, or may determine whether the railway vehicle 200 has departed by recognizing the presence or absence of movement of the railway vehicle 200 and the opening / closing status of the platform door, etc. from the latest captured image of the imaging unit 110. If the object recognition unit 170 determines that the railway vehicle 200 has not departed, it returns to step S402 and executes the process of step S402 again. On the other hand, if the object recognition unit 170 determines that the railway vehicle 200 has departed, it ends the process, and the end of this process ends the process of this flowchart.

[0243] In this way, in this example, similar to the above-described first example (FIG. 6), the monitoring system 100 monitors whether there is a signal from the platform staff to the driver from the stop to the departure of the railway vehicle 200, and when a signal to the driver is recognized, it can notify the driver to that effect.

[0244] [Another Example of the Monitoring System] Next, another example of the monitoring system 100 according to the present embodiment will be described.

[0245] Appropriate modifications and changes may be made to the monitoring system 100 (first example to fourth example) according to the above-described embodiment. Hereinafter, an example in which modifications and changes are made to the monitoring system 100 according to the above-described embodiment will be conveniently referred to as a "modified example".

[0246] For example, in the first example of the above-described monitoring system 100, the functions of the person recognition unit 120, the person classification unit 130, the posture estimation unit 140, the behavior recognition unit 150, and the notification unit 160 may be realized by one information processing device. Also, in the first example of the above-described monitoring system 100, the functions of the person recognition unit 120, the person classification unit 130, the posture estimation unit 140, the behavior recognition unit 150, and the notification unit 160 are realized in a distributed manner by five information processing devices, but may be realized in a distributed manner by two, three, or four information processing devices, or may be realized in a distributed manner by six or more information processing devices.

[0247] Also, in the second example of the above-described monitoring system 100, the functions of the person recognition unit 120, the posture estimation unit 140, the candidate recognition unit 152, the area extraction unit 154, the determination unit 156, and the notification unit 160 may be realized by one information processing device. Also, in the second example of the above-described monitoring system 100, the functions of the person recognition unit 120, the posture estimation unit 140, the candidate recognition unit 152, the area extraction unit 154, the determination unit 156, and the notification unit 160 are realized in a distributed manner by four information processing devices, but may also be realized in a distributed manner by two or three information processing devices, or may be realized in a distributed manner by five or more information processing devices.

[0248] Also, in the third example of the above-described monitoring system 100, the functions of the person recognition unit 120, the person classification unit 130, the posture estimation unit 140, the determination unit 153, the area extraction unit 154, the determination unit 156, and the notification unit 160 may be realized by one information processing device. Also, in the above-described monitoring system 100, the functions of the person recognition unit 120, the person classification unit 130, the posture estimation unit 140, the determination unit 153, the area extraction unit 154, the determination unit 156, and the notification unit 160 are realized in a distributed manner by five information processing devices, but may also be realized in a distributed manner by two, three, or four information processing devices, or may be realized in a distributed manner by six or more information processing devices.

[0249] Also, in the fourth example of the above-described monitoring system 100, the functions of the object recognition unit 170, the object tracking unit 180, the action recognition unit 150, and the notification unit 160 may be realized by one information processing device. Also, in the fourth example of the above-described monitoring system 100, the functions of the object recognition unit 170, the object tracking unit 180, the action recognition unit 150, and the notification unit 160 are realized in a distributed manner by four information processing devices, but may also be realized in a distributed manner by two or three information processing devices, or may be realized in a distributed manner by five or more information processing devices.

[0250] Also, in the first example of the above-described monitoring system 100 and its modified examples, the action recognition unit 150 may directly recognize a specific action of a person based on the data of a partial image including the person classified into a specific type by the person classification unit 130. In this case, for example, the posture estimation unit 140 may be omitted, and the output of the person classification unit 130 is taken in from the person classification unit 130 to the action recognition unit 150. Also, in this case, for example, the process of step S112 in FIG. 6 is omitted. For example, the action recognition unit 150 may recognize a specific action of a person using a trained model that has undergone supervised learning based on local feature amounts obtained from the data of the partial image or the image data itself, or may recognize a specific action of a person by a rule-based method such as pattern matching.

[0251] Also, in the second example of the above-described monitoring system 100 and its modified examples, the posture estimation unit 140 may directly estimate the posture state of a person from the image captured by the imaging unit 110. In this case, for example, the person recognition unit 120 may be omitted, and the output (image data) of the imaging unit 110 is taken in from the imaging unit 110 to the posture estimation unit 140. Also, in this case, for example, the processes of steps S204 and S206 in FIG. 9 are omitted.

[0252] Also, in the first example of the above-described monitoring system 100 and its modified examples, the action recognition unit 150 may have a configuration including a candidate recognition unit 152, a region extraction unit 154, and a determination unit 156, similar to the case of the second example described above.

[0253] Also, in the modified example of the first example of the above-described monitoring system 100, the second example, and its modified examples, the determination unit 156 may determine whether the similar action recognized by the candidate recognition unit 152 corresponds to a specific action unique to a specific type of person by tracking the movement of a specific object. In this case, for example, the determination unit 156 can track the movement of a specific object in the same manner as the object tracking unit 180 in the fourth example of the above-described monitoring system 100.

[0254] Also, in the third example of the above-described monitoring system 100 and its modified examples, the human classification unit 130 may perform classification processing on the person corresponding to the head recognized by the human recognition unit 120 based on a partial image of an extended area obtained by expanding the area including the head of the person recognized by the human recognition unit 120 to the periphery within the entire image acquired by the imaging unit 110. For example, the extended area is an area obtained by expanding the area including the head of the person recognized by the human recognition unit 120 downward to include the person's neck and shoulders. This is because the human classification unit 130 can classify the attributes of the person by using features such as the clothing of the person whose head has been recognized by the human recognition unit 120.

[0255] Also, in the third example of the above-described monitoring system 100 and its modified examples, the determination as to whether a person may be performing a specific action may be omitted. In this case, for example, the functions of the posture estimation unit 140 and the determination unit 153 are omitted, and the output of the human classification unit 130 is taken in by the region extraction unit 154 of the action recognition unit 150 from the human classification unit 130. Also, in this case, for example, the processes in steps S312 and S314 in FIG. 13 are omitted, and the process in step S310 is executed by the region extraction unit 154.

[0256] Also, in the third example of the above-described monitoring system 100 and its modified examples, the classification processing on the person corresponding to the head recognized by the human recognition unit 120 may be omitted. In this case, for example, the function of the human classification unit 130 is omitted, and the output of the human recognition unit 120 is taken in by the posture estimation unit 140 or the region extraction unit 154. Also, in this case, for example, the processes in steps S308 and S310 in FIG. 13 are omitted, and in step S312, posture estimation processing is executed for all of the heads recognized by the human recognition unit 120. Also, when the processes in steps S312 and S314 in FIG. 13 are omitted, in step S316, region extraction processing is executed for all of the heads recognized by the human recognition unit 120.

[0257] Also, in the fourth example of the above-described monitoring system 100 and its modified examples, the action recognition unit 150 may recognize specific actions unique to a specific type of person based on the presence or absence of a specific object in a specific area, similar to the determination unit 156 in the second example of the above-described monitoring system 100. In this case, for example, the object tracking unit 180 is omitted, and the output of the object recognition unit 170 is taken in from the object recognition unit 170 to the action recognition unit 150. Also, in this case, for example, the processes in steps S406, S408, and S410 in FIG. 17 are omitted, and in step S412, based on the result of the object recognition process of the object recognition unit 170, an action recognition process for the signal given by the station staff to the crew of the railway vehicle 200 is executed. More specifically, for example, in step S412 in FIG. 17, by adopting the determination processes corresponding to steps S406 and S408, the presence or absence of the action of the signal given by the station staff to the crew of the railway vehicle 200 is determined.

[0258] Also, in the fourth example of the above-described monitoring system 100 and its modified examples, the object recognition unit 170 may perform object recognition processing on a specific area that is a part of the entire image acquired by the imaging unit 110. In this case, since the imaging range of the imaging unit 110 is fixed in advance, the specific area is defined in advance as a fixed range within the entire image acquired by the imaging unit 110. Also, in this case, for example, the process in step S408 in FIG. 17 is omitted.

[0259] Also, in the first example, the second example of the above-described monitoring system 100 and their modified examples, although mainly described is the case where the signal given by the station staff to the crew of the railway vehicle 200 is the action to be recognized as an action unique to a specific type of person, actions unique to other types of people on the platform PF may also be the action to be recognized. For example, specific actions of a person requiring assistance (also referred to as a "person requiring care") using a wheelchair or a white cane (for example, the action of approaching the side where the railway vehicle 200 on the platform PF enters the line) may be the action to be recognized.

[0260] In addition, in the third and fourth examples of the above-described monitoring system 100 and their modified examples, although mainly described is the case where a signal given by a station staff member to the crew of the railway vehicle 200 is recognized as an action specific to a particular type of person, actions specific to other types of people on the platform PF may also be the recognition targets. For example, the action of a passenger on the platform PF raising their hand to call for help, the action of a passenger on the platform PF squatting down to look into another passenger who has fallen from the platform PF, the sitting or collapsing of a drunken passenger or the like on the platform PF may be the recognition targets.

[0261] [Function] Next, the functions of the monitoring system, monitoring device, monitoring method, and program according to the present embodiment will be described.

[0262] In the first aspect of the present embodiment, the monitoring system includes an imaging unit, an action recognition unit, and a notification unit. The monitoring system is, for example, the above-described monitoring system 100. The imaging unit is, for example, the above-described imaging unit 110. The action recognition unit is, for example, the above-described action recognition unit 150. The notification unit is, for example, the above-described notification unit 160. Specifically, the imaging unit is installed on the platform of the station and acquires an image of the platform. The station is, for example, the above-described station ST. The platform is, for example, the above-described platform PF. Also, the action recognition unit recognizes the actions of a particular type of person based on the image. The particular type of person is, for example, the station staff member on the above-described platform PF. Then, the notification unit notifies the recognition result by the action recognition unit.

[0263] Also, in the first aspect of the present embodiment, the monitoring device may include the person recognition unit, the person classification unit, the action recognition unit, and the notification unit.

[0264] Also, in the first aspect of the present embodiment, the information processing apparatus may execute a monitoring method. The monitoring method includes an action recognition step and a notification step. Specifically, in the action recognition step, the information processing apparatus recognizes the actions of a specific type of person based on an image of the platform obtained by an imaging unit installed on the platform of the station. Then, in the notification step, the information processing apparatus notifies the recognition result in the action recognition step.

[0265] Also, in the first aspect of the present embodiment, the information processing apparatus may be caused to execute a program. Specifically, the program causes the information processing apparatus to execute an action recognition step and a notification step. In the action recognition step, based on an image of the platform obtained by an imaging unit installed on the platform of the station, the actions of the person classified into the specific type in the person classification step are recognized. Then, in the notification step, the recognition result in the action recognition step is notified.

[0266] Thereby, a monitoring system, a monitoring apparatus, an information processing apparatus (hereinafter, "monitoring system etc.") can recognize (i.e., detect) and notify the specific actions of a specific type of person on the platform of the station. Therefore, for example, a person related to the operation of a railway vehicle can appropriately operate the railway vehicle while grasping the specific actions of a specific type of person on the platform according to the notification content.

[0267] Also, in the second aspect of the present embodiment, on the premise of the first aspect described above, the monitoring system may include a person recognition unit and a person classification unit. The person recognition unit is, for example, the person recognition unit 120 described above. The person classification unit is, for example, the person classification unit 130 described above. Specifically, the person recognition unit recognizes the person included in the image. Also, the person classification unit classifies the person recognized by the person recognition unit based on the image. Then, the action recognition unit may recognize the actions of the person classified into a specific type by the person classification unit based on the image.

[0268] Also, in the second aspect of the present embodiment, the monitoring apparatus may include the person recognition unit and the person classification unit.

[0269] Also, in the second aspect of the present embodiment, the monitoring method may include a human recognition step and a human classification step. Specifically, in the human recognition step, the information processing device recognizes a person included in the image. Further, in the human classification step, the information processing device classifies the person recognized in the human recognition step based on the image. Then, in the action recognition step, the information processing device may recognize the action of the person classified into a specific type in the human classification step based on the image.

[0270] Also, in the second aspect of the present embodiment, the program may cause the information processing device to execute the human recognition step and the human classification step. Then, the program may cause the information processing device to execute the action recognition step of recognizing the action of the person classified into a specific type in the human classification step based on the image.

[0271] Thereby, a monitoring system or the like can detect the unique actions of a specific type of person on the platform of a station.

[0272] Also, in the third aspect of the present embodiment, on the premise of the above-described second aspect, the monitoring system may include a posture estimation unit. The posture estimation unit is, for example, the above-described posture estimation unit 140. Specifically, the posture estimation unit may estimate the posture of the person classified into a specific type by the human classification unit based on the image. Then, the action recognition unit may recognize the action of the person classified into the specific type by the human classification unit based on the estimation result of the posture estimation unit.

[0273] Thereby, a monitoring system or the like can detect the unique actions of a specific type of person based on the posture state of the specific type of person included in the image.

[0274] Also, in the fourth aspect of the present embodiment, on the premise of the above-described second or third aspect, the action recognition unit may include a candidate recognition unit, a region extraction unit, and a determination unit. The candidate recognition unit is, for example, the above-described candidate recognition unit 152. The region extraction unit is, for example, the above-described region extraction unit 154. The determination unit is, for example, the above-described determination unit 156. Specifically, the candidate recognition unit may recognize candidates for the specific action by a person classified into the specific type by the person classification unit based on the image. Further, the region extraction unit may extract a first region corresponding to a body part characteristic of the specific action of the person performing the candidate action recognized by the candidate recognition unit from the image. The first region corresponding to the body part characteristic of the specific action is, for example, the above-described rectangular region AR3. Then, the determination unit may recognize the specific action by a person classified into the specific type by the person classification unit by determining whether the candidate action recognized by the candidate recognition unit corresponds to the specific action based on a partial image of the first region.

[0275] Thereby, a monitoring system or the like can more accurately detect an action unique to a specific type of person included in an image of a home.

[0276] Also, in the fifth aspect of the present embodiment, on the premise of the above-described fourth aspect, the determination unit may determine whether the candidate action recognized by the candidate recognition unit corresponds to the specific action by determining the presence or absence of a predetermined object related to the specific action in a partial image of the first region in the image. The predetermined object is, for example, a hand flag, a signal lamp, or the like that a station staff raises above the head when signaling to a crew member of a railway vehicle 200.

[0277] Thereby, a monitoring system or the like can more accurately detect an action unique to a specific type of person included in an image of a home.

[0278] Further, in the sixth aspect of the present embodiment, on the premise of the above-described second aspect, the human recognition unit may recognize a person included in the image by recognizing the head of the person based on the image.

[0279] Thereby, even in a case where, for example, a platform of a station is crowded and a part of the body of a person in the image is hidden by the body of another person, a monitoring system or the like can recognize the person by recognizing the head of the person in the image.

[0280] Further, in the seventh aspect of the present embodiment, on the premise of the above-described sixth aspect, the human classification unit may classify the person recognized by the human recognition unit based on a partial image of a second region corresponding to the head of the person recognized by the human recognition unit in the image.

[0281] Thereby, even in a case where, for example, a platform of a station is crowded and there is a person who can only be seen by the head in the image acquired by the imaging unit, a monitoring system or the like can classify the attributes of the person.

[0282] Further, in the eighth aspect of the present embodiment, on the premise of the above-described sixth or seventh aspect, the action recognition unit may recognize the action of the person classified as the specific type of person by the human classification unit based on a partial image of a third region defined with reference to a second region corresponding to the head of the person classified as the specific type of person by the human classification unit in the image. The second region is, for example, the rectangular region AR10 described above, and the third region is, for example, the rectangular region AR20 described above.

[0283] Thereby, even in a case where, for example, a monitoring system or the like can recognize the action of a specific type of person by using a partial image of a third region including characteristic body parts.

[0284] Further, in the ninth aspect of the present embodiment, on the premise of the above-described eighth aspect, the action recognition unit determines whether a person classified as the specific type of person by the person classification unit is likely to perform a specific action based on a partial image of the second region in the image. When there is a possibility that the person is performing the specific action, the action of the person classified as the specific type of person by the person classification unit may be recognized based on a partial image of the third region in the image.

[0285] Thereby, a monitoring system or the like can more accurately detect an action specific to a specific type of person included in an image of a home.

[0286] Further, in the tenth aspect of the present embodiment, on the premise of the above-described ninth aspect, a monitoring system or the like may include a posture estimation unit that estimates the posture of the head of a person classified as the specific type based on a partial image of the second region in the image by the person classification unit. The posture estimation unit is, for example, the above-described posture estimation unit 140. Then, the action recognition unit may determine whether a person classified as the specific type of person by the person classification unit is likely to perform a specific action based on the estimation result of the posture estimation unit.

[0287] Thereby, a monitoring system or the like can determine whether a person classified as a specific type of person is likely to perform a specific action.

[0288] Further, in the eleventh aspect of the present embodiment, on the premise of the above-described first aspect, in a monitoring system or a monitoring device, the action recognition unit may recognize an action specific to a specific type of person included in the image.

[0289] Further, in the eleventh aspect of the present embodiment, in the action recognition step of the monitoring method, an information processing device may recognize an action specific to a specific type of person included in the image.

[0290] Further, in the eleventh aspect of the present embodiment, the program may cause the information processing apparatus to execute the action recognition step of recognizing actions specific to a specific type of person included in the image.

[0291] As a result, a monitoring system or the like can recognize (i.e., detect) and notify actions specific to a specific type of person on the platform of a station. Therefore, for example, a person related to the operation of a railway vehicle can appropriately operate the railway vehicle while grasping the actions specific to a specific type of person on the platform according to the notification content.

[0292] Further, in the twelfth aspect of the present embodiment, on the premise of the above-described eleventh aspect, a monitoring system or the like may include a person recognition unit. The person recognition unit is, for example, the above-described person recognition unit 120. Specifically, the person recognition unit recognizes a person included in the image. Then, the action recognition unit may recognize the specific action by the person recognized by the person recognition unit based on the image.

[0293] As a result, a monitoring system or the like can recognize a person included in an image of a platform and, for the recognized person, recognize the presence or absence of an action specific to a specific type of person.

[0294] Further, in the thirteenth aspect of the present embodiment, on the premise of the above-described eleventh or twelfth aspect, a monitoring system or the like may include a posture estimation unit. The posture estimation unit is, for example, the above-described posture estimation unit 140. Specifically, the posture estimation unit may estimate the posture of a person included in the image. Then, the action recognition unit may recognize the specific action included in the image based on the estimation result of the posture estimation unit.

[0295] As a result, a monitoring system or the like can detect an action specific to a specific type of person based on the posture state of the specific type of person included in the image.

[0296] Also, in the 14th aspect of the present embodiment, on the premise of any one of the 11th to 13th aspects described above, the action recognition unit may include a candidate recognition unit, a region extraction unit, and a determination unit. The candidate recognition unit is, for example, the candidate recognition unit 152 described above. The region extraction unit is, for example, the region extraction unit 154 described above. The determination unit is, for example, the determination unit 156 described above. Specifically, the candidate recognition unit may recognize candidates for the specific action included in the image. Also, the region extraction unit may extract a fourth region corresponding to a body part characteristic of the specific action of the person performing the candidate action from the image. The fourth region corresponding to the body part characteristic of the specific action is, for example, the rectangular region AR3 described above. Then, the determination unit may recognize the specific action included in the image by determining whether the candidate action recognized by the candidate recognition unit corresponds to the specific action based on the partial image of the region.

[0297] As a result, a monitoring system or the like can more accurately detect specific actions unique to a specific type of person included in a home image.

[0298] Also, in the 15th aspect of the present embodiment, on the premise of the 14th aspect described above, the determination unit may determine whether the candidate action recognized by the candidate recognition unit corresponds to the specific action by determining the presence or absence of a predetermined object related to the specific action in the partial image. The predetermined object is, for example, a hand flag or a signal lamp that a station staff raises above their head when signaling to the crew of the railway vehicle 200.

[0299] As a result, a monitoring system or the like can more accurately detect specific actions unique to a specific type of person included in a home image.

[0300] Further, in the 16th aspect of the present embodiment, the monitoring system or the like may include an object recognition unit that recognizes a predetermined object related to the specific behavior based on the image. The object recognition unit is, for example, the above-described object recognition unit 170. The predetermined object is, for example, a hand flag, a signal lamp, or the like that a station staff raises above the head when giving a signal to the crew of the railway vehicle 200. Then, the behavior recognition unit may recognize the specific behavior based on whether the predetermined object is recognized by the object recognition unit.

[0301] Thereby, the monitoring system or the like can recognize the behavior specific to a specific type of person.

[0302] Further, in the 17th aspect of the present embodiment, on the premise of the above-described 16th aspect, the monitoring system or the like may include an object tracking unit that tracks the time-series movement of the predetermined object recognized by the object recognition unit. The object tracking unit is, for example, the above-described object tracking unit 180. Then, the behavior recognition unit may recognize the specific behavior based on the tracking result of the predetermined object by the object tracking unit.

[0303] Thereby, the monitoring system or the like can more accurately detect the behavior specific to a specific type of person included in the image of the home.

[0304] Further, in the 18th aspect of the present embodiment, on the premise of the above-described 17th aspect, the specific behavior may be a behavior performed by raising the predetermined object above the head. Then, the object tracking unit may track the time-series movement of the predetermined object recognized by the object recognition unit in a fifth region corresponding to above the head of the person on the home in the image by the object recognition unit. The fifth region corresponding to above the head of the person on the home is, for example, the above-described specific region AR40.

[0305] Thereby, the monitoring system or the like can more accurately detect the behavior specific to a specific type of person included in the image of the home.

[0306] Also, in the 19th aspect of the present embodiment, on the premise of any one of the above-described 16th to 18th aspects, the unique action may be an action of lifting the predetermined object above the head. And the action recognition unit may recognize the unique action based on a partial image of a fifth region corresponding to above the head of the person on the home in the image.

[0307] Thereby, a monitoring system or the like can detect a unique action of a specific type of person included in an image of a home with higher accuracy and a lower processing load.

[0308] Also, in the 20th aspect of the present embodiment, on the premise of any one of the above-described 1st to 19th aspects, the specific type of person may be a station staff member. And the action recognition unit may recognize an action for sending a predetermined signal by the station staff member.

[0309] Thereby, a monitoring system or the like can recognize (i.e., detect) and notify the actions of a specific type of person on the home. Therefore, for example, a person related to the operation of a railway vehicle can appropriately operate the railway vehicle while grasping the actions of a specific type of person on the home according to the notification content.

[0310] Also, in the 21st aspect of the present embodiment, on the premise of any one of the above-described 1st to 20th aspects, the notification unit may notify the crew of the railway vehicle of the recognition result by the action recognition unit through an output device inside the railway vehicle that is stopped at the home. The railway vehicle is, for example, the above-described railway vehicle 200. The output device inside the railway vehicle is, for example, a display device installed on the driver's cab of the above-described railway vehicle 200 or a portable terminal device that a crew member brings into the railway vehicle 200.

[0311] As a result, the crew of a railway vehicle can surely confirm the behavior of specific types of people on the platform of a station inside the railway vehicle. In particular, in the case of a single-person operated railway vehicle 200, the crew member (i.e., the driver) cannot leave the driver's cab, and there may be a situation where it is difficult to check the situation on the platform. However, the crew member can surely grasp the behavior of specific types of people on the platform inside the railway vehicle.

[0312] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims.

Description of Reference Numerals

[0313] 100 Monitoring system 110 Imaging unit 120 Person recognition unit 130 Person classification unit 140 Posture estimation unit 150 Action recognition unit 152 Candidate recognition unit 153 Judgment unit 154 Region extraction unit 156 Judgment unit 160 Notification unit 170 Object recognition unit 180 Object tracking unit 200 Railway vehicle PF Platform ST Station

Claims

1. An imaging unit installed on a station platform for acquiring an image of the platform; An action recognition unit for recognizing the actions of a specific type of person based on the image; A notification unit for notifying the recognition result by the action recognition unit; A monitoring system.

2. A person recognition unit for recognizing a person included in the image; A person classification unit for classifying the person recognized by the person recognition unit based on the image; The action recognition unit recognizes the actions of the person classified into a specific type by the person classification unit based on the image; The monitoring system according to Claim 1.

3. A posture estimation unit for estimating the posture of the person classified into the specific type by the person classification unit based on the image; The action recognition unit recognizes the actions of the person classified into the specific type by the person classification unit based on the estimation result of the posture estimation unit; The monitoring system according to Claim 2.

4. The action recognition unit includes a candidate recognition unit, a region extraction unit, and a determination unit; The candidate recognition unit recognizes candidates for a specific action by the person classified into the specific type by the person classification unit based on the image; The region extraction unit extracts a first region corresponding to a body part characteristic of the specific action of the person performing the candidate action recognized by the candidate recognition unit from the image; The determination unit recognizes the specific action by the person classified into the specific type by the person classification unit by determining whether the candidate action recognized by the candidate recognition unit corresponds to the specific action based on a partial image of the first region in the image; The monitoring system according to Claim 2 or 3.

5. The determination unit determines whether the candidate action recognized by the candidate recognition unit corresponds to the specific action by determining the presence or absence of a predetermined object related to the specific action in the partial image of the first region in the image; The monitoring system according to Claim 4.

6. The person recognition unit recognizes a person included in the image by recognizing the head of the person based on the image; The monitoring system according to Claim 2.

7. The person classification unit classifies the person recognized by the person recognition unit based on a partial image of a second region corresponding to the head of the person recognized by the person recognition unit in the image; The monitoring system according to Claim 6.

8. The action recognition unit recognizes the action of a person classified as the specific type of person by the person classification unit based on a partial image of a third region defined with reference to a second region corresponding to the head of the person classified as the specific type of person by the person classification unit in the image. The monitoring system according to claim 6 or 7.

9. The action recognition unit determines whether a person classified as the specific type of person by the person classification unit may be performing a specific action based on a partial image of the second region in the image, and when there is a possibility that the specific action is being performed, recognizes the action of the person classified as the specific type of person by the person classification unit based on a partial image of the third region in the image. The monitoring system according to claim 8.

10. A posture estimation unit is provided that estimates the posture of the head of a person classified as the specific type by the person classification unit based on a partial image of the second region in the image. The action recognition unit determines whether a person classified as the specific type of person by the person classification unit may be performing a specific action based on the estimation result of the posture estimation unit. The monitoring system according to claim 9.

11. The action recognition unit recognizes an action unique to the specific type of person included in the image. The monitoring system according to claim 1.

12. A person recognition unit is provided that recognizes a person included in the image. The action recognition unit recognizes the unique action by the person recognized by the person recognition unit based on the image. The monitoring system according to claim 11.

13. A posture estimation unit is provided that estimates the posture of a person included in the image. The action recognition unit recognizes the unique action included in the image based on the estimation result of the posture estimation unit. The monitoring system according to claim 11 or 12.

14. The action recognition unit includes a candidate recognition unit, a region extraction unit, and a determination unit. The candidate recognition unit recognizes a candidate for the unique action included in the image. The region extraction unit extracts a fourth region corresponding to a body part characteristic of the unique action of a person performing the candidate action from the image. The determination unit recognizes the unique action included in the image by determining whether the candidate action recognized by the candidate recognition unit corresponds to the specific action based on a partial image of the fourth region. The monitoring system according to claim 11 or 12.

15. The determination unit determines whether the candidate action recognized by the candidate recognition unit corresponds to the specific action by determining the presence or absence of a predetermined object related to the specific action in the partial image. The monitoring system according to claim 14.

16. Comprising an object recognition unit that recognizes a predetermined object related to the specific action based on the image. The action recognition unit recognizes the specific action based on whether or not the predetermined object is recognized by the object recognition unit. The monitoring system according to claim 11.

17. Comprising an object tracking unit that tracks the temporal movement of the predetermined object recognized by the object recognition unit. The action recognition unit recognizes the specific action based on the tracking result of the predetermined object by the object tracking unit. The monitoring system according to claim 16.

18. The specific action is an action performed by lifting the predetermined object above the head. The object tracking unit tracks the temporal movement of the predetermined object recognized by the object recognition unit in a fifth region corresponding to above the head of the person on the platform in the image recognized by the object recognition unit. The monitoring system according to claim 17.

19. The specific action is an action performed by lifting the predetermined object above the head. The action recognition unit recognizes the specific action based on a partial image of a fifth region corresponding to above the head of the person on the platform in the image. The monitoring system according to any one of claims 16 to 18.

20. The specific type of person is a station staff member. The action recognition unit recognizes an action for sending a predetermined signal by the station staff member. The monitoring system according to claim 1, 2, 3, 6, 7, 11, 12, 16, 17 or 18.

21. The notification unit notifies the crew of the railway vehicle of the recognition result by the action recognition unit through an output device inside the railway vehicle that is stopped at the platform. The monitoring system according to claim 1, 2, 3, 6, 7, 11, 12, 16, 17 or 18.

22. An action recognition unit that recognizes the actions of a specific type of person based on an image of the platform acquired by an imaging unit installed on the platform of the station, and A notification unit that notifies the recognition result by the action recognition unit. Monitoring device.

23. An action recognition step in which an information processing device recognizes the actions of a specific type of person based on an image of the platform acquired by an imaging unit installed on the platform of the station. An information processing apparatus includes a notification step of notifying the recognition result in the action recognition step. Monitoring method.

24. An information processing apparatus An action recognition step of recognizing the actions of a specific type of person based on an image of the platform obtained by an imaging unit installed on the platform of the station, and A notification step of notifying the recognition result in the action recognition step, are executed. Program.

Images