Monitoring systems, monitoring devices, monitoring methods, programs

Abstract

This technology provides the ability to detect the operation of special signal emitters. [Solution] A monitoring system 1 according to one embodiment includes a camera 10 that images the front of a railway vehicle 100, a partial image extraction unit 201 that extracts a partial image of a light-emitting region including a light-emitting object based on the image captured by the camera 10, a light-emitting state determination unit 202 that determines the light-emitting state of a light-emitting object visible in the light-emitting region based on a plurality of time-series partial images of light-emitting regions extracted by the partial image extraction unit 201, and an operation determination unit 203 that determines whether or not the special signal light-emitting device 150 is operating based on the light-emitting state determined by the light-emitting state determination unit 202.

Description

Technical Field

[0001] The present disclosure relates to a monitoring system and the like.

Background Art

[0002] Conventionally, a technique for detecting the operation (light emission) of a special signal emitter that instructs an immediate stop of a railway vehicle to the driver of the railway vehicle based on an imaging image in front of the railway vehicle is known (see, for example, Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is room for improvement in the technique for detecting the operation of the special signal emitter.

[0005] Therefore, in view of the above problems, an object is to provide a technique capable of detecting the operation of a special signal emitter.

Means for Solving the Problems

[0006] To achieve the above object, in one embodiment of the present disclosure, an imaging unit that images the front of a railway vehicle; an extraction unit that extracts a partial image of a first region including a light-emitting object based on the imaging image of the imaging unit; a light emission state determination unit that determines the light emission state of the object based on the time-series history of the partial image of the first region extracted by the extraction unit; The system includes an operation determination unit that determines whether or not the special signal light emitter is operating based on the light emission state determined by the light emission state determination unit. A monitoring system will be provided.

[0007] In other embodiments of this disclosure, An extraction unit extracts a partial image of a first region containing a luminescent object based on the image captured by an imaging unit that images the front of a railway vehicle, A light emission state determination unit determines the light emission state of the object based on the time-series history of the partial image of the first region extracted by the extraction unit, The system includes an operation determination unit that determines whether or not the special signal light emitter is operating based on the light emission state determined by the light emission state determination unit. A monitoring device will be provided.

[0008] Furthermore, in yet another embodiment of this disclosure, A monitoring method performed by a monitoring device or monitoring system, Based on the image captured by the imaging unit that images the front of the railway vehicle, an extraction step is performed to extract a partial image of a first region containing a luminescent object, A light emission state determination step, which determines the light emission state of the object based on the time-series history of the partial image of the first region extracted in the extraction step, The process includes an operation determination step that determines whether or not the special signal light emitter is operating based on the light emission state determined in the light emission state determination step, A monitoring method will be provided.

[0009] Furthermore, in yet another embodiment of this disclosure, In an information processing device, Based on the image captured by the imaging unit that images the front of the railway vehicle, an extraction step is performed to extract a partial image of a first region containing a luminescent object, A light emission state determination step, which determines the light emission state of the object based on the time-series history of the partial image of the first region extracted in the extraction step, Based on the light emission state determined in the light emission state determination step, execute an operation determination step for determining the presence or absence of the operation of the special signal emitter. A program is provided.

Effect of the Invention

[0010] According to the above embodiment, the operation of the special signal emitter can be detected.

Brief Description of the Drawings

[0011] [Figure 1] It is a diagram showing the configuration of the first example of the monitoring system. [Figure 2] It is a diagram showing a specific example of the captured image of the camera in which the special signal emitter is shown. [Figure 3] It is a diagram for explaining an example of the processing of the partial image extraction unit. [Figure 4] It is a diagram for explaining an example of the processing of the light emission state determination unit. [Figure 5] It is a flowchart schematically showing the first example of the processing related to the monitoring of the operation of the special signal emitter by the monitoring device. [Figure 6] It is a diagram showing the configuration of the second example of the monitoring system. [Figure 7] It is a diagram showing an example of the area corresponding to the special signal emitter detected by the object detection unit in the captured image of the camera. [Figure 8] It is a diagram for explaining an example of the processing of the background separation unit. [Figure 9] It is a flowchart schematically showing the second example of the processing related to the monitoring of the operation of the special signal emitter by the monitoring device. [Figure 10] It is a diagram showing the configuration of the third example of the monitoring system. [Figure 11] It is a diagram for explaining an example of the processing of the light emission state determination unit. [Figure 12] It is a flowchart schematically showing the third example of the processing related to the monitoring of the operation of the special signal emitter by the monitoring device.

Modes for Carrying Out the Invention

[0012] The embodiments will be described below with reference to the drawings.

[0013] [First example of a monitoring system] A first example of the monitoring system 1 according to this embodiment will be described with reference to Figures 1 to 4.

[0014] Figure 1 shows the configuration of a first example of the monitoring system 1. Figure 2 shows a specific example of an image captured by camera 10, showing the special signal light emitter 150. Figure 3 is a diagram illustrating an example of the processing of the partial image extraction unit 201. Figure 4 is a diagram illustrating an example of the processing of the light emission state determination unit 202.

[0015] Specifically, Figure 2 includes Figures 2A and 2B. Figure 2A is an image P21 showing an example of the special signal emitter 150 in a non-operating state (i.e., when not emitting light), and Figure 2B is an image P22 showing an example of the special signal emitter 150 in an operating state (i.e., when emitting light).

[0016] The monitoring system 1 monitors whether or not the special signal light emitter 150 (see Figure 2, etc.), which is installed along the track TK on which the railway vehicle 100 travels, is operating (i.e., emitting light).

[0017] The monitoring system 1 includes a camera 10 and a monitoring device 20.

[0018] Camera 10 is mounted on the railway vehicle 100 and captures images of the area in front of the railway vehicle 100, including the areas adjacent to the left and right sides of the track TK, and outputs the captured images. As a result, as shown in Figure 2, camera 10 can acquire captured images showing the special signal lighters 150 that are positioned along the track TK in front of the railway vehicle 100 while the railway vehicle 100 is in motion.

[0019] Camera 10 is, for example, a monocular camera. Alternatively, camera 10 may be a three-dimensional camera, such as a stereo camera or a depth camera, capable of acquiring information representing the distance to objects in the captured image in addition to the two-dimensional image.

[0020] For example, camera 10 is mounted on the leading car of the railway vehicle 100. Camera 10 may be placed inside or outside the railway vehicle 100, as long as it can capture images of the area in front of the railway vehicle 100. In the former case, camera 10 captures images of the area in front of the railway vehicle 100 through a transparent window.

[0021] The output image (i.e., captured image) from camera 10 is received by the monitoring device 20 via a predetermined communication line.

[0022] For example, when the monitoring device 20 is mounted on a railway vehicle 100, the specified communication lines include one-to-one communication lines, a local area network (LAN) mounted on the railway vehicle 100, and short-range communication lines based on wireless standards such as WiFi and Bluetooth®. Also, for example, when the monitoring device 20 is installed outside the railway vehicle 100, the specified communication lines include, for example, a mobile communication network with a base station as its endpoint, a satellite communication network using communication satellites, and a wide area network (WAN) such as the Internet.

[0023] The monitoring device 20 monitors whether or not the special signal light emits light, which is positioned along the track TK on which the railway vehicle 100 travels, is operating (emitting light), based on the output image (imaging image) captured from the camera 10.

[0024] The functions of the monitoring device 20 can be realized by any configuration, such as any hardware or any combination of hardware and software. For example, the monitoring device 20 is centered around a computer that includes a processor, a memory device (also called "main memory"), an auxiliary storage device, and an interface device for input / output with the outside world. This allows the monitoring device 20 to realize an emergency monitoring function, for example, by loading a program installed in the auxiliary storage device into the memory device and executing it with the processor. The processor includes, for example, a CPU (Central Processing Unit). The processor may also include, for example, a GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), or ASIC (Application Specific Integrated Circuit). The memory device is, for example, SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The auxiliary storage device is, for example, EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The interface device includes, for example, an interface for communicating with external devices and an interface for reading data from a portable recording medium. This allows the monitoring device 20 to download processing programs and various data from external sources via communication, and to read processing programs and various data from a portable recording medium.

[0025] For example, the monitoring device 20 is a control device (controller) mounted on the railway vehicle 100. The monitoring device 20 is a dedicated control device that performs processing specifically for the function of monitoring whether or not the special signal light 150 is operating (hereinafter, for convenience, referred to as the "special signal monitoring function") among the various functions of the railway vehicle 100. Alternatively, the monitoring device 20 may be a general-purpose control device that performs processing related to the special signal monitoring function along with processing related to other functions among the various functions of the railway vehicle 100.

[0026] Furthermore, the monitoring device 20 may be installed outside the railway vehicle 100 in a manner that enables communication with the railway vehicle 100. The monitoring device 20 is, for example, a server device with relatively high processing power installed outside the railway vehicle 100. The server device may be an on-premise server, a cloud server, or an edge server. The monitoring device 20 may also be a terminal device with relatively low processing power. The terminal device may be, for example, a stationary terminal device such as a desktop PC (Personal Computer), or a portable terminal device (i.e., a mobile terminal) such as a smartphone, tablet, or laptop PC.

[0027] As shown in Figure 2, the special signal light 150 is of the rotary type. The rotary type special signal light 150 is a special signal light in which five light-emitting parts 151 (see Figure 3) are arranged to form the vertices of a pentagon. When the rotary type special signal light 150 is in operation, two adjacent light-emitting parts 151 among the five light-emitting parts 151 light up (illuminate), and the combination of the two light-emitting parts 151 that light up changes in a counterclockwise direction. In the rotary type special signal light 150, the frequency at which the combination of the two light-emitting parts 151 that light up changes is, for example, about 1 Hz. In this case, the combination of the two light-emitting parts 151 in the rotary type special signal light 150 changes sequentially in a counterclockwise direction at a frequency of about 60 times per minute.

[0028] Furthermore, the special signal light 150 may be of the flashing type. The flashing type special signal light 150 has a vertically elongated light-emitting part that extends vertically and flashes at a unique frequency that is completely different from that of a typical LED (Light Emitted Diode) light. In the flashing type special signal light 150, the flashing frequency of the light-emitting part is, for example, 8.3 Hz, in which case the flashing type special signal light 150 flashes at a frequency of approximately 500 times / minute.

[0029] The monitoring device 20 includes, as functional units, a partial image extraction unit 201, a light emission state determination unit 202, an operation determination unit 203, and a notification unit 204.

[0030] Furthermore, the functions of the partial image extraction unit 201, the light emission state determination unit 202, the operation determination unit 203, and the notification unit 204 may be implemented by two or more devices in a distributed manner. For example, the functions of the partial image extraction unit 201, the light emission state determination unit 202, the operation determination unit 203, and the notification unit 204 may be implemented by four different devices. Alternatively, the functions of the partial image extraction unit 201, the light emission state determination unit 202, the operation determination unit 203, and the notification unit 204 may be implemented by two or three devices in a distributed manner, or by five or more devices in a distributed manner.

[0031] The partial image extraction unit 201 extracts a partial image based on the output image (captured image) of the camera 10 to determine whether or not the special signal light emitter 150 is operating (emitting light). Specifically, the partial image extraction unit 201 extracts a partial image based on the output image of the camera 10 of the region containing a candidate object that emits light (hereinafter referred to as the "emitted region"), which corresponds to the light-emitting part 151 of the special signal light emitter 150 when it emits light.

[0032] For example, the partial image extraction unit 201 applies known image processing techniques based on the image captured by the camera 10 to extract a partial image of the light-emitting region, specifically the region of the same color (red) as the light-emitting color of the light-emitting unit 151 in the special signal light-emitting device 150. Specifically, for example, the partial image extraction unit 201 extracts a partial image of the region of the same color as the light-emitting color of the light-emitting unit 151 in the special signal light-emitting device 150, based on a comparison between the brightness value of each pixel in the image captured by the camera 10 and the unique brightness value corresponding to the light-emitting color of the light-emitting unit 151. This allows for the extraction of partial images P32 of the light-emitting region corresponding to the two light-emitting units 151 that are emitting light out of the five light-emitting units 151 in the special signal light-emitting device 150, from the captured image P31 showing the special signal light-emitting device 150 in operation (light emission), as shown in Figure 3.

[0033] Alternatively, the partial image extraction unit 201 may use the trained model LM1 to extract a partial image of the region corresponding to the light-emitting part 151 of the special signal light emitter 150 that is emitting light, based on the image captured by the camera 10, as a partial image of the light-emitting region.

[0034] The trained model LM1 takes, for example, local features obtained from an image or the image itself as input and outputs a region corresponding to the emitting light-emitting part 151 in the image (for example, a rectangular region surrounding the emitting light-emitting part 151). If no region corresponding to the emitting light-emitting part 151 in the image is output, it means that the emitting light-emitting part 151 was not detected (i.e., recognized) in the image. The trained model LM1 is obtained, for example, by machine learning (specifically, supervised learning) of the base training model using a training dataset. The trained model LM1 may be generated by machine learning of the base training model in the monitoring device 20, or by machine learning of the base training model in an information processing device other than the monitoring device 20. For example, each training data included in the training dataset consists of a combination of local features obtained from an image or the image itself as input data and data specifying the region corresponding to the emitting light-emitting part 151 on the image as output data. As a result, the trained model LM1 can take local features obtained from the image or the image itself as input to detect (recognize) the light-emitting parts 151 on the image and output data for the region (e.g., a rectangular region) corresponding to the light-emitting parts 151 on the image. For example, the trained model LM1 is mainly composed of a deep neural network (DNN), and the machine learning of the DNN is made more efficient by applying backpropagation (error backpropagation) based on training data. The trained model LM1 may also include a U-Net that can recognize the light-emitting parts 151 from the image when the image is taken as input.

[0035] Furthermore, the partial image extraction unit 201 may detect (recognize) the external shape of the special signal light emitter 150 in a front view, specifically a pentagon shape, and extract a partial image of the region containing the pentagon shape as a partial image of the light-emitting region. Alternatively, the partial image extraction unit 201 may detect (recognize) two adjacent circular shapes corresponding to the two light-emitting parts 151 of the special signal light emitter 150, and extract a partial image of the region containing the two circular shapes as a partial image of the light-emitting region.

[0036] For example, the partial image extraction unit 201 uses a trained model LM2 to detect specific shapes in the image captured by the camera 10, such as a pentagon or two circular shapes, based on local features obtained from the image captured by the camera 10 or the image itself.

[0037] The trained model LM2 takes local features obtained from an image or the image itself as input and outputs data specifying a region corresponding to a specific shape in the image (for example, a rectangular region surrounding a specific shape). If the trained model LM2 does not output a region corresponding to a specific shape in the image, it means that the specific shape was not detected in the image. The trained model LM2 is obtained, for example, by machine learning (specifically, supervised learning) of the base trained model using a training dataset. The trained model LM2 may be generated by machine learning of the base trained model in the monitoring device 20, or by machine learning of the base trained model in an information processing device separate from the monitoring device 20. For example, each training data included in the training dataset consists of a combination of local features obtained from an image or the image data itself as input data and data specifying a region corresponding to a specific shape on the image as output data. As a result, the trained model LM2 can take local features obtained from an image or the image itself as input, detect (recognize) a specific shape on the image, and output data of a region corresponding to the specific shape on the image (for example, a rectangular region). For example, the pre-trained model LM2 is primarily composed of a deep neural network (DNN), and the DNN's machine learning is made more efficient by applying backpropagation based on training data. Furthermore, the pre-trained model LM2 may also include a U-Net capable of recognizing specific shapes from images taken as input.

[0038] Furthermore, the partial image extraction unit 201 may not be able to extract a partial image of the light-emitting region. This is because the light-emitting object (specifically, the special signal light emitter 150) may not be visible in the image captured by the camera 10.

[0039] The emission state determination unit 202 determines the emission state of a candidate object shown in a partial image based on a group of partial images (i.e., multiple partial images) of the emission region in a time series over a predetermined period, extracted by the partial image extraction unit 201. The predetermined period is, for example, a few seconds to about ten seconds.

[0040] For example, the light emission state determination unit 202 determines whether two circular candidate objects that are emitting light are rotating counterclockwise at a unique frequency or period, based on a group of partial images (i.e., multiple partial images) extracted by the partial image extraction unit 201 over the most recent predetermined period. The unique frequency or period is the frequency or period at which the combination of the two light-emitting units 151 in the rotating special signal light emitter 150 changes counterclockwise. For example, the light emission state determination unit 202 acquires the brightness value for each pixel for each of the multiple time-series partial images over the most recent predetermined period, and determines whether two circular candidate objects that are emitting light are changing to rotate counterclockwise at a unique frequency or period, based on the change in brightness value on the partial images over time (specifically, the distribution of that brightness value). This allows the operation determination unit 203 to determine whether the candidate objects are the light-emitting units 151 of the rotating special signal light emitter 150.

[0041] For example, as shown in Figure 4, when the rotating special signal light emitter 150 is operating (emitting light), the light emission state determination unit 202 can determine, based on multiple time-series partial images P41, P42, and P43, that the two light-emitting parts 151 (the textured areas in the figure) are rotating counterclockwise at a specific frequency or period.

[0042] Furthermore, the light emission state determination unit 202 may use a machine learning-based or rule-based classifier to classify each of the multiple partial images into one of several types of light emission patterns, including five types of light emission patterns of the rotating special signal light emitter 150 corresponding to the light emission region. The five types of light emission patterns of the rotating special signal light emitter 150 refer to the combinations of two adjacent light emitters among the five light emitters 151 that are emitting light. Based on the classification results for each partial image of the light emission region in the time series, the light emission state determination unit 202 determines whether the two circular candidate objects that are emitting light are rotating counterclockwise at a unique frequency or period.

[0043] The machine learning-based classifier is a pre-trained model LM3 that classifies the emission patterns of candidate luminescent objects on an image, taking local features obtained from the image or the image itself as input. The pre-trained model LM3 is obtained, for example, by machine learning (specifically, supervised learning) of the base training model using a training dataset. The pre-trained model LM3 may be generated by machine learning of the base training model in the monitoring device 20, or by machine learning of the base training model in an information processing device separate from the monitoring device 20. For example, each training data included in the training dataset consists of a combination of local features obtained from the image as input data or the image data itself, and a classification result as output data. This allows the pre-trained model LM3 to classify the emission patterns of candidate luminescent objects on an image, taking local features obtained from the image or the image itself as input. For example, the pre-trained model LM3 is mainly composed of a deep neural network (DNN), and the machine learning of the DNN is made more efficient by applying backpropagation based on training data. Alternatively, the pre-trained model LM3 may be a support vector machine.

[0044] Rule-based classifiers use rule-based algorithms, such as pattern matching, to classify the emission patterns of candidate luminescent objects in an image.

[0045] Furthermore, the emission state determination unit 202 may determine whether the two emitting circular candidate objects are rotating counterclockwise at a specific frequency or period by applying methods for analyzing the movement of objects, such as background subtraction or optical flow, based on multiple time-series partial images extracted by the partial image extraction unit 201.

[0046] Furthermore, the light emission state determination unit 202 may determine whether or not a candidate object with a vertical shape is flashing at a unique frequency based on a plurality of time-series partial images extracted by the partial image extraction unit 201 over a predetermined period. The unique frequency is the flashing frequency unique to the flashing type special signal light emitter 150. For example, the light emission state determination unit 202 acquires the distribution of luminance values ​​for each of the plurality of time-series partial images over a predetermined period, and determines whether or not a candidate object with a vertical shape is flashing at a unique frequency based on the change in the luminance values ​​(distribution) of the partial images over time. This allows the operation determination unit 203 to determine whether or not the candidate object is a flashing type special signal light emitter 150.

[0047] Furthermore, as described above, there are cases where a candidate object that is emitting light is not captured in the image, and a partial image of the emitting region in the image is not extracted. Therefore, if there is no history data of a partial image for the most recent predetermined period, the light emission state determination unit 202 may determine that there is no candidate object that is emitting light in the first place.

[0048] The operation determination unit 203 determines whether or not the special signal light emitter 150 is operating based on the determination result of the light emission state determination unit 202.

[0049] For example, the operation determination unit 203 determines that the special signal light emitter 150 is operating (emitting light) if the light emission state determination unit 202 determines that the two circular candidate objects that are emitting light are rotating counterclockwise at a specific frequency. Alternatively, the operation determination unit 203 may determine in this case that the rotating type special signal light emitter 150 is operating (emitting light).

[0050] Furthermore, for example, if the light emission state determination unit 202 determines that a vertically elongated candidate object that is emitting light is blinking at a specific frequency, the operation determination unit 203 determines that the special signal light emitter 150 is operating (emitting light). Alternatively, the operation determination unit 203 may determine in this case that a blinking type special signal light emitter 150 is operating (emitting light).

[0051] If the operation determination unit 203 determines that the special signal light 150 is operating (emitting light), the notification unit 204 notifies the driver of the railway vehicle 100 that the special signal light 150 is operating (emitting light).

[0052] For example, the notification unit 204 outputs a control signal to a monitor or indicator installed in the driver's cab of the railway vehicle 100 via a predetermined communication line. This allows the notification unit 204 to visually notify the driver that the special signal light 150 is operating (emitting light) via the monitor or indicator in the driver's cab of the railway vehicle 100.

[0053] Furthermore, the notification unit 204 outputs a control signal to a speaker, buzzer, etc., installed in the driver's cab of the railway vehicle 100 via a predetermined notification line. This allows the notification unit 204 to notify the driver, through the speaker, buzzer, etc., in the driver's cab of the railway vehicle 100, in an audible manner that the special signal lighter 150 is operating (emitting light).

[0054] Thus, in this example, the monitoring device 20 can determine whether or not the special signal emitter 150 is operating (emitting light) by extracting partial images of the light-emitting region from the image captured by the camera 10 and analyzing the extracted time-series partial image group.

[0055] Furthermore, in this example, the monitoring device 20 can determine whether or not the special signal emitter 150 is operating (emitting light) by analyzing the extracted time-series partial image group, regardless of whether the special signal emitter 150 is of the rotating or flashing type.

[0056] [Example 1 of a process for monitoring the operation status of a special signal light] Referring to Figure 5, a first example of the monitoring process for the operation (illumination) of the special signal emitter 150 by the monitoring device 20 will be described.

[0057] Figure 5 is a flowchart illustrating a schematic example of the process for monitoring whether the special signal emitter 150 is operating using the monitoring device 20.

[0058] This flowchart is executed repeatedly at predetermined processing cycles, for example, while the railway vehicle 100 is in motion. The same applies to the flowcharts in Figures 9 and 12 described later.

[0059] In step S102, the partial image extraction unit 201 acquires the latest image captured by the camera 10.

[0060] Once the process in step S102 is complete, the monitoring device 20 proceeds to step S104.

[0061] In step S104, the partial image extraction unit 201 extracts a partial image of the light-emitting region based on the most recent image captured by the camera 10.

[0062] Once the process in step S104 is complete, the monitoring device 20 proceeds to step S106.

[0063] In step S106, the light emission state determination unit 202 determines the light emission state of a candidate object captured in the image captured by the camera 10 based on historical data of partial images of the light emission region in the image captured by the camera 10 over a predetermined period.

[0064] Once the process in step S106 is complete, the monitoring device 20 proceeds to step S108.

[0065] In step S108, the operation determination unit 203 determines whether the special signal light emitter 150 is operating (emitting light) based on the determination result in step S106. If the special signal light emitter 150 is operating (emitting light), the operation determination unit 203 proceeds to step S110. If the special signal light emitter 150 is not operating (emitting light), the process of this flowchart is terminated.

[0066] In step S110, the notification unit 204 notifies the driver of the railway vehicle 100 that the special signal light 150 is operating by outputting a control signal to the driver's cab of the railway vehicle 100.

[0067] Once the process in step S110 is complete, the monitoring device 20 terminates the process in this flowchart.

[0068] [Second example of a monitoring system] Referring to Figures 6 to 8, a second example of the monitoring system 1 according to this embodiment will be described.

[0069] In the following examples, components identical to or corresponding to those in the first example (Figure 1) described above will be denoted by the same reference numerals. The explanation will focus on the parts that differ from the first example, and explanations of parts that are the same as or corresponding to the first example may be omitted.

[0070] Figure 6 shows the configuration of a second example of the monitoring system 1. Figure 7 shows an example of a region in the image captured by camera 10 that corresponds to a special signal emitter 150 detected by the object detection unit 205. Figure 8 is a diagram illustrating an example of the processing of the background separation unit 206.

[0071] As shown in Figure 6, this example differs from the first example described above in that the monitoring device 20 includes an object detection unit 205 and a background separation unit 206 as functional units.

[0072] Furthermore, the functions of the partial image extraction unit 201, the light emission state determination unit 202, the operation determination unit 203, the notification unit 204, the object detection unit 205, and the background separation unit 206 may be implemented by two or more devices in a distributed manner. For example, the functions of the partial image extraction unit 201, the light emission state determination unit 202, the operation determination unit 203, the notification unit 204, the object detection unit 205, and the background separation unit 206 may be implemented by six different devices. Alternatively, the functions of the partial image extraction unit 201, the light emission state determination unit 202, the operation determination unit 203, the notification unit 204, the object detection unit 205, and the background separation unit 206 may be implemented by two to five devices in a distributed manner, or by seven or more devices.

[0073] The object detection unit 205 detects (recognizes) the special signal emitter 150 based on the output image (captured image) from the camera 10.

[0074] For example, the object detection unit 205 uses a trained model LM4 to detect the special signal emitter 150 based on local features obtained from the image captured by the camera 10 or the image data itself, and estimates the region in the image captured by the camera 10 that corresponds to the special signal emitter 150. As a result, for example, as shown in Figure 7, the object detection unit 205 can extract a partial image P72 of the region corresponding to the special signal emitter 150 from the image captured by the camera 10 P71 in which the special signal emitter 150 is visible.

[0075] The trained model LM4 takes, for example, local features obtained from an image or the image itself as input and outputs data specifying the region in the image corresponding to the special signal emitter 150 (for example, a rectangular region containing the special signal emitter 150). If the region corresponding to the special signal emitter 150 in the image is not output from the trained model LM4, it means that the special signal emitter 150 was not detected in the image. The trained model LM4 is obtained, for example, by machine learning (specifically, supervised learning) of the base training model using a training dataset. The trained model LM4 may be generated by machine learning of the base training model in the monitoring device 20, or by machine learning of the base training model in an information processing device other than the monitoring device 20. For example, each training data included in the training dataset consists of a combination of local features obtained from an image or the image data itself as input data and data specifying the region on the image corresponding to the special signal emitter 150 as output data. As a result, the trained model LM4 can take local features obtained from the image or the image itself as input to detect (recognize) the special signal emitters 150 on the image and output data for the region (e.g., a rectangular region) corresponding to the special signal emitters 150 on the image. For example, the trained model LM4 is mainly composed of a deep neural network (DNN), and the machine learning of the DNN is made more efficient by applying backpropagation based on training data. The trained model LM4 may also include a U-Net that can recognize the special signal emitters 150 from the image when the image is taken as input.

[0076] Furthermore, the object detection unit 205 may apply a rule-based algorithm such as pattern matching based on the local features of the image captured by the camera 10 or the image itself to detect (recognize) the special signal emitters 150 in the image and estimate the region in the image corresponding to the special signal emitters 150.

[0077] For example, if the object detection unit 205 detects a special signal emitter 150 in the image captured by the camera 10, it outputs a partial image of the region in the image captured by the camera 10 that corresponds to the special signal emitter 150.

[0078] The background separation unit 206 performs segmentation on the partial image output from the object detection unit 205, separating it into a region corresponding to the special signal emitter 150 and a region corresponding to the background. As a result, for example, as shown in Figure 8, the background separation unit 206 can separate the partial image P72 corresponding to the special signal emitter 150 into a partial image P81 of the region corresponding to the special signal emitter 150 and a partial image P82 of the background region P72A.

[0079] For example, the background separation unit 206 performs semantic segmentation on each individual pixel of a partial image in the region corresponding to the special signal emitter 150, classifying whether each pixel represents a person or the background. The background separation unit 206 performs semantic segmentation using, for example, a trained model LM5.

[0080] The trained model LM5 is obtained, for example, by performing machine learning (specifically, supervised learning) on ​​the base trained model using a training dataset. The trained model LM5 may be generated by performing machine learning on the base trained model in the monitoring device 20, or by performing machine learning on the base trained model in an information processing device separate from the monitoring device 20. For example, each training data included in the training dataset is a combination of image data showing the special signal light emitter 150 as input, and data (feature map data) that specifies the classification of each pixel in the image data as either a pixel representing the special signal light emitter 150 or a pixel representing the background, as output. As a result, the trained model LM5 can take the image data itself as input and output classification data for each pixel on the image, indicating whether it represents the special signal light emitter 150 or a pixel representing the background. For example, the trained model LM5 is mainly composed of a DNN, and the machine learning of the DNN is made more efficient by applying backpropagation (error backpropagation method) based on the training data. A DNN is, for example, a Fully Convolutional Network (FCN) that takes image data as input and can output a feature map that represents the classification of each pixel of the image data.

[0081] Alternatively, the background separation unit 206 may apply a rule-based algorithm to search for the boundary between the special signal light emitter 150 and the background, and separate a portion of the image of the area corresponding to the special signal light emitter 150 into an area corresponding to the special signal light emitter 150 and an area corresponding to the background.

[0082] The background separation unit 206 outputs a partial image of the region corresponding to the special signal emitter 150, obtained as a result of segmentation of the partial image of the region corresponding to the special signal emitter 150 detected by the object detection unit 205. In addition, the background separation unit 206 may output a partial image of the background region, obtained as a result of segmentation of the partial image of the region corresponding to the special signal emitter 150 detected by the object detection unit 205.

[0083] The background separation section 206 may be omitted.

[0084] The partial image extraction unit 201 extracts a partial image of the light-emitting region based on the partial image of the region corresponding to the special signal light emitter 150 output from the background separation unit 206. This allows the partial image extraction unit 201 to extract a partial image corresponding to the light-emitting section 151 of the special signal light emitter 150 that is emitting light.

[0085] The partial image extraction unit 201 extracts a partial image corresponding to the light-emitting part 151 of the special signal light-emitting unit 150 that is emitting light, using the same method as in the first example described above, based on the partial image of the area corresponding to the special signal light-emitting unit 150 output from the background separation unit 206.

[0086] Furthermore, the partial image extraction unit 201 may detect a circular shape based on the partial image of the region corresponding to the special signal light emitter 150 output from the background separation unit 206, and extract a partial image of the region corresponding to the circular shape as a partial image of the light-emitting region. This is because the circular shape detectable from the partial image of the region corresponding to the special signal light emitter 150 clearly corresponds to the light-emitting unit 151 that is emitting light.

[0087] Furthermore, the partial image extraction unit 201 may set regions corresponding to each of the five light-emitting units 151 based on the partial image of the region corresponding to the special signal light-emitting device 150 output from the background separation unit 206. This is because the shape, position, and size of the light-emitting units 151 within the region corresponding to the special signal light-emitting device 150 are known. The partial image extraction unit 201 may then extract partial images of the regions corresponding to two of the set regions corresponding to the five light-emitting units 151 whose brightness is above or greater than a threshold, as partial images of the light-emitting regions.

[0088] If the background separation unit 206 is omitted, the partial image extraction unit 201 extracts a partial image of the light-emitting region based on the partial image of the region corresponding to the special signal emitter 150 output from the object detection unit 205.

[0089] Thus, in this example, the monitoring device 20 can detect the special signal emitter 150 from the image captured by the camera 10, and determine whether or not the special signal emitter 150 is operating (emitting light) based on the portion of the image in which the detected special signal emitter 150 is visible. Therefore, the monitoring device 20 can suppress the influence of noise such as disturbances in unnecessary areas of the image captured by the camera 10 other than the portion in which the special signal emitter 150 is visible, and more reliably determine whether or not the special signal emitter 150 is operating.

[0090] [Second example of processing related to monitoring the operation status of special signal lights] Referring to Figure 9, a second example of the monitoring process for the operation (illumination) of the special signal emitter 150 by the monitoring device 20 will be described.

[0091] Figure 9 is a flowchart illustrating a second example of the process for monitoring whether the special signal emitter 150 is operating using the monitoring device 20.

[0092] As shown in Figure 9, step S202 is the same as the process in step S102 in Figure 5 described above, so its explanation is omitted.

[0093] Once the process in step S202 is complete, the monitoring device 20 proceeds to step S204.

[0094] In step S204, the object detection unit 205 performs a process to detect the special signal emitter 150 based on the latest image captured by the camera 10.

[0095] Once the processing in step S204 is complete, the monitoring device 20 proceeds to step S206.

[0096] In step S206, the monitoring device 20 determines whether the special signal emitter 150 was detected in the process of step S204. If the special signal emitter 150 is detected, the monitoring device 20 proceeds to step S208. If the special signal emitter 150 is not detected, the process of this flowchart is terminated.

[0097] In step S208, the background separation unit 206 separates the area corresponding to the background from the partial image of the area corresponding to the special signal emitter 150 output from the object detection unit 205, and outputs a partial image of the area corresponding to the special signal emitter 150.

[0098] Once the process in step S208 is complete, the monitoring device 20 proceeds to step S210.

[0099] In step S210, the partial image extraction unit 201 extracts a partial image of the light-emitting region based on the partial image of the region corresponding to the special signal light emitter 150 output from the background separation unit 206.

[0100] Once the process in step S210 is complete, the monitoring device 20 proceeds to step S212.

[0101] Steps S212, S214, and S216 are the same as steps S106, S108, and S110 in the flowchart of Figure 5 above, so their explanation is omitted.

[0102] Once the process in step S216 is complete, the monitoring device 20 terminates the process in this flowchart.

[0103] [Third example of a monitoring system] A third example of the monitoring system 1 according to this embodiment will be described with reference to Figures 10 and 11.

[0104] In the following, the same reference numerals are used for components that are the same as or correspond to the first example (Figure 1) and the second example (Figure 6) described above. The explanation will focus on the parts that differ from the first and second examples, and may omit explanations of parts that are the same as or correspond to the first and second examples.

[0105] Figure 10 shows the configuration of a third example of the monitoring system 1. Figure 11 is a diagram illustrating an example of the processing of the light emission state determination unit 202.

[0106] As shown in Figure 10, in this example, the monitoring device 20 differs from the first and second examples described above in that the partial image extraction unit 201 includes an object detection unit 201A and a background separation unit 201B.

[0107] Furthermore, the functions of the object detection unit 201A, background separation unit 201B, light emission state determination unit 202, operation determination unit 203, and notification unit 204 may be implemented by two or more devices in a distributed manner. For example, the functions of the object detection unit 201A, background separation unit 201B, light emission state determination unit 202, operation determination unit 203, and notification unit 204 may be implemented by five different devices. Alternatively, the functions of the object detection unit 201A, background separation unit 201B, light emission state determination unit 202, operation determination unit 203, and notification unit 204 may be implemented by two to four devices in a distributed manner, or by six or more devices.

[0108] The object detection unit 201A and the background separation unit 201B have the same functions as the object detection unit 205 and the background separation unit 206 in the second example described above.

[0109] Similar to the object detection unit 205, the object detection unit 201A detects (recognizes) the special signal emitter 150 based on the output image (captured image) of the camera 10, and outputs a partial image of the region in the captured image of the camera 10 that corresponds to the special signal emitter 150.

[0110] Similar to the background separation unit 206, the background separation unit 201B separates the area corresponding to the background from the partial image output from the object detection unit 201A and outputs a partial image of the area corresponding to the special signal emitter 150.

[0111] Thus, in this example, the partial image extraction unit 201 extracts and outputs a partial image of the region corresponding to the detected special signal light emitter 150 as a partial image of the light-emitting region in the image captured by the camera 10.

[0112] The background separation unit 201B may be omitted. In this case, the partial image extraction unit 201 extracts and outputs a partial image corresponding to the detected special signal emitter 150 as a partial image of the light-emitting region in the image captured by the camera 10.

[0113] The light emission state determination unit 202 determines the light emission state of the special signal emitter 150 based on the partial image of the region corresponding to the special signal emitter 150, which is output from the partial image extraction unit 201 (specifically, the background separation unit 201B).

[0114] For example, the light emission state determination unit 202 obtains the brightness value for each pixel of the target partial image for each group of partial images (multiple partial images) of the light emission region in a time series over a predetermined period, which have been extracted by the partial image extraction unit 201. Then, based on the change in brightness values ​​(or their distribution) on each of the multiple partial images in the time series, the light emission state determination unit 202 determines whether the two light-emitting units 151 are transitioning to rotate counterclockwise at a unique frequency or period. This allows the operation determination unit 203 to determine whether the candidate object is a light-emitting unit 151 of a rotating special signal light emitter 150.

[0115] For example, as shown in Figure 11, the time-series partial images P111, P112, and P113 extracted by the partial image extraction unit 201 show the rotating special signal light emitter 150 emitting light. In this case, the light emission state determination unit 202 can determine that the two light-emitting units 151 are rotating counterclockwise at a unique frequency or period by acquiring the distribution of brightness values ​​P111A, P112A, and P113A for each pixel of the partial images P111, P112, and P113.

[0116] Furthermore, the light emission state determination unit 202 may determine whether or not a vertically elongated candidate object is flashing at a specific frequency based on the change in the luminance value (distribution) on each of the multiple partial images in a time series. This allows the operation determination unit 203 to determine whether or not the candidate object is a flashing type special signal light emitter 150.

[0117] Thus, in this example, the monitoring device 20 can detect the special signal emitter 150 based on the image captured by the camera 10, and determine whether or not the special signal emitter 150 is operating based on the time-series change in the brightness value on the partial image corresponding to the detected special signal emitter 150. Therefore, the monitoring device 20 can suppress the influence of noise such as disturbances in unnecessary areas of the image captured by the camera 10 other than the part in which the special signal emitter 150 is visible, and more reliably determine whether or not the special signal emitter 150 is operating.

[0118] Furthermore, in this example, the monitoring device 20 can determine whether the special signal emitter 150 is operating or not by analyzing the time-series changes in brightness values ​​on the partial image corresponding to the special signal emitter 150, without having to identify the area corresponding to the light-emitting unit 151. Therefore, it is possible to reduce the effort required for parameter adjustments, such as setting thresholds that match the light emission color of the light-emitting unit 151.

[0119] [Third example of a process for monitoring the operation status of a special signal light] Referring to Figure 12, a third example of the monitoring process for the operation (illumination) of the special signal emitter 150 by the monitoring device 20 will be described.

[0120] Figure 12 is a flowchart illustrating a third example of the process for monitoring whether the special signal emitter 150 is operating using the monitoring device 20.

[0121] As shown in Figure 12, step S302 is the same as the process in step S102 in Figure 5 described above, so its explanation is omitted.

[0122] Once the process in step S302 is complete, the monitoring device 20 proceeds to step S304.

[0123] In step S304, the object detection unit 201A performs a process to detect the special signal emitter 150 based on the latest image captured by the camera 10.

[0124] Once the process in step S304 is complete, the monitoring device 20 proceeds to step S306.

[0125] In step S306, the monitoring device 20 determines whether the special signal emitter 150 was detected in step S304. If the special signal emitter 150 is detected, the monitoring device 20 proceeds to step S308. If the special signal emitter 150 is not detected, the process in this flowchart is terminated.

[0126] In step S308, the background separation unit 201B separates the area corresponding to the background from the partial image of the area corresponding to the special signal emitter 150 output from the object detection unit 201A, and outputs a partial image of the area corresponding to the special signal emitter 150.

[0127] Once the process in step S308 is complete, the monitoring device 20 proceeds to step S310.

[0128] In step S310, the light emission state determination unit 202 determines the light emission state of the special signal emitter 150 based on the partial image of the area corresponding to the special signal emitter 150 output from the background separation unit 201B.

[0129] Once the processing in step S310 is complete, the monitoring device 20 proceeds to step S312.

[0130] Steps S312 and S314 are the same as steps S108 and S110 in Figure 5 above, so their explanation is omitted.

[0131] Once step S314 is completed, the monitoring device 20 terminates the process in this flowchart.

[0132] [Effect] Next, the operation of the monitoring system, monitoring device, monitoring method, and program according to this embodiment will be described.

[0133] In the first aspect of this embodiment, the monitoring system comprises an imaging unit, an extraction unit, a light emission state determination unit, and an operation determination unit. The monitoring system is, for example, the monitoring system 1 described above. The imaging unit is, for example, the camera 10 described above. The extraction unit is, for example, the partial image extraction unit 201 described above. The light emission state determination unit is, for example, the light emission state determination unit 202 described above. The operation determination unit is, for example, the operation determination unit 203 described above. Specifically, the imaging unit images the area in front of the railway vehicle. The railway vehicle is, for example, the railway vehicle 100 described above. The extraction unit extracts a partial image of a first region containing a light-emitting object based on the image captured by the imaging unit. The first region is, for example, the light-emitting region described above. The light emission state determination unit determines the light emission state of the object based on a plurality of time-series partial images of the first region extracted by the extraction unit. The operation determination unit then determines whether or not the special signal light emitter is operating based on the light emission state determined by the light emission state determination unit. The special signal light emitter is, for example, the special signal light emitter 150 described above.

[0134] Furthermore, in the first aspect of this embodiment, the monitoring device may include the extraction unit, the light emission state determination unit, and the operation determination unit. The monitoring device is, for example, the monitoring device 20 described above.

[0135] Furthermore, in the first aspect of this embodiment, a monitoring method to be performed by the monitoring device or the monitoring system may be provided. Specifically, the monitoring method includes an extraction step, a light emission state determination step, and an operation determination step. The extraction step is, for example, step S104 in Figure 5, step S210 in Figure 9, or steps S304 and S308 in Figure 12. The light emission state determination step is, for example, step S106 in Figure 5, step S212 in Figure 9, or step S310 in Figure 12. The operation determination step is, for example, step S108 in Figure 5, step S214 in Figure 9, or step S312 in Figure 12. More specifically, in the extraction step, a partial image of a first region including a light-emitting object is extracted based on the image captured by the imaging unit that images the front of the railway vehicle. In the light emission state determination step, the light emission state of the object is determined based on a plurality of time-series partial images of the first region extracted in the extraction step. Then, in the operation determination step, it is determined whether or not the special signal light emitter is operating based on the light emission state determined in the light emission state determination step.

[0136] Furthermore, in the first aspect of this embodiment, a program may be provided that causes the information processing device to perform the extraction step, the light emission state determination step, and the operation determination step. The information processing device is, for example, the monitoring device 20 described above.

[0137] As a result, monitoring systems, monitoring devices, and information processing devices (hereinafter referred to as "monitoring systems, etc.") can determine whether or not a special signal light is operating by extracting a first region containing a light-emitting object from the image captured by the imaging unit and analyzing the extracted time-series partial image group. Furthermore, monitoring systems, etc. can determine whether or not a special signal light is operating by analyzing the extracted partial image group of the first region, regardless of the type of special signal light (for example, whether it is a rotating type or a flashing type). Therefore, monitoring systems, etc. can appropriately detect the operation of a special signal light.

[0138] Furthermore, in a second aspect of this embodiment, based on the first aspect described above, the extraction unit may extract a partial image of the first region corresponding to the emitting object based on the image captured by the imaging unit.

[0139] Furthermore, in a second aspect of this embodiment, based on the first aspect described above, the extraction step may involve extracting a partial image of the first region corresponding to the emitting object based on the image captured by the imaging unit.

[0140] This allows the monitoring system to determine whether or not the special signal emitter is operating based on the time series of partial images of the first region corresponding to the emitting object.

[0141] Furthermore, in a third aspect of this embodiment, based on the second aspect described above, the extraction unit may extract a partial image of the first region corresponding to the object of the same color as the light emitted by the special signal light emitter, based on the image captured by the imaging unit.

[0142] Furthermore, in a third aspect of this embodiment, based on the second aspect described above, the extraction step may involve extracting a partial image of the first region corresponding to the object of the same color as the light emitted by the special signal light emitter, based on the image captured by the imaging unit.

[0143] This allows monitoring systems to determine whether or not a special signal emitter is operating based on a time series of partial images of areas corresponding to objects of the same color as the light emitted by the special signal emitter.

[0144] Furthermore, in a fourth aspect of this embodiment, based on the second aspect described above, the monitoring system and the monitoring device may include a detection unit that detects a special signal light emitter from the image captured by the imaging unit. The detection unit is, for example, the object detection unit 205 described above. The extraction unit may then extract a partial image of the first region corresponding to the light-emitting part of the special signal light emitter, based on a partial image of the second region in the image captured by the imaging unit that includes the special signal light emitter detected by the detection unit.

[0145] Furthermore, in a fourth aspect of this embodiment, based on the second aspect described above, the monitoring method may include a detection step, and the program may cause the information processing device to execute the detection step. The detection step is, for example, step S204 in Figure 9 described above. Specifically, in the detection step, a special signal light emitter may be detected from the image captured by the imaging unit. Then, in the extraction step, a partial image of the first region corresponding to the light-emitting part of the special signal light emitter may be extracted based on a partial image of the second region in the image captured by the imaging unit that includes the special signal light emitter detected in the detection step. The light-emitting part is, for example, the light-emitting part 151 described above.

[0146] As a result, the monitoring system can extract a partial image of the first region corresponding to the light-emitting part of the special signal emitter based on a partial image of the second region corresponding to the special signal emitter. Therefore, the monitoring system can suppress the effects of disturbances and noise compared to, for example, extracting a partial image of the first region from the entire image captured by the imaging unit. Consequently, the monitoring system can more appropriately detect the operation of the special signal emitter.

[0147] Furthermore, in a fifth aspect of this embodiment, based on the fourth aspect described above, the monitoring system and the monitoring device may include a background separation unit. The background separation unit is, for example, the background separation unit 206 described above. Specifically, the background separation unit may separate the background of the special signal light emitter from the partial image of the second region and extract a partial image of the third region corresponding to the special signal light emitter. The extraction unit may then extract the first region corresponding to the light-emitting part of the special signal light emitter based on the partial image of the third region.

[0148] Furthermore, in a fifth aspect of this embodiment, based on the fourth aspect described above, the monitoring method may include a background separation step, and the program may cause the information processing device to execute the background separation step. The background separation step is, for example, step S208 in Figure 5 described above. Specifically, in the background separation step, the background of the special signal light emitter may be separated from the partial image of the second region, and a partial image of the third region corresponding to the special signal light emitter may be extracted. Then, in the extraction step, the first region corresponding to the light-emitting part of the special signal light emitter may be extracted based on the partial image of the third region.

[0149] As a result, the monitoring system can suppress the effects of disturbances and noise compared to when it extracts a partial image of the first region, including the background of the special signal emitter. Therefore, the monitoring system can more accurately detect the operation of the special signal emitter.

[0150] Furthermore, in a sixth aspect of this embodiment, based on the first aspect described above, the extraction unit may extract the first region including the special signal light emitter as the object from the image captured by the imaging unit.

[0151] Furthermore, in a sixth aspect of this embodiment, based on the first aspect described above, the extraction step may involve extracting the first region including the special signal light emitter as the object from the image captured by the imaging unit.

[0152] As a result, the monitoring system can determine whether or not the special signal emitter is operating by analyzing a time-series partial image of the first region including the special signal emitter. Therefore, the monitoring system can suppress situations such as misinterpreting the emission of light from an object other than the special signal emitter as the operation of the special signal emitter, and can more appropriately detect the operation of the special signal emitter.

[0153] Furthermore, in a seventh aspect of this embodiment, based on the sixth aspect described above, the extraction unit may include a detection unit and a background separation unit. The detection unit is, for example, the object detection unit 201A described above. The background separation unit is, for example, the background separation unit 201B described above. Specifically, the detection unit may detect a special signal emitter from the image captured by the imaging unit. The background separation unit may separate the background portion of the special signal emitter from a partial image of a second region in the image captured by the imaging unit that includes the special signal emitter detected by the detection unit, and extract a partial image of the first region corresponding to the special signal emitter.

[0154] Furthermore, in a seventh aspect of this embodiment, based on the sixth aspect described above, the extraction step may include a detection step and a background separation step. The detection step is, for example, step S304 in Figure 12 described above. The background separation unit is, for example, step S308 in Figure 12 described above. Specifically, in the detection step, a special signal light emitter may be detected from the image captured by the imaging unit. Then, in the background separation step, the background portion of the special signal light emitter may be separated from the partial image of the second region containing the special signal light emitter detected in the detection step in the image captured by the imaging unit, and the partial image of the first region corresponding to the special signal light emitter may be extracted.

[0155] As a result, monitoring systems, for example, can suppress the occurrence of situations where the emission of light from an object other than the special signal emitter is mistakenly identified as the operation of the special signal emitter, compared to when partial images including the background are used, and can more accurately detect the operation of the special signal emitter.

[0156] Furthermore, in the eighth aspect of this embodiment, based on any one of the first to seventh aspects described above, the light emission state determination unit may determine whether or not the light-emitting portion of the object is moving in a rotating manner based on a plurality of time-series partial images of the first region.

[0157] Furthermore, in the eighth aspect of this embodiment, based on any one of the first to seventh aspects described above, the light emission state determination step may determine whether or not the light-emitting portion of the object is moving in a rotating manner based on a plurality of time-series partial images of the first region.

[0158] This allows monitoring systems and other equipment to detect the operation of the rotating special signal emitter.

[0159] Furthermore, in the ninth aspect of this embodiment, based on any one of the first to eighth aspects described above, the light emission state determination unit may determine whether or not the object is flashing based on a plurality of time-series partial images of the first region.

[0160] Furthermore, in the ninth aspect of this embodiment, based on any one of the first to eighth aspects described above, the light emission state determination step may determine whether or not the object is flashing based on a plurality of time-series partial images of the first region.

[0161] This allows monitoring systems and the like to detect the operation of a special flashing signal light.

[0162] Furthermore, in the tenth aspect of this embodiment, based on any one of the first to ninth aspects described above, the light emission state determination unit may determine the light emission state of the object based on the time-series change in brightness in a plurality of time-series partial images of the first region.

[0163] Furthermore, in the tenth aspect of this embodiment, based on any one of the first to ninth aspects described above, the light emission state determination step may determine the light emission state of the object based on the time-series change in brightness in a plurality of time-series partial images of the first region.

[0164] As a result, the monitoring system can determine the emission state of light-emitting objects included in the partial images of the first region by analyzing the partial images of the first region in a time series.

[0165] Furthermore, in the eleventh aspect of this embodiment, the monitoring system and the monitoring device may be equipped with a notification unit, based on any one of the first to tenth aspects described above. The notification unit is, for example, the notification unit 204 described above. Specifically, the notification unit may notify the driver of the railway vehicle when the operation determination unit determines that the special signal light emitter is operational.

[0166] Furthermore, in the eleventh aspect of this embodiment, based on any one of the first to tenth aspects described above, the monitoring method may include a notification step, and the program may cause the information processing device to execute the notification step. The notification step is, for example, step S110 in Figure 5, step S216 in Figure 9, and step S314 in Figure 12. Specifically, in the notification step, if the operation determination step determines that the special signal light emitter is operating, the driver of the railway vehicle may be notified.

[0167] This allows monitoring systems to notify train drivers when they detect the activation of special signal lights, thereby supporting the assurance of train safety.

[0168] Although embodiments have been described in detail above, this disclosure is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims. [Explanation of Symbols]

[0169] 1. Monitoring System 10 Cameras 20 Monitoring equipment 100 Railway Vehicles 150 Special signal flashing devices 151 Light-emitting part 201 Partial image extraction section 201A Object Detection Unit 201B Background separation part 202 Light emission state determination unit 203 Operation determination unit 204 Notification Department 205 Object detection unit 206 Background separation part P21, P22 acquired images P31 Image P32 Partial image P41,P42,P43 partial images P71 captured image P72 Partial Image P72A area Partial images on pages 81 and 82. P111,P112,P113 partial images

Claims

1. An imaging unit that images the front of a railway vehicle, An extraction unit extracts a partial image of a first region containing a light-emitting object based on the image captured by the imaging unit, A light emission state determination unit determines the light emission state of the object based on a plurality of time-series partial images of the first region extracted by the extraction unit, The system includes an operation determination unit that determines whether or not the special signal light emitter is operating based on the light emission state determined by the light emission state determination unit. Monitoring system.

2. The extraction unit extracts a partial image of the first region corresponding to the emitting object based on the image captured by the imaging unit. The monitoring system according to claim 1.

3. The extraction unit extracts a partial image of the first region corresponding to the object, which has the same color as the light emitted by the special signal emitter, based on the image captured by the imaging unit. The monitoring system according to claim 2.

4. The system includes a detection unit that detects a special signal emitter from the image captured by the imaging unit, The extraction unit extracts a partial image of the first region corresponding to the light-emitting part of the special signal emitter, based on a partial image of the second region containing the special signal emitter detected by the detection unit within the image captured by the imaging unit. The monitoring system according to claim 2.

5. The system includes a background separation unit that separates the background of the special signal emitter from the partial image of the second region and extracts a partial image of the third region corresponding to the special signal emitter. The extraction unit extracts the first region corresponding to the light-emitting part of the special signal light emitter based on the partial image of the third region. The monitoring system according to claim 4.

6. The extraction unit extracts the first region, which includes the special signal light emitter as the object, from the image captured by the imaging unit. The monitoring system according to claim 1.

7. The extraction unit is A detection unit for detecting a special signal emitter from the image captured by the imaging unit, The system includes a background separation unit that separates the background portion of a special signal emitter from a partial image of a second region containing the special signal emitter detected by the detection unit within the image captured by the imaging unit, and extracts a partial image of the first region corresponding to the special signal emitter. The monitoring system according to claim 6.

8. The light emission state determination unit determines, based on a plurality of time-series partial images of the first region, whether or not the light-emitting portion of the object is moving in a rotating manner. The monitoring system according to any one of claims 1 to 7.

9. The light emission state determination unit determines whether or not the object is flashing based on a plurality of time-series partial images of the first region. The monitoring system according to any one of claims 1 to 7.

10. The light emission state determination unit determines the light emission state of the object based on the time-series change in brightness in a plurality of partial images of the first region in a time series. The monitoring system according to any one of claims 1 to 7.

11. The system includes a notification unit that notifies the driver of the railway vehicle when the operation determination unit determines that the special signal light emitter is activated. The monitoring system according to any one of claims 1 to 7.

12. An extraction unit extracts a partial image of a first region containing a luminescent object based on the image captured by an imaging unit that images the front of a railway vehicle, A light emission state determination unit determines the light emission state of the object based on a plurality of time-series partial images of the first region extracted by the extraction unit, The system includes an operation determination unit that determines whether or not the special signal light emitter is operating based on the light emission state determined by the light emission state determination unit. monitoring equipment.

13. A monitoring method performed by a monitoring device or monitoring system, Based on the image captured by the imaging unit that images the front of the railway vehicle, an extraction step is performed to extract a partial image of a first region containing a luminescent object, A light emission state determination step, which determines the light emission state of the object based on a plurality of time-series partial images of the first region extracted in the extraction step, The process includes an operation determination step that determines whether or not the special signal light emitter is operating based on the light emission state determined in the light emission state determination step, Monitoring method.

14. In an information processing device, Based on the image captured by the imaging unit that images the front of the railway vehicle, an extraction step is performed to extract a partial image of a first region containing a luminescent object, A light emission state determination step, which determines the light emission state of the object based on a plurality of time-series partial images of the first region extracted in the extraction step, Based on the light emission state determined in the light emission state determination step, an operation determination step is performed to determine whether or not the special signal light emitter is operating. program.

Images