Monitoring systems, monitoring devices, monitoring methods, programs

Abstract

This technology provides the ability to properly monitor the open / closed state of a door. [Solution] A monitoring system 1 according to one embodiment includes: a camera 10 that captures an image of a predetermined range including the movable part (door panel 121) of a door 120 to be monitored; a storage unit 202 that stores a trained model LM that takes an image as input and outputs an index value (depth estimate DP) that represents the distance of an object in the image from the viewpoint; a depth estimation unit 201 that uses the trained model LM to estimate an index value (depth estimate DP) that represents the distance of an object in a target pixel from the camera 10 for each of a plurality of pixels spanning a predetermined range in the image captured by the camera 10; and an open / closed state monitoring unit 203 that monitors the open / closed state of the door 120 to be monitored based on the index value (depth estimate DP) estimated by the depth estimation unit 201.

Description

Technical Field

[0001] The present disclosure relates to a monitoring system and the like.

Background Art

[0002] For example, technologies for monitoring the opening and closing state of doors of railway vehicles and the like are known (see, for example, Patent Documents 1 to 3).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is room for improvement in the technology for monitoring the opening and closing state of doors.

[0005] Therefore, in view of the above problems, an object is to provide a technology capable of appropriately monitoring the opening and closing state of a door.

Means for Solving the Problems

[0006] To achieve the above object, in one embodiment of the present disclosure, an imaging unit that images a predetermined range including a movable part of a door to be monitored; a storage unit that stores a learned model that takes an image as an input and outputs an index value representing the distance from the viewpoint of an object shown in the image; an estimation unit that uses the learned model to estimate the index value representing the distance from the imaging unit of an object shown in a target section for each of a plurality of sections across the predetermined range in the captured image of the imaging unit; The system includes a monitoring unit that monitors the open / closed state of the door being monitored based on the index value estimated by the estimation unit, A monitoring system will be provided.

[0007] In other embodiments of this disclosure, A storage unit that stores a trained model that takes an image as input and outputs an index value representing the distance of an object in the image from the viewpoint, An estimation unit that uses the trained model to estimate an index value representing the distance of an object from the imaging unit to a target area for each of several sections spanning the predetermined range in the image captured by the imaging unit, which captures a predetermined range including the movable part of the door to be monitored. The system includes a monitoring unit that monitors the open / closed state of the door being monitored based on the index value estimated by the estimation unit, A monitoring device will be provided.

[0008] Furthermore, in yet another embodiment of this disclosure, The monitoring device takes an image as input and uses a trained model that outputs an index value representing the distance of an object in the image from the viewpoint to the object in the image to estimate the index value representing the distance of an object in a target area to the imaging unit for each of several sections spanning the predetermined range in the image captured by the imaging unit that captures the predetermined range including the movable part of the door to be monitored. The monitoring device includes a monitoring step of monitoring the open / closed state of the door being monitored based on the index value estimated in the estimation step, A monitoring method will be provided.

[0009] Furthermore, in yet another embodiment of this disclosure, In an information processing device, An estimation step in which, using a trained model that takes an image as input and outputs an index value representing the distance of an object in the image from the viewpoint, estimates the index value representing the distance of an object in a target section from the imaging unit for each of several sections spanning the predetermined range in the image captured by the imaging unit that captures the predetermined range including the movable part of the door to be monitored. Based on the index value estimated in the estimation step, a monitoring step of monitoring the open / closed state of the door to be monitored is executed. A program is provided.

Advantages of the Invention

[0010] According to the above embodiment, the open / closed state of the door can be appropriately monitored.

Brief Description of the Drawings

[0011] [Figure 1] It is a diagram showing the configuration of the first example of the monitoring system. [Figure 2] It is a diagram showing the configuration of an example of the monitoring device. [Figure 3] It is a diagram showing an example of the depth estimation value output from the depth estimation unit. [Figure 4] It is a diagram showing an example of the depth estimation value for each pixel coordinate on the depth monitoring line. [Figure 5] It is a diagram for explaining the first example of the method for monitoring the open / closed state of the door. [Figure 6] It is a diagram for explaining the first example of the method for monitoring the open / closed state of the door. [Figure 7] It is a flowchart schematically showing the first example of the processing of the monitoring device during the opening operation of the door. [Figure 8] It is a flowchart schematically showing the first example of the processing of the monitoring device during the closing operation of the door. [Figure 9] It is a flowchart schematically showing the first example of the processing of the monitoring device during the closing operation of the door. [Figure 10] It is a diagram for explaining the second example of the method for monitoring the open / closed state of the door. [Figure 11] It is a flowchart schematically showing the second example of the processing of the monitoring device during the opening operation of the door. [Figure 12] It is a flowchart schematically showing the second example of the processing of the monitoring device during the closing operation of the door. [Figure 13]This figure shows another example of depth estimates output from the depth estimation unit. [Figure 14] This figure shows another example of depth estimates for each pixel coordinate on the depth monitoring line. [Figure 15] This diagram illustrates a third example of a method for monitoring the open / closed state of a door. [Figure 16] This flowchart schematically illustrates a third example of how a monitoring device processes information when a door is opened. [Figure 17] This flowchart schematically illustrates a third example of how a monitoring device processes information during door closing. [Figure 18] This flowchart schematically illustrates a third example of how a monitoring device processes information during door closing. [Figure 19] This diagram shows the configuration of a second example of a monitoring system. [Figure 20] This figure shows an example of an image captured by a camera. [Figure 21] This figure illustrates the first example of a method for correcting depth estimates. [Figure 22] This figure illustrates a second example of a method for correcting depth estimates. [Figure 23] This diagram shows the configuration of the third example of a monitoring system. [Figure 24] This diagram illustrates an example of how to set monitoring criteria. [Modes for carrying out the invention]

[0012] The embodiments will be described below with reference to the drawings.

[0013] [First example of a monitoring system] A first example of the monitoring system 1 according to this embodiment will be described with reference to Figures 1 and 2.

[0014] Figure 1 shows the configuration of a first example of monitoring system 1. Figure 2 shows the configuration of an example of monitoring device 20.

[0015] As shown in Figure 1, the monitoring system 1 includes a camera 10 and a monitoring device 20. Based on the image captured by the camera 10, the monitoring system 1 monitors the open / closed state of the door 120 that is the object to be monitored and is visible in the captured image.

[0016] In this example, the door 120 is provided in an opening 111 formed on the side of the body 110 of the railway vehicle 100 for passengers to board and alight. Hereafter, the position and orientation of the door 120 may be described using the up, down, left, and right directions shown in Figure 1.

[0017] Furthermore, as shown in Figure 1, a machine learning system 2 is a component associated with the monitoring system 1. The machine learning system 2 includes a camera 30, a distance sensor 40, and a machine learning device 50.

[0018] Camera 10 captures a predetermined imaging range, including the portion of the vehicle body 110 around the door panel 121 and opening 111 of the door 120 of a railway vehicle 100 entering the station platform. Camera 10 is a monocular camera, for example, fixed so as to be suspended from the ceiling of the station platform. Camera 10 acquires and outputs captured images at intervals of 1 / 30th of a second, for example, during the station's operating hours.

[0019] The door 120 whose open / closed state is monitored may be one or multiple doors.

[0020] If there are multiple doors 120 whose open / closed state is to be monitored, the imaging range of the camera 10 may include one or more doors 120. Also, if there are multiple doors 120 whose open / closed state is to be monitored, the monitoring system 1 may include multiple cameras 10.

[0021] Furthermore, the door 120 whose open / closed state is monitored may be a double-leaf type, for example, as shown in Figure 1, where the opening and closing operation is performed by two door panels 121 moving in opposite directions, or a single-leaf type, where the opening and closing operation is performed by the movement of one door panel 121. The following explanation will focus mainly on the case where the door 120 is a double-leaf type.

[0022] Camera 10 is connected to the monitoring device 20 via a predetermined communication line, and the output of camera 10, i.e., the captured image data, is received by the monitoring device 20.

[0023] The monitoring device 20 monitors the open / closed state of the door 120 based on the images captured sequentially from the camera 10.

[0024] The functions of the monitoring device 20 may be realized by any hardware, or any combination of hardware and software. For example, as shown in Figure 2, the monitoring device 20 includes an external interface 21, an auxiliary storage device 22, a memory device 23, a CPU 24, a high-speed processing unit 25, a communication interface 26, an input device 27, a display device 28, and a sound output device 29. The components of the monitoring device 20 are connected communicably by bus BS2.

[0025] The external interface 21 functions as an interface for reading data from and writing data to the recording medium 21A. The recording medium 21A includes, for example, flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), BDs (Blu-ray® Discs), SD memory cards, USB memory, etc. This allows the monitoring device 20 to read various data used in processing through the recording medium 21A, store it in the auxiliary storage device 22, and install programs that realize various functions.

[0026] Furthermore, the monitoring device 20 may acquire various data and programs for processing from external devices (for example, the machine learning device 50 described later) via the communication interface 26.

[0027] The auxiliary storage device 22 stores various installed programs, as well as files and data necessary for various processes. The auxiliary storage device 22 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Disk), or flash memory.

[0028] When a program startup command is received, the memory device 23 reads the program from the auxiliary storage device 22 and stores it. The memory device 23 includes, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory).

[0029] The CPU 24 executes various programs loaded from the auxiliary storage device 22 into the memory device 23, and implements various functions related to the monitoring device 20 according to the programs.

[0030] The high-speed computing unit 25 works in conjunction with the CPU 24 to perform calculations at a relatively high speed. The high-speed computing unit 25 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).

[0031] Furthermore, the high-speed computing unit 25 may be omitted depending on the required processing speed.

[0032] The communication interface 26 is used as an interface for connecting to external devices in a communicative manner. This allows the monitoring device 20 to communicate with external devices such as a camera 10 or a machine learning device 50 through the communication interface 26. Furthermore, the communication interface 26 may have multiple types of communication interfaces depending on the communication method between the connected devices.

[0033] The input device 27 receives various inputs from the user.

[0034] The input device 27 includes, for example, an input device that accepts mechanical operation input from a user (hereinafter referred to as "mechanical input device"). The mechanical input device includes, for example, buttons, toggles, levers, keyboards, mice, touch panels implemented on the display device 28, touch pads provided separately from the display device 28, and the like.

[0035] Furthermore, the input device 27 may include a voice input device capable of receiving voice input from the user. The voice input device may include, for example, a microphone capable of collecting the user's voice.

[0036] Furthermore, the input device 27 may include a gesture input device capable of receiving gesture input from the user. The gesture input device may include, for example, a camera capable of capturing images of the user's gestures.

[0037] Furthermore, the input device 27 may include a biometric input device capable of receiving biometric input from the user. The biometric input device may include, for example, a camera capable of acquiring image data containing information about the user's fingerprints or iris.

[0038] The display device 28 displays information screens and operation screens to the user of the monitoring device 20. The display device 28 is, for example, a liquid crystal display or an organic EL (electroluminescence) display.

[0039] The sound output device 29 transmits various information to the user of the monitoring device 20 by sound. The sound output device 29 may be, for example, a buzzer, alarm, or speaker.

[0040] As shown in Figure 1, the monitoring device 20 includes, as functional units, a depth estimation unit 201, a storage unit 202, an open / closed state monitoring unit 203, and a monitoring result output unit 204. These functional units are realized, for example, by loading a program installed in the auxiliary storage device 22 into the memory device 23 and executing it on the CPU 24. Furthermore, the function of the storage unit 202 is realized by a predetermined storage area in the auxiliary storage device 22, etc.

[0041] The depth estimation unit 201 estimates the depth of a portion or the entirety of the captured image input from the camera 10, for each of several sections that span the entire image of the target object, representing the distance from the viewpoint, i.e., the camera 10, to the object in the section of the target object. The depth estimation unit 201 then outputs an estimated depth value (depth estimate DP) for each of the multiple sections. For example, if a section is a pixel, the depth estimation unit 201 estimates the depth of an object for each of several pixels that span the entire image of the target object, for a portion or the entirety of the image captured by the camera 10, and outputs the depth estimate DP.

[0042] The depth estimate DP may be a value that increases as the distance from the viewpoint to the object increases, or it may be a value that decreases as the distance from the viewpoint to the object increases. The following explanation will proceed under the assumption that the depth estimate DP increases as the distance from the viewpoint to the object increases.

[0043] Specifically, the depth estimation unit 201 uses a trained model LM to estimate the depth of objects in multiple sections that span the entire image, based on the image captured by the camera 10, for part or all of the image captured by the camera 10.

[0044] The trained model LM can take an image as input and output depth estimates DP for multiple sections across the entire image. The trained model LM is delivered from the machine learning device 50 and pre-stored in the memory unit 202.

[0045] The open / closed state monitoring unit 203 monitors the open / closed state of the door 120 based on the depth estimate value DP output from the depth estimation unit 201.

[0046] For example, the open / closed state monitoring unit 203 detects the start and completion of the door 120's opening operation when the door 120 is opened.

[0047] The monitoring result output unit 204 outputs the monitoring results regarding the open / closed state of the door 120, which are output from the open / closed state monitoring unit 203, to the outside.

[0048] For example, the monitoring result output unit 204 outputs the monitoring results regarding the open / closed state of the door 120 to the control device for the platform doors installed on the station platform. This allows the control device to control the platform doors in accordance with the open / closed state of the door 120.

[0049] The machine learning system 2 includes a camera 30, a distance sensor 40, and a machine learning device 50.

[0050] The machine learning system 2 performs machine learning on the base learning model M based on the image captured by the camera 30 and the output of the distance sensor 40, and generates a trained model LM.

[0051] Camera 30 captures a predetermined imaging range, including the door panel 221 of a door 220 provided in an opening 211 formed on the side of the body 210 of the railway vehicle 200, and the portion of the body 210 surrounding the opening 211. The output of camera 30, i.e., the image captured by camera 30, is taken into the machine learning device 50 via a predetermined communication line.

[0052] The railway vehicle 200 may be an actual railway vehicle in operation, or it may be a dedicated railway vehicle used for opening and closing the door 220 for generating a trained model LM. Alternatively, a simulation device may be used in place of the railway vehicle 200, in which the door 220 and the surrounding part of the vehicle body 210 are simulated.

[0053] The distance sensor 40 is attached to the camera 30 or placed near the camera 30 and measures the distance to an object within the detection range, which includes the imaging range of the camera 30. The distance sensor 40 is, for example, a distance image sensor. The output of the distance sensor 40 is received by the machine learning device 50 via a predetermined communication line.

[0054] The machine learning device 50 performs machine learning on the learning model M based on the outputs of both the camera 30 and the distance sensor 40, and generates a trained model LM.

[0055] The functions of the machine learning device 50 may be realized by any hardware, or any combination of hardware and software. For example, the machine learning device 50 has the same configuration as the monitoring device 20 (Figure 2).

[0056] The machine learning device 50 includes, as functional units, a training data generation unit 501, a machine learning unit 502, a storage unit 503, and a distribution unit 504. These functional units are realized, for example, by loading a program installed in an auxiliary storage device into a memory device and executing it on the CPU. Furthermore, the function of the storage unit 503 is realized by a predetermined storage area in the auxiliary storage device or the like.

[0057] The training data generation unit 501 generates training data based on the outputs of the camera 30 and the distance sensor 40, and outputs the training data set TRD.

[0058] Each training data point included in the training dataset TRD is a combination of an image captured by camera 30 as input data and depth values ​​for multiple sections (e.g., pixels) across the entire image as ground truth data. The depth values ​​for multiple sections in the image captured by camera 30 are obtained based on the output of distance sensor 40 acquired at the same time.

[0059] The machine learning unit 502 uses the training dataset TRD to perform machine learning on the base training model M and generate a trained model LM. The machine learning unit 502 may also use additional training dataset TRD to perform additional training or retraining on the trained model LM and generate an updated trained model LM. As a result, the trained model LM can output depth estimates DP for multiple sections across the entire image from the image as input data.

[0060] The memory unit 503 stores the trained model LM generated by the machine learning unit 502. When the machine learning unit 502 generates an updated version of the trained model LM, the trained model LM in the memory unit 503 is updated, for example, by overwriting the trained model LM before the update. Alternatively, the trained model LM in the memory unit 503 may be updated in such a way that the updated version of the trained model LM is stored in the memory unit 503 in addition to the trained model LM before the update.

[0061] The distribution unit 504 distributes the trained model LM to the monitoring device 20 via the communication interface. The distribution unit 504 may automatically distribute the trained model LM to the monitoring device 20, or it may transmit the trained model LM to the monitoring device 20 in response to a request received from the monitoring device 20. For example, when the trained model LM in the storage unit 503 is updated, the distribution unit 504 distributes the latest trained model LM, and the monitoring device 20 distributes the latest trained model LM to the monitoring device 20.

[0062] [Example 1 of a method for monitoring the open / closed state of a door] In addition to Figures 1 and 2, a first example of a method for monitoring the open / closed state of door 120 will be described with reference to Figures 3 to 6.

[0063] Figure 3 shows an example of depth estimates DP output from the depth estimation unit 201. Figure 4 shows an example of depth estimates DP for each pixel coordinate on the depth monitoring line ML. Figures 5 and 6 illustrate a first example of a method for monitoring the open / closed state of the door 120.

[0064] Specifically, Figure 3 includes Figures 3A to 3C. Figure 3A is a diagram showing the heatmap 311 of the depth estimate value DP output from the depth estimation unit 201 when an image 301 of the door 120 in a completely closed state (fully closed state) is input. Figure 3B is a diagram showing the heatmap 312 of the depth estimate value DP output from the depth estimation unit 201 when an image 302 of the door 120 in an opening or closing operation is input. Figure 3C is a diagram showing the heatmap 313 of the depth estimate value DP output from the depth estimation unit 201 when an image 303 of the door 120 in a completely open state (fully open state) is input.

[0065] Figure 4 shows the distribution of depth estimates DP on the depth monitoring line ML for the fully closed state of the door 120, the opening or closing operation of the door 120, and the fully open state of the door 120. As shown in Figure 3, the depth monitoring line ML is a straight line extending horizontally on the image captured by the camera 10, connecting the left and right ends of the door panel 121 of the door 120.

[0066] Figure 5 shows a specific example of the time change of the average value of the depth estimate DP on the depth monitoring line ML (average depth DPa) during the opening operation of door 120. The operating points P501 to P503 in Figure 5 correspond to the fully closed state before the opening operation of door 120, the opening operation of door 120, and the fully open state after the opening operation of door 120 is completed, respectively.

[0067] Figure 6 shows a specific example of the time change of the average depth value DPa on the depth monitoring line ML during the closing operation of door 120. More specifically, Figure 6 shows a specific example of the time change of the average depth value DPa on the depth monitoring line ML when, after the closing operation of door 120 has started, door 120 reverses and opens due to some reason (for example, the occurrence of a foreign object getting caught in door 120).

[0068] <When the door is opened> As shown in Figures 3A and 4, in the state before the door 120 opens, the depth estimate DP on the depth monitoring line ML takes a relatively small value, indicating that it is relatively close to the camera 10.

[0069] As shown in Figures 3B and 4, when the door 120 begins to open, it starts opening from the center in the left-right direction. Therefore, on the depth monitoring line ML, the depth estimate DP of the center in the left-right direction takes a relatively large value, indicating that it is relatively far from the camera 10. The range on the depth monitoring line ML where the depth estimate DP takes a relatively large value expands as time progresses due to the opening of the door 120.

[0070] As shown in Figures 3C and 4, once the door 120 is fully opened, the depth estimate DP on the depth monitoring line ML takes on a relatively large value across the entire range corresponding to the opening of the vehicle body 110.

[0071] Therefore, as shown in Figure 5, the average depth DPa on the depth monitoring line ML is relatively small and almost constant before the opening operation of the door 120 begins (see operating point P501), and increases over time after the opening operation of the door 120 begins (see operating point P502). Then, the average depth DPa converges to a relatively large and almost constant state after the closing operation of the door 120 is completed (see operating point P503).

[0072] Therefore, for example, the opening / closing state monitoring unit 203 determines that the opening operation of the door 120 has started, that is, the door 120 has begun to open, when the average depth DPa exceeds the threshold DPth1 in the direction of increasing depth during the opening operation of the door 120. The threshold DPth1 is located between the operating point P501 and the operating point P503, and is a value relatively close to the operating point P501, and is predetermined through prior experiments or computer simulations.

[0073] Furthermore, for example, the open / closed state monitoring unit 203 determines that the door 120 has completed its opening operation, i.e., the door 120 is fully open, when the average depth DPa exceeds a threshold DPth2 (>DPth1) in the direction of increasing during the opening operation of the door 120, and then becomes almost constant. The threshold DPth2 is defined in advance through prior experiments or computer simulations as a value between the operating point P501 and the operating point P503, and relatively close to the operating point P503. For example, the open / closed state monitoring unit 203 determines that the average depth DPa is almost constant when the absolute value of the difference ΔDPa between the latest average depth DPa calculated sequentially and the previous average depth DPa (previous value DPa_lt) remains below a threshold ΔDPth1, which is set as a very small value, for a predetermined time t0 or longer. The threshold value ΔDPth1 is predetermined through prior experiments or computer simulations as the maximum expected variation in the difference ΔDPa when the door 120 is fully closed or fully open.

[0074] Thus, in this example, the open / closed state monitoring unit 203 can monitor the open / closed state of the door 120 based on the time change of the average depth DPa when the door 120 is opened.

[0075] <When the door is closing> During the closing operation of door 120, the depth-average DPa changes in the opposite direction to that during the opening operation of door 120. Specifically, the depth-average DPa is relatively large and almost constant before the closing operation of door 120 begins (see operating point P503), and decreases over time after the closing operation of door 120 begins (see operating point P502). Then, after the closing operation of door 120 is completed, the depth-average DPa converges to a relatively small and almost constant value.

[0076] Therefore, for example, the open / closed state monitoring unit 203 determines that the closing operation of the door 120 has started, that is, the door 120 has begun to close, when the average depth DPa exceeds the threshold DPth2 in the direction that decreases during the closing operation of the door 120.

[0077] Furthermore, for example, the open / closed state monitoring unit 203 determines that the closing operation of the door 120 is complete, i.e., the door 120 is fully closed, when the average depth DPa exceeds the threshold DPth1 in the direction of decreasing during the closing operation of the door 120, and then becomes almost constant thereafter. For example, the unit determines that the average depth DPa is almost constant when the absolute value of the difference ΔDPa between the latest average depth DPa and the previous value DPa_lt is greater than or equal to the threshold ΔDPth1 for a predetermined time t0 or longer.

[0078] Furthermore, if foreign objects become trapped in the door 120 during its closing operation, the door 120 may reverse direction and open, returning to its fully open position.

[0079] In this case, as shown in Figure 6, when the closing operation of the door 120 begins, the depth-average DPa exceeds the threshold DPth2 in the direction of decreasing (operating point P601). Subsequently, when the door 120 reverses and begins the opening operation, the depth-average DPa takes a local minimum value and begins to increase. Then, after the depth-average DPa exceeds the threshold DPth2 in the direction of increasing, the opening operation of the door 120 is completed, and it converges to a relatively large value that is almost constant.

[0080] Therefore, for example, the open / closed state monitoring unit 203 determines that the door 120 has reversed and started opening when the average depth DPa exceeds the threshold DPth2 in the direction of decreasing, then changes from decreasing to increasing, and the amount of increase from the minimum value (specifically, the difference ΔDPai described later) becomes greater than or equal to the threshold ΔDPth2 (see operating point P602).

[0081] Thus, in this example, the open / closed state monitoring unit 203 can monitor the open / closed state of the door 120 based on the time change of the average depth DPa when the door 120 is closing.

[0082] [Example 1 of the processing by the monitoring device when a door is opened] Referring to Figure 7, a first example of the processing of the monitoring device 20 when the door 120 is opened will be described. Specifically, a concrete example of the processing of the monitoring device 20 corresponding to the first example of the method for monitoring the open / closed state of the door 120 described above will be explained.

[0083] Figure 7 is a flowchart illustrating a schematic example of the processing performed by the monitoring device 20 when the door 120 is opened.

[0084] This flowchart is initiated, for example, when a signal indicating that the railway vehicle 100 is about to arrive at the platform is input to the monitoring device 20 from an external source.

[0085] As shown in Figure 7, in step S102, the monitoring device 20 initializes flag F1 and counter C1. Flag F1 indicates whether the door 120 has started opening. Counter C1 is used to measure the time after the average depth DPa exceeds the threshold DPth2 in an increasing direction. Specifically, the monitoring device 20 sets the values ​​of both flag F1 and counter C1 to "0" (zero).

[0086] When the monitoring device 20 completes the process in step S102, it proceeds to step S104.

[0087] In step S104, the depth estimation unit 201 acquires the latest captured image input from the camera 10.

[0088] When the monitoring device 20 completes the processing in step S104, it proceeds to step S106.

[0089] In step S106, the depth estimation unit 201 uses the trained model LM to estimate the depth of each pixel of the latest captured image based on the latest captured image, and obtains the depth estimate value DP.

[0090] When the monitoring device 20 completes the process in step S106, it proceeds to step S108.

[0091] In step S108, the open / closed state monitoring unit 203 calculates the average depth DPa on the depth monitoring line ML in the most recent captured image.

[0092] When the monitoring device 20 completes the process in step S108, it proceeds to step S110.

[0093] In step S110, the open / closed state monitoring unit 203 determines whether the value of flag F1 is "1". If the value of flag F1 is not "1", i.e., the value of flag F1 is "0", the unit proceeds to step S112, and if the value of flag F1 is "1", it proceeds to step S118.

[0094] In step S112, the open / closed state monitoring unit 203 determines whether the average depth DPa is greater than the threshold DPth1. If the average depth DPa is greater than the threshold DPth1, the open / closed state monitoring unit 203 determines that the door 120 has started to open and proceeds to step S114; otherwise, it returns to step S104.

[0095] In step S114, the monitoring result output unit 204 outputs a notification to the outside indicating that the door 120 has started to open.

[0096] When the monitoring device 20 completes the processing in step S114, it proceeds to step S116.

[0097] In step S116, the monitoring device 20 sets the value of flag F1 to "1".

[0098] When the monitoring device 20 completes the process in step S116, it returns to step S104.

[0099] Meanwhile, in step S118, the open / closed state monitoring unit 203 determines whether the average depth DPa is greater than the threshold DPth2. If the average depth DPa is greater than the threshold DPth2, the open / closed state monitoring unit 203 proceeds to step S120; otherwise, it returns to step S104.

[0100] In step S120, the opening / closing state monitoring unit 203 determines whether the absolute value of the difference ΔDPa between the latest depth average value DPa and the previous value DPa_lt is less than or equal to the threshold value ΔDPth1. If the absolute value of the difference ΔDPa is less than or equal to the threshold value ΔDPth1, the opening / closing state monitoring unit 203 proceeds to step S122; otherwise, it returns to step S104.

[0101] In step S122, the opening / closing state monitoring unit 203 increments the counter C1 by "1".

[0102] When the processing of step S122 is completed, the monitoring device 20 proceeds to step S124.

[0103] In step S124, the opening / closing state monitoring unit 203 determines whether the value of the counter C1 is greater than or equal to the threshold value C1th (>1). The threshold value C1th corresponds to the above-mentioned predetermined time t0. If the value of the counter C1 is greater than or equal to the threshold value C1th, the opening / closing state monitoring unit 203 determines that the opening operation of the door 120 is completed and proceeds to step S126; otherwise, it returns to step S104.

[0104] In step S126, the monitoring result output unit 204 outputs a notification to the outside indicating that the opening operation of the door 120 is completed.

[0105] When the processing of step S126 is completed, the monitoring device 20 ends the processing of the current flowchart.

[0106] [First Example of the Processing of the Monitoring Device during the Closing Operation of the Door] Next, referring to FIGS. 8 and 9, a first example of the processing of the monitoring device 20 during the closing operation of the door 120 will be described. Specifically, a specific example of the processing of the monitoring device 20 corresponding to the first example of the method for monitoring the opening / closing state of the door 120 during the closing operation of the door 120 will be described.

[0107] FIGS. 8 and 9 are flowcharts schematically showing a first example of the processing of the monitoring device 20 during the closing operation of the door 120.

[0108] This flowchart starts, for example, when the flowchart in Figure 7 is completed.

[0109] As shown in Figure 8, in step S202, the monitoring device 20 initializes flags F2 to F4, counter C1, and variable DPa_lmin. Flag F2 indicates whether the closing operation of door 120 has started. Flag F3 indicates whether the opening operation of door 120 is possible. Flag F4 indicates whether the opening operation of door 120 has started. The variable DPa_lmin is used to hold the minimum value of the depth average DPa when door 120 reverses from closing operation and starts opening operation. Specifically, the monitoring device 20 sets the values ​​of flags F2 to F4, counter C1, and variable DPa_lmin to "0".

[0110] Steps S204 to S208 are the same as steps S104 to S108 in Figure 7, so their explanation is omitted.

[0111] When step S208 is completed, the monitoring device 20 proceeds to step S210.

[0112] In step S210, the open / closed state monitoring unit 203 determines whether the value of flag F2 is "1". If the value of flag F2 is not "1", i.e., "0", the open / closed state monitoring unit 203 proceeds to step S212; if it is "1", it proceeds to step S218.

[0113] In step S212, the open / closed state monitoring unit 203 determines whether the average depth DPa is less than the threshold DPth2. If the average depth DPa is less than the threshold DPth2, the open / closed state monitoring unit 203 determines that the closing operation of the door 120 has started and proceeds to step S214; otherwise, it returns to step S204.

[0114] In step S214, the monitoring result output unit 204 outputs a notification to the outside indicating that the closing operation of the door 120 has started.

[0115] When the monitoring device 20 completes the processing in step S214, it proceeds to step S216.

[0116] In step S216, the monitoring device 20 sets the value of flag F2 to "1".

[0117] When the monitoring device 20 completes the processing in step S216, it returns to step S204.

[0118] On the other hand, in step S218, the open / closed state monitoring unit 203 determines whether the value of flag F3 is "1". If the value of flag F3 is not "1", i.e., "0", the open / closed state monitoring unit 203 proceeds to step S220, and if the value of flag F3 is "1", it proceeds to step S238 as shown in Figure 9.

[0119] In step S220, the open / closed state monitoring unit 203 determines whether the value of flag F4 is "1". If the value of flag F4 is not "1", i.e., "0", the open / closed state monitoring unit 203 proceeds to step S222, and if the value of flag F4 is "1", it proceeds to step S248.

[0120] In step S222, the open / closed state monitoring unit 203 determines whether the average depth DPa is less than the threshold DPth1. If the average depth DPa is less than the threshold DPth1, the open / closed state monitoring unit 203 proceeds to step S224; otherwise, it proceeds to step S232.

[0121] In step S224, the open / closed state monitoring unit 203 determines whether the absolute value of the difference ΔDPa between the latest depth average value DPa and the previous value DPa_lt is less than or equal to the threshold ΔDPth1. If the absolute value of the difference ΔDPa is less than or equal to the threshold ΔDPth1, the unit proceeds to step S226; otherwise, it returns to step S204.

[0122] In step S226, the open / closed state monitoring unit 203 increments the value of counter C1 by "1".

[0123] When the monitoring device 20 completes the processing in step S226, it proceeds to step S228.

[0124] In step S228, the open / closed state monitoring unit 203 determines whether the value of counter C1 is equal to or greater than the threshold C1th. If the value of counter C1 is equal to or greater than the threshold C1th, the open / closed state monitoring unit 203 determines that the closing operation of the door 120 is complete and proceeds to step S230; otherwise, it returns to step S204.

[0125] In step S230, the monitoring result output unit 204 outputs a notification to the outside indicating that the closing operation of the door 120 has been completed.

[0126] When the process in step S230 is completed, the monitoring device 20 terminates the process in this flowchart.

[0127] Meanwhile, in step S232, the open / closed state monitoring unit 203 determines whether the difference ΔDPa between the latest average depth DPa and the previous value DPa_lt is greater than 0. If the difference ΔDPa is greater than 0, the open / closed state monitoring unit 203 determines that the door 120 may have reversed and started opening, and proceeds to step S234; otherwise, it returns to step S204.

[0128] In step S234, the open / closed state monitoring unit 203 stores the previous value DPa_lt in the variable DPa_lmin.

[0129] When the monitoring device 20 completes the processing in step S234, it proceeds to step S236.

[0130] In step S236, the open / closed state monitoring unit 203 sets the value of flag F3 to "1".

[0131] When the monitoring device 20 completes the processing in step S236, it returns to step S204.

[0132] Meanwhile, in step S238, the open / closed state monitoring unit 203 determines whether the difference ΔDPa between the latest depth average value DPa and the previous value DPa_lt is greater than 0. If the difference ΔDPa is greater than 0, the open / closed state monitoring unit 203 determines that the state in which the door 120 may have started opening continues, and proceeds to step S240. Otherwise, it determines that the door 120 may not have started opening, and proceeds to step S246.

[0133] In step S240, the open / closed state monitoring unit 203 determines whether the difference ΔDPai between the latest depth average value DPa and the variable DPa_lmin corresponding to the minimum value is greater than or equal to the threshold ΔDPth2. If the difference DPai is greater than or equal to the threshold ΔDPth2, the open / closed state monitoring unit 203 determines that the door 120 has made the determination and started the opening operation, and proceeds to step S242; otherwise, it returns to step S204.

[0134] In step S242, the monitoring result output unit 204 outputs a notification to the outside indicating that the door 120 has reversed and started opening.

[0135] When the monitoring device 20 completes the processing in step S242, it proceeds to step S244.

[0136] In step S244, the open / closed state monitoring unit 203 sets the value of flag F4 to "1".

[0137] When the processing in step S244 is completed, the monitoring device 20 returns to step S204.

[0138] Meanwhile, in step S248, the open / closed state monitoring unit 203 determines whether the average depth DPa is greater than the threshold DPth2. If the average depth DPa is greater than the threshold DPth2, the open / closed state monitoring unit 203 determines that the opening operation of the reversed door 120 is complete and proceeds to step S250; otherwise, it returns to step S204.

[0139] In step S250, the monitoring result output unit 204 outputs a notification to the outside indicating that the opening operation of the door 120 after reversal has been completed.

[0140] When the monitoring device 20 completes the processing in step S250, it proceeds to step S252.

[0141] In step S252, the open / closed state monitoring unit 203 sets the values ​​of both flag F2 and flag F4 to "0".

[0142] When the monitoring device 20 completes the process in step S252, it returns to step S204.

[0143] [Second example of a method for monitoring the open / closed state of a door] Next, with reference to Figures 1 and 2, as well as Figure 10, a second example of a method for monitoring the open / closed state of door 120 will be described.

[0144] In the following example, we will focus on explaining the differences from the first example of the monitoring method described above, and may omit explanations of content that is the same as or corresponds to the first example of the monitoring method.

[0145] Figure 10 illustrates a second example of a method for monitoring the open / closed state of door 120.

[0146] Specifically, Figure 10 shows a concrete example of the time change of the average depth value DPa on the depth monitoring line ML during the opening operation of the door 120.

[0147] In this example, the method for determining the start of the opening operation and the start of the closing operation of the door 120 differs from that of the first example described above.

[0148] <When the door is opened> As shown in Figure 10, in this example, the opening / closing state monitoring unit 203 determines that the door 120 has started to open when the average depth DPa exceeds the threshold DPth1 in an increasing direction, and this state continues for a time t1 (>0) or longer.

[0149] This makes it possible to distinguish between cases where, for example, the depth average value DPa temporarily exceeds the threshold DPth1 due to disturbances, such as the operating point P1001 in Figure 10, and cases where the door 120 actually begins to open. Therefore, the monitoring device 20 can determine the start of the door 120's opening operation with greater accuracy.

[0150] <When the door is closing> In this example, the open / closed state monitoring unit 203 determines that the closing operation of the door 120 has started when the average depth DPa exceeds the threshold DPth2 in a decreasing direction, and this state continues for a time t1 or longer.

[0151] This makes it possible to distinguish between a situation where the depth average value DPa temporarily exceeds the threshold DPth2 in a negative direction due to disturbances, similar to the case when the door 120 is opened, and the actual start of the door 120 closing operation. Therefore, the monitoring device 20 can determine the start of the door 120 closing operation with greater accuracy.

[0152] [Second example of monitoring device processing during door opening operation] Referring to Figure 11, a second example of the processing of the monitoring device 20 when the door 120 is opened will be described. Specifically, a concrete example of the processing of the monitoring device 20 corresponding to the second example of the method for monitoring the open / closed state of the door 120 described above will be explained.

[0153] Figure 11 is a flowchart illustrating a second example of the processing performed by the monitoring device 20 when the door 120 is opened.

[0154] This flowchart is initiated, for example, when a signal indicating that the railway vehicle 100 is about to arrive at the platform is input to the monitoring device 20 from an external source.

[0155] As shown in Figure 11, this flowchart differs from the flowchart of the first example (Figure 7) described above in that the processes in steps S102 and S116 are replaced by steps S102A and S116A, and the processes in steps S113A to S113C are added. The following explanation will focus on the differences from the flowchart of the first example described above.

[0156] In step S102A, the monitoring device 20 initializes flag F1 and counters C1 and C2. Specifically, it sets all values ​​of flag F1 and counters C1 and C2 to "0".

[0157] When the monitoring device 20 completes the processing in step S102A, it proceeds to step S104.

[0158] In step S112, the open / closed state monitoring unit 203 proceeds to step S213A if the average depth DPa is greater than the threshold DPth1, and to step S113 otherwise.

[0159] In step S113A, the open / closed state monitoring unit 203 increments the value of counter C2 by "1".

[0160] When the monitoring device 20 completes the processing in step S113A, it proceeds to step S113B.

[0161] In step S113B, the open / closed state monitoring unit 203 determines whether the counter C2 is greater than or equal to the threshold C2th. The threshold C2th is a positive value and corresponds to the time t1 described above. If the counter C2 is greater than or equal to the threshold C2th, the open / closed state monitoring unit 203 determines that the door 120 has started to open and proceeds to step S114; otherwise, it returns to step S104.

[0162] When the monitoring device 20 completes the processing in step S114, it proceeds to step S116A.

[0163] In step S116A, the open / closed state monitoring unit 203 sets the value of flag F1 to "1" and sets the value of counter C2 to "0".

[0164] When the processing in step S116A is completed, the monitoring device 20 returns to step S104.

[0165] Meanwhile, in step S113C, the open / closed state monitoring unit 203 sets the value of counter C2 to "0".

[0166] This prevents the monitoring device 20 from mistakenly determining that the door 120 has started opening, even if the determination condition in step S112 is temporarily met due to disturbances or the like.

[0167] When the monitoring device 20 completes the processing in step S113C, it returns to step S104.

[0168] [Second example of monitoring device processing during door closing operation] Referring to Figure 12, a second example of the processing of the monitoring device 20 during the closing operation of the door 120 will be described. Specifically, a concrete example of the processing of the monitoring device 20 corresponding to the second example of the method for monitoring the open / closed state of the door 120 described above will be explained.

[0169] Figure 12 is a flowchart illustrating a second example of the processing performed by the monitoring device 20 when the door 120 is closed.

[0170] This flowchart starts, for example, when the flowchart in Figure 11 is completed.

[0171] As shown in Figure 12, this flowchart differs from the flowchart of the first example (Figures 8 and 9) described above in that the processes in steps S202 and S216 are replaced by steps S202A and S216A, and the processes in steps S213A to S213C are added. The following explanation will focus on the differences from the flowchart of the first example described above.

[0172] Since there are no changes to the processing in the section shown in Figure 9, the illustration is omitted.

[0173] As shown in Figure 12, in step S202A, the monitoring device 20 initializes the flags F2 to F4, counters C1 and C2, and the variable DPa_lmin. Specifically, the monitoring device 20 sets all values ​​of the flags F2 to F4, counters C1 and C2, and the variable DPa_lmin to "0".

[0174] In step S212, the open / closed state monitoring unit 203 proceeds to step S213A if the average depth DPa is less than the threshold DPth2, and to step S213C otherwise.

[0175] In step S213A, the open / closed state monitoring unit 203 increments the counter C2 by "1".

[0176] When the monitoring device 20 completes the processing in step S213A, it proceeds to step S213B.

[0177] In step S213B, the open / closed state monitoring unit 203 determines whether the counter C2 is equal to or greater than the threshold C2th. If the counter C2 is equal to or greater than the threshold C2th, the open / closed state monitoring unit 203 determines that the closing operation of the door 120 has started and proceeds to step S214; otherwise, it returns to step S204.

[0178] When the monitoring device 20 completes the processing in step S214, it proceeds to step S216A.

[0179] In step S216A, the open / closed state monitoring unit 203 sets the value of flag F2 to "1" and the value of counter C2 to "0".

[0180] When the processing in step S216A is completed, the monitoring device 20 returns to step S204.

[0181] Meanwhile, in step S213C, the open / closed state monitoring unit 203 sets the value of counter C2 to "0".

[0182] This prevents the monitoring device 20 from mistakenly determining that the closing operation of the door 120 has started, even if the determination condition in step S212 is temporarily met due to disturbances or the like.

[0183] When the monitoring device 20 completes the processing in step S213C, it returns to step S204.

[0184] [Third example of a method for monitoring the open / closed state of a door] Next, with reference to Figures 13 to 15, in addition to Figures 1 and 2, a third example of a method for monitoring the open / closed state of door 120 will be described.

[0185] Figure 13 shows another example of the depth estimate DP output from the depth estimation unit 201. Figure 14 shows another example of the depth estimate DP for each pixel coordinate on the depth monitoring line ML. Figure 15 illustrates a third example of a method for monitoring the open / closed state of the door 120.

[0186] Specifically, Figure 13 includes Figures 13A to 13C. Figure 13A is a heatmap 1311 of the depth estimate value DP output from the depth estimation unit 201 when an image 1301 of the door 120 in a completely closed state (fully closed state) is input. Figure 13B is a heatmap 1312 of the depth estimate value DP output from the depth estimation unit 201 when an image 1302 of the door 120 in an opening or closing operation is input. Figure 13C is a heatmap 1313 of the depth estimate value DP output from the depth estimation unit 201 when an image 1303 of the door 120 in a completely open state (fully open state) is input.

[0187] Figures 14 and 15 show the distribution of depth estimates DP on the depth monitoring line ML for the fully closed state of the door 120, the opening or closing operation of the door 120, and the fully open state of the door 120, respectively.

[0188] <When the door is opened> As shown in Figure 13, in this example, the passenger 130 is standing in a position visible from the camera 10 through the opening 111 of the vehicle body 110. In this case, as shown in Figures 13B, 13C, and 14, the depth estimate DP of the portion of the opening 111 where the passenger 130 is located, which is open (i.e., not covered by the door panel 121), is smaller than that of the other portions (see range 1400 in Figure 14). However, the depth estimate DP of the portion where the passenger 130 is located is still larger than the depth estimate DP of the door panel 121 and the side portion of the vehicle body 110.

[0189] Therefore, as shown in Figure 15, a threshold DPth3 is set that is sufficiently larger than the upper limit of the depth estimate DP assumed to be for the door panel 121 and the side portion of the vehicle body 110, and sufficiently smaller than the lower limit of the depth estimate DP assumed to be for the passengers 130 inside the vehicle body 110. This allows the opening / closing state monitoring unit 203 to determine that, when the door 120 is opening, pixels in which the depth estimate DP exceeds the threshold DPth3 in the direction of increasing over time are the parts of the opening 111 that are not covered by the door panel 121. For example, when the door 120 is opening, the opening / closing state monitoring unit 203 determines that the door 120 has started to open when the number of pixels w1 in which the depth estimate DP exceeds the threshold DPth3 in the direction of increasing becomes greater than or equal to the threshold wth1 (>0). The threshold wth1 is predetermined, for example, taking into consideration the accuracy of the depth estimate DP output from the trained model LM.

[0190] Furthermore, the opening / closing state monitoring unit 203 determines that the opening operation of the door 120 is complete when, during the opening operation of the door 120, the number of pixels w1 in which the depth estimate DP exceeds the threshold DPth3 in the direction of increasing over time reaches a certain number corresponding to the opening 111 of the vehicle body 110. Specifically, for example, the opening / closing state monitoring unit 203 determines that the opening operation of the door 120 is complete when the number of pixels w1 in which the depth estimate DP exceeds the threshold DPth3 in the direction of increasing over time becomes equal to or greater than the threshold wth2 (>wth1). The threshold wth2 is predetermined based on the width of the opening 111 of the vehicle body 110 on the depth monitoring line ML.

[0191] Furthermore, the open / closed state monitoring unit 203 may determine that the door 120 has opened when the number of pixels w1 in which the depth estimate DP exceeds the threshold DPth3 in an increasing direction over time remains almost constant, that is, hardly changes. Specifically, for example, the open / closed state monitoring unit 203 determines that the door 120 has opened when the difference Δw between the latest value and the previous value of the number of pixels w1 in which the depth estimate DP exceeds the threshold DPth3 in an increasing direction over time remains less than or equal to the threshold Δwth1 for a period of time t2 or longer. The threshold Δwth1 is predetermined as the maximum expected error in the difference Δw after the door 120 has opened, taking into consideration the accuracy of the depth estimate DP output from the trained model LM.

[0192] Thus, in this example, the open / closed state monitoring unit 203 can monitor the open / closed state of the door 120 when the door 120 is opened, based on the number of pixels w1 in which the depth estimate DP exceeds the threshold DPth3 in the direction of increasing over time.

[0193] <When the door is closing> For example, the open / closed state monitoring unit 203 determines that the door 120 has started closing when the depth estimate DP exceeds the threshold DPth3 in a way that decreases over time, and the number of pixels w2 is equal to or greater than the threshold wth1.

[0194] Furthermore, for example, the open / closed state monitoring unit 203 determines that the closing operation of the door 120 is complete when the number of pixels w2 in which the depth estimate DP exceeds the threshold DPth3 in a way that decreases over time becomes equal to or greater than the threshold wth2 during the closing operation of the door 120.

[0195] Furthermore, the open / closed state monitoring unit 203 determines that, during the closing operation of the door 120, if the number of pixels w2 in which the depth estimate DP exceeds the threshold DPth3 in a way that decreases over time becomes greater than or equal to the threshold wth1, and then the number of pixels w2 decreases and the amount of decrease Δw2i becomes greater than or equal to the threshold Δwth, the door 120 has reversed and begun to open.

[0196] Thus, in this example, the open / closed state monitoring unit 203 can monitor the open / closed state of the door 120 during the closing operation of the door 120, based on the number of pixels w2 in which the depth estimate DP exceeds the threshold DPth3 in a direction that decreases over time.

[0197] [Third example of how a monitoring device processes information when a door is opened] Referring to Figure 16, a third example of the processing of the monitoring device 20 when the door 120 is opened will be described. Specifically, a concrete example of the processing of the monitoring device 20 corresponding to the third example of the method for monitoring the open / closed state of the door 120 described above will be explained.

[0198] Figure 16 is a flowchart illustrating a third example of the processing performed by the monitoring device 20 when the door 120 is opened.

[0199] This flowchart is initiated, for example, when a signal indicating that the railway vehicle 100 is about to arrive at the platform is input to the monitoring device 20 from an external source.

[0200] As shown in Figure 16, in step S302, the monitoring device 20 initializes flag F1. Specifically, the monitoring device 20 sets the value of flag F1 to "0" (zero).

[0201] When the monitoring device 20 completes the processing in step S302, it proceeds to step S304.

[0202] Steps S304 and S306 are the same as steps S104 and S106 in Figure 7, so their explanation is omitted.

[0203] When the processing in step S306 is completed, the monitoring device 20 will move in step S308. In step S308, the number of pixels w1 on the depth monitoring line ML whose depth estimate DP is greater than the threshold DPth3 is counted.

[0204] When the monitoring device 20 completes the process in step S308, it proceeds to step S310.

[0205] In step S310, the open / closed state monitoring unit 203 determines whether the value of flag F1 is "1". If the value of flag F1 is not "1", i.e., the value of flag F1 is "0", the unit proceeds to step S312, and if the value of flag F1 is "1", it proceeds to step S318.

[0206] In step S312, the open / closed state monitoring unit 203 determines whether the number of pixels w1 is equal to or greater than the threshold wth1. If the number of pixels w1 is equal to or greater than the threshold wth1, the open / closed state monitoring unit 203 proceeds to step S314; otherwise, it returns to step S304.

[0207] Furthermore, between steps S312 and S314, additional processing similar to that in steps S113A and S113B may be added, as in the case of the second example of the processing of the monitoring device 20 described above (Figure 11).

[0208] Steps S314 and S316 are the same as steps S114 and S116 in Figure 7, so their explanation is omitted.

[0209] When the monitoring device 20 completes the processing in step S316, it returns to step S304.

[0210] Meanwhile, in step S318, the open / closed state monitoring unit 203 determines whether the number of pixels w1 is equal to or greater than the threshold wth2. If the number of pixels w1 is equal to or greater than the threshold wth2, the open / closed state monitoring unit 203 proceeds to step S320; otherwise, it returns to step S304.

[0211] Step S320 is the same as the process in step S126 in Figure 7, so its explanation is omitted.

[0212] When the process in step S320 is completed, the monitoring device 20 terminates the process in this flowchart.

[0213] [Third example of how a monitoring device processes information during door closing] Referring to Figures 17 and 18, a third example of the processing of the monitoring device 20 during the closing operation of the door 120 will be described. Specifically, a concrete example of the processing of the monitoring device 20 corresponding to the third example of the method for monitoring the open / closed state of the door 120 described above will be described during the closing operation of the door 120.

[0214] Figures 17 and 18 are flowcharts illustrating a third example of the processing performed by the monitoring device 20 when the door 120 is closed.

[0215] This flowchart starts, for example, when the flowchart in Figure 16 is completed.

[0216] As shown in Figure 17, in step S402, the monitoring device 20 initializes the flags F2 to F4 and the variable w2_lmax. Specifically, the monitoring device 20 sets all values ​​of the flags F2 to F4 and the variable w2_lmax to "0" (zero).

[0217] When the monitoring device 20 completes the processing in step S402, it proceeds to step S404.

[0218] Steps S404 and S406 are the same as steps S204 and S206 in Figure 8, so their explanation is omitted.

[0219] When the monitoring device 20 completes the processing in step S406, it proceeds to step S408.

[0220] In step S408, the open / closed state monitoring unit 203 counts the number of pixels w2 on the depth monitoring line ML whose depth estimate DP is less than the threshold DPth3.

[0221] When the monitoring device 20 completes the processing in step S408, it proceeds to step S410.

[0222] In step S410, the open / closed state monitoring unit 203 determines whether the value of flag F1 is "1". If the value of flag F1 is not "1", i.e., "0", the open / closed state monitoring unit 203 proceeds to step S412, and if the value of flag F1 is "1", it proceeds to step S418.

[0223] In step S412, the open / closed state monitoring unit 203 determines whether the number of pixels w2 is equal to or greater than the threshold wth1. If the number of pixels w2 is equal to or greater than the threshold wth1, the open / closed state monitoring unit 203 proceeds to step S414; otherwise, it returns to step S404.

[0224] Furthermore, between steps S412 and S414, additional processing similar to that in the second example of the monitoring device 20's processing described above (Figure 12) may be added, as in steps S213A and S213B.

[0225] Steps S414 and S416 are the same as steps S214 and S216 in Figure 8, so their explanation is omitted.

[0226] When the processing in step S416 is completed, the monitoring device 20 returns to step S404.

[0227] Meanwhile, in step S418, the open / closed state monitoring unit 203 determines whether the value of flag F3 is "1". If the value of flag F3 is not "1", i.e., "0", the open / closed state monitoring unit 203 proceeds to step S420, and if the value of flag F3 is "1", it proceeds to step S432.

[0228] In step S420, the open / closed state monitoring unit 203 determines whether the value of flag F4 is "1". If the value of flag F4 is not "1", i.e., "0", the open / closed state monitoring unit 203 proceeds to step S422, and if the value of flag F4 is "1", it proceeds to step S442.

[0229] In step S422, the open / closed state monitoring unit 203 determines whether the number of pixels w2 is equal to or greater than the threshold wth2. If the number of pixels w2 is equal to or greater than the threshold wth2, the open / closed state monitoring unit 203 determines that the closing operation of the door 120 is complete and proceeds to step S424; otherwise, it returns to step S404.

[0230] Step S424 is the same as the process in step S230 in Figure 8, so its explanation is omitted.

[0231] When the process in step S424 is completed, the monitoring device 20 terminates the process in this flowchart.

[0232] Meanwhile, in step S426, the open / closed state monitoring unit 203 determines whether the difference Δw2 (=w2-w2_lt) between the latest pixel count w2 and the previous value w2_lt is less than 0. If the difference Δw2 is less than 0, the open / closed state monitoring unit 203 proceeds to step S428; otherwise, it returns to step S404.

[0233] In step S428, the open / closed state monitoring unit 203 stores the previous value w2_lt in the variable w2_lmax.

[0234] When the monitoring device 20 completes the processing in step S428, it proceeds to step S430.

[0235] In step S430, the open / closed state monitoring unit 203 sets the value of flag F3 to "1".

[0236] When the monitoring device 20 completes the processing in step S430, it returns to step S404.

[0237] On the other hand, as shown in Figure 18, in step S432, the open / closed state monitoring unit 203 determines whether the difference Δw2 (=w2-w2_lt) between the latest pixel count w2 and the previous value w2_lt is less than 0. If the difference Δw2 is less than 0, the open / closed state monitoring unit 203 determines that the decrease in the pixel count w2 over time is continuing and proceeds to step S434. Otherwise, it determines that the decrease in the pixel count w2 over time is not continuing and proceeds to step S440.

[0238] In step S434, the open / closed state monitoring unit 203 determines whether the amount of decrease Δw2i (=w2_lmax-w2) since the pixel count w2 began to decrease is greater than or equal to the threshold Δwth. If the amount of decrease Δw2i is greater than or equal to the threshold Δwth, the open / closed state monitoring unit 203 determines that the door 120 has reversed and started opening, and proceeds to step S436; otherwise, it returns to step S404.

[0239] Steps S436 and S438 are the same processes as steps S242 and S244 in Figure 9, so their explanation is omitted.

[0240] When the processing in step S438 is complete, the monitoring device 20 returns to step S404.

[0241] On the other hand, since step S440 is the same as the process in step S246 in Figure 9, its explanation will be omitted.

[0242] When the monitoring device 20 completes the processing in step S440, it returns to step S404.

[0243] Meanwhile, in step S442, the open / closed state monitoring unit 203 determines whether the number of pixels w2 is less than or equal to the threshold wth1. If the number of pixels w2 is less than or equal to the threshold wth1, the open / closed state monitoring unit 203 proceeds to step S444; otherwise, it returns to step S404.

[0244] Steps S444 and S446 are the same as steps S250 and S252 in Figure 9, so their explanation is omitted.

[0245] When the processing in step S446 is completed, the monitoring device 20 returns to step S404.

[0246] [Second example of a monitoring system] A second example of the monitoring system 1 according to this embodiment will be described with reference to Figure 19.

[0247] In the following examples, components identical to or corresponding to those in the first example (Figure 1) above will be denoted by the same reference numerals. The explanation will focus on the parts that differ from the first example, and the explanation of parts that are the same as or corresponding to the first example may be omitted.

[0248] Figure 19 shows the configuration of the first example of monitoring system 1.

[0249] As shown in Figure 19, the monitoring system 1 in this example differs from the first example described above in that the monitoring device 20 further includes a depth correction unit 205 as a functional unit, but may be the same as the first example described above in other respects. The function of the depth correction unit 205 is realized, for example, by loading a program installed in the auxiliary storage device 22 into the memory device 23 and executing it on the CPU 24.

[0250] The depth correction unit 205 corrects the depth estimate value DP output from the depth estimation unit 201 and outputs the corrected depth estimate value DP.

[0251] The open / closed state monitoring unit 203 monitors the open / closed state of the door 120 based on the depth estimate DP corrected by the depth correction unit 205. The open / closed state monitoring unit 203 can adopt any one of the first to third examples of the door 120 monitoring method described above.

[0252] [First example of a method for correcting depth estimates] In addition to Figure 19, we will now describe a first example of a method for correcting the depth estimate DP with reference to Figures 20 and 21.

[0253] Figure 20 shows an example of an image captured by camera 10 (image 2000). Figure 21 illustrates a first example of a method for correcting the depth estimate DP.

[0254] In this example, as shown in Figure 20, objects 140 and 150 are present within the imaging range of camera 10, and objects 140 and 150 are visible in the image 2000 captured by camera 10.

[0255] Objects 140 and 150 are fixed objects whose distance from camera 10 does not change. Objects 140 and 150 are suspended, for example, from the ceiling of a train station platform. Objects 140 and 150 may be dedicated objects for correcting the depth estimate DP, or they may be any objects already present on the station platform, such as signs.

[0256] The distances of objects 140 and 150 from camera 10 are known, and information representing the respective distances of objects 140 and 150 from camera 10 is pre-stored in an auxiliary storage device 22, etc. This allows the depth correction unit 205 to correct the depth estimate DP of any section in the captured image 2000 based on the information representing the respective distances of objects 140 and 150 from camera 10 and the depth estimate DP of the section corresponding to objects 140 and 150 in the captured image 2000. Therefore, the monitoring device 20 can improve the estimation accuracy of the depth estimate DP, and as a result, can more appropriately monitor the open / closed state of door 120.

[0257] For example, as shown in Figure 21, the relationship between the uncorrected depth estimate DP and the distance x from camera 10 is expressed by the following equations (1) to (3) as a linear function corresponding to the line 2101, using the distance m of object 140 from camera 10, the depth estimate qm corresponding to object 140, the distance n of object 150 from camera 10, and the depth estimate qn of the pixel corresponding to object 150.

[0258]

number

[0259] Still, the depth estimation value qn corresponding to the object 140 is, for example, the maximum value, minimum value, most frequent value, median value, average value, etc. of the depth estimation value DP among the pixels corresponding to the object 140 in the captured image 2000. Also, the depth estimation value qn corresponding to the object 140 may be the depth estimation value DP of the pixel corresponding to the pre-specified reference point in the object 140 among the pixels corresponding to the object 140 in the captured image 2000. The same may apply to the depth estimation value qn corresponding to the object 150.

[0260] Also, the relationship between the corrected depth estimation value DP and the distance x from the camera 10 is pre-specified by the following formula (4) as a linear function corresponding to the straight line 2102, using pre-specified constants a and b.

[0261]

Equation

[0262] From formulas (1) to (4), the corrected depth estimation value qB of an arbitrary object at a distance p from the camera 10 and with a pre-correction depth estimation value qA is represented by the following formula (5).

[0263]

Equation

[0264] Thereby, the depth correction unit 205 can calculate the corrected depth estimation value DP (depth estimation value qB) from the pre-correction depth estimation value DP (depth estimation value qA) using formula (5).

[0265] Still, in this example, the depth correction unit 205 may correct the depth estimation value DP output from the depth estimation unit 201 using one or more objects with a known distance from the camera 10 in addition to the objects 140 and 150. That is, the depth correction unit 205 may correct the depth estimation value DP output from the depth estimation unit 201 based on the depth estimation values DP corresponding to three or more objects with a known distance from the camera 10.

[0266] In this example, the depth correction unit 205 may also correct the depth estimate DP output from the depth estimation unit 201 based on depth estimates DP corresponding to two or more different locations on the same object whose distance from the camera 10 is known.

[0267] In this example, the depth estimate DP output from the depth estimation unit 201 may be corrected based on the depth estimate DP corresponding to the fixed part of the vehicle body 110 whose distance from the camera 10 is known, instead of replacing part or all of two or more objects different from the railway vehicle 100.

[0268] Furthermore, in this example, the relationship between the depth estimate DP and the distance x from camera 10 may be defined by a function other than a linear function.

[0269] [Second example of a method for correcting depth estimates] In addition to Figure 19, a second example of a method for correcting the depth estimate DP will be described with reference to Figure 22.

[0270] Figure 22 illustrates a second example of a method for correcting depth estimates (DP).

[0271] Specifically, Figure 22 includes Figures 22A and 22B. Figure 22A shows an example of the distribution of the uncorrected depth estimate DP on the depth monitoring line ML when the door 120 is fully closed. Figure 22B shows an example of the distribution of the corrected depth estimate DP on the depth monitoring line ML when the door 120 is fully closed.

[0272] For example, due to deviations in the stopping position of the railway vehicle 100 or limitations on the placement of the camera 10, the camera 10 may not be able to image the door 120 from the front. In this case, for example, as shown in Figure 22A, the depth estimate DP may change along the left-right direction of the door 120.

[0273] Therefore, in this example, as shown in FIG. 22B, the depth correction unit 205 corrects the depth estimated value DP so that the change in the left-right direction of the depth estimated value DP is in a state where the door 120 is virtually viewed from the front. Thereby, the monitoring device 20 can more appropriately monitor the opening / closing state of the door 120 by using the first to third examples of the above-described door 120 monitoring method.

[0274] For example, based on the depth estimated values DP of the portions of the vehicle body 110 adjacent to the opening 111 on the depth monitoring line ML (regions AR1 and AR2 in FIG. 22), the depth correction unit 205 obtains a linear function (Equation (6)) corresponding to the straight line 2201 representing the relationship between the depth estimated value DP (depth estimated value y) before correction and the coordinate x in the left-right direction. Specifically, the depth correction unit 205 performs linear approximation on the depth estimated values DP before correction in the regions AR1 and AR2, and obtains the slope t and the intercept u of Equation (6).

[0275]

Equation

[0276] Thereby, the depth correction unit 205 can calculate the depth estimated value DP (depth estimated value DP′) after correction by using the following Equation (7) based on Equation (6) and a predetermined value qf corresponding to the baseline after correction.

[0277]

Equation

[0278] Note that the predetermined value qf may be defined in advance, or may be defined based on the depth estimated values DP before correction in the regions AR1 and AR2.

[0279] [Third Example of Monitoring System] Referring to FIG. 23, a third example of the monitoring system 1 according to the present embodiment will be described.

[0280] In the following, the same or corresponding components as in the first example (Figure 1) and the second example (Figure 19) described above will be denoted by the same reference numerals. The explanation will focus on the parts that differ from the first and second examples, and may omit explanations of parts that are the same or corresponding to the first and second examples.

[0281] Figure 23 shows the configuration of the third example of monitoring system 1.

[0282] As shown in Figure 23, the monitoring system 1 in this example differs from the first example described above in that the monitoring device 20 further includes a monitoring standard setting unit 206 as a functional unit, but may be the same as the first example described above in other respects.

[0283] The monitoring standard setting unit 206 sets monitoring standards for the open / closed state monitoring unit 203 to monitor the open / closed state of the door 120 based on the depth estimate value DP output from the depth estimation unit 201. For example, the monitoring standard setting unit 206 sets monitoring standard values ​​for the depth estimate value DP and the average depth value DPa. These monitoring standard values ​​are, for example, the threshold values ​​DPth1, DPth2, DPth3, etc.

[0284] The open / closed state monitoring unit 203 monitors the open / closed state of the door 120 using the monitoring criteria set by the monitoring criteria setting unit 206 based on the depth estimate value DP.

[0285] [Example of how to set monitoring criteria] Referring to Figure 24, an example of how to set monitoring criteria for the open / closed state of door 120 will be described.

[0286] Figure 24 illustrates an example of how to set monitoring criteria.

[0287] Specifically, figure 24 is a diagram showing an example of the distribution of depth estimates DP on the depth monitoring line ML.

[0288] In this example, the monitoring standard setting unit 206 sets the monitoring standard for the open / closed state of the door 120 based on the depth estimate value DP of a fixed part whose distance from the camera 10 does not change.

[0289] For example, as shown in FIG. 24, the monitoring standard setting unit 206 sets the above-described threshold values DPth1, DPth2, and threshold value DPth3 based on the depth estimated value DP of the portion of the vehicle body 110 adjacent to the opening 111 on the depth monitoring line ML (regions AR1 and AR2 in FIG. 24). Specifically, the monitoring standard setting unit 206 calculates the average value DPa_AR of the depth estimated values DP of the regions AR1 and AR2, and sets the threshold values DPth1 and DPth2 by adding an offset amount g to the average value. Specifically, offset amounts g1 and g2 (g1 < g2) are defined in advance for the threshold values DPth1 and DPth2, respectively, and the monitoring standard setting unit 206 sets the threshold values DPth1 and DPth2 using the following equations (8) and (9).

[0290] [Equation]

[0291] Similarly, the monitoring standard setting unit 206 sets the threshold value DPth3 using the following equation (10) on the premise of an offset amount g3 defined in advance for the threshold value DPth3.

[0292] [Equation]

[0293] Thereby, even when a drift or the like occurs in the depth estimated value DP and the accuracy as an absolute value decreases, the monitoring device 20 can appropriately monitor the open / closed state of the door 120 by using the depth estimated value DP of the portion of the vehicle body 110 as a fixed part as a reference.

[0294] [Other Embodiments] Other embodiments will be described.

[0295] In the above-described embodiments, modifications and changes may be added as appropriate. Hereinafter, examples of modifications and changes added to the above-described embodiments will be referred to as "modified examples" for convenience.

[0296] For example, the second and third examples of the monitoring system 1 described above may be combined. For example, the depth correction unit 205 corrects the depth estimated value DP output from the depth estimation unit 201 using the second example of the depth estimated value DP correction method described above, and the monitoring standard setting unit 206 sets thresholds DPth1 and DPth2 from the depth estimated value DP output from the depth estimation unit 201 using the example of the monitoring standard setting method described above. Then, the open / closed state monitoring unit 203 may monitor the open / closed state of the door 120 using the first or second example of the monitoring method described above, based on the depth estimated value DP corrected by the depth correction unit 205 and the thresholds DPth1 and DPth2 set by the monitoring standard setting unit 206.

[0297] Furthermore, in the embodiments described above and their variations, the functions of the monitoring device 20 may be implemented in a distributed manner by multiple information processing devices.

[0298] Furthermore, in the embodiments described above and their modifications, the functions of the monitoring device 20 and the machine learning device 50 may be realized by a single information processing device.

[0299] [Effect] Next, the operation of the monitoring system, monitoring device, monitoring method, and program according to this embodiment will be described.

[0300] In the first aspect of this embodiment, the monitoring system comprises an imaging unit, a storage unit, an estimation unit, and a monitoring unit. The monitoring system is, for example, the monitoring system 1 described above. The imaging unit is, for example, the camera 10 described above. The storage unit is, for example, the storage unit 202 described above. The estimation unit is, for example, the depth estimation unit 201 described above. The monitoring unit is, for example, the open / closed state monitoring unit 203 described above. Specifically, the imaging unit images a predetermined range including the movable part of the door to be monitored. The door to be monitored is, for example, the door 120 described above. The movable part is, for example, the door panel 121 described above. The predetermined range is, for example, the range corresponding to the depth monitoring line ML described above. The storage unit also stores a trained model that takes an image as input and outputs an index value representing the distance from the viewpoint of an object shown in the image. The index value is, for example, the depth described above. The trained model is, for example, the trained model LM described above. Furthermore, the estimation unit uses the trained model to estimate an index value representing the distance from the imaging unit to an object in a target area for each of a plurality of areas spanning a predetermined range in the image captured by the imaging unit. The plurality of areas are a plurality of pixels on the depth monitoring line ML in the image captured by the camera 10. The monitoring unit then monitors the open / closed state of the door being monitored based on the index value estimated by the estimation unit. The index value estimated by the estimation unit is, for example, the depth estimate value DP described above.

[0301] Furthermore, in the first aspect of this embodiment, the monitoring device may include the storage unit, the estimation unit, and the monitoring unit. The monitoring device is, for example, the monitoring device 20 described above.

[0302] Furthermore, in the first aspect of this embodiment, a monitoring method performed by a monitoring device may be provided. The information processing device is, for example, the monitoring device 20 described above. The monitoring method includes an estimation step and a monitoring step. The estimation step is, for example, steps S106 and S206 described above. The monitoring step is, for example, steps S110 to S126 and steps S210 to S252. Specifically, in the estimation step, the monitoring device takes an image as input and uses a trained model that outputs an index value representing the distance of an object in the image from the viewpoint to the image to estimate the index value representing the distance of an object in a target section from the imaging unit for each of a plurality of sections spanning the predetermined range in the image captured by the imaging unit that images the predetermined range including the movable part of the door to be monitored. Then, in the monitoring step, the monitoring device monitors the open / closed state of the door to be monitored based on the index value estimated in the estimation step.

[0303] Furthermore, in the first aspect of this embodiment, the information processing device may be provided with a program that causes it to execute the above-described monitoring method, specifically, the estimation step and the monitoring step. The information processing device is, for example, the monitoring device 20 described above.

[0304] For example, as described in Patent Document 1 above, the open / closed state of a door may be monitored using a distance sensor (e.g., an optical distance meter). In this case, depending on the type of door, the reflected wave may not be received properly. For example, in the case of a highly reflective door such as a stainless steel door, the laser may not be reflected back, and as a result, the open / closed state of the door may not be properly monitored.

[0305] Furthermore, as in the aforementioned Patent Document 2, for example, retroreflective bodies are installed on both the movable and fixed parts of the door, and the door's open / closed state is monitored from the positional relationship between the retroreflective bodies in the captured image, which includes both retroreflective bodies. In this case, if the reflectors are hidden from the camera's view, become dirty, or fall off, it may not be possible to properly monitor the door's open / closed state.

[0306] Furthermore, as described in Patent Document 3 above, for example, there are cases where multiple cameras are installed, and the opening and closing state of the door is monitored from the change in the depth direction of the captured image calculated from the parallax of the multiple cameras. In this case, if the multiple cameras are not properly coordinated, or if there is a discrepancy in the coordination for some reason, it may not be possible to properly monitor the opening and closing state of the door. Moreover, since multiple cameras are required, this may lead to an increase in initial costs, and the coordination of the multiple cameras may become time-consuming.

[0307] Furthermore, it is possible to monitor the open / closed state of a door by applying known image processing techniques to an image that includes the door. However, the color of the door and its surroundings may interfere with the accurate monitoring of the door's open / closed state.

[0308] In contrast, the monitoring system, monitoring device, and information processing device (hereinafter referred to as "monitoring system, etc.") of this embodiment can use a trained model to estimate an index value representing the distance from the camera to an object captured in an image, based on the captured image including the movable part of the door. Therefore, the monitoring system, etc. can acquire an index value representing the distance from the camera to an object captured in each of multiple sections within a predetermined range of the captured image, in a manner that is less affected by, for example, the material and color of the door. Thus, the monitoring system, etc. can appropriately monitor the open / closed state of the door based on the index value representing the distance from the camera to an object captured in each of multiple sections within a predetermined range of the captured image. Furthermore, the monitoring system, etc. does not require the use of multiple cameras, thus reducing the cost and effort required to monitor the open / closed state of the door.

[0309] Furthermore, in a second aspect of this embodiment, based on the first aspect described above, the estimation unit may sequentially estimate the index value for each of the plurality of sections. The monitoring unit may then monitor the open / closed state of the monitored door based on the time change of the average value of the index value in the plurality of sections. The average value is, for example, the depth average value DPa described above.

[0310] This allows monitoring systems to monitor the door's open / closed state by using the average value of indicator values ​​for multiple sections, which change over time as the door opens and closes. This is because there is a difference in indicator values ​​between the section corresponding to the open part of the door and the section corresponding to the closed or fixed part of the door, and the ratio of the former to the latter sections changes with the door's opening and closing operation, resulting in a change in the average value of the indicator values ​​over time.

[0311] Furthermore, in a third aspect of this embodiment, based on the second aspect described above, the monitoring unit may determine that the monitored door has started to open or close when the average value exceeds a first threshold in a predetermined direction as time progresses. The first threshold is, for example, the threshold DPth1 when the door 120 is opened and the threshold DPth2 when the door 120 is closed.

[0312] This allows monitoring systems to detect the start of door opening and closing operations.

[0313] Furthermore, in a fourth aspect of this embodiment, based on the third aspect described above, the monitoring unit may determine that the door being monitored has started to open or close if the average value exceeds the first threshold in the predetermined direction as time progresses, and this condition continues for a predetermined time or longer. The predetermined time is, for example, the time t1 described above.

[0314] This allows monitoring systems, for example, to avoid determining that a door has started to open or close even if the average value of the indicator temporarily exceeds a predetermined threshold in a specific direction due to disturbances. As a result, monitoring systems can more accurately monitor the open and closed state of the door.

[0315] Furthermore, in a fifth aspect of this embodiment, assuming any one of the second to fourth aspects described above, the monitoring unit may determine that the monitored door is fully open or fully closed when the average value exceeds the second threshold in the predetermined direction, and thereafter the average value is maintained within a certain range of fluctuation. The second threshold is, for example, the threshold DPth2 during the opening operation of the door 120 and the threshold DPth1 during the closing operation of the door 120. The certain range of fluctuation is, for example, the threshold ΔDPth1 described above.

[0316] This allows monitoring systems to detect when a door is opened or closed.

[0317] Furthermore, in a sixth aspect of this embodiment, assuming either the third or fourth aspect described above, the monitoring unit may determine that the door being monitored has reversed and begun to open when the average value, after exceeding the first threshold in the predetermined direction corresponding to the closing direction of the door being monitored as time progresses, changes in the opposite direction to the predetermined direction as time progresses, and the amount of change becomes greater than or equal to the third threshold. The amount of change is, for example, the difference ΔDPai described above. The third threshold is the ΔDPth2 described above.

[0318] This allows monitoring systems, for example, to detect when a door reverses and opens due to an object getting caught in it during the closing operation.

[0319] Furthermore, in a seventh aspect of this embodiment, based on the first aspect described above, the estimation unit may sequentially estimate the index value for each of the plurality of sections. The monitoring unit may then monitor the open / closed state of the door to be monitored based on the number of sections in which the index value exceeds the fourth threshold in a predetermined direction as time progresses. The number of sections is, for example, the number of pixels w1, w2 described above. The fourth threshold is, for example, the threshold DPth3 described above.

[0320] This allows monitoring systems to monitor the door's open / closed state by utilizing the number of sections whose index values, which change over time due to the door's opening and closing operations, exceed a fourth threshold in a predetermined direction.

[0321] Furthermore, in the eighth aspect of this embodiment, based on the seventh aspect described above, the monitoring unit may determine that the door being monitored has started to open or close if the number of sections among the plurality of sections in which the index value exceeds the fourth threshold in the predetermined direction as time progresses becomes equal to or greater than the fifth threshold. The fifth threshold is, for example, the threshold wth1 described above.

[0322] This allows monitoring systems to detect the start of door opening and closing operations.

[0323] Furthermore, in the ninth aspect of this embodiment, based on the seventh or eighth aspect described above, the monitoring unit may determine that the door being monitored is fully open or fully closed when the number of compartments among the plurality of compartments in which the index value exceeds the fourth threshold in the predetermined direction over time reaches a predetermined value, or when the number stops changing over time. The predetermined value is, for example, the threshold wth2 described above.

[0324] This allows monitoring systems to detect when a door is opened or closed.

[0325] Furthermore, in the tenth embodiment of this embodiment, based on the eighth embodiment described above, the monitoring unit may determine that the door being monitored has reversed and begun to open when, among the plurality of compartments, the number of compartments in which the index value exceeds the fourth threshold in the predetermined direction corresponding to the closing direction of the door being monitored as time progresses becomes equal to or greater than the fifth threshold, and then decreases as time progresses, and the amount of decrease becomes equal to or greater than the sixth threshold. The amount of decrease is, for example, the amount of decrease Δw2i described above. The sixth threshold is Δwth described above.

[0326] Furthermore, in the eleventh aspect of this embodiment, a correction unit may be provided that corrects the index value estimated by the estimation unit, based on any one of the first to tenth aspects described above. The correction unit is, for example, the depth correction unit 205 described above. Specifically, the captured image may include a predetermined object whose actual distance from the imaging unit is known. The predetermined object is, for example, the objects 140 and 150 described above. The correction unit may also correct the index value for each of the plurality of sections based on the index value corresponding to the predetermined object estimated by the estimation unit and the actual distance from the imaging unit to the predetermined object. The monitoring unit may then monitor the open / closed state of the door being monitored based on the index value corrected by the correction unit.

[0327] This allows the monitoring system to appropriately correct the index value by comparing the actual distance from the imaging unit to a designated object with an index value (estimated value) representing that distance. Therefore, by using the corrected index value, the door's open / closed state can be monitored more accurately.

[0328] Furthermore, in the twelfth aspect of this embodiment, based on any one of the first to tenth aspects described above, the monitoring system, etc., may include a setting unit that sets a reference value for monitoring the open / closed state of the door to be monitored, based on the index value corresponding to the fixed part of the door to be monitored other than the movable part of the door to be monitored among the plurality of compartments. The setting unit is, for example, the monitoring reference setting unit 206 described above. The reference value is, for example, the threshold values ​​DPth1, DPth2, and DPth3 described above. The monitoring unit may then monitor the open / closed state of the door based on the index value and the reference value.

[0329] As a result, even if there is a certain degree of error between the estimated and actual values ​​of the indicator values, the monitoring system can appropriately monitor the door's open / closed state using the indicator value (estimated value) of a fixed part whose distance from the imaging unit does not change as a reference.

[0330] Furthermore, in the 13th aspect of this embodiment, based on any one of the first to ten aspects described above, the monitoring system, etc., may include a correction unit that corrects the index values ​​for each of the plurality of compartments based on the index values ​​corresponding to the fixed parts on both sides adjacent to the left and right sides of the opening of the door to be monitored when the door to be monitored is viewed from the front. The correction unit is, for example, the depth correction unit 205 described above. The monitoring unit may then monitor the open / closed state of the door to be monitored based on the index values ​​corrected by the correction unit.

[0331] For example, if the imaging unit cannot image the door from the front, the index value will change according to the left-right position of the door, regardless of whether the door is open or closed. As a result, the door's open or closed state cannot be monitored solely based on the relative magnitudes of the index value and a fixed reference value.

[0332] In contrast, in this embodiment, even if the imaging unit cannot image the door from the front, the monitoring system can use index values ​​corresponding to the fixed parts on both the left and right sides of the door to correct the index values ​​so that they do not change according to the left and right position of the door when there is no change in the door's open / closed state. Therefore, the monitoring system can appropriately monitor the door's open / closed state.

[0333] Although embodiments have been described in detail above, this disclosure is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims. [Explanation of Symbols]

[0334] 1. Monitoring System 2. Machine Learning Systems 10 Cameras 20 Monitoring equipment 30 Cameras 40 Distance Sensor 50 Machine Learning Devices 100 Railway Vehicles 110 vehicle body 111 Aperture 120 doors 121 Door Panel 130 passengers 140 Object 150 Object 200 railway vehicles 201 Depth estimation section 202 Storage section 203 Open / Close Status Monitoring Unit 204 Monitoring Result Output Section 205 Depth correction section 206 Monitoring standard setting section 210 vehicle body 211 Aperture 220 doors 221 Door Panel 501 Training Data Generation Unit 502 Machine Learning Department 503 Storage section 504 Distribution Department LM pre-trained model ML depth monitoring line

Claims

1. An imaging unit that images a predetermined range including the movable part of the door being monitored, A storage unit that stores a trained model that takes an image as input and outputs an index value representing the distance of an object in the image from the viewpoint, An estimation unit that uses the trained model to estimate the index value representing the distance from the imaging unit to an object in a target area for each of a plurality of areas spanning a predetermined range in the image captured by the imaging unit, The system includes a monitoring unit that monitors the open / closed state of the door being monitored based on the index value estimated by the estimation unit, Monitoring system.

2. The estimation unit sequentially estimates the index value for each of the plurality of sections, The monitoring unit monitors the open / closed state of the door being monitored based on the time change of the average value of the index value in the plurality of compartments. The monitoring system according to claim 1.

3. The monitoring unit determines that the monitored door has started to open or close when the average value exceeds a first threshold in a predetermined direction as time progresses. The monitoring system according to claim 2.

4. The monitoring unit determines that the door being monitored has started to open or close if the average value exceeds the first threshold in the predetermined direction as time progresses, and this condition continues for a predetermined time or longer. The monitoring system according to claim 3.

5. The monitoring unit determines that the monitored door is fully open or fully closed when the average value exceeds a second threshold in the predetermined direction, and thereafter the average value is maintained within a certain range of fluctuation. The monitoring system according to any one of claims 2 to 4.

6. The monitoring unit determines that the door being monitored has started to reverse and open when the average value exceeds the first threshold in the predetermined direction corresponding to the closing direction of the door being monitored as time progresses, and then changes in the opposite direction to the predetermined direction as time progresses, and the amount of change becomes equal to or greater than the third threshold. The monitoring system according to claim 3 or 4.

7. The estimation unit sequentially estimates the index value for each of the plurality of sections, The monitoring unit monitors the open / closed state of the door being monitored based on the number of compartments in which the index value exceeds a fourth threshold in a predetermined direction as time progresses. The monitoring system according to claim 1.

8. The monitoring unit determines that the door being monitored has started to open or close when the number of compartments in which the index value exceeds the fourth threshold in a predetermined direction over time reaches or exceeds the fifth threshold. The monitoring system according to claim 7.

9. The monitoring unit determines that the door being monitored is fully open or fully closed when the number of compartments in which the index value exceeds the fourth threshold in the predetermined direction over time reaches a predetermined value, or when the number of compartments in which the index value stops changing over time. The monitoring system according to claim 7 or 8.

10. The monitoring unit determines that the door being monitored has begun to reverse and open when, among the plurality of compartments, the number of compartments in which the index value exceeds the fourth threshold in the predetermined direction corresponding to the closing direction of the door being monitored, after the number of compartments has exceeded the fifth threshold, decreases over time, and the amount of decrease exceeds the sixth threshold. The monitoring system according to claim 8.

11. The system includes a correction unit that corrects the index value estimated by the estimation unit, The captured image shows a predetermined object whose actual distance from the imaging unit is known. The correction unit corrects the index value for each of the plurality of sections based on the index value corresponding to the predetermined object estimated by the estimation unit and the actual distance from the imaging unit to the predetermined object. The monitoring unit monitors the open / closed state of the door being monitored based on the index value corrected by the correction unit. A monitoring system according to any one of claims 1 to 4, 7, 8, and 10.

12. The system includes a setting unit that sets a reference value for monitoring the open / closed state of the door to be monitored, based on the index value corresponding to the fixed part of the door to be monitored other than the movable part of the door to be monitored among the plurality of compartments, The monitoring unit monitors the open / closed state of the door based on the index value and the reference value. A monitoring system according to any one of claims 1 to 4, 7, 8, and 10.

13. The system includes a correction unit that corrects the index values ​​for each of the multiple sections based on the index values ​​corresponding to the fixed parts on both sides adjacent to the left and right sides of the opening of the door to be monitored when the door to be monitored is viewed from the front among the multiple sections, The monitoring unit monitors the open / closed state of the door being monitored based on the index value corrected by the correction unit. A monitoring system according to any one of claims 1 to 4, 7, 8, and 10.

14. A storage unit that stores a trained model that takes an image as input and outputs an index value representing the distance of an object in the image from the viewpoint, An estimation unit that uses the trained model to estimate an index value representing the distance of an object from the imaging unit to a target area for each of several sections spanning the predetermined range in the image captured by the imaging unit, which captures a predetermined range including the movable part of the door to be monitored. The system includes a monitoring unit that monitors the open / closed state of the door being monitored based on the index value estimated by the estimation unit, monitoring equipment.

15. The monitoring device takes an image as input and uses a trained model that outputs an index value representing the distance of an object in the image from the viewpoint to the object in the image to estimate the index value representing the distance of an object in a target area to the imaging unit for each of several sections spanning the predetermined range in the image captured by the imaging unit that captures the predetermined range including the movable part of the door to be monitored. The monitoring device includes a monitoring step of monitoring the open / closed state of the door being monitored based on the index value estimated in the estimation step, Monitoring method.

16. In an information processing device, An estimation step in which, using a trained model that takes an image as input and outputs an index value representing the distance of an object in the image from the viewpoint, estimates the index value representing the distance of an object in a target section from the imaging unit for each of several sections spanning the predetermined range in the image captured by the imaging unit that captures the predetermined range including the movable part of the door to be monitored. Based on the index value estimated in the estimation step, a monitoring step is performed to monitor the open / closed state of the door being monitored. program.

Images