Information processing method, computer program, and information processing device

Abstract

Provided are an information processing method, a computer program, and an information processing device that can be expected to realize safety monitoring or the like based on an image captured by a camera.　An information processing method according to one embodiment of the present invention involves, by means of an information processing device: acquiring a captured image in which the inside of a room is captured by a camera; and detecting, on the basis of the acquired captured image, an opening in the floor of the room. An information processing method according to one embodiment of the present invention involves, by means of an information processing device: acquiring a captured image in which the inside of a room is captured by a camera; detecting, on the basis of the acquired captured image, a person in the room; inferring the posture of the detected person; acquiring an operating state of a device installed in the room; and determining, on the basis of the inferred posture of the person and the acquired operating state of the device, an error regarding a device that a person is to work on.

Description

Information processing method, computer program, and information processing device

[0001] The present disclosure relates to an information processing method, a computer program, and an information processing device.

[0002] Patent Document 1 proposes a surveillance system in which an authentication controller performs authentication processing based on authentication information received from an authentication terminal via electric field communication, an image recognition server recognizes people included in the shooting area of ​​an image captured by a camera, and the person authenticated by the authentication controller is compared with the person image-recognized by the image recognition server based on the position information of the authentication terminal and the coordinate information of the camera's shooting area.

[0003] JP 2012-8802 A

[0004] The present disclosure provides an information processing method, a computer program, and an information processing device that are expected to realize safety monitoring based on images captured by a camera.

[0005] In one embodiment, an information processing method includes an information processing device acquiring an image of a room taken by a camera, and detecting an opening in a floor of the room based on the acquired image. In another embodiment, an information processing method includes an information processing device acquiring an image of a room taken by a camera, detecting a person in the room based on the acquired image, estimating a posture of the detected person, acquiring an operating state of a device installed in the room, and determining an error in a device on which the person is working based on the estimated posture of the person and the acquired operating state of the device.

[0006] According to the present disclosure, it is expected that safety monitoring based on images captured by cameras will be realized.

[0007] FIG. 1 is a schematic diagram for explaining an overview of an information processing system according to embodiment 1. FIG. 2 is a block diagram showing an example of a configuration of an information processing device according to embodiment 1. FIG. 3 is a schematic diagram showing an example of a learning model according to embodiment 1. FIG. 4 is a block diagram showing an example of a configuration of a management device according to embodiment 1. FIG. 5 is a flowchart showing an example of a procedure for opening detection processing performed by the information processing device according to embodiment 1. FIG. 6 is a flowchart showing an example of a procedure for opening detection processing performed by the information processing device according to embodiment 1. FIG. 7 is a schematic diagram for explaining an overview of an information processing system according to embodiment 2. FIG. 8 is a block diagram showing an example of a configuration of an information processing device according to embodiment 2. FIG. 9 is a block diagram showing an example of a configuration of a management device according to embodiment 2. FIG. 10 is a flowchart showing an example of a procedure for schedule management processing performed by the management device according to embodiment 2. FIG. 11 is a flowchart showing an example of a procedure for device error prevention processing performed by the information processing device according to embodiment 2. FIG. 12 is a flowchart showing an example of a procedure for device error prevention processing performed by the information processing device according to embodiment 2.

[0008] Specific examples of information processing systems according to embodiments of the present disclosure will be described below with reference to the drawings. Note that the present disclosure is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0009] 1 is a schematic diagram for explaining an overview of an information processing system according to embodiment 1. The information processing system according to embodiment 1 is a system for monitoring safety within a clean room 100 in which, for example, a plurality of substrate processing apparatuses 101 are installed. The information processing system is configured to include a plurality of cameras 3 arranged at appropriate positions within the clean room 100, a plurality of information processing apparatuses 1 that control the cameras 3, and a management apparatus 5 that manages information obtained from the plurality of information processing apparatuses 1.

[0010] The floor of the clean room 100 is made up of, for example, approximately rectangular floor tiles laid in rows and columns. Below the floor on which these floor tiles are laid, there may be a space for accommodating various pieces of equipment, or it may also lead to a lower floor. One or several of the laid floor tiles may be removed, for example, when inspecting or maintaining the equipment below. Where a floor tile is removed, an opening is formed in the floor surface, and safety measures are required to prevent workers and the like in the clean room 100 from falling through the opening.

[0011] In the information processing system according to the first embodiment, a plurality of cameras 3 disposed at appropriate locations within the clean room 100 capture images of the floor of the clean room 100, and each information processing device 1 performs a process of detecting an opening in the floor based on the images captured by each camera 3. When the information processing device 1 detects an opening based on the images of the floor captured by the cameras 3, it issues an alarm, for example, using light or sound. Furthermore, when the information processing device 1 detects an opening, it notifies, for example, a management device 5, that an opening has been detected, and transmits the captured image of the detected opening to the management device 5. Upon receiving notification of the detection of an opening from the information processing device 1, the management device 5 notifies, for example, a terminal device carried by a worker within the clean room 100, of information such as the location of the opening. Furthermore, the management device 5 obtains the captured image when the opening is detected from the information processing device 1 and stores it in a database together with information such as the date and time and the location of the opening.

[0012] Furthermore, in the first embodiment, the camera 3 placed in the clean room 100 is a camera capable of capturing images in both the visible light and infrared regions, i.e., a so-called RGB-IR camera. Therefore, the image data obtained by the camera 3 includes RGB color data and data indicating infrared intensity. Among the multiple floor tiles laid on the floor of the clean room 100, some may be made of translucent glass or synthetic resin to enhance downward visibility. When non-translucent floor tiles and translucent floor tiles are mixed, there is a possibility that the translucent floor tiles may be mistakenly detected as openings using only images captured by the camera in the visible light region. Therefore, the information processing system according to the first embodiment uses images captured in both the visible light region and the infrared region to prevent the translucent floor tiles from being mistakenly detected as openings.

[0013] Furthermore, even if an opening is formed in the floor of the clean room 100, if sufficient protective measures have been taken, there is no need to notify workers in the clean room 100. In the information processing system according to the first embodiment, even if an opening is detected in the floor of the clean room 100, no notification is issued if it is determined that sufficient protective measures have been taken, such as by installing road cones, safety bars, or safety fences around the opening. Each information processing device 1 detects safety devices, such as road cones or safety fences, from images captured by the camera 3. The information processing device 1 determines whether a predetermined condition is met for the opening detected in the captured image, such as whether a safety device, such as a road cone, is present within a predetermined range, whether a predetermined number of safety devices are present, or whether multiple safety devices are installed surrounding the opening. If the information processing device 1 determines that a safety device is placed around the opening detected in the captured image in a manner that satisfies the predetermined condition, it determines that a protective measure has been taken for the opening. If it determines that no safety device is present in the captured image in which an opening is detected, or that the safety devices are not placed in a manner that satisfies the predetermined condition, it determines that no protective measure has been taken for the opening.

[0014] The information processing system may also detect an opening in the floor of the clean room 100 and issue a notification when a person, such as a worker, approaches the opening. In this case, the information processing device 1 detects a person, for example, from an image captured by the camera 3. For example, when the information processing device 1 detects an opening and a person from the captured image, or when it detects a person within a predetermined distance from the detected opening, the information processing device 1 notifies the management device 5 and issues an alarm using light or sound, etc. Furthermore, for example, when the information processing device 1 detects only an opening from the captured image, it may only notify the management device 5, and when it detects both an opening and a person from the captured image, it may both notify the management device 5 and issue an alarm using light, etc.

[0015] <Device Configuration> Fig. 2 is a block diagram showing an example configuration of an information processing device 1 according to embodiment 1. The information processing device 1 according to embodiment 1 is, for example, an information processing device called an edge computer in the field of IoT (Internet of Things). The information processing device 1 is configured to include a processing unit 11, a storage unit 12, a communication unit 13, an interface unit 14, etc. Note that in this embodiment, one camera 3 is connected to one information processing device 1, but this is not limited thereto, and multiple cameras 3 may be connected to one information processing device 1.

[0016] The processing unit 11 is configured using an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit) or a quantum processor, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processing unit 11 reads and executes a program 12a stored in the storage unit 12 to perform various processes such as detecting an opening in the floor inside the clean room 100 from an image captured by the camera 3 and notifying an operator or the like according to the detection result.

[0017] The storage unit 12 is configured using a large-capacity storage device such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11 and various data necessary for the processing of the processing unit 11. In this embodiment, the storage unit 12 stores a program 12a executed by the processing unit 11. The storage unit 12 also includes a learning model storage unit 12b that stores information about one or more learning models that have been previously subjected to machine learning.

[0018] In this embodiment, the program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disc, and the information processing device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may also be written to the storage unit 12, for example, during the manufacturing stage of the information processing device 1. Alternatively, the program 12a may be distributed by a remote server device or the like and acquired by the information processing device 1 via communication. For example, the program 12a may be read from the recording medium 99 by a writing device and written to the storage unit 12 of the information processing device 1. The program 12a may be provided in a form distributed via a network or in a form recorded on the recording medium 99.

[0019] The learning model storage unit 12b stores in advance information about the learning model used in the various processes described above performed by the information processing device 1 according to embodiment 1. The information about the learning model stored in the learning model storage unit 12b may include, for example, information indicating the structure of the learning model and values ​​of internal parameters of the learning model determined in advance by machine learning. Fig. 3 is a schematic diagram showing an example of a learning model according to embodiment 1.

[0020] The segmentation model 21 shown in the upper part of Figure 3 is a learning model that has undergone machine learning in advance to accept data of a captured image as input and output data indicating whether each pixel of the captured image corresponds to an opening, a safety device, or something other than those. The segmentation model 21 may be a learning model configured to handle images, such as a convolutional neural network (CNN). Note that the information processing system according to the first embodiment uses a camera 3 capable of capturing images in the visible light and infrared light regions, and therefore the captured image data input to the segmentation model 21 is image data of four channels: R (red), G (green), B (blue), and IR (infrared). The segmentation result data output by the segmentation model 21 is, for example, a two-dimensional array of the same size as the input captured image, with each element of the array storing a value corresponding to the classification result. The segmentation model 21 is generated in advance by performing so-called supervised machine learning processing using learning data (teacher data) that associates image data of four channels (R, G, B, and IR) with correct value data that indicates whether each pixel of this image data corresponds to an opening, a safety device, or something other than those. Note that supervised machine learning is an existing technology, so a detailed description will be omitted.

[0021] The human detection model 22 shown in the lower part of Figure 3 is a learning model that has undergone machine learning in advance to accept captured image data as input and detect people in the captured image. The human detection model 22 may employ a learning model configured to handle images, such as CNN. The captured image data input to the human detection model 22 may be image data with four channels (R, G, B, and IR), or may be image data with three channels (R, G, and B). The human detection result output by the human detection model 22 may be the coordinates of a rectangle, or so-called bounding box, that surrounds the area in the captured image in which a person is captured. The human detection model 22 is generated in advance by performing so-called supervised machine learning processing using training data (teacher data) that associates image data with ground truth data indicating the area in the image data in which a person is captured. The human detection model 22 may also utilize an existing trained learning model, such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

[0022] The communication unit 13 communicates with the management device 5 via a wired or wireless network. The communication unit 13 transmits data provided by the processing unit 11 to the management device 5, and also receives data transmitted from the management device 5 and provides the data to the processing unit 11. In this embodiment, the communication unit 13 transmits notification data based on the processing result of the opening detection, data of the image captured by the camera 3, etc. to the management device 5.

[0023] The interface unit 14 is connected to devices such as the camera 3 and the LED (Light Emitting Diode) lamp 4 via a communication line or a signal line, and exchanges data, signals, and the like with these devices. In the present embodiment, the interface unit 14 acquires captured image data output by the camera 3 and provides the data to the processing unit 11. The interface unit 14 also outputs an on / off control signal provided by the processing unit 11 to the LED lamp 4, switching the LED lamp 4 on and off. Note that in the present embodiment, the camera 3 is connected to the interface unit 14 of the information processing device 1, but this is not limited thereto. For example, if the camera 3 has a communication function, the information processing device 1 may be configured to communicate with the camera 3 via the communication unit 13. In this case, the information processing device 1 may acquire captured image data output from the camera 3 from the communication unit 13 via a wired or wireless network. The same applies to the LED lamp 4.

[0024] The storage unit 12 may be an external storage device connected to the information processing device 1. The information processing device 1 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The information processing device 1 is not limited to the above configuration, and may include, for example, a display unit such as a liquid crystal display and an operation unit that accepts user operations.

[0025] In addition, in the information processing device 1 according to this embodiment, the processing unit 11 reads and executes the program 12a stored in the memory unit 12, whereby the captured image acquisition unit 11a, opening detection unit 11b, protective measure determination unit 11c, person detection unit 11d, notification processing unit 11e, etc. are realized in the processing unit 11 as software functional units.

[0026] The captured image acquiring unit 11a acquires the captured image data output by the camera 3 via the interface unit 14, thereby performing a process of acquiring captured images of the floor in the clean room 100. The camera 3 takes images and outputs the captured images, for example, several times to several tens of times per second, and the captured image acquiring unit 11a acquires the captured images, for example, at the same frequency as the images taken by the camera 3, and temporarily stores the acquired captured images in the storage unit 12. Note that if the camera 3 has a communication function, the captured image acquiring unit 11a may communicate with the camera 3 via the communication unit 13 and acquire the captured images from the camera 3.

[0027] The opening detection unit 11b performs processing to detect openings present on the floor in the clean room 100 based on the captured images acquired by the captured image acquisition unit 11a from the camera 3. In the first embodiment, the opening detection unit 11b detects openings using the segmentation model 21 stored in the learning model storage unit 12b. The opening detection unit 11b inputs data (R, G, B, IR) of the captured images acquired by the captured image acquisition unit 11a to the segmentation model 21, and acquires the segmentation result output by the segmentation model 21 in response to this. The segmentation result includes flag information and the like that classifies what is captured in each pixel of the captured image as an opening, a safety device, or something else, and the opening detection unit 11b detects openings by identifying image areas in the captured image that are classified as containing an opening.

[0028] The protective measure determination unit 11c performs processing to determine whether protective measures have been taken for openings in the floor of the clean room 100, based on the captured image acquired by the captured image acquisition unit 11a from the camera 3. First, the protective measure determination unit 11c detects safety devices, such as traffic cones, safety bars, or safety fences, that appear in the captured image. In the first embodiment, the protective measure determination unit 11c detects safety devices using the segmentation model 21. However, the protective measure determination unit 11c may also use the segmentation results acquired from the segmentation model 21 by the opening detection unit 11b in response to the input of the captured image. The segmentation results include flag information and the like that classifies what appears in each pixel of the captured image, and the protective measure determination unit 11c identifies image areas classified as containing safety devices. Note that instead of detecting safety devices using the segmentation model 21, the protective measure determination unit 11c may also detect safety devices using a separate learning model that has been previously machine-learned to detect safety devices based on captured images. Furthermore, some or all of the above processing by the protective measure determination unit 11c may be performed only when an opening is detected by the opening detection unit 11b, or may be performed at all times regardless of whether an opening is present or not.

[0029] Next, if no safety device is present in the captured image, the protective measure determination unit 11c determines that no protective measure has been taken. If a safety device is present in the captured image, the protective measure determination unit 11c determines whether the protective measure has been taken based on the relationship between the position of the opening detected by the opening detection unit 11b and the position of the safety device detected by the protective measure determination unit 11c itself. In the first embodiment, the protective measure determination unit 11c determines whether the correct protective measure has been taken by, for example, determining whether two or more safety devices are present within a predetermined distance from the opening. However, the criteria for determining whether the protective measure has been taken are not limited to these and can be set as appropriate by the designer or administrator of the system. Various conditions can be set as conditions for the correct protective measure, such as that the safety devices are installed around the opening, that the safety devices are installed so as to separate the opening from the passageway in the clean room 100, or that two or more types of safety devices are installed. The protective measure determination unit 11c may, for example, perform a predetermined calculation process based on the coordinate information of the opening and safety device in the captured image to determine whether the above conditions are met, or may make a determination using a learning model that has been machine-learned in advance to determine whether the correct protective measures have been taken based on this coordinate information.

[0030] The human detection unit 11d performs processing to detect people in the clean room 100 based on the captured images acquired by the captured image acquisition unit 11a from the camera 3. In the first embodiment, the human detection unit 11d detects people using the human detection model 22 stored in the learning model storage unit 12b. The human detection unit 11d inputs data of the captured images acquired by the captured image acquisition unit 11a to the human detection model 22, and acquires a human detection result output by the human detection model 22 in response to this. The human detection result includes coordinate information of a bounding box surrounding a person shown in the captured image, and the human detection unit 11d detects people by acquiring information about this bounding box.

[0031] Furthermore, when the opening detection unit 11b detects an opening on the floor in the clean room 100, the person detection unit 11d compares the position of the opening with the position of each of the one or more people detected by the person detection unit 11d, and determines whether or not the person is present within a predetermined distance from the opening. For example, the information processing device 1 can make a person detected within a predetermined distance from an opening the target of a notification, which will be described later.

[0032] Furthermore, the human detection unit 11d detects the orientation of the face or body of the detected person and determines whether the person is facing the opening. For example, the information processing device 1 can set a person who is present within a predetermined range from the opening and facing the opening as the target of a notification, as described below. Furthermore, the human detection unit 11d determines the movement direction of the detected person, for example, based on multiple captured images acquired in chronological order from the camera 3. For example, the information processing device 1 can set a person who is moving toward the opening as the target of a notification, as described below. Note that the conditions for determining whether or not a person is the target of a notification are not limited to those described above and may be any conditions, and can be set as appropriate by the designer or administrator of the system.

[0033] The notification processing unit 11e issues notifications regarding safety within the clean room 100 based on the opening detection result by the opening detection unit 11b, the protective measure determination result by the protective measure determination unit 11c, and the person detection result by the person detection unit 11d. In the first embodiment, the notification processing unit 11e can issue two types of notifications: a notification to the management device 5 and a notification by lighting the LED lamp 4. For example, when the opening detection unit 11b detects an opening in the floor of the clean room 100, the notification processing unit 11e transmits a message notifying the management device 5 of this. At this time, the notification processing unit 11e transmits data of the captured image in which the opening was detected and the determination result by the protective measure determination unit 11c regarding the presence or absence of a protective measure to the management device 5. Furthermore, when the person detection unit 11d detects a person within a predetermined distance from the opening, a person facing the opening, or a person approaching the opening, the notification processing unit 11e outputs a control signal from the interface unit 14 to light the LED lamp 4 and notify nearby people of the presence of the opening. The above-mentioned two types of notifications are merely examples, and the present invention is not limited to these examples. The conditions under which notifications are sent can be appropriately set by the designer or administrator of the system. Furthermore, if the information processing device 1 is configured to be able to communicate with the user's terminal device 7, the notification processing unit 11e may send a notification message to the terminal device 7.

[0034] 4 is a block diagram showing an example configuration of the management device 5 according to the first embodiment. The management device 5 according to the first embodiment can be realized by installing a predetermined application program or the like in a general-purpose information processing device such as a personal computer or a server computer. The management device 5 is configured to include a processing unit 51, a storage unit 52, a communication unit 53, a display unit 54, and an operation unit 55. Note that although the first embodiment will be described assuming that processing is performed by a single device, the processing of the management device 5 may be distributed among a plurality of devices.

[0035] The processing unit 51 is configured using an arithmetic processing device such as a CPU, an MPU, a GPU, or a quantum processor, a ROM, a RAM, etc. The processing unit 51 reads and executes a program 52a stored in the storage unit 52 to perform various processes such as collecting information from a plurality of information processing devices 1 and notifying terminal devices 7 carried by workers or the like in the clean room 100.

[0036] The storage unit 52 is configured using a large-capacity storage device such as a hard disk, and stores a program 52a executed by the processing unit 51. The storage unit 52 also includes a safety management DB (database) 52b that stores and accumulates safety-related image data captured by a plurality of cameras 3 installed in the clean room 100.

[0037] In this embodiment, the program (computer program, program product) 52a is provided in a form recorded on a recording medium 98 such as a memory card or an optical disc, and the management device 5 reads the program 52a from the recording medium 98 and stores it in the storage unit 52. However, the program 52a may also be written to the storage unit 52, for example, during the manufacturing stage of the management device 5. Alternatively, for example, the program 52a may be distributed by a remote server device or the like and acquired by the management device 5 via communication. For example, the program 52a may be read by a writing device from the recording medium 98 and written to the storage unit 52 of the management device 5. The program 52a may be provided in a form distributed via a network or in a form recorded on the recording medium 98.

[0038] The safety management DB 52b is a database for collecting and storing safety-related images from among the images captured by the multiple cameras 3 installed in the clean room 100. In the first embodiment, the management device 5 collects, for example, images captured of openings detected in the floor of the clean room 100 and stores them in the safety management DB 52b. The safety management DB 52b stores, for example, images captured by the cameras 3 in association with various related information such as the date, time, and location at which the images were captured, identification information of the camera 3 that captured the images or the information processing device 1 that controls the camera 3, and the presence or absence of protective measures.

[0039] The communication unit 53 transmits and receives data to and from the plurality of information processing devices 1 and the plurality of terminal devices 7 via a wired or wireless network. In the first embodiment, the communication unit 53 receives a notification message based on the detection of an opening and the associated captured image data, etc., transmitted from the information processing device 1, and transmits these to the processing unit 51. The communication unit 53 also transmits the notification message regarding the opening transmitted from the processing unit 51 to the terminal device 7. The communication unit 53 also communicates with the camera 3 having a communication function via a wired or wireless network, and can directly acquire captured images from the camera 3.

[0040] The display unit 54 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 51. In the first embodiment, the display unit 54 displays, for example, notification messages and captured images received from the information processing device 1. The management device 5 can also display various information including captured images stored in the safety management DB 52b on the display unit 54. The management device 5 can also display captured images of the camera 3 obtained via the communication unit 53 on the display unit 54 in real time.

[0041] The operation unit 55 accepts user operations and notifies the processing unit 51 of the accepted operations. For example, the operation unit 55 may be input devices such as a mouse and a keyboard, and these input devices may be configured to be detachable from the management device 5. Furthermore, for example, the operation unit 55 accepts user operations via an input device such as a mechanical button or a touch panel provided on the surface of the display unit 54.

[0042] The storage unit 52 may be an external storage device connected to the management device 5. The management device 5 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The management device 5 is not limited to the above configuration, and may not include, for example, the display unit 54 and the operation unit 55.

[0043] In addition, in the management device 5 according to this embodiment, the processing unit 51 reads and executes the program 52a stored in the memory unit 52, thereby realizing the information acquisition unit 51a, the notification processing unit 51b, etc. as software functional units in the processing unit 51.

[0044] The information acquisition unit 51a communicates with a plurality of information processing devices 1 via the communication unit 53, and performs processing to acquire information such as notifications regarding openings and photographed image data from the information processing devices 1. The information acquisition unit 51a associates the acquired information such as notifications and photographed images with information such as date and time, location, identification information, and the presence or absence of protective measures, and stores the information in the safety management DB 52b.

[0045] The notification processing unit 51b performs processing to notify a terminal device 7 carried by a worker or the like in the clean room 100 that an opening has been detected in the floor of the clean room 100. When the notification processing unit 51b is notified by the information processing device 1 of the detection of an opening and is notified of the determination result that protective measures for the detected opening have not been properly taken, the notification processing unit 51b transmits a message to one or more terminal devices 7 notifying that an opening exists in the floor of the clean room 100. At this time, the notification processing unit 51b may transmit a message including information on the location of the opening, etc., in the message, and notify the worker or the like carrying the terminal device 7 of the location of the opening.

[0046] 5 and 6 are flowcharts showing an example of the procedure of the opening detection process performed by the information processing device 1 according to embodiment 1. The captured image acquisition unit 11a of the processing unit 11 of the information processing device 1 according to embodiment 1 acquires a captured image of the floor in the clean room 100 captured by the camera 3 by exchanging information with the camera 3 via the interface unit 14 (step S1). The opening detection unit 11b of the processing unit 11 inputs data of the captured image acquired in step S1 to the segmentation model 21 stored in the learning model storage unit 12b (step S2). The opening detection unit 11b acquires the segmentation result output by the segmentation model 21 in response to the input in step S2 (step S3).

[0047] The segmentation result obtained in step S3 is flag information that classifies each pixel of the photographed image as depicting an opening, a safety device, or something else. The opening detection unit 11b determines whether an opening exists on the floor of the clean room 100 based on whether the photographed image contains any pixels classified as depicting an opening from the segmentation result (step S4). If no opening exists (S4: NO), the processing unit 11 returns to step S1, acquires the next photographed image, and performs the same process.

[0048] If an opening is present (S4: YES), the protective measure determination unit 11c of the processing unit 11 determines whether or not a safety device such as a road cone, a safety bar, or a safety fence is present in the clean room 100 based on whether or not the captured image contains pixels depicting the safety device from the segmentation result acquired in step S3 (step S5). If a safety device is present (S5: YES), the protective measure determination unit 11c determines whether or not the correct protective measure for the opening is being taken based on, for example, the relationship between the position of the opening and the position of the safety device (step S6).

[0049] If the correct protective measures have been taken (S6: YES), the notification processing unit 11e of the processing unit 11 notifies the management device 5 that an opening exists in the floor in the clean room 100 and that the correct protective measures have been taken for this opening (step S7). Note that in this flowchart, the notification processing unit 11e issues a notification when the correct protective measures for the opening have been taken, but this is not limited to this. If the correct protective measures for the opening have been taken, the state is safe, and if notification is unnecessary, the notification processing unit 11e may be configured not to issue a notification in this case.

[0050] Furthermore, if there is no safety device (S5: NO) or if the correct protective measures have not been taken (S6: NO), the notification processing unit 11e notifies the management device 5 that an opening exists in the floor of the clean room 100 and that the correct protective measures for this opening have not been taken (step S8). After making the notification in step S7 or S8, the notification processing unit 11e transmits the data of the captured image acquired in step S1 to the management device 5 (step S9).

[0051] Next, the human detection unit 11d of the processing unit 11 inputs the captured image data acquired in step S1 to the human detection model 22 stored in the learning model storage unit 12b (step S10). The human detection unit 11d acquires a human detection result output by the human detection model 22 in response to the input in step S10 (step S11). The human detection unit 11d determines whether a person is captured in the captured image from the human detection result acquired in step S11 (step S12).

[0052] If a person is captured in the captured image (S12: YES), the person detection unit 11d compares the position of the person with the position of the opening and determines whether or not a person is approaching the opening based on whether or not the person is present within a predetermined distance from the opening (step S13). Note that in this flowchart, the person detection unit 11d determines whether or not a person is approaching the opening by determining whether or not a person is present within a predetermined distance from the opening, but this is not limited to this. The person detection unit 11d may also determine, for example, the person's orientation or the direction of movement to determine whether or not a person is approaching the opening.

[0053] If a person is approaching the opening (S13: YES), the notification processing unit 11e outputs a control signal from the interface unit 14 to the LED lamp 4 to turn on the LED lamp 4 (step S14), and the process returns to step S1. If no person is captured in the captured image (S12: NO) or if no person is approaching the opening (S13: NO), the processing unit 11 returns to step S1.

[0054] 7 is a flowchart showing an example of the procedure of the opening detection process performed by the management device 5 according to the first embodiment. The information acquisition unit 51a of the processing unit 51 of the management device 5 according to the first embodiment determines whether or not the communication unit 53 has received a notification that an opening has been detected from any of the information processing devices 1 (step S21). If no notification has been received (S21: NO), the information acquisition unit 51a waits until a notification is received from any of the information processing devices 1. If a notification has been received (S21: YES), the information acquisition unit 51a stores various information included in the notification from the information processing device 1 and information received together with the notification, such as the presence or absence of protective measures and captured images, in the safety management DB 52b (step S22).

[0055] Next, the notification processing unit 51b of the processing unit 51 determines whether the content of the notification indicates that an opening exists and that no protective measures are taken, based on the information accompanying the notification from the information processing device 1 (step S23). If the content of the notification indicates that an opening exists and that no protective measures are taken (S23: YES), the notification processing unit 51b sends a message notifying one or more terminal devices 7 of the existence of an opening via the communication unit 53 (step S24), and ends the processing. If the content of the notification does not indicate that an opening exists and that no protective measures are taken (S23: NO), the notification processing unit 51b ends the processing without sending the notification message.

[0056] <Summary> In the information processing system according to the first embodiment configured as described above, the information processing device 1 acquires images captured by the camera 3 of the interior of a room such as the clean room 100, and detects an opening in the floor of the clean room 100 based on the acquired images. As a result, the information processing system is expected to realize safety monitoring and the like within the clean room 100 based on the images captured by the camera 3.

[0057] Furthermore, in the information processing system according to the first embodiment, the information processing device 1 determines, based on the image captured by the camera 3, whether or not a protective measure using a safety device such as a traffic cone, a safety bar, or a safety fence has been taken for an opening in the floor of the clean room 100. As a result, when an opening exists in the floor of the clean room 100, the information processing system can be expected to determine whether or not the opening is dangerous based on the presence or absence of a protective measure.

[0058] Furthermore, in the information processing system according to the first embodiment, the information processing device 1 notifies the workers in the clean room 100 by sending a message to the terminal device 7 carried by the workers, for example, in the clean room 100 via the management device 5 based on the opening detection result and the determination result of whether or not a protective measure is taken. In this way, the information processing system notifies the workers, for example, in the clean room 100 of the presence of a dangerous opening, which is expected to reduce accidents, etc.

[0059] Alternatively, the information processing device 1 may directly notify the terminal device 7. In this case, the notification can be sent to the worker more quickly. Whether the notification to the terminal device 7 is sent from the management device 5 or from the information processing device 1 may be determined appropriately from the viewpoint of the size of the clean room 100, optimization of the information system, etc.

[0060] Furthermore, in the information processing system according to the first embodiment, the information processing device 1 detects people based on images captured by the camera 3, and when it detects a person approaching an opening, when it detects a person facing an opening, or when it detects a person approaching an opening, it issues a notification by, for example, lighting up the LED lamp 4. In this way, the information processing system is expected to make people approaching a dangerous opening aware of the existence of the opening and to help them avoid danger.

[0061] The information processing system according to the first embodiment detects an opening in the floor of a clean room 100 in which a plurality of floor tiles are laid. The floor of the clean room 100, in which floor tiles are laid in rows and columns, has a space below it for accommodating various pieces of equipment, and an opening may occur when a floor tile is removed during inspection or maintenance of the equipment. The information processing system according to the first embodiment is suitable for safety measures in a clean room 100 or the like in which openings are likely to occur in such a floor.

[0062] Furthermore, in the information processing system according to the first embodiment, the camera 3 that takes images based on infrared light is used to take images of the inside of the clean room 100. As a result, the information processing system is expected to be able to accurately detect openings even when floor tiles made of a translucent material such as glass are used on the floor of the clean room 100.

[0063] In this embodiment, a plurality of cameras 3 and information processing devices 1 are installed inside the clean room 100, but this is not limiting, and at least one camera 3 and one information processing device 1 may be installed. The information processing device 1 may also be installed outside the clean room 100. In this embodiment, one camera 3 is connected to one information processing device 1, but this is not limiting, and a plurality of cameras 3 may be connected to one information processing device 1.

[0064] Furthermore, in the first embodiment, the camera 3 is configured to capture images in both the visible light range and the infrared range, but this is not limited thereto, and the camera 3 may capture images only in the visible light range without capturing images in the infrared range. Alternatively, a camera that captures images in the visible light range and a camera that captures images in the infrared range may be installed separately, and the information processing device 1 may acquire images captured by both cameras to detect openings. In the information processing system according to the first embodiment, some or all of the processing performed by the information processing device 1 may be performed by the management device 5, and conversely, some or all of the processing performed by the management device 5 may be performed by the information processing device 1.

[0065] [Embodiment 2] <System Overview> Figure 8 is a schematic diagram illustrating an overview of an information processing system according to embodiment 2. The information processing system according to embodiment 2 is a system that prevents an operator from mistakenly selecting the substrate processing apparatus 101 to be used for work among multiple substrate processing apparatuses 101 installed in a clean room 100. The information processing system according to embodiment 2 can prevent an operator from mistakenly performing work such as maintenance on an operating substrate processing apparatus 101 that is performing substrate processing such as etching on substrates such as wafers. To this end, the information processing system according to embodiment 2 determines the operating status of each substrate processing apparatus 101 based on schedule information input by an operator or the like, and issues a notification when it detects that an operator or the like has approached an operating substrate processing apparatus 101 based on images captured by multiple cameras 3 installed in the clean room 100.

[0066] Similar to the information processing system according to the first embodiment, the information processing system according to the second embodiment is configured to include a plurality of cameras 3 arranged at appropriate locations within the clean room 100, a plurality of information processing devices 1 that control the cameras 3, and a management device 5 that manages information obtained from the plurality of information processing devices 1. However, in the second embodiment, one camera 3 is associated with one substrate processing apparatus 101. Each camera 3 is installed in or near the corresponding substrate processing apparatus 101, and by photographing from the substrate processing apparatus 101 toward its surroundings, it is possible to photograph, for example, a worker approaching the substrate processing apparatus 101 from the front. However, this installation position of the camera 3 is merely an example and is not limited thereto.

[0067] In the information processing system according to the second embodiment, a plurality of cameras 3 installed corresponding to a plurality of substrate processing apparatuses 101 capture images of the periphery of each substrate processing apparatus 101, and an information processing apparatus 1 that acquires the images captured by each camera 3 detects a worker or the like approaching the substrate processing apparatus 101, i.e., a worker or the like about to perform work on the substrate processing apparatus 101. In addition, in the information processing system according to the second embodiment, a management apparatus 5 manages schedule information related to the operation of a plurality of substrate processing apparatuses 101 installed in a clean room 100, and transmits information regarding whether each substrate processing apparatus 101 is operating or stopped from the management apparatus 5 to the information processing apparatus 1 connected to the corresponding camera 3. When each information processing apparatus 1 detects the approach of a worker or the like to an operating substrate processing apparatus 101, it notifies the management apparatus 5 and also notifies the worker or the like by light, sound, or the like.

[0068] In the information processing system according to the second embodiment, the plurality of cameras 3 installed in the clean room 100 may be cameras that take images in the visible light range but do not take images in the infrared range.

[0069] In the information processing system according to the second embodiment, the management device 5 manages the schedules of each substrate processing apparatus 101 and accepts input of information such as work schedules from workers via the terminal device 7. The terminal device 7 may be, for example, a tablet terminal device or a smartphone. The terminal device 7 displays an input screen for the work schedule on a display unit, accepts information input from workers, and transmits the accepted information to the management device 5. The terminal device 7 may also transmit to the management device 5 an image of handwritten notes by the workers on a sheet (paper) for entering the work schedule, captured with an attached camera. The management device 5 may acquire information such as the work schedule by reading character strings written in the captured image of the sheet received from the terminal device 7 using a so-called OCR (Optical Character Recognition) function. The management device 5 manages schedules, such as the time periods during which each substrate processing apparatus 101 operates, based on the information acquired via the terminal device 7, and notifies each information processing apparatus 1 of the operating status of the corresponding substrate processing apparatus 101.

[0070] <Device Configuration> Fig. 9 is a block diagram showing an example configuration of the information processing device 1 according to embodiment 2. Like the information processing device 1 according to embodiment 1, the information processing device 1 according to embodiment 2 is configured to include a processing unit 11, a storage unit 12, a communication unit 13, and an interface unit 14. The hardware configuration of the information processing device 1 according to embodiment 2 is the same as that of the information processing device 1 according to embodiment 1, and therefore detailed description thereof will be omitted.

[0071] The learning model storage unit 12b of the storage unit 12 of the information processing device 1 according to the second embodiment stores information related to a human detection model 22 similar to that of the first embodiment and a posture estimation model for estimating a human posture. The posture estimation model is a learning model that has undergone machine learning in advance to receive an image of a person as input and output an estimation result of the person's posture. The estimation result output by the posture estimation model is, for example, information representing the head, waist, joints of the limbs, etc. of the person's body using a plurality of points, and the posture estimation model outputs coordinate information of these multiple points. Based on the coordinates of these multiple points, the information processing device 1 according to the second embodiment determines the direction of the face (gaze) of the person captured in the captured image, the direction in which the arms are extended, etc. The posture estimation model may utilize an existing learning model such as OpenPose. Since the posture estimation model is an existing technology, detailed description of the learning method, etc. will be omitted.

[0072] The storage unit 12 of the information processing device 1 according to the second embodiment is provided with an operation status information storage unit 12c that stores information regarding the operation status of the substrate processing device 101 associated with the information processing device 1 or the camera 3 connected thereto. In the present embodiment, the operation status of the substrate processing device 101 is either operating or stopped, but this is merely an example, and operation statuses other than these two may also be adopted. The operation status information storage unit 12c stores information indicating whether the corresponding substrate processing device 101 is operating or stopped.

[0073] In addition, in the information processing device 1 according to the second embodiment, the processing unit 11 reads and executes the program 12a stored in the memory unit 12, whereby the captured image acquisition unit 11a, the human detection unit 11d, the notification processing unit 11e, the posture estimation unit 11f, the operation status acquisition unit 11g, the device error determination unit 11h, etc. are realized in the processing unit 11 as software functional units.

[0074] The captured image acquisition unit 11a acquires data of images captured by the camera 3 via the interface unit 14 and temporarily stores the acquired captured images in the storage unit 12. The human detection unit 11d performs processing to detect people in the clean room 100 using the human detection model 22 stored in the learning model storage unit 12b, based on the captured images acquired by the captured image acquisition unit 11a from the camera 3. When the device error determination unit 11h, which will be described later, determines that a device error has occurred, the notification processing unit 11e transmits a message notifying the management device 5 of this, and also outputs a control signal from the interface unit 14 to turn on the LED lamp 4 to notify an operator, etc. Furthermore, when the information processing device 1 is configured to be able to communicate with a user's terminal device 7, the notification processing unit 11e may transmit a notification message to the terminal device 7.

[0075] When the human detection unit 11d detects a person, such as a worker, from a captured image, the posture estimation unit 11f performs a process of estimating the posture of the person. In the second embodiment, the posture estimation unit 11f estimates the posture of the person using a posture estimation model stored in the learning model storage unit 12b. Based on the person detection result by the human detection unit 11d, the posture estimation unit 11f extracts an image area in which a person is captured from the captured image, inputs data of the extracted image area to the posture estimation model, and obtains a posture estimation result output by the posture estimation model. Note that if the posture estimation model is capable of estimating the postures of multiple people from a single captured image, the posture estimation unit 11f may directly input the captured image of the camera 3 to the posture estimation model and obtain a posture estimation result of one or more people captured in the captured image, rather than extracting image areas of people from the captured image based on the person detection result and inputting them to the posture estimation model. In the second embodiment, the posture estimation unit 11f determines the gaze and arm direction of the detected person based on the posture estimation result obtained by the posture estimation model.

[0076] The operating status acquisition unit 11g performs processing to acquire the operating status of the substrate processing apparatus 101 associated with the information processing apparatus 1 or the camera 3 connected thereto. In this embodiment, the management device 5 manages information relating to the operation schedule of each substrate processing apparatus 101, and based on this schedule information, the management device 5 notifies each information processing apparatus 1 of information relating to the operating status of the corresponding substrate processing apparatus 101. The operating status acquisition unit 11g of the information processing apparatus 1 communicates with the management device 5 via the communication unit 13 and acquires the operating status of the corresponding substrate processing apparatus 101 by receiving the information relating to the operating status transmitted by the management device 5. The operating status acquisition unit 11g stores the acquired information relating to the operating status in the operating status information storage unit 12c.

[0077] The apparatus error determination unit 11h performs processing to determine whether or not the detected person is attempting to perform maintenance or other work on the wrong substrate processing apparatus 101, based on the person detection result by the person detection unit 11d, the person posture estimation result by the posture estimation unit 11f, and the operating state of the substrate processing apparatus 101 acquired by the operating state acquisition unit 11g. In this embodiment, the apparatus error determination unit 11h determines that an apparatus error has occurred when a worker or the like in the clean room 100 is attempting to perform some work on the substrate processing apparatus 101 that is in operation.

[0078] In this embodiment, for example, when a person, such as a worker, detected from an image captured by the camera 3 continues to look (stare) at the operating substrate processing apparatus 101 for a predetermined time or more (e.g., 10 seconds or more), the apparatus error determination unit 11h determines that the person is about to perform some kind of work on the substrate processing apparatus 101 and determines that an apparatus error has occurred. Furthermore, for example, when a detected person moves their hand closer to the operating substrate processing apparatus 101, the apparatus error determination unit 11h determines that the person is about to perform some kind of work on the substrate processing apparatus 101 and determines that an apparatus error has occurred. In this embodiment, since the camera 3 is installed in each substrate processing apparatus 101, for example, when a person is photographed extending their hand toward the camera 3 or gazing at the camera 3, it can be determined that the person has performed an apparatus error. To implement the above-described determination method, the apparatus error determination unit 11h tracks the detected person not only in images captured at a single point in time but also in images captured at multiple consecutive points in time, and determines whether the person's gaze continues for a predetermined time or more, whether the person is moving their hand closer or farther away, etc. Furthermore, the apparatus error determination unit 11h may perform the above determination process when the corresponding substrate processing apparatus 101 is in operation, but may not perform the determination process when the corresponding substrate processing apparatus 101 is stopped.

[0079] 10 is a block diagram showing an example of the configuration of the management device 5 according to embodiment 2. Like the management device 5 according to embodiment 1, the management device 5 according to embodiment 2 is configured to include a processing unit 51, a storage unit 52, a communication unit 53, a display unit 54, and an operation unit 55. The hardware configuration of the management device 5 according to embodiment 2 is the same as that of the management device 5 according to embodiment 1, and therefore detailed description thereof will be omitted.

[0080] The storage unit 52 of the management device 5 according to the second embodiment is provided with a schedule DB 52c that stores schedule information related to the operation of a plurality of substrate processing apparatuses 101 installed in the clean room 100. The schedule DB 52c stores information related to the date and time when the substrate processing apparatus 101 is operating or stopped, for example, in association with the identification information of each substrate processing apparatus 101. The schedule DB 52c may further store various information, such as the content of processing performed in the substrate processing apparatus 101 and the person who inputted the information, in association with each other.

[0081] In addition, in the management device 5 according to the second embodiment, the processing unit 51 reads and executes a program 52a stored in the storage unit 52, thereby realizing an information acquisition unit 51a, a notification processing unit 51b, a schedule information management unit 51c, a sheet reading unit 51d, and the like as software functional units in the processing unit 51. The information acquisition unit 51a acquires information such as notifications regarding an incorrect device and data on captured images from the information processing device 1 via the communication unit 53, and stores the acquired information in the safety management DB 52b. When notified of an incorrect device by the information processing device 1, the notification processing unit 51b transmits a message notifying one or more terminal devices 7 of the detection of an incorrect device. Note that, if it is possible to distinguish the terminal device 7 held by the person determined to have the incorrect device, the notification processing unit 51b may transmit the message only to this terminal device 7.

[0082] The schedule information management unit 51c manages schedule information relating to the operation of a plurality of substrate processing apparatuses 101 installed in the clean room 100. The schedule information management unit 51c communicates with a terminal device 7 used by a user such as an operator via the communication unit 53, and acquires information relating to work performed in the substrate processing apparatus 101 that is input by the user via the terminal device 7. Based on the information acquired from the terminal device 7, the schedule information management unit 51c extracts or calculates the operating and stopped times of the substrate processing apparatus 101, and stores the information in the schedule DB 52c in association with identification information of the substrate processing apparatus 101, etc.

[0083] In this embodiment, a user such as an operator can handwrite information about work performed in the substrate processing apparatus 101 on a predetermined sheet, and the sheet can be photographed by a camera of the terminal device 7 and the photographed image can be transmitted to the management device 5. In this case, the schedule information management unit 51c of the management device 5 can provide the photographed image obtained from the terminal device 7 to a sheet reading unit 51d (described later), and the sheet reading unit 51d can obtain information read from the sheet. In this embodiment, the management device 5 reads characters and the like written on the sheet, but this is not limiting, and the sheet can also be read by the terminal device 7, and the information read by the terminal device 7 can be transmitted to the management device 5.

[0084] Furthermore, the schedule information management unit 51c determines the operating status of each substrate processing apparatus 101 in the clean room 100 based on the information stored in the schedule DB 52c, and performs processing to notify the information processing apparatus 1 corresponding to each substrate processing apparatus 101 of the operating status. The schedule information management unit 51c may notify the operating status at a predetermined cycle, for example, once every few seconds or once every few minutes, or may notify the operating status when the operating status changes, or may notify the operating status in response to a request from the information processing apparatus 1, for example.

[0085] The sheet reading unit 51d reads characters written on a captured image of a sheet acquired by the schedule information management unit 51c from the terminal device 7. The sheet reading unit 51d reads characters written on a sheet using a so-called OCR function. Reading characters using OCR may be performed, for example, by extracting the characteristics of the written characters and comparing them with a dictionary prepared in advance, or may be performed using a learning model that has undergone machine learning to infer characters from images of handwritten characters. Reading handwritten characters using OCR is an existing technology, so a detailed description will be omitted.

[0086] 11 is a flowchart showing an example of the procedure of schedule management processing performed by management device 5 according to embodiment 2. In the information processing system according to embodiment 2, when substrate processing or the like is performed in substrate processing apparatus 101 in clean room 100, an operator inputs information such as a work schedule in advance using terminal device 7. The information input by terminal device 7 is transmitted to management device 5. In addition, an operator can input information by handwriting information such as a work schedule on a predetermined sheet and photographing the handwritten information using terminal device 7, and terminal device 7 transmits the photographed image of the sheet to management device 5.

[0087] The schedule information management unit 51c of the processing unit 51 of the management device 5 according to the second embodiment determines whether information regarding the work schedule in the clean room 100 has been received from the terminal device 7 that has accepted input from the worker (step S41). If information regarding the work schedule has been received (S41: YES), the schedule information management unit 51c determines whether the received information is a photographed image of a sheet (step S42). If the received information is a photographed image of a sheet (S42: YES), the sheet reading unit 51d of the processing unit 51 performs OCR processing on the received photographed image of the sheet to read the information written on the sheet (step S43), and the process proceeds to step S44. If the received information is not a photographed image of a sheet (S42: NO), the processing unit 51 proceeds to step S44. The schedule information management unit 51c stores the information received from the terminal device 7 or the information read from the received photographed image in the schedule DB 52c (step S44), and returns the process to step S41.

[0088] If no information regarding the work schedule has been received from the terminal device 7 (S41: NO), the schedule information management unit 51c determines whether it is time to notify the information processing device 1 (step S45). In this example, the management device 5 notifies the information processing device 1 corresponding to each substrate processing device 101 of the operating status of the substrate processing devices 101 in the clean room 100 at an appropriate interval, for example, once every few seconds or once every few minutes. The schedule information management unit 51c can determine whether it is time to notify based on whether a predetermined interval has elapsed since the previous notification. If it is not time to notify (S45: NO), the schedule information management unit 51c returns the process to step S41.

[0089] If the timing for notification has arrived (S45: YES), the schedule information management unit 51c reads out the schedule information of each substrate processing apparatus 101 stored in the schedule DB 52c (step S46). The schedule information management unit 51c determines the operating status of each substrate processing apparatus 101 based on the read schedule information (step S47). For example, if a schedule is set for a substrate processing apparatus 101 to operate from midnight to 2:00 and the current time is 1:00, the schedule information management unit 51c can determine that the substrate processing apparatus 101 is operating. The schedule information management unit 51c notifies the operating status of each substrate processing apparatus 101 by transmitting the determined operating status of each substrate processing apparatus 101 to the information processing apparatus 1 corresponding to each substrate processing apparatus 101 (step S48), and the process returns to step S41.

[0090] 12 and 13 are flowcharts showing an example of the procedure of the device error prevention process performed by the information processing device 1 according to the second embodiment. In this example, the information processing device 1 receives the operating status notified from the management device 5 in step S48 of the flowchart shown in FIG. 11 and stores the received information in the operating status information storage unit 12c. The operating status acquisition unit 11g of the processing unit 11 of the information processing device 1 according to the second embodiment reads the operating status of the substrate processing device 101 stored in the operating status information storage unit 12c (step S61). The operating status acquisition unit 11g determines whether the operating status of the corresponding substrate processing device 101 is operating (step S62). If the operating status of the substrate processing device 101 is not operating (S62: NO), i.e., if it is stopped, the operating status acquisition unit 11g returns the process to step S61.

[0091] If the substrate processing apparatus 101 is in operation (S62: YES), the captured image acquisition unit 11a of the processing unit 11 acquires the captured image from the camera 3 by exchanging information with the camera 3 via the interface unit 14 (step S63). The human detection unit 11d of the processing unit 11 inputs the captured image data acquired in step S63 to the human detection model 22 stored in the learning model storage unit 12b (step S64). The human detection unit 11d acquires the human detection result output by the human detection model 22 in response to the input in step S64 (step S65). The human detection unit 11d determines whether a human is captured in the captured image based on the human detection result acquired in step S65 (step S66). If a human is not captured in the captured image (S66: NO), the human detection unit 11d returns the process to step S61.

[0092] If a person is captured in the captured image (S66: YES), the posture estimation unit 11f of the processing unit 11 extracts an image area in which the detected person is captured from the captured image and inputs data of the extracted image area of ​​the person to the posture estimation model stored in the learning model storage unit 12b (step S67).The posture estimation unit 11f then obtains a posture estimation result output by the posture estimation model in response to the input in step S67 (step S68).

[0093] In this flowchart, posture estimation unit 11f extracts an image area in which a person is captured from the captured image and inputs the extracted image area to the posture estimation model, but this is not limited to this. If the posture estimation model is capable of estimating the postures of multiple people from a single captured image, posture estimation unit 11f can directly input the image captured by camera 3 to the posture estimation model and obtain the estimation results of the postures of one or more people output by the posture estimation model.

[0094] The apparatus error determination unit 11h of the processing unit 11 determines whether the person in the captured image is in a posture with their hand outstretched toward the substrate processing apparatus 101 based on the posture estimation result acquired in step S68 (step S69). If the person in the captured image is not in a posture with their hand outstretched toward the substrate processing apparatus 101 (step S69), the apparatus error determination unit 11h determines whether the person in the captured image is in a posture with their hand staring at the substrate processing apparatus 101 (step S70). If the person is not in a posture with their hand outstretched toward the substrate processing apparatus 101 (step S70), the apparatus error determination unit 11h returns the process to step S61.

[0095] If the person in the photographed image is in a position where they are extending their hand toward the substrate processing apparatus 101 (S69: YES), or if the person in the photographed image is in a position where they are staring at the substrate processing apparatus 101 (S70: YES), the notification processing unit 11e of the processing unit 11 notifies the management device 5 that an incorrect substrate processing apparatus 101 has been selected (step S71). The notification processing unit 11e also transmits the data of the photographed image acquired in step S63 to the management device 5 (step S72). The notification processing unit 11e then outputs a control signal from the interface unit 14 to the LED lamp 4 to turn on the LED lamp 4 (step S73), and the process returns to step S61.

[0096] 14 is a flowchart showing an example of the procedure of the device error prevention process performed by the management device 5 according to the second embodiment. The information acquisition unit 51a of the processing unit 51 of the management device 5 according to the second embodiment determines whether or not a notification indicating a device error has occurred has been received from any of the information processing devices 1 via the communication unit 53 (step S81). If no notification has been received (S81: NO), the information acquisition unit 51a waits until a notification has been received from any of the information processing devices 1. If a notification has been received (S81: YES), the information acquisition unit 51a stores information included in the notification from the information processing device 1 and information received with the notification, such as various information such as captured images, in the safety management DB 52b (step S82). Next, the notification processing unit 51b of the processing unit 51 sends a message notifying one or more terminal devices 7 of the device error via the communication unit 53 (step S83), and the process ends.

[0097] <Summary> In the information processing system according to the second embodiment configured as described above, the information processing device 1 acquires images captured by the camera 3 of the interior of a room such as the clean room 100, and estimates the posture of a person in the clean room 100 based on the acquired images. The information processing device 1 also acquires the operating status of devices such as the substrate processing device 101 installed in the clean room 100. The information processing device 1 determines whether there is an error in the substrate processing device 101 on which the person is working, based on the estimated posture and the acquired operating status. As a result, the information processing system is expected to realize safety monitoring and the like based on the images captured by the camera 3.

[0098] Furthermore, in the information processing system according to the second embodiment, when the information processing device 1 estimates that a person is in a posture of reaching out to or gazing at the substrate processing device 101 while the substrate processing device 101 is in operation, the information processing device 1 determines that the substrate processing device 101 that the person is working on is incorrect. This makes it possible for the information processing system to determine that the wrong device has been selected, such as when a worker or other person touches the substrate processing device 101 that is performing substrate processing such as etching in order to perform maintenance or other work.

[0099] In the information processing system according to the second embodiment, the management device 5 acquires schedule information relating to the operation of the substrate processing apparatus 101 via the terminal device 7 and stores the acquired schedule information in the operation status information storage unit 12c. The management device 5 determines the operation status of the substrate processing apparatus 101 based on the schedule information stored in the operation status information storage unit 12c. This allows the information processing system to determine the operation status of the substrate processing apparatus 101 and determine whether an apparatus is being used incorrectly, even when the operation status of the substrate processing apparatus 101 cannot be directly obtained.

[0100] In the information processing system according to the second embodiment, the management device 5 acquires schedule information by reading information from a sheet on which information about work in the substrate processing apparatus 101 is written. In the information processing system according to the second embodiment, the management device 5 acquires schedule information by accepting input of information about work in the substrate processing apparatus 101 from a worker or the like via the terminal device 7. As a result, the information processing system is expected to make it easier for workers or the like to input information.

[0101] In the information processing system according to the second embodiment, when the information processing device 1 determines that the substrate processing device 101 to be worked on is the wrong device, it notifies the user by turning on the LED lamp 4 and sending a message to the management device 5. This enables the information processing system to notify the worker in the clean room 100 of the wrong device, and is expected to prevent danger due to the wrong device before it occurs.

[0102] In the information processing system according to the second embodiment, when the information processing device 1 determines that the substrate processing device 101 to be worked on is an incorrect device, the photographed image used for the determination is sent to the management device 5, and the photographed image is stored in the safety management DB 52b. This allows the information processing system to save the state of the interior of the clean room 100 when the incorrect device occurs as a photographed image, which is expected to contribute to safety measures and the like within the clean room 100.

[0103] In the second embodiment, the operating status of the substrate processing apparatus 101 is determined based on schedule information managed by the management device 5, but this is not limiting. For example, the information processing apparatus 1 may exchange information with the substrate processing apparatus 101 and obtain the operating status directly from the substrate processing apparatus 101. In the information processing system according to the second embodiment, the camera 3 is installed on or near the substrate processing apparatus 101, but the location of the camera 3 is not limited to this and the camera 3 may be installed in any location. In the information processing system according to the second embodiment, some or all of the processing performed by the information processing apparatus 1 may be performed by the management device 5, and conversely, some or all of the processing performed by the management device 5 may be performed by the information processing apparatus 1.

[0104] The information processing system according to the second embodiment may further perform the opening detection process of the information processing system according to the first embodiment. That is, the information processing system may be configured to perform both the opening detection process described in the first embodiment and the device error prevention process described in the second embodiment.

[0105] The other configurations of the information processing system according to the second embodiment are the same as those of the information processing system according to the first embodiment, so the same reference numerals are used for the same parts and detailed description thereof will be omitted.

[0106] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0107] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, the claims use a format in which a claim references two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used.

[0108] REFERENCE SIGNS LIST 1 Information processing device (computer) 3 Camera 4 LED lamp 5 Management device 7 Terminal device 11 Processing unit 11a Photographed image acquisition unit 11b Opening detection unit 11c Protective measure determination unit 11d Human detection unit 11e Notification processing unit 11f Posture estimation unit 11g Operation status acquisition unit 11h Device error determination unit 12 Memory unit 12a Program (computer program) 12b Learning model memory unit 12c Operation status information memory unit 13 Communication unit 14 Interface unit 21 Segmentation model 22 Human detection model 51 Processing unit 51a Information acquisition unit 51b Notification processing unit 51c Schedule information management unit 51d Sheet reading unit 52 Memory unit 52a Program 52b Safety management DB 53 Communication unit 54 Display unit 55 Operation unit 98, 99 Recording medium 100 Clean room 101 Substrate processing equipment (equipment)

Claims

1. An information processing method, comprising: an information processing device acquiring an image of a room captured by a camera; and detecting an opening in a floor of the room based on the acquired image.

2. The information processing method according to claim 1, further comprising determining whether or not a protective measure is taken for the opening based on the captured image.

3. The information processing method according to claim 2, further comprising the step of: issuing a notification based on the result of detection of the opening and the result of determination of the presence or absence of the protective measure.

4. The information processing method of claim 3, further comprising: detecting a person based on the captured image; and issuing a notification when a person is detected within a predetermined distance from the opening, when a person is detected facing the opening, or when a person is detected approaching the opening.

5. The information processing method according to claim 1, further comprising detecting an opening in a floor within a clean room in which a plurality of floor tiles are laid.

6. The information processing method according to claim 1, wherein the camera is a camera that takes pictures based on infrared rays.

7. The information processing method of claim 1, further comprising: estimating the posture of a person in the room based on the acquired captured image; acquiring the operating status of a device installed in the room; and determining whether or not the device being operated by the person is incorrect based on the estimated posture of the person and the acquired operating status of the device.

8. An information processing method in which an information processing device acquires an image captured by a camera inside a room, estimates the posture of a person in the room based on the acquired image, acquires the operating status of a device installed in the room, and determines whether or not there is an error in the device on which the person is working based on the estimated posture of the person and the acquired operating status of the device.

9. The information processing method according to claim 7 or 8, further comprising determining that the device being worked on by the person is incorrect when the operating state is a posture in which the person extends his / her hand toward a device in operation or the person stares at a device in operation.

10. An information processing method according to claim 7 or claim 8, wherein an information processing device acquires schedule information relating to the operation of the device, stores the acquired schedule information in a memory unit, and determines the operating status of the device based on the stored schedule information.

11. The information processing method according to claim 10, further comprising the steps of: reading information relating to work on said device from a sheet on which said information is written; and acquiring schedule information for said device based on said read information.

12. The information processing method according to claim 10, further comprising the steps of: receiving an input of information relating to work on said device via a terminal device; and acquiring schedule information for said device based on said received information.

13. The information processing method according to claim 7 or 8, further comprising the step of: issuing a notification when it is determined that the device on which the person is performing the work is incorrect.

14. The information processing method according to claim 7 or 8, further comprising storing the captured image used for the judgment in a memory unit when it is judged that the device on which the person is working is incorrect.

15. A computer program that causes a computer to execute a process of acquiring an image of a room captured by a camera, and detecting an opening in a floor of the room based on the acquired image.

16. A computer program that causes a computer to execute the following processes: acquire images taken by a camera inside a room; estimate the posture of a person in the room based on the acquired images; acquire the operating status of equipment installed in the room; and determine whether there is an error in the equipment on which the person is working based on the estimated posture of the person and the acquired operating status of the equipment.

17. An information processing device comprising a processing unit, the processing unit acquiring an image captured by a camera of a room, and detecting an opening in a floor of the room based on the acquired captured image.

18. An information processing device having a processing unit, which acquires an image captured by a camera inside a room, estimates a posture of a person in the room based on the acquired captured image, acquires an operating status of a device installed in the room, and determines an error in a device on which the person is working based on the estimated posture of the person and the acquired operating status of the device.

Images