Information processing method, computer program, and information processing device

Abstract

The invention provides an information processing method, a computer program, and an information processing device, which are expected to realize safe monitoring and the like based on images captured by a camera. In an information processing method according to the present embodiment, an information processing device acquires a captured image obtained by capturing an indoor image by a camera, and detects an opening of a floor in the indoor image on the basis of the acquired captured image. Furthermore, in the information processing method according to the present embodiment, an information processing device acquires a captured image in which a camera captures a room, detects a person in the room on the basis of the acquired captured image, estimates the posture of the detected person, acquires the operating state of a device provided in the room, and transmits the acquired operating state to the camera. Based on the estimated posture of the person and the acquired operating state of the device, it is determined that the person is erroneous as a device to be operated.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing method, a computer program, and an information processing apparatus. BACKGROUND

[0002] In Patent Literature 1, a monitoring system is proposed in which an authentication controller performs authentication processing based on authentication information received from an authentication terminal through electric field communication, an image recognition server recognizes a person included in a captured region of an image captured by a camera, and the person authenticated by the authentication controller and the person recognized by the image recognition server are collated based on position information of the authentication terminal and coordinate information of the captured region of the camera.

[0003] Patent Literature 1: Japanese Patent Application Publication No. 2012-8802 SUMMARY

[0004] The present disclosure provides an information processing method, a computer program, and an information processing apparatus that can achieve monitoring and the like based on a safe camera-captured image.

[0005] In an information processing method according to one embodiment, an information processing apparatus acquires a captured image obtained by a camera capturing an indoor space, detects an opening of a floor in the indoor space based on the acquired captured image, and detects a person in the indoor space based on the acquired captured image. In an information processing method according to another embodiment, an information processing apparatus acquires a captured image obtained by a camera capturing an indoor space, detects a person in the indoor space based on the acquired captured image, estimates a posture of the detected person, acquires a state of operation of an apparatus provided in the indoor space, and determines that the person has mistaken the apparatus as an operation target based on the estimated posture of the person and the acquired state of operation of the apparatus.

[0006] According to the present disclosure, it is possible to achieve monitoring and the like based on a safe camera-captured image. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a schematic view for explaining an outline of an information processing system according to Embodiment 1.

[0008] Figure 2 is a block diagram showing one configuration example of an information processing apparatus according to Embodiment 1.

[0009] Figure 3 is a schematic view showing one example of a learning model according to Embodiment 1.

[0010] Figure 4 is a block diagram showing one configuration example of a management apparatus according to Embodiment 1.

[0011] Figure 5This is a flowchart illustrating an example of the steps of the opening detection process performed by the information processing apparatus according to Embodiment 1.

[0012] Figure 6 This is a flowchart illustrating an example of the steps of the opening detection process performed by the information processing apparatus according to Embodiment 1.

[0013] Figure 7 This is a flowchart illustrating an example of the steps involved in the opening detection process performed by the management device according to Embodiment 1.

[0014] Figure 8 This is a schematic diagram illustrating the outline of the information processing system involved in Embodiment 2.

[0015] Figure 9 This is a block diagram illustrating a structural example of the information processing apparatus according to Embodiment 2.

[0016] Figure 10 This is a block diagram illustrating a structural example of the management device involved in Embodiment 2.

[0017] Figure 11 This is a flowchart illustrating an example of the schedule management process performed by the management device involved in Implementation 2.

[0018] Figure 12 This is a flowchart illustrating an example of the steps involved in the device error prevention process performed by the information processing apparatus according to Embodiment 2.

[0019] Figure 13 This is a flowchart illustrating an example of the steps involved in the device error prevention process performed by the information processing apparatus according to Embodiment 2.

[0020] Figure 14 This is a flowchart illustrating an example of the steps involved in the device error prevention process performed by the management device according to Embodiment 2. Detailed Implementation

[0021] The following description, with reference to the accompanying drawings, illustrates specific examples of the information processing system according to embodiments of the present disclosure. Furthermore, the present disclosure is not limited to these examples, as indicated by the claims, and is intended to include all equivalents and modifications within the scope of the claims.

[0022] [Implementation Method 1]

[0023] <System Overview>

[0024] Figure 1is a schematic view for explaining an outline of an information processing system according to Embodiment 1. The information processing system according to Embodiment 1 is, for example, a system that performs security monitoring in a clean room 100 provided with a plurality of substrate processing apparatuses 101. The information processing system is configured to include a plurality of cameras 3 disposed at appropriate positions in the clean room 100, a plurality of information processing apparatuses 1 that control the respective cameras 3, and a management apparatus 5 that manages information obtained from the plurality of information processing apparatuses 1.

[0025] The floor of the clean room 100 is laid with, for example, substantially rectangular floor tiles in a lattice pattern. Below the floor laid with the plurality of floor tiles, there are sometimes, for example, spaces that house various equipment, or, for example, cases where there is a passage to a lower floor. The plurality of floor tiles laid are removed one or several tiles at a time, for example, when checking or maintaining the equipment below, and the like. The position where the floor tiles are removed forms an opening in the floor surface, and a safety countermeasure is required to prevent a worker or the like in the clean room 100 from falling from the opening.

[0026] The information processing system according to Embodiment 1 captures the floor of the clean room 100 by the plurality of cameras 3 disposed at appropriate positions in the clean room 100, and each information processing apparatus 1 detects an opening in the floor based on a captured image captured by each camera 3. The information processing apparatus 1, when an opening is detected based on the captured image of the floor captured by the camera 3, for example, issues an alarm based on light or sound, or the like. In addition, the information processing apparatus 1, when an opening is detected, for example, notifies the management apparatus 5 of the gist of the detection of the opening, and transmits a captured image in which the opening is detected to the management apparatus 5. The management apparatus 5 notified of the detection of the opening from the information processing apparatus 1, for example, notifies a terminal apparatus or the like held by a worker in the clean room 100 of information such as the position of the opening. In addition, the management apparatus 5 acquires the captured image at the time of the detection of the opening from the information processing apparatus 1, and stores the captured image in a database together with information such as the date and time and the place of the opening, for example.

[0027] In addition, in Embodiment 1, the camera 3 disposed in the clean room 100 is a camera that can simultaneously perform capturing in the visible light region and capturing in the infrared region, and is a so-called RGB-IR camera. Therefore, the data of the captured image obtained by the capturing of the camera 3 includes data of the colors of RGB and data indicating the intensity of infrared light. Among the plurality of floor tiles laid in the floor of the clean room 100, in order to improve the visibility of the lower portion, sometimes a portion thereof uses a floor tile made of glass or synthetic resin or the like that has light transmittance. In a case where a floor tile that is not light-transmissive and a floor tile that is light-transmissive are mixed together, it is possible to erroneously detect the floor tile that is light-transmissive as an opening, based on only the captured image of the visible light region of the camera. Therefore, in the information processing system according to Embodiment 1, by using the captured image in the infrared region in addition to the captured image in the visible light region, it is possible to prevent the floor tile that is light-transmissive from being erroneously detected as an opening.

[0028] In addition, even in a case where an opening is formed in the floor of the clean room 100, notification to a worker or the like in the clean room 100 is not required if sufficient protection measures are taken. In the information processing system according to Embodiment 1, even in a case where an opening is detected in the floor of the clean room 100, notification is not performed if, for example, it is determined that sufficient protection measures are taken by providing traffic cones, safety bars, or safety fences or the like around the opening. The information processing device 1 detects safety appliances such as traffic cones or safety fences or the like from the captured image of the camera 3. For an opening detected from the captured image, the information processing device 1 determines whether or not a prescribed condition is satisfied, for example, whether or not a safety appliance such as a traffic cone is present within a prescribed range, whether or not a safety appliance is present in a prescribed number or more, or whether or not a plurality of safety appliances are provided so as to surround the opening. For an opening detected from the captured image, the information processing device 1 determines that the safety appliance is configured so as to satisfy the prescribed condition if it is determined that the safety appliance satisfies the prescribed condition. In a case where it is determined that the safety appliance is not present in the captured image of the detected opening or that the safety appliance is not configured so as to satisfy the prescribed condition, the information processing device 1 determines that protection measures are not taken for the opening.

[0029] In addition, the information processing system can detect an opening in the floor of the clean room 100 and perform notification in a case where a worker or the like approaches the opening. In this case, the information processing device 1 detects a person from the captured image of the camera 3, for example. The information processing device 1 performs notification to the management device 5 and issues an alarm based on light or sound or the like, for example, in a case where an opening is detected from the captured image and a person is detected, or in a case where a person is detected within a prescribed distance from the detected opening or the like. In addition, for example, the information processing device 1 can perform notification to the management device 5 only in a case where an opening is detected from the captured image, and perform notification to the management device 5 and an alarm based on light or the like simultaneously in a case where both an opening and a person are detected from the captured image.

[0030] <Device Configuration>

[0031] Figure 2 is a block diagram that shows one configuration example of the information processing device 1 according to Embodiment 1. The information processing device 1 according to Embodiment 1 is an information processing device that is referred to as an edge computer in the field of IoT (Internet of Things), for example. The information processing device 1 is configured to include a processing unit 11, a storage unit 12, a communication unit 13, and an interface unit 14, and the like. Note that, in the present embodiment, one camera 3 is connected to one information processing device 1, but the present embodiment is not limited thereto, and a plurality of cameras 3 can be connected to one information processing device 1.

[0032] The processing section 11 is configured to use an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or a quantum processor, a ROM (Read Only Memory), and a RAM (Random Access Memory). The processing section 11 performs various processes such as a process of detecting an opening of a floor in the clean room 100 from a captured image of the camera 3 and a process of notifying a worker or the like of a detection result, by reading out and executing a program 12a stored in the storage section 12.

[0033] The storage section 12 is configured to use a large-capacity storage device such as a hard disk. The storage section 12 stores various programs executed by the processing section 11 and various data required for the processing of the processing section 11. In the present embodiment, the storage section 12 stores the program 12a executed by the processing section 11. In addition, a learning model storage section 12b that stores information related to one or a plurality of learning models that have been subjected to machine learning in advance is provided in the storage section 12.

[0034] In the present embodiment, the program (computer program, program product) 12a is provided in the form of being recorded on a recording medium 99 such as a memory card or an optical disc, and the information processing device 1 reads out the program 12a from the recording medium 99 and stores it in the storage section 12. However, the program 12a may, for example, be written to the storage section 12 at the manufacturing stage of the information processing device 1. In addition, for example, the program 12a can be acquired by the information processing device 1 through communication from a program distributed by a remote server device or the like. For example, the program 12a can be read out from the recording medium 99 by a writing device and written to the storage section 12 of the information processing device 1. The program 12a can be provided in the form of being distributed via a network, or in the form of being recorded on the recording medium 99.

[0035] The learning model storage section 12b stores information related to a learning model used for the above-described various processes performed by the information processing device 1 according to Embodiment 1 in advance. The information related to the learning model stored in the learning model storage section 12b may, for example, include information indicating what kind of structure the structure of the learning model is, and values of internal parameters of the learning model determined in advance through machine learning. Figure 3 is a schematic view indicating one example of a learning model according to Embodiment 1.

[0036] Figure 3The segmentation model 21 shown in the upper segment is a learning model that is obtained by performing machine learning in advance in a manner that accepts data of a captured image as input and outputs data indicating which of an opening, a safety appliance, or other than them each pixel of the captured image corresponds to. The segmentation model 21 employs, for example, a learning model that processes the structure of an image such as a CNN (Convolutional Neural Network). Further, in the information processing system related to Embodiment 1, the camera 3 that can capture a visible light region and an infrared light region is used, and thus the data of the captured image input to the segmentation model 21 is four-channel image data of R (red), G (green), B (blue), and IR (infrared light). The data of the segmentation result output by the segmentation model 21 is, for example, a two-dimensional array of the same size as the input captured image, and a value corresponding to the classification result is stored in each element of the array. The segmentation model 21 is generated in advance by performing a so-called supervised machine learning process using learning data (teacher data) in which four-channel image data of R, G, B, and IR and correct answer value data indicating which of an opening, a safety appliance, or other than them each pixel of the image data corresponds to are associated with each other. Further, supervised machine learning is a known technique, and thus detailed description thereof is omitted.

[0037] Figure 3 The person detection model 22 shown in the lower segment is a learning model that is obtained by performing machine learning in advance in a manner that accepts data of a captured image as input and detects a person appearing in the captured image. The person detection model 22 employs, for example, a learning model that processes the structure of an image such as a CNN. Further, the data of the captured image input to the person detection model 22 can be four-channel image data of R, G, B, and IR, but can also be three-channel image data of R, G, and B. The person detection result output by the person detection model 22 can be set as the coordinates of a so-called bounding box that is a rectangle that encloses a region in which a person appears in the captured image. The person detection model 22 is generated in advance by performing a so-called supervised machine learning process using learning data (teacher data) in which image data and correct answer value data indicating a region in which a person appears in the image data are associated with each other. Further, the person detection model 22 can utilize a known learning model that is completed by learning, such as YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector).

[0038] The communication section 13 communicates with the management device 5 via a wired or wireless network. The communication section 13 transmits data given from the processing section 11 to the management device 5, and receives data transmitted from the management device 5 and gives it to the processing section 11. In the present embodiment, the communication section 13 transmits data of a notification of a processing result based on the opening detection, data of a captured image of the camera 3, and the like to the management device 5.

[0039] The interface section 14 is connected to the camera 3 and the LED (Light Emitting Diode) lamp 4 and the like via a communication line or a signal line and the like, and performs transmission and reception of data or signals and the like with these devices. In the present embodiment, the interface section 14 acquires data of a captured image output from the camera 3 and gives it to the processing section 11. In addition, the interface section 14 outputs a control signal of on / off given from the processing section 11 to the LED lamp 4, and switches the lighting / off of the LED lamp 4. Further, in the present embodiment, the camera 3 is connected to the interface section 14 of the information processing device 1, but is not limited thereto. For example, in a case where the camera 3 has a communication function, the information processing device 1 can be a structure that communicates with the camera 3 via the communication section 13. In this case, the information processing device 1 can also acquire data of a captured image output from the camera 3 from the communication section 13 via a wired or wireless network. In addition, the same applies to the LED lamp 4.

[0040] Further, the storage section 12 can be an external storage device connected to the information processing device 1. In addition, the information processing device 1 can be a multi-computer configured to include a plurality of computers, or can be a virtual machine virtually constructed by software. In addition, the information processing device 1 is not limited to the above-described structure, and for example, can have a display section such as a liquid crystal display and an operation section that accepts an operation of a user and the like.

[0041] In addition, in the information processing device 1 related to the present embodiment, the processing section 11 reads out the program 12a stored in the storage section 12 and executes, and thereby the captured image acquisition section 11a, the opening detection section 11b, the protection measure determination section 11c, the person detection section 11d, and the notification processing section 11e and the like are realized as software functional sections in the processing section 11.

[0042] The captured image acquisition unit 11a acquires data of the captured image output from the camera 3 via the interface unit 14, and thereby performs processing of acquiring the captured image obtained by capturing the floor in the clean room 100. The camera 3, for example, captures and outputs the captured image at a frequency of several to several tens of times per second, and the captured image acquisition unit 11a acquires the captured image at the same frequency as the capturing of the camera 3, for example, and temporarily stores the acquired captured image in the storage unit 12. Further, in the case where the camera 3 has a communication function, the captured image acquisition unit 11a can also perform communication with the camera 3 via the communication unit 13, and acquire the captured image from the camera 3.

[0043] The opening detection unit 11b performs processing of detecting the opening present on the floor in the clean room 100, on the basis of the captured image acquired by the captured image acquisition unit 11a from the camera 3. In Embodiment 1, the opening detection unit 11b performs detection of the opening using the segmentation model 21 stored in the learning model storage unit 12b. The opening detection unit 11b inputs data (R, G, B, IR) of the captured image acquired by the captured image acquisition unit 11a to the segmentation model 21, and thereby acquires the segmentation result output from the segmentation model 21. The segmentation result contains flag information or the like that classifies which of the object represented by each pixel of the captured image corresponds to the opening, the safety appliance, or an object other than them, and the opening detection unit 11b detects the opening by determining the image region classified as representing the opening in the captured image.

[0044] The protection measure determination unit 11c performs processing of determining whether or not the opening present on the floor in the clean room 100 is subjected to the protection measure, on the basis of the captured image acquired by the captured image acquisition unit 11a from the camera 3. First, the protection measure determination unit 11c detects the safety appliance such as the traffic cone, the safety bar, or the safety fence represented in the captured image. In Embodiment 1, the protection measure determination unit 11c performs detection of the safety appliance using the segmentation model 21, but can also use the segmentation result acquired from the segmentation model 21 by the opening detection unit 11b on the basis of the input of the captured image. The segmentation result contains flag information or the like that classifies which of the object represented by each pixel of the captured image corresponds to the safety appliance, and the protection measure determination unit 11c determines the image region classified as representing the safety appliance. Further, the protection measure determination unit 11c can perform detection of the safety appliance without using the segmentation model 21, and using another learning model obtained by performing machine learning in a manner that detects the safety appliance on the basis of the captured image. In addition, part or all of the above-described processing performed by the protection measure determination unit 11c can be performed only in the case where the opening is detected by the opening detection unit 11b, or can be performed all the time regardless of the presence or absence of the opening.

[0045] Next, in a case where the safety instrument is not present in the captured image, the protection measure determination unit 11c determines that the protection measure is not performed. In a case where the safety instrument is present in the captured image, the protection measure determination unit 11c determines whether the protection measure is correct or not, based on a relationship between the position of the opening detected by the opening detection unit 11b and the position of the safety instrument detected by itself. In Embodiment 1, the protection measure determination unit 11c determines whether the correct protection measure is performed or not, for example, by determining whether two or more safety instruments are present within a prescribed distance from the opening. However, the determination criterion of success or failure of the protection measure is not limited to this, and can be appropriately set by a designer or a manager of the present system, or the like. As the condition of the correct protection measure, various conditions can be set, such as, for example, the safety instrument being disposed so as to surround the periphery of the opening, or, for example, the safety instrument being disposed so as to separate a passage within the clean room 100 from the opening, or, for example, two or more safety instruments being disposed. The protection measure determination unit 11c can determine whether the above-described conditions are satisfied, for example, based on coordinate information of the opening and the safety instrument in the captured image, by performing a prescribed arithmetic process, or, for example, can determine using a learning model obtained by performing machine learning in advance in a manner such that whether the correct protection measure is performed or not is determined based on the coordinate information.

[0046] The person detection unit 11d performs a process of detecting a person within the clean room 100, based on the captured image acquired by the captured image acquisition unit 11a from the camera 3. In Embodiment 1, the person detection unit 11d performs detection of a person using the person detection model 22 stored in the learning model storage unit 12b. The person detection unit 11d inputs data of the captured image acquired by the captured image acquisition unit 11a to the person detection model 22, and thereby acquires a person detection result output from the person detection model 22. The person detection result includes coordinate information of a bounding box that surrounds a person appearing in the captured image, and the person detection unit 11d detects a person by acquiring information of the bounding box.

[0047] Further, in a case where the opening is detected on the floor within the clean room 100 by the opening detection unit 11b, the person detection unit 11d compares the position of the opening and the position of the person detected by itself, for each person, and determines whether the person is present within a prescribed distance from the opening. For example, the information processing apparatus 1 can set a person detected within a prescribed distance from the opening as an object of notification described later.

[0048] Further, the person detection unit 1 Id detects the orientation of the face or the body of the detected person, and determines whether the person is facing the opening. For example, the information processing apparatus 1 can set a person who exists within a prescribed range from the opening and faces the opening as the object of the notification described later. Further, the person detection unit 1 Id determines the moving direction of the detected person, for example, based on a plurality of captured images acquired in time series from the camera 3. For example, the information processing apparatus 1 can set a person who moves in a manner approaching the opening as the object of the notification described later. Note that the condition for determining the presence or absence of a person as the object of the notification is not limited to the above, and can be an arbitrary condition, and can be appropriately set by a designer or a manager of the present system, or the like.

[0049] The notification processing unit 1 Ie performs a notification related to safety within the clean room 100 based on the detection result of the opening by the opening detection unit 1 Ib, the determination result of the protective measure by the protective measure determination unit 1 Ic, and the detection result of the person by the person detection unit 1 Id. In the embodiment 1, the notification processing unit 1 Ie can perform both of the notification to the management apparatus 5 and the notification of the lighting of the LED lamp 4. For example, in a case where the opening detection unit 1 Ib detects the opening on the floor within the clean room 100, the notification processing unit 1 Ie transmits a message notifying the gist to the management apparatus 5. At this time, the notification processing unit 1 Ie transmits the data of the captured image in which the opening is detected, and the determination result related to the presence or absence of the protective measure by the protective measure determination unit 1 Ic at this time to the management apparatus 5. Further, in a case where the person detection unit 1 Id detects a person existing within a prescribed distance from the opening, in a case where a person facing the opening is detected, or in a case where a person approaching the opening is detected, or the like, the notification processing unit 1 Ie lights the LED lamp 4 by outputting a control signal from the interface unit 14, and notifies the presence of the opening to the surrounding people. Note that the above-described distinction of the two kinds of notifications is one example, and is not limited thereto, and the conditions under which the notifications are performed can be appropriately set by a designer or a manager of the present system, or the like. Further, in a case where the information processing apparatus 1 is configured to be able to communicate with the terminal apparatus 7 of the user, the notification processing unit 1 Ie can also transmit a message of the notification to the terminal apparatus 7.

[0050] Figure 4 is a block diagram illustrating one configuration example of the management apparatus 5 according to the embodiment 1. The management apparatus 5 according to the embodiment 1 can be realized by, for example, installing a prescribed application program or the like in a general-purpose information processing apparatus such as a personal computer or a server computer. The management apparatus 5 is configured to include a processing unit 51, a storage unit 52, a communication unit 53, a display unit 54, an operation unit 55, and the like. Note that in the embodiment 1, the processing is described as being performed by one apparatus, but the processing of the management apparatus 5 can be performed by a plurality of apparatuses in a distributed manner.

[0051] The processing section 51 is configured using an arithmetic processing device such as a CPU, an MPU, a GPU, or a quantum processor, a ROM, a RAM, and the like. The processing section 51 performs various processes such as a process of collecting information from the plurality of information processing devices 1 and a process of notifying a terminal device 7 held by a worker or the like in the clean room 100, by reading out and executing the program 52a stored in the storage section 52.

[0052] The storage section 52 is configured using a large-capacity storage device such as a hard disk. The storage section 52 stores the program 52a executed by the processing section 51. In addition, a security management DB (database) 52b that stores and accumulates data of a security-related captured image and the like captured by the plurality of cameras 3 provided in the clean room 100 is provided in the storage section 52.

[0053] In the present embodiment, the program (computer program, program product) 52a is provided in a form of being recorded in a recording medium 98 such as a memory card or an optical disk, and the management device 5 reads out the program 52a from the recording medium 98 and stores it in the storage section 52. However, the program 52a may, for example, be written in the storage section 52 at the manufacturing stage of the management device 5. In addition, for example, the program 52a can be acquired by the management device 5 through communication from a program distributed by a remote server device or the like. For example, the program 52a can be read out by a writing device from the recording medium 98 and written in the storage section 52 of the management device 5. The program 52a can be provided in a form of being distributed via a network, or in a form of being recorded in the recording medium 98.

[0054] The security management DB 52b is a database for collecting and storing a security-related image from among captured images obtained by the plurality of cameras 3 provided in the clean room 100. In Embodiment 1, the management device 5 collects and stores, for example, a captured image related to an opening detected on the floor in the clean room 100 in the security management DB 52b. The security management DB 52b stores, for example, a captured image of the camera 3, various related information such as a date and time and a place at which the image was captured, identification information of the camera 3 that captured the image or the information processing device 1 that controls it, and presence or absence of a protective measure, in correspondence.

[0055] The communication section 53 performs transmission and reception of data between the plurality of information processing apparatuses 1 and the plurality of terminal apparatuses 7 via a wired or wireless network. In Embodiment 1, the communication section 53 receives a message of notification of opening detection based on the transmission from the information processing apparatus 1 and the data of the photographed image and the like accompanying this and gives them to the processing section 51. In addition, the communication section 53 transmits a message of notification of opening related to the opening given from the processing section 51 to the terminal apparatus 7. In addition, the communication section 53 communicates with the camera 3 having a communication function via a wired or wireless network, and can acquire the photographed image directly from the camera 3.

[0056] The display section 54 is configured using a liquid crystal display or the like and displays various images, characters, and the like based on the processing of the processing section 51. In Embodiment 1, the display section 54 displays, for example, the message of notification and the photographed image and the like received from the information processing apparatus 1. In addition, the management apparatus 5 can display various information including the photographed image stored in the security management DB 52b on the display section 54. In addition, the management apparatus 5 can also display the photographed image of the camera 3 obtained via the communication section 53 in real time on the display section 54.

[0057] The operation section 55 accepts the operation of the user and notifies the processing section 51 of the accepted operation. For example, the operation section 55 can be an input device such as a mouse and a keyboard, and these input devices can also be a structure that can be removed with respect to the management apparatus 5. In addition, for example, the operation section 55 accepts the operation of the user by a mechanical button or an input device such as a touch panel provided on the surface of the display section 54.

[0058] Further, the storage section 52 can be an external storage apparatus connected to the management apparatus 5. In addition, the management apparatus 5 can be a multi-computer configured to include a plurality of computers, or can be a virtual machine virtually constructed by software. In addition, the management apparatus 5 is not limited to the above-described structure, and, for example, can not have the display section 54 and the operation section 55 and the like.

[0059] In addition, in the management apparatus 5 related to the present embodiment, the processing section 51 reads out the program 52a stored in the storage section 52 and executes, so that the information acquisition section 51a and the notification processing section 51b and the like are realized as software functional sections in the processing section 51.

[0060] The information acquisition section 51a performs communication with the plurality of information processing apparatuses 1 through the communication section 53, and performs processing of acquiring information such as the notification of opening and the data of the photographed image from the information processing apparatus 1. The information acquisition section 51a stores the information of the date and time, the place, the identification information, and the presence or absence of the protective measure in correspondence with the acquired notification and the photographed image and the like in the security management DB 52b.

[0061] The notification processing section 51b performs processing of notifying the terminal device 7 held by the worker or the like in the clean room 100 of the gist that the floor in the clean room 100 detects an opening. The notification processing section 51b, in a case where the determination result of the gist that the opening is detected and that the protective measure for the detected opening is not properly performed is notified from the information processing device 1, transmits a message of the gist that the floor in the clean room 100 has an opening to one or a plurality of terminal devices 7. At this time, the notification processing section 51b can also transmit the message with information related to the place where the opening exists or the like included therein, to notify the worker or the like holding the terminal device 7 of the place where the opening exists.

[0062] <Opening detection processing>

[0063] Figure 5 And Figure 6 is a flowchart showing one example of steps of the opening detection processing performed by the information processing device 1 according to Embodiment 1. The captured image acquisition section 11a of the processing section 11 of the information processing device 1 according to Embodiment 1 acquires a captured image of the floor in the clean room 100 captured by the camera 3 by the exchange of information between the interface section 14 and the camera 3 (step S1). The opening detection section 11b of the processing section 11 inputs data of the captured image acquired in step S1 to the segmentation model 21 stored in the learning model storage section 12b (step S2). According to the input in step S2, the opening detection section 11b acquires a segmentation result output by the segmentation model 21 (step S3).

[0064] The segmentation result acquired in step S3 is flag information that classifies whether each pixel of the captured image represents an existing opening, a safety appliance, or any one other than them. The opening detection section 11b determines whether there is an opening in the floor in the clean room 100 based on whether a pixel classified as representing an existing opening is included in the captured image according to the segmentation result (step S4). In a case where there is no opening (S4: No), the processing section 11 returns the processing to step S1, acquires the next captured image, and performs the same processing.

[0065] In a case where there is an opening (S4: Yes), the protective measure determination section 11c of the processing section 11 determines whether there is a safety appliance such as a traffic cone, a safety bar, or a safety fence in the clean room 100 based on whether a pixel representing an existing safety appliance is included in the captured image according to the segmentation result acquired in step S3 (step S5). In a case where there is a safety appliance (S5: Yes), the protective measure determination section 11c determines whether a proper protective measure is performed for the opening based on, for example, the relationship between the position of the opening and the position of the safety appliance (step S6).

[0066] If proper protective measures have been taken (S6: Yes), the notification processing unit 11e will notify the management device 5 that an opening exists in the floor of the cleanroom 100 and that proper protective measures have been taken for the opening (step S7). Furthermore, in this flowchart, the notification processing unit 11e makes the notification when proper protective measures have been taken for the opening, but it is not limited to this. If proper protective measures have been taken for the opening, and the situation is safe, then notification is not required; in this case, the notification processing unit 11e may also be configured not to make a notification.

[0067] Additionally, if no safety equipment is available (S5: No), or if proper protective measures are not taken (S6: No), the notification processing unit 11e notifies the management device 5 that there is an opening in the floor of the cleanroom 100 and that the opening is not properly protected (step S8). After notification in step S7 or S8, the notification processing unit 11e sends the data of the captured image obtained in step S1 to the management device 5 (step S9).

[0068] Next, the person detection unit 11d of the processing unit 11 inputs the data of the captured image obtained in step S1 into the person detection model 22 stored in the learning model storage unit 12b (step S10). Based on the input in step S10, the person detection unit 11d obtains the person detection result output by the person detection model 22 (step S11). Based on the person detection result obtained in step S11, the person detection unit 11d determines whether a person is reflected in the captured image (step S12).

[0069] If a person is reflected in the captured image (S12: Yes), the person detection unit 11d compares the person's position with the position of the opening, and determines whether the person is approaching the opening based on whether a person exists within a predetermined distance from the opening (step S13). Furthermore, in this flowchart, the person detection unit 11d determines whether a person is approaching the opening by determining whether a person exists within a predetermined distance from the opening, but it is not limited to this. For example, the person detection unit 11d may further determine whether the person is approaching the opening by determining the person's orientation or direction of movement.

[0070] If a person approaches the opening (S13: Yes), the processing unit 11e is notified to output a control signal to the LED light 4 from the interface unit 14, thereby turning on the LED light 4 (step S14), and the processing returns to step S1. If no person is shown in the captured image (S12: No), or if no person approaches the opening (S13: No), the processing unit 11 returns the processing to step S1.

[0071] Figure 7is a flowchart showing one example of a step of the opening detection process performed by the management device 5 according to Embodiment 1. The information acquisition section 51a of the processing section 51 of the management device 5 according to Embodiment 1 determines whether or not a notification of the gist of detection of an opening is accepted from any one of the information processing devices 1 via the communication section 53 (step S21). In the case where the notification is not accepted (S21: No), the information acquisition section 51a stands by until the notification from any one of the information processing devices 1 is accepted. In the case where the notification is accepted (S21: Yes), the information acquisition section 51a stores the information included in the notification from the information processing device 1 and various information received together with the notification, such as the presence or absence of a protective measure and a captured image, in the security management DB 52b (step S22).

[0072] Next, the notification processing section 51b of the processing section 51 determines whether or not the content of the notification is an opening with no protective measure based on the information accompanying the notification from the information processing device 1 (step S23). In the case where the content of the notification is an opening with no protective measure (S23: Yes), the notification processing section 51b transmits a message notifying the presence of the opening to one or more of the terminal devices 7 via the communication section 53 (step S24), and ends the process. In the case where the content of the notification is not an opening with no protective measure (S23: No), the notification processing section 51b does not transmit the notification message, and ends the process.

[0073] <Summary>

[0074] In the information processing system according to Embodiment 1 having the above structure, the information processing device 1 acquires a captured image obtained by capturing the interior of the clean room 100 or the like by the camera 3, and detects an opening in the floor of the clean room 100 based on the acquired captured image. Thus, the information processing system can be expected to realize monitoring of the safety in the clean room 100 based on the captured image by the camera 3 or the like.

[0075] In addition, in the information processing system according to Embodiment 1, the information processing device 1 determines the presence or absence of a protective measure of a safety appliance, such as a traffic cone, a safety bar, or a safety fence, for the opening in the floor of the clean room 100 based on the captured image by the camera 3. Thus, the information processing system can be expected to determine whether or not the opening is dangerous based on the presence or absence of the protective measure in the case where the opening exists in the floor of the clean room 100.

[0076] In addition, in the information processing system according to Embodiment 1, the information processing device 1 performs notification based on message transmission to the terminal device 7 held by a worker or the like in the clean room 100, for example, via the management device 5 based on the detection result of the opening and the determination result of the presence or absence of the protective measure. Thus, the information processing system notifies the worker or the like in the clean room 100 of the dangerous opening, and can be expected to reduce accidents or the like.

[0077] Alternatively, the notification can also be made directly to the terminal device 7 from the information processing device 1. In this case, the notification can be made to the worker more quickly. Whether the notification is made to the terminal device 7 from the management device 5 or from the information processing device 1 can be appropriately decided from the viewpoint of the scale of the clean room 100 or optimization of the information system, or the like.

[0078] Further, in the information processing system according to Embodiment 1, the information processing device 1 detects a person based on the captured image of the camera 3, and makes a notification such as lighting based on the LED lamp 4, in a case where the person is detected to approach the opening, in a case where the person is detected to face the opening, or in a case where the person is detected to approach the opening, or the like. Thereby, the information processing system can be expected to make the person who approaches the dangerous opening recognize the existence of the opening, and avoid the danger.

[0079] Further, in the information processing system according to Embodiment 1, the opening of the floor in the clean room 100 in which a plurality of tiles are laid is detected. The floor of the clean room 100 in which the tiles are laid in a longitudinal and lateral arrangement is provided with a space below the floor to accommodate various equipment, and an opening is sometimes generated by removing the tiles at the time of inspection or maintenance of the equipment, or the like. The information processing system according to Embodiment 1 is applicable to safety measures in a clean room 100 or the like in which the opportunity of generation of an opening in the floor is high.

[0080] Further, in the information processing system according to Embodiment 1, the camera 3 that performs capturing based on infrared rays is used to perform capturing in the clean room 100. Thereby, even in a case where the tiles of the floor of the clean room 100 use a light-transmissive material such as glass, the information processing system can be expected to perform detection of the opening with high accuracy.

[0081] Further, in the present embodiment, a plurality of cameras 3 and information processing devices 1 are provided in the clean room 100, but the present application is not limited thereto, and the cameras 3 and the information processing devices 1 can be provided with at least one. Further, the information processing device 1 can be provided outside the clean room 100. Further, in the present embodiment, one information processing device 1 is connected to one camera 3, but the present application is not limited thereto, and a plurality of cameras 3 can be connected to one information processing device 1.

[0082] In addition, in Embodiment 1, the camera 3 simultaneously performs photographing in the visible light region and photographing in the infrared region, but is not limited thereto, and the camera 3 can perform photographing in the visible light region only without performing photographing in the infrared region. In addition, a camera that performs photographing in the visible light region and a camera that performs photographing in the infrared region can be provided individually, and the information processing apparatus 1 acquires photographed images of the two cameras and performs opening detection. In addition, in the information processing system according to Embodiment 1, a part or all of the processing performed by the information processing apparatus 1 can be performed by the management apparatus 5, or a part or all of the processing performed by the management apparatus 5 can be performed by the information processing apparatus 1.

[0083] [Embodiment 2]

[0084] <Outline of System>

[0085] Figure 8 is a schematic view for explaining an outline of the information processing system according to Embodiment 2. The information processing system according to Embodiment 2 is a system for preventing a worker from confusing a substrate processing apparatus 101 that is an operation target, with respect to a plurality of substrate processing apparatuses 101 provided in a clean room 100. The information processing system according to Embodiment 2 can prevent a worker from erroneously performing maintenance or the like, with respect to a substrate processing apparatus 101 in operation, for example, in substrate processing such as etching of a substrate such as a wafer. Therefore, the information processing system according to Embodiment 2 determines an operation state of each substrate processing apparatus 101 on the basis of schedule information or the like input by a worker or the like, and performs notification when it is detected that a worker or the like approaches a substrate processing apparatus 101 in operation, on the basis of photographed images of a plurality of cameras 3 provided in the clean room 100.

[0086] The information processing system according to Embodiment 2 is configured to have, similarly to the information processing system according to Embodiment 1, a plurality of cameras 3 disposed at appropriate positions in the clean room 100, a plurality of information processing apparatuses 1 that control the respective cameras 3, and a management apparatus 5 that manages information obtained from the plurality of information processing apparatuses 1. However, in Embodiment 2, one substrate processing apparatus 101 corresponds to one camera 3. Each camera 3 is disposed at or near the corresponding substrate processing apparatus 101, and performs photographing toward the periphery of the substrate processing apparatus 101, so that, for example, a worker or the like approaching the substrate processing apparatus 101 can be photographed from the front. However, the disposition position of the camera 3 is one example, and is not limited thereto.

[0087] The information processing system according to Embodiment 2 detects an approach of a worker or the like to the substrate processing apparatus 101, that is, a worker or the like who is to perform work on the substrate processing apparatus 101, by the information processing apparatus 1 that acquires information of captured images captured by the plurality of cameras 3 respectively provided corresponding to the plurality of substrate processing apparatuses 101 by capturing the periphery of the substrate processing apparatus 101. In the information processing system according to Embodiment 2, the schedule information related to work of the plurality of substrate processing apparatuses 101 provided in the clean room 100 is managed by the management apparatus 5, and information related to whether each of the substrate processing apparatuses 101 is in operation or in stop is transmitted from the management apparatus 5 to the information processing apparatus 1 to which the corresponding camera 3 is connected. In a case where the approach of a worker or the like is detected by the information processing apparatus 1 with respect to the substrate processing apparatus 101 in operation, the information processing apparatus 1 notifies the management apparatus 5, and performs notification to the worker or the like based on light or sound or the like.

[0088] Further, in the information processing system according to Embodiment 2, the plurality of cameras 3 provided in the clean room 100 can be cameras that perform capturing in a visible light region and do not perform capturing in an infrared light region.

[0089] Further, in the information processing system according to Embodiment 2, since the schedule of each of the substrate processing apparatuses 101 is managed by the management apparatus 5, input of information such as work scheduling from a worker or the like is accepted via the terminal apparatus 7. The terminal apparatus 7 can use, for example, a tablet terminal apparatus or a smartphone or the like. The terminal apparatus 7, for example, displays an input screen of work scheduling or the like on a display portion and accepts input of information from a worker, and transmits the accepted information to the management apparatus 5. Further, the terminal apparatus 7 can transmit an image obtained by capturing a content of a form (paper) on which work scheduling or the like is written by a worker or the like by hand using an attached camera to the management apparatus 5. The management apparatus 5 can also read a character string described in a captured image of the form received from the terminal apparatus 7 using a function of so-called OCR (Optical Character Recognition), and thereby acquire information of work scheduling or the like. The management apparatus 5 manages a schedule such as a period of work of each of the substrate processing apparatuses 101 based on the information acquired via the terminal apparatus 7, and notifies each of the information processing apparatuses 1 of an action state of the corresponding substrate processing apparatus 101.

[0090] <Device Structure>

[0091] Figure 9is a block diagram showing one configuration example of the information processing apparatus 1 according to Embodiment 2. The information processing apparatus 1 according to Embodiment 2 is configured to have the processing section 11, the storage section 12, the communication section 13, the interface section 14, and the like, like the information processing apparatus 1 according to Embodiment 1. The hardware configuration of the information processing apparatus 1 according to Embodiment 2 is the same as that of the information processing apparatus 1 according to Embodiment 1, and thus detailed description thereof is omitted.

[0092] The learning model storage section 12b of the storage section 12 of the information processing apparatus 1 according to Embodiment 2 stores information related to the same person detection model 22 as that of Embodiment 1, and a posture estimation model that estimates a posture of a person. The posture estimation model is a learning model obtained by performing machine learning in advance in such a manner that an image of an existing person is accepted as input, and an estimation result of a posture of the person is output. The estimation result output by the posture estimation model is information in which, for example, joints of a head, a waist, hands, and feet of a person's body are represented by a plurality of points, and the posture estimation model outputs coordinate information of the plurality of points. The information processing apparatus 1 according to Embodiment 2 determines a direction of a face (line of sight) of a person and a direction in which an arm is extended, and the like, in the imaged image, on the basis of the coordinates of the plurality of points. The posture estimation model can utilize an existing learning model such as OpenPose. The posture estimation model is an existing technology, and thus detailed description of a learning method and the like is omitted.

[0093] The action state information storage section 12c that stores information related to an action state of the substrate processing apparatus 101 corresponding to the information processing apparatus 1 or the camera 3 connected to the information processing apparatus 1 is provided in the storage section 12 of the information processing apparatus 1 according to Embodiment 2. In the present embodiment, the action state of the substrate processing apparatus 101 is either one of in operation and in stop, which is one example, and an action state other than these two can also be adopted. The action state information storage section 12c stores information indicating which one of in operation and in stop the corresponding substrate processing apparatus 101 is in.

[0094] In addition, in the information processing apparatus 1 according to Embodiment 2, the processing section 11 reads out the program 12a stored in the storage section 12 and executes, so that the imaged image acquisition section 11a, the person detection section 11d, the notification processing section 11e, the posture estimation section 11f, the action state acquisition section 11g, and the apparatus error determination section 11h, and the like are realized as software functional sections in the processing section 11.

[0095] The captured image acquisition section 11a acquires data of a captured image of the camera 3 via the interface section 14, and temporarily stores the acquired captured image in the storage section 12. The person detection section lid performs processing of detecting a person in the clean room 100 based on the captured image acquired by the captured image acquisition section 11a from the camera 3, using the person detection model 22 stored in the learning model storage section 12b. In a case where it is determined by the device error determination section 11h described later that a device error has occurred, the notification processing section 11e transmits a message notifying of this to the management device 5, and also outputs a control signal from the interface section 14, thereby causing the LED lamp 4 to light up and performing notification to a worker or the like. In addition, in a case where the information processing device 1 is configured to be able to communicate with the terminal device 7 of the user, the notification processing section 11e can also transmit a message for notification to the terminal device 7.

[0096] In a case where the person detection section lid detects a worker or the like from the captured image, the posture estimation section 11f performs processing of estimating the posture of the person. In Embodiment 2, the posture estimation section 11f estimates the posture of the person using the posture estimation model stored in the learning model storage section 12b. The posture estimation section 11f extracts an image region in which the person is imaged from the captured image based on the detection result of the person by the person detection section lid, inputs data of the extracted image region to the posture estimation model, and acquires an estimation result of the posture output by the posture estimation model. Further, in a case where the posture estimation model is a model capable of estimating the postures of a plurality of persons from one captured image, the posture estimation section 11f can also acquire an estimation result of the posture of one or a plurality of persons imaged in the captured image by inputting the captured image of the camera 3 directly to the posture estimation model, without extracting an image region of the person from the captured image based on the detection result of the person and inputting it to the posture estimation model. In Embodiment 2, the posture estimation section 11f determines the direction of the line of sight and the arm of the detected person based on the estimation result of the posture of the person by the posture estimation model.

[0097] The operation state acquisition section 11g performs processing of acquiring an operation state of the substrate processing device 101 corresponding to the information processing device 1 or the camera 3 connected to the information processing device 1. In the present embodiment, the management device 5 manages information related to a schedule of work of each substrate processing device 101, and based on the information related to the schedule, the management device 5 notifies each information processing device 1 of information related to the operation state of the corresponding substrate processing device 101. The operation state acquisition section 11g of the information processing device 1 communicates with the management device 5 through the communication section 13, receives the information related to the operation state transmitted by the management device 5, and thereby acquires the operation state of the corresponding substrate processing device 101. The operation state acquisition section 11g stores information related to the acquired operation state in the operation state information storage section 12c.

[0098] The device error determination section 11h determines whether or not the substrate processing device 101 in which the detected person is to perform a maintenance or the like has an error, based on the detection result of the person by the person detection section 11d, the estimation result of the posture of the person by the posture estimation section 11f, and the operation state of the substrate processing device 101 acquired by the operation state acquisition section 11g. In the present embodiment, in a case where the worker or the like in the clean room 100 is to perform some operation on the substrate processing device 101 in operation, the device error determination section 11h determines that a device error occurs.

[0099] In the present embodiment, for example, in a case where the worker or the like detected from the captured image of the camera 3 is looking at (gazing at) the substrate processing device 101 in operation for a prescribed time or more (for example, 10 seconds or more, etc.), the device error determination section 11h determines that the person is to perform some operation on the substrate processing device 101, and determines that a device error occurs. Also, for example, in a case where the detected person performs a hand-approaching operation on the substrate processing device 101 in operation, the device error determination section 11h determines that the person is to perform some operation on the substrate processing device 101, and determines that a device error occurs. In the present embodiment, the camera 3 is provided to each substrate processing device 101, and thus, for example, in a case where a person in a posture of extending a hand toward the camera 3 or gazing at the camera 3 is captured, it can be determined that the person is confusing the devices. Furthermore, in order to implement the above-described determination method, the device error determination section 11h tracks the detected person in the captured images of a plurality of continuous time points, and determines whether or not the line of sight is continued for a prescribed time or more, or whether or not the hand is approaching or retreating, etc. Also, the device error determination section 11h can perform the above-described determination processing in a case where the operation state of the corresponding substrate processing device 101 is in operation, and can not perform the determination processing in a case where the operation state is in stop.

[0100] Figure 10 is a block diagram showing one configuration example of the management device 5 according to Embodiment 2. The management device 5 according to Embodiment 2 is configured to have a processing section 51, a storage section 52, a communication section 53, a display section 54, an operation section 55, and the like, like the management device 5 according to Embodiment 1. The hardware configuration of the management device 5 according to Embodiment 2 is the same as that of the management device 5 according to Embodiment 1, and thus detailed description thereof is omitted.

[0101] In the storage section 52 of the management apparatus 5 according to Embodiment 2, a schedule DB 52c that stores schedule information related to the operation of the plurality of substrate processing apparatuses 101 provided in the clean room 100 is provided. The schedule DB 52c stores, for example, information related to the date and time of operation or the date and time of stop of each of the substrate processing apparatuses 101 in correspondence with the identification information of each of the substrate processing apparatuses 101. In addition, in the schedule DB 52c, various information such as the content of processing performed by the substrate processing apparatuses 101 and the inputter of the information can be stored in correspondence.

[0102] In addition, in the management apparatus 5 according to Embodiment 2, the processing section 51 reads out and executes the program 52a stored in the storage section 52, whereby the information acquisition section 51a, the notification processing section 51b, the schedule information management section 51c, the form reading section 51d, and the like are realized as software functional sections in the processing section 51. The information acquisition section 51a acquires information such as the notification of the apparatus error and the data of the captured image from the information processing apparatus 1 via the communication section 53, and stores the acquired information in the security management DB 52b. In a case where the apparatus error is notified from the information processing apparatus 1, the notification processing section 51b transmits a message notifying of the detection of the apparatus error to one or a plurality of terminal apparatuses 7. In addition, in a case where the terminal apparatus 7 held by the person who is determined to have made the mistake can be distinguished, the notification processing section 51b can also transmit the message only to the terminal apparatus 7.

[0103] The schedule information management section 51c performs processing of managing schedule information related to the operation of the plurality of substrate processing apparatuses 101 provided in the clean room 100. The schedule information management section 51c communicates with the terminal apparatus 7 used by the user such as the worker via the communication section 53, and acquires information related to the job and the like in the substrate processing apparatus 101 inputted from the user by the terminal apparatus 7. The schedule information management section 51c extracts or calculates the time of operation and the time of stop of the substrate processing apparatus 101 and the like based on the information acquired from the terminal apparatus 7, and stores in the schedule DB 52c in correspondence with the identification information of the substrate processing apparatus 101 and the like.

[0104] In addition, in the present embodiment, a user such as a worker can handwrite information related to a job or the like in the substrate processing apparatus 101 on a prescribed form, and transmit a captured image obtained by capturing the form using a camera of the terminal apparatus 7 to the management apparatus 5. In this case, the schedule information management section 51c of the management apparatus 5 is given the captured image acquired from the terminal apparatus 7 to the form reading section 51d described later, and can acquire information read from the form by the form reading section 51d. Further, in the present embodiment, reading of characters or the like described on the form is performed by the management apparatus 5, but is not limited thereto, and the reading of the form can be performed by the terminal apparatus 7, and information read by the terminal apparatus 7 can be transmitted to the management apparatus 5.

[0105] In addition, the schedule information management section 51c performs processing of determining an operation state of each substrate processing apparatus 101 in the clean room 100 based on information stored in the schedule DB 52c, and notifying the operation state to the information processing apparatus 1 corresponding to each substrate processing apparatus 101. The schedule information management section 51c can notify the operation state at a prescribed period such as once every several seconds or once every several minutes, for example, in addition to, for example, can notify at a timing at which the operation state changes, and in addition to, for example, can notify the operation state according to a request from the information processing apparatus 1.

[0106] The form reading section 51d performs reading of characters described on a captured image of a form acquired from the terminal apparatus 7 by the schedule information management section 51c. The form reading section 51d performs reading of characters described on the form by a function of so-called OCR. Reading of characters based on OCR can be performed by, for example, extracting features of the described characters and collating with a dictionary prepared in advance, and in addition to, for example, can be performed using a learning model obtained by machine learning in a manner of estimating characters from an image of handwritings. Reading of handwritings based on OCR is a conventional technology, and detailed description is omitted.

[0107] <Device Error Prevention Processing>

[0108] Figure 11 is a flowchart showing one example of steps of schedule management processing performed by the management apparatus 5 according to Embodiment 2. In the information processing system according to Embodiment 2, in a case where the substrate processing apparatus 101 in the clean room 100 performs substrate processing or the like, a worker pre-enters information of job scheduling or the like in the terminal apparatus 7. The information accepted to be input in the terminal apparatus 7 is transmitted to the management apparatus 5. In addition, the worker can perform input of information by capturing contents obtained by handwriting information of job scheduling or the like on a prescribed form using the terminal apparatus 7, and the terminal apparatus 7 transmits a captured image of the form to the management apparatus 5.

[0109] The schedule information management section 51c of the processing section 51 of the management device 5 involved in the embodiment 2 determines whether information related to the work scheduled in the clean room 100 is received from the terminal device 7 that has accepted the input of the worker (step S41). In the case where the information related to the work scheduled is received (S41: YES), the schedule information management section 51c determines whether the received information is a captured image of the form (step S42). In the case where the received information is a captured image of the form (S42: YES), the form reading section 5 Id of the processing section 51 performs OCR processing on the received captured image of the form, reads the information described on the form (step S43), and causes the processing to proceed to step S44. In the case where the received information is not a captured image of the form (S42: NO), the processing section 51 causes the processing to proceed to step S44. The schedule information management section 51c stores the information received from the terminal device 7 or the information read from the received captured image in the schedule DB 52c (step S44), and causes the processing to return to step S41.

[0110] In the case where the information related to the work scheduled is not received from the terminal device 7 (S41: NO), the schedule information management section 51c determines whether it is the notification timing to the information processing device 1 (step S45). In this example, the management device 5 notifies the action state of the substrate processing device 101 in the clean room 100 to the information processing device 1 corresponding to each substrate processing device 101 at an appropriate period, such as once every several seconds or once every several minutes. The schedule information management section 51c can determine whether it is the notification timing based on whether a prescribed period has passed since the last notification. In the case where it is not the notification timing (S45: NO), the schedule information management section 51c returns the processing to step S41.

[0111] In the case where it is the notification timing (S45: YES), the schedule information management section 51c reads out the schedule information of each substrate processing device 101 stored in the schedule DB 52c (step S46). The schedule information management section 51c determines the action state of each substrate processing device 101 based on the read schedule information (step S47). For example, assuming that the schedule of the substrate processing device 101 is from 0 o'clock to 2 o'clock, if the current time is 1 o'clock, the schedule information management section 51c can determine that the substrate processing device 101 is in operation. The schedule information management section 51c notifies the action state of the substrate processing device 101 by transmitting the determined action state of each substrate processing device 101 to the information processing device 1 corresponding to each substrate processing device 101 (step S48), and returns the processing to step S41.

[0112] Figure 12 and Figure 13This is a flowchart illustrating an example of the steps involved in the device error prevention processing performed by the information processing apparatus 1 according to Embodiment 2. In this example, the information processing apparatus 1 receives... Figure 11 In step S48 of the flowchart shown, the operation status is notified from the management device 5, and the received information is stored in the operation status information storage unit 12c. The operation status acquisition unit 11g of the processing unit 11 of the information processing device 1 according to Embodiment 2 reads the operation status of the substrate processing device 101 stored in the operation status information storage unit 12c (step S61). The operation status acquisition unit 11g determines whether the corresponding operation status of the substrate processing device 101 is active (step S62). If the operation status of the substrate processing device 101 is not active (S62: No), that is, in the case of being stopped, the operation status acquisition unit 11g returns the processing to step S61.

[0113] When the substrate processing apparatus 101 is in operation (S62: Yes), the image acquisition unit 11a of the processing unit 11 acquires an image captured by the camera 3 through the interface unit 14. The person detection unit 11d of the processing unit 11 inputs the image data acquired in step S63 into the person detection model 22 stored in the learning model storage unit 12b (step S64). The person detection unit 11d acquires the person detection result output by the person detection model 22 based on the input in step S64 (step S65). Based on the person detection result acquired in step S65, the person detection unit 11d determines whether a person is reflected in the captured image (step S66). If no person is reflected in the captured image (S66: No), the person detection unit 11d returns the processing to step S61.

[0114] If a person is reflected in the captured image (S66: Yes), the pose estimation unit 11f of the processing unit 11 extracts the image region reflecting the detected person from the captured image, and inputs the data of the extracted image region of the person into the pose estimation model stored in the learning model storage unit 12b (step S67). Based on the input in step S67, the pose estimation unit 11f obtains the pose estimation result output by the pose estimation model (step S68).

[0115] Furthermore, in this flowchart, the pose estimation unit 11f extracts the image region reflecting people from the captured image and inputs it into the pose estimation model, but is not limited to this. If the pose estimation model is capable of estimating the poses of multiple people based on a single captured image, the pose estimation unit 11f can directly input the captured image from the camera 3 into the pose estimation model to obtain the pose estimation results of one or more people output by the pose estimation model.

[0116] The device error determination section 11h of the processing section 11 determines whether the person appearing in the captured image is a posture of extending a hand toward the substrate processing device 101 based on the posture estimation result acquired in step S68 (step S69). In the case of a posture other than extending a hand (S69: No), the device error determination section 11h determines whether the person appearing in the captured image is a posture of gazing at the substrate processing device 101 (step S70). In the case of a posture other than gazing (S70: No), the device error determination section 11h returns the processing to step S61.

[0117] In the case of the person appearing in the captured image being a posture of extending a hand toward the substrate processing device 101 (S69: Yes), or in the case of the person appearing in the captured image being a posture of gazing at the substrate processing device 101 (S70: Yes), the notification processing section 11e of the processing section 11 notifies the gist of the occurrence of a device error of the substrate processing device 101 to the management device 5 (step S71). In addition, the notification processing section 11e transmits the data of the captured image acquired in step S63 to the management device 5 (step S72). Further, the notification processing section 11e outputs a control signal to the LED lamp 4 from the interface section 14, thereby lighting the LED lamp 4 (step S73), and returns the processing to step S61.

[0118] Figure 14 This is a flowchart showing one example of the steps of the device error prevention processing performed by the management device 5 according to Embodiment 2. The information acquisition section 51a of the processing section 51 of the management device 5 according to Embodiment 2 determines whether a notification of the gist of the occurrence of a device error is accepted from any one of the information processing devices 1 through the communication section 53 (step S81). In the case of not accepting the notification (S81: No), the information acquisition section 51a stands by until the notification is accepted from any one of the information processing devices 1. In the case of accepting the notification (S81: Yes), the information acquisition section 51a stores the information included in the notification from the information processing device 1 and various information such as a captured image received together with the notification in the security management DB 52b (step S82). Next, the notification processing section 51b of the processing section 51 transmits a message notifying of a device error to one or more of the terminal devices 7 through the communication section 53 (step S83), and ends the processing.

[0119] <Summary>

[0120] In the information processing system according to Embodiment 2, the information processing apparatus 1 acquires a captured image obtained by the camera 3 capturing the inside of the clean room 100 or the like, and estimates a posture of a person in the clean room 100 on the basis of the acquired captured image. In addition, the information processing apparatus 1 acquires an operation state of the device such as the substrate processing device 101 provided in the clean room 100. The information processing apparatus 1 determines that the person has mistaken the substrate processing device 101 as a work target on the basis of the estimated posture and the acquired operation state. Thus, the information processing system can be expected to realize safe monitoring or the like on the basis of the captured image of the camera 3.

[0121] In addition, in the information processing system according to Embodiment 2, in a case where it is estimated that the person has a posture of stretching a hand toward the substrate processing device 101 or gazing at the substrate processing device 101 in an operation state of working, the information processing apparatus 1 determines that the person has mistaken the substrate processing device 101 as a work target. Thus, the information processing system can be expected to determine that a worker or the like has erroneously touched the device such as the substrate processing device 101 in which substrate processing such as etching is being performed, for work such as maintenance.

[0122] In addition, in the information processing system according to Embodiment 2, the management apparatus 5 acquires schedule information relating to work of the substrate processing device 101 via the terminal apparatus 7, and stores the acquired schedule information in the operation state information storage section 12c. The management apparatus 5 determines the operation state of the substrate processing device 101 on the basis of the schedule information stored in the operation state information storage section 12c. Thus, even in a case where it is not possible to directly obtain the operation state of the substrate processing device 101, the information processing system can be expected to determine the operation state of the substrate processing device 101 to perform determination of a device error.

[0123] In addition, in the information processing system according to Embodiment 2, the management apparatus 5 acquires the schedule information by reading information from a form in which information relating to work in the substrate processing device 101 is written. In addition, in the information processing system according to Embodiment 2, the management apparatus 5 acquires the schedule information by accepting input of information relating to work in the substrate processing device 101 from a worker or the like via the terminal apparatus 7. Thus, the information processing system can be expected to facilitate input of information by the worker or the like.

[0124] In addition, in the information processing system according to Embodiment 2, the information processing apparatus 1 performs notification such as lighting of the LED lamp 4 and message transmission to the management apparatus 5 in a case where it is determined that there is a device error in the substrate processing device 101 as a work target. Thus, the information processing system can notify the worker or the like in the clean room 100 of the device error, and can be expected to prevent danger due to the device error in advance.

[0125] In addition, in the information processing system according to Embodiment 2, the information processing apparatus 1 transmits the captured image used for the determination to the management apparatus 5 in a case where it is determined that the substrate processing apparatus 101 as a work target has an apparatus error, and stores the captured image in the security management DB 52b. Thus, the information processing system can save the situation in the clean room 100 when the apparatus error occurs as a captured image, and can contribute to a safety countermeasure in the clean room 100 or the like.

[0126] Further, in Embodiment 2, the operation state of the substrate processing apparatus 101 is determined based on the schedule information managed by the management apparatus 5, but is not limited thereto. For example, the information processing apparatus 1 can directly acquire the operation state from the substrate processing apparatus 101 by performing information exchange between the substrate processing apparatus 101. In addition, in the information processing system according to Embodiment 2, the camera 3 is provided at or near the substrate processing apparatus 101, but the position where the camera 3 is provided is not limited thereto, and can be provided at any place. In addition, in the information processing system according to Embodiment 2, a part or all of the processing performed by the information processing apparatus 1 can be performed by the management apparatus 5, or a part or all of the processing performed by the management apparatus 5 can be performed by the information processing apparatus 1.

[0127] In addition, the information processing system according to Embodiment 2 can also perform the opening detection processing of the information processing system according to Embodiment 1. That is, the information processing system can be a structure that performs both the opening detection processing described in Embodiment 1 and the apparatus error prevention processing described in Embodiment 2.

[0128] Further, the other structures of the information processing system according to Embodiment 2 are the same as those of the information processing system according to Embodiment 1, and thus the same reference numerals are attached to the same parts, and detailed description is omitted.

[0129] It should be understood that the embodiments disclosed herein are illustrative in all points and are not restrictive. The scope of the present disclosure is not the above-described meaning, but is indicated by the claims, and is intended to include the meaning equivalent to the claims and all modifications within the scope thereof.

[0130] The matters described in each of the embodiments can be combined with each other. In addition, the independent claims and the dependent claims described in the claims can be combined with each other in all combinations regardless of the citation form. Also, the claims are described in a form in which the claims citing two or more other claims are used (multiple claim form), but are not limited thereto. The claims can be described in a form in which a multiple claim citing at least one multiple claim is described (multiple citation multiple claim).

[0131] Reference Signs

[0132] 1…information processing apparatus (computer); 3…camera; 4…LED lamp; 5…management apparatus; 7…terminal apparatus; 11…processing section; 11a…captured image acquisition section; 11b…opening detection section; 11c…protection measure determination section; 11d…person detection section; 11e…notification processing section; 11f…posture estimation section; 11g…motion state acquisition section; 11h…apparatus error determination section; 12…storage section; 12a…program (computer program); 12b…learning model storage section; 12c…motion state information storage section; 13…communication section; 14…interface section; 21…segmentation model; 22…person detection model; 51…processing section; 51a…information acquisition section; 51b…notification processing section; 51c…schedule information management section; 51d…form reading section; 52…storage section; 52a…program; 52b…security management DB; 53…communication section; 54…display section; 55…operation section; 98, 99…recording medium; 100…clean room; 101…substrate processing apparatus (apparatus).

Claims

1. An information processing method, wherein the information processing device performs the following processing: Acquire images taken by the camera indoors; and Based on the acquired images, the openings in the floor of the aforementioned room are detected.

2. The information processing method according to claim 1, wherein, Based on the images captured above, determine whether there are any protective measures for the aforementioned opening.

3. The information processing method according to claim 2, wherein, Notification will be issued based on the test results of the aforementioned openings and the determination of the presence or absence of the aforementioned protective measures.

4. The information processing method according to claim 3, wherein, Based on the above-mentioned image capture, human detection A notification will be issued if a person is detected within a specified distance from the opening, if a person is detected facing the opening, or if a person is detected approaching the opening.

5. The information processing method according to claim 1, wherein, The openings in the floor of a cleanroom with multiple floor tiles were inspected.

6. The information processing method according to claim 1, wherein, The camera mentioned above is an infrared-based camera.

7. The information processing method according to claim 1, wherein, Based on the acquired images, the posture of the people indoors is estimated. Obtain the operating status of the device installed in the aforementioned room. Based on the estimated posture of the person and the obtained operating state of the device, it is determined that the person has mistakenly identified the device as the object of the operation.

8. An information processing method, wherein an information processing device performs processing: Acquire images captured by a camera inside an indoor space; Based on the acquired images, the posture of the people indoors is estimated; Acquire the operational status of the device installed in the aforementioned room; and Based on the estimated posture of the person and the obtained operating state of the device, it is determined that the person has mistakenly identified the device as the object of the operation.

9. The information processing method according to claim 7 or 8, wherein, If the aforementioned person extends their hand to the device that is in operation or stares at the device in operation, it is determined that the aforementioned person has mistaken the device as the object of operation.

10. The information processing method according to claim 7 or 8, wherein, The information processing device processes the information. Obtain schedule information related to the operation of the aforementioned devices; The obtained schedule information is stored in the storage unit; and Based on the stored schedule information, the operating status of the device is determined.

11. The information processing method according to claim 10, wherein, The information is read from a form containing information related to the operation of the aforementioned device. Based on the information read above, the schedule information of the aforementioned device is obtained.

12. The information processing method according to claim 10, wherein, The terminal device receives input of information related to the operations in the aforementioned device. Based on the received information, obtain the schedule information of the aforementioned device.

13. The information processing method according to claim 7 or 8, wherein, If it is determined that the aforementioned person has mistakenly identified the device as the object of the operation, a notification will be issued.

14. The information processing method according to claim 7 or 8, wherein, If it is determined that the person has mistakenly identified the device as the object of the operation, the captured image used for the determination will be stored in the storage unit.

15. A computer program that causes a computer to perform the following processes: Acquire images taken by the camera indoors; and Based on the acquired images, the openings in the floor of the aforementioned room are detected.

16. A computer program that causes a computer to perform the following processes: Acquire images captured by a camera inside an indoor space; Based on the acquired images, estimate the posture of the people indoors. Acquire the operational status of the device installed in the aforementioned room; and Based on the estimated posture of the person and the obtained operating state of the device, it is determined that the person has mistakenly identified the device as the object of the operation.

17. An information processing apparatus, comprising a processing unit. The above-mentioned processing department will handle the following: Acquire images taken by the camera indoors; and Based on the acquired images, the openings in the floor of the aforementioned room are detected.

18. An information processing apparatus, comprising a processing unit, The above-mentioned processing department will handle the following: Acquire images captured by a camera inside an indoor space; Based on the acquired images, the posture of the people indoors is estimated; Acquire the operational status of the device installed in the aforementioned room; and Based on the estimated posture of the person and the obtained operating state of the device, it is determined that the person has mistakenly identified the device as the object of the operation.

Images