Information processing method, computer program, and information processing device

Abstract

Provided are an information processing method, a computer program, and an information processing device that can be expected to realize safety monitoring or the like based on a captured image of a camera.　An information processing method according to the present embodiment involves: acquiring a captured image obtained by imaging a room with a camera; detecting protective equipment worn by a person appearing in the captured image on the basis of the acquired captured image; determining the suitability of the protective equipment on the basis of a combination of the detected protective equipment and work to be performed by the person; and notifying of a determination result.

Description

Information processing method, computer program, and information processing device

[0001] The present disclosure relates to an information processing method, a computer program, and an information processing device.

[0002] Patent Document 1 proposes a surveillance system in which an authentication controller performs authentication processing based on authentication information received from an authentication terminal via electric field communication, an image recognition server recognizes people included in the shooting area of ​​an image captured by a camera, and the person authenticated by the authentication controller is compared with the person image-recognized by the image recognition server based on the position information of the authentication terminal and the coordinate information of the camera's shooting area.

[0003] JP 2012-8802 A

[0004] The present disclosure provides an information processing method, a computer program, and an information processing device that are expected to realize safety monitoring based on images captured by a camera.

[0005] An information processing method according to one embodiment acquires an image captured by a camera inside a room, detects protective equipment worn by a person appearing in the image based on the acquired image, determines the suitability of the protective equipment based on a combination of the detected protective equipment and the work being performed by the person, and notifies the user of the determination result.

[0006] According to the present disclosure, it is expected that safety monitoring based on images captured by a camera can be realized.

[0007] FIG. 1 is a schematic diagram for explaining an overview of an information processing system according to the present embodiment. FIG. 2 is a block diagram showing an example of a configuration of an information processing device according to the present embodiment. FIG. 3 is a schematic diagram showing an example of a necessary protective equipment table provided in the information processing device according to the present embodiment. FIG. 4 is a block diagram showing an example of a configuration of a management device according to the present embodiment. FIG. 5 is a flowchart showing an example of a procedure for protective equipment determination processing performed by the information processing device according to the present embodiment. FIG. 6 is a flowchart showing an example of a procedure for schedule management processing performed by the management device according to the present embodiment. FIG. 7 is a flowchart showing an example of a procedure for protective equipment determination processing performed by the management device according to the present embodiment.

[0008] Specific examples of information processing systems according to embodiments of the present disclosure will be described below with reference to the drawings. Note that the present disclosure is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0009] 1 is a schematic diagram for explaining an overview of an information processing system according to this embodiment. The information processing system according to this embodiment is, for example, a system for monitoring safety within a clean room 100 in which a plurality of substrate processing apparatuses 101 are installed. The information processing system is configured to include a plurality of cameras 3 arranged at appropriate locations within the clean room 100, a plurality of information processing apparatuses 1 that control the cameras 3, and a management apparatus 5 that manages information obtained from the plurality of information processing apparatuses 1.

[0010] Various operations are performed in the clean room 100, such as supplying gas to the substrate processing apparatus 101, supplying chemicals, and performing maintenance on the apparatus. When performing these operations, the workers are required to properly wear various protective equipment, such as protective glasses, safety belts, helmets, goggles, gas masks, aprons, protective gloves, or protective foot covers, in order to ensure their safety.

[0011] In the information processing system according to this embodiment, multiple cameras 3 arranged at appropriate locations within the clean room 100 capture images of the inside of the clean room 100, and each information processing device 1 performs a process of determining the appropriateness of protective equipment worn by a worker based on the images captured by each camera 3. The information processing device 1 detects people within the clean room 100 based on the images captured by the cameras 3 and determines what protective equipment the detected people are wearing. In addition, in the information processing system according to this embodiment, the management device 5 manages schedule information related to work to be performed within the clean room 100, and transmits information related to work being performed in the clean room 100 at that time from the management device 5 to the information processing device 1. Each information processing device 1 compares the protective equipment required for the work at that time with the protective equipment worn by the person captured in the captured image based on the information from the management device 5, and determines the appropriateness of the protective equipment.

[0012] When the information processing device 1 determines that the protective equipment worn by a person working in the clean room 100 is inappropriate, it notifies the management device 5 of this fact and also notifies the worker, etc. by light, sound, etc. In this case, the information processing device 1 transmits a captured image of the protective equipment determined to be inappropriate to the management device 5 along with the notification that the protective equipment is inappropriate. When the management device 5 is notified by the information processing device 1 that the protective equipment is inappropriate, it notifies, for example, a terminal device carried by a worker in the clean room 100, that the protective equipment is inappropriate. Furthermore, the management device 5 obtains from the information processing device 1 the captured image when the protective equipment is determined to be inappropriate, and stores it in a database together with information such as the date and time and the work content, for example.

[0013] In the present embodiment, the camera 3 disposed in the clean room 100 is a so-called depth camera, which captures images in a general visible light range and can measure the distance from the camera 3. Therefore, the captured image data obtained by the camera 3 includes RGB color data and data indicating the distance from the camera 3. However, the camera 3 may only capture images in the visible light range without measuring the distance. Alternatively, a camera capturing images in the visible light range and a camera measuring the distance (which may be a sensor or LiDAR (Light Detection and Ranging)) may be separately installed, and the information processing device 1 may separately acquire and process data from both cameras. In the following description, for example, when a description is made of an image captured by the camera 3, the captured image data may include RGB data captured in the visible light range and data indicating the distance (depth) from the camera 3.

[0014] In the information processing system according to this embodiment, the management device 5 manages schedule information related to work in the clean room 100. The management device 5 notifies each information processing device 1 of the work being performed in the clean room 100 based on this schedule information and the current date and time, etc. Each information processing device 1 stores in advance, for example, a correspondence between the types of work that can be performed in the clean room 100 and the protective equipment required for each work. The information processing device 1 determines the protective equipment required according to the type of work at that time notified by the management device 5, and determines whether the protective equipment worn by the person working in the clean room 100 is appropriate.

[0015] In the information processing system according to the present embodiment, the management device 5 manages the schedule of work in the clean room 100, and accepts input of information such as work schedules from workers via the terminal device 7. The terminal device 7 may be, for example, a tablet terminal device or a smartphone. The terminal device 7 displays an input screen for the work schedule on a display unit, accepts information input from workers, and transmits the accepted information to the management device 5. The terminal device 7 may also transmit to the management device 5 an image captured with an attached camera of handwritten notes written by workers on a sheet (paper) on which the work schedule is to be entered. The management device 5 may acquire information such as the work schedule by reading character strings written in the captured image of the sheet received from the terminal device 7 using a so-called OCR (Optical Character Recognition) function. The management device 5 generates and stores schedule information for work to be performed in the clean room 100 based on the information acquired via the terminal device 7, and notifies each information processing device 1 of information regarding work being performed in the clean room 100.

[0016] <Device Configuration> Fig. 2 is a block diagram showing an example configuration of an information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment is, for example, an information processing device called an edge computer in the field of IoT (Internet of Things). The information processing device 1 is configured to include a processing unit 11, a storage unit 12, a communication unit 13, an interface unit 14, etc. Note that in this embodiment, one camera 3 is connected to one information processing device 1, but this is not limited thereto, and multiple cameras 3 may be connected to one information processing device 1.

[0017] The processing unit 11 is configured using an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit) or a quantum processor, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processing unit 11 reads and executes a program 12a stored in the storage unit 12 to perform various processes, such as a process of detecting people and protective equipment in the clean room 100 from images captured by the camera 3 and a process of determining whether the protective equipment worn by the detected people is appropriate.

[0018] The storage unit 12 is configured using a large-capacity storage device such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11 and various data necessary for the processing of the processing unit 11. In this embodiment, the storage unit 12 stores a program 12a executed by the processing unit 11. The storage unit 12 also includes a learning model storage unit 12b that stores information about one or more learning models that have been previously machine-learned, and a work information storage unit 12c that stores information about work being performed in the clean room 100.

[0019] In this embodiment, the program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disc, and the information processing device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may also be written to the storage unit 12, for example, during the manufacturing stage of the information processing device 1. Alternatively, the program 12a may be distributed by a remote server device or the like and acquired by the information processing device 1 via communication. For example, the program 12a may be read from the recording medium 99 by a writing device and written to the storage unit 12 of the information processing device 1. The program 12a may be provided in a form distributed via a network or in a form recorded on the recording medium 99.

[0020] The learning model storage unit 12b stores in advance information about learning models used in the various processes described above performed by the information processing device 1 according to this embodiment. The information about the learning models stored in the learning model storage unit 12b may include, for example, information indicating the structure of the learning model and values ​​of internal parameters of the learning model determined in advance by machine learning. In this embodiment, the information processing device 1 performs a process of determining the suitability of protective equipment using multiple learning models, and for this purpose, stores information about multiple learned learning models that have been previously subjected to machine learning in the learning model storage unit 12b.

[0021] For example, the information processing device 1 stores information about a human detection model that detects people from images captured by the camera 3 in the learning model storage unit 12b. The human detection model is a learning model that has undergone machine learning in advance to accept captured image data as input and detect people appearing in the captured image. For example, a learning model configured to handle images such as CNN is employed as the human detection model. Note that in this embodiment, the captured image data input to the human detection model is four-channel data, including RGB data and depth data, but may also be three-channel RGB data. The human detection result output by the human detection model can be the coordinates of a rectangle, or so-called bounding box, that surrounds an area in the captured image in which a person is captured. The human detection model is generated in advance by performing so-called supervised machine learning processing using learning data (teacher data) that associates image data with ground truth data indicating the area in the image data in which a person is captured. Furthermore, the human detection model may utilize an existing trained learning model, such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

[0022] For example, the information processing device 1 stores information about a protective equipment detection model that detects protective equipment worn by a person from a captured image of the person in the learning model storage unit 12b. The protective equipment detection model is a learning model that has undergone machine learning in advance to receive data of a captured image of a person as input and detect the protective equipment worn by the person. In this embodiment, the data input to the protective equipment detection model is data obtained by extracting an image area in which the person detected by the human detection model is captured from an image captured by the camera 3. The output of the protective equipment detection model is, for example, an estimated value indicating whether or not a specific protective equipment (such as protective glasses) is being worn. In this embodiment, the information processing device 1 detects multiple types of protective equipment, so it may include multiple protective equipment detection models for each type of protective equipment, or a configuration in which one protective equipment detection model outputs multiple estimated values ​​corresponding to each type of protective equipment is adopted. The protective equipment detection model is generated in advance by performing so-called supervised machine learning processing using learning data (teacher data) that associates image data of a person with a correct value flag indicating whether or not the person captured in the image data is wearing specific protective equipment. The protective gear detection model may be a learning model that may be called a classification model or a multi-value classification model.

[0023] For example, the information processing device 1 stores information about a posture estimation model that estimates the posture of a person from a captured image of the person in the learning model storage unit 12b. The posture estimation model is a learning model that has undergone machine learning in advance to accept data of a captured image of a person as input and output an estimation result of the posture of the person. Note that in this embodiment, the data input to the posture estimation model is data obtained by extracting an image area in which a person detected by a human detection model is captured from an image captured by the camera 3. The estimation result output by the posture estimation model is, for example, information representing the head, waist, joints of the limbs, etc. of the person's body using multiple points, and the posture estimation model outputs coordinate information of these multiple points. The posture estimation model may utilize an existing learning model such as OpenPose. Since the posture estimation model is an existing technology, detailed description of the learning method, etc. will be omitted.

[0024] For example, the information processing device 1 stores information about an activity estimation model that estimates the activity being performed by a person based on the estimated results of the person's posture in the learning model storage unit 12b. The activity estimation model is a learning model that has undergone machine learning in advance to receive as input the estimated results of the person's posture by the posture estimation model, i.e., information representing the person's body joints and the like using multiple points, and output an estimated result of the activity being performed by the person. The estimated result output by the activity estimation model is, for example, an estimated value indicating whether or not a specific activity (e.g., supplying gas to the substrate processing device 101) is being performed. In this embodiment, since the information processing device 1 performs estimation for multiple types of activities, a configuration is adopted in which multiple activity estimation models are provided for each activity type, or one activity estimation model outputs multiple estimated values ​​corresponding to each activity. The activity estimation model is generated in advance by performing so-called supervised machine learning processing using learning data (teacher data) that associates data representing the person's posture with a correct value flag indicating whether or not the person is performing a specific activity. A learning model that may be called a classification model or a multi-value classification model may be used as the activity estimation model.

[0025] The communication unit 13 communicates with the management device 5 via a wired or wireless network. The communication unit 13 transmits data provided from the processing unit 11 to the management device 5, and also receives data transmitted from the management device 5 and provides the data to the processing unit 11. In this embodiment, the communication unit 13 receives information relating to work being performed in the clean room 100 from the management device 5, and transmits to the management device 5 the results of the judgment on the suitability of the protective equipment, data on images captured by the camera 3, etc.

[0026] The interface unit 14 is connected to devices such as the camera 3 and the LED (Light Emitting Diode) lamp 4 via a communication line or a signal line, and exchanges data, signals, and the like with these devices. In the present embodiment, the interface unit 14 acquires captured image data output by the camera 3 and provides the data to the processing unit 11. The interface unit 14 also outputs an on / off control signal provided by the processing unit 11 to the LED lamp 4, switching the LED lamp 4 on and off. Note that in the present embodiment, the camera 3 is connected to the interface unit 14 of the information processing device 1, but this is not limited thereto. For example, if the camera 3 has a communication function, the information processing device 1 may be configured to communicate with the camera 3 via the communication unit 13. In this case, the information processing device 1 may acquire captured image data output from the camera 3 from the communication unit 13 via a wired or wireless network. The same applies to the LED lamp 4.

[0027] The storage unit 12 may be an external storage device connected to the information processing device 1. The information processing device 1 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The information processing device 1 is not limited to the above configuration, and may include, for example, a display unit such as a liquid crystal display and an operation unit that accepts user operations.

[0028] In addition, in the information processing device 1 according to this embodiment, the processing unit 11 reads and executes the program 12a stored in the memory unit 12, whereby the captured image acquisition unit 11a, the person detection unit 11b, the protective equipment detection unit 11c, the work judgment unit 11d, the protective equipment judgment unit 11e, the notification processing unit 11f, etc. are realized in the processing unit 11 as software functional units.

[0029] The captured image acquiring unit 11a acquires the captured image data output by the camera 3 via the interface unit 14, thereby performing a process of acquiring captured images of the floor in the clean room 100. The camera 3 takes images and outputs the captured images, for example, several times to several tens of times per second, and the captured image acquiring unit 11a acquires the captured images, for example, at the same frequency as the images taken by the camera 3, and temporarily stores the acquired captured images in the storage unit 12. Note that if the camera 3 has a communication function, the captured image acquiring unit 11a may communicate with the camera 3 via the communication unit 13 and acquire the captured images from the camera 3.

[0030] The human detection unit 11b performs processing to detect people in the clean room 100 based on the captured images acquired by the captured image acquisition unit 11a from the camera 3. In this embodiment, the human detection unit 11b detects people using a human detection model stored in the learning model storage unit 12b. The human detection unit 11b inputs data of the captured images acquired by the captured image acquisition unit 11a to the human detection model, and acquires a human detection result output by the human detection model in response to this. The human detection result includes coordinate information of a bounding box surrounding a person captured in the captured image, and the human detection unit 11b detects people by acquiring information about this bounding box.

[0031] The protective equipment detection unit 11c performs a process of detecting protective equipment worn by people in the clean room 100 based on the captured images acquired by the captured image acquisition unit 11a from the camera 3 and the person detection results by the person detection unit 11b. In the present embodiment, the protective equipment for which the information processing device 1 determines suitability includes, for example, a face guard, goggles, a facepiece, a gas mask, protective glasses, a helmet, an apron, a safety belt, protective gloves, and protective foot covers. However, the protective equipment is not limited to these, and any other protective equipment may be used as the target for suitability determination. In the present embodiment, the protective equipment detection unit 11c detects the protective equipment worn by each person using a protective equipment detection model stored in the learning model storage unit 12b. The protective equipment detection unit 11c extracts an image area in which the person detected by the person detection unit 11b is captured from the captured image data acquired by the captured image acquisition unit 11a, inputs the data of the extracted image area to the protective equipment detection model, and acquires a protective equipment detection result output by the protective equipment detection model in response to the extracted image area. When multiple people are detected in the captured image, the protective equipment detection unit 11c detects the protective equipment for each person separately.

[0032] The work determination unit 11d determines the work being performed in the clean room 100 and the work being performed by people in the clean room 100. In the information processing system according to this embodiment, the management device 5 manages schedule information for the work being performed in the clean room 100, and the management device 5 periodically transmits information about the work being performed in the clean room 100 to each information processing device 1. The work determination unit 11d of the information processing device 1 receives this information from the management device 5 and stores it in the work information storage unit 12c. When a determination becomes necessary, the work determination unit 11d reads the information stored in the work information storage unit 12c and determines which work is being performed in the clean room 100.

[0033] Furthermore, the work determination unit 11d uses the posture estimation model and the work determination model stored in the learning model storage unit 12b to individually determine the work being performed by people in the clean room 100. First, the work determination unit 11d extracts an image area in which a person detected by the person detection unit 11b is captured from the captured image data acquired by the captured image acquisition unit 11a, inputs the data of the extracted image area to the posture estimation model, and acquires a posture estimation result output by the posture estimation model in response to this. Next, the work determination unit 11d inputs the posture estimation result to the work determination model, and acquires a work determination result output by the work determination model in response to this. Note that if multiple people are detected in the captured image, the work determination unit 11d separately determines the work for each person.

[0034] In the information processing system according to the present embodiment, the information processing device 1 performs both task determination based on schedule information managed by the management device 5 and task determination based on images captured by the camera 3. However, this is not limited to this configuration, and the information processing device 1 may perform only one of these task determinations. However, by having the task determination unit 11d perform both task determinations and combine the determination results, it is expected that the task content can be determined in more detail. For example, even if it is determined based on images captured by the camera 3 that a person in the clean room 100 is operating the substrate processing apparatus 101, the required protective equipment may differ depending on whether the person is performing maintenance work on the substrate processing apparatus 101 or gas replacement work. Even in such cases, by combining task determination based on schedule information, it is expected that the task content, which is difficult to determine based on a person's posture alone, can be accurately determined. Furthermore, even if it is determined based on schedule information that gas replacement work is being performed, the required protective equipment may differ between the person actually performing the work and the person assisting the work. Even in such cases, by combining task determination based on images captured by the camera 3, it is expected that the task content can be determined individually for multiple people in the clean room 100.

[0035] The protective equipment determination unit 11e performs processing to determine whether the protective equipment worn by each person is appropriate, based on the detection result of the protective equipment for each person by the protective equipment detection unit 11c and the determination result of the work for each person by the work determination unit 11d. The information processing device 1 according to this embodiment stores, for example, a table in the memory unit 12, a correspondence between the types of work performed in the clean room 100 and the protective equipment required for each work. The protective equipment determination unit 11e obtains information on the required protective equipment for each person in the clean room 100 by referring to the table based on the work determined by the work determination unit 11d, and compares it with the protective equipment detected by the protective equipment detection unit 11c to determine whether each person is wearing appropriate protective equipment.

[0036] FIG. 3 is a schematic diagram illustrating an example of a required protective equipment table included in the information processing device 1 according to the present embodiment. The illustrated required protective equipment table stores information on "type of work" and "required protective equipment" in association with each other. "Type of work" may include, for example, "ladder work," "work involving gas," and "work involving liquid." For "ladder work," for example, "protective glasses," "safety harness," and "helmet" are associated as required protective equipment. For "work involving gas," for example, "protective glasses" (or "goggles") and "gas mask" are associated as required protective equipment. For "work involving liquid," for example, "goggles," "face mask," "apron," "protective gloves," and "protective foot covers" are associated as required protective equipment. Furthermore, for work types that do not fall under any of the above-mentioned "ladder work," "work involving gas," and "work involving liquid," for example, "protective glasses" are associated as required protective equipment for "work other than these." Note that the contents of the illustrated required protective equipment table are merely an example and are not limited thereto. The required protection table is determined in advance by, for example, the designer, administrator, or user of this information processing system.

[0037] For example, when the task determination unit 11d determines that the task being performed by a person in the clean room 100 is "work involving handling liquid," the protective equipment determination unit 11e acquires information from the required protective equipment table shown in the figure that the required protective equipment is five items: "goggles," "facepiece," "apron," "protective gloves," and "protective foot covers." The protective equipment determination unit 11e compares the protective equipment detected by the protective equipment detection unit 11c with the protective equipment stored in the required protective equipment table, and determines that the protective equipment for this person is appropriate if the person being determined is not wearing even one of the required protective equipment. The protective equipment determination unit 11e determines that the protective equipment for this person is inappropriate.

[0038] The notification processing unit 11f performs processing to notify the determination result of the protective equipment determiner 11e. In the present embodiment, the notification processing unit 11f can perform two types of notification: notification to the management device 5 and notification by lighting the LED lamp 4. For example, if the protective equipment determiner 11e determines that the protective equipment worn by a person in the clean room 100 is inappropriate, the notification processing unit 11f transmits a message notifying the management device 5 of this fact. At this time, the notification processing unit 11f transmits, along with the notification message, data on the captured image that was the basis for determining that the protective equipment is inappropriate to the management device 5. Furthermore, if the notification processing unit 11f determines that the protective equipment is inappropriate, the notification processing unit 11f outputs a control signal from the interface unit 14 to light the LED lamp 4 and notify the person in the clean room 100 that the protective equipment is inappropriate. Note that, although the information processing device 1 performs both notification to the management device 5 and notification by the LED lamp 4 in the present embodiment, this is not limited thereto, and the information processing device 1 may be configured to perform only one of the two types of notification. Furthermore, the notification processing unit 11f may transmit a notification message to the terminal device 7 when the information processing device 1 is configured to be able to communicate with the terminal device 7 of the user.

[0039] 4 is a block diagram showing an example of the configuration of the management device 5 according to this embodiment. The management device 5 according to this embodiment can be realized by installing a predetermined application program or the like in a general-purpose information processing device such as a personal computer or a server computer. The management device 5 is configured to include a processing unit 51, a storage unit 52, a communication unit 53, a display unit 54, and an operation unit 55. Note that although this embodiment will be described assuming that processing is performed by a single device, the processing of the management device 5 may be distributed among multiple devices.

[0040] The processing unit 51 is configured using an arithmetic processing device such as a CPU, an MPU, a GPU, or a quantum processor, a ROM, a RAM, etc. The processing unit 51 reads and executes a program 52a stored in the storage unit 52, thereby performing various processes such as a process of managing a work schedule in the clean room 100, a process of collecting information from a plurality of information processing devices 1, and a process of notifying that protective equipment is inappropriate.

[0041] The storage unit 52 is configured using a large-capacity storage device such as a hard disk, and stores a program 52a executed by the processing unit 51. The storage unit 52 also includes a safety management DB (database) 52b that stores and accumulates safety-related image data captured by a plurality of cameras 3 installed in the clean room 100, and a schedule DB 52c that stores schedule information related to work performed in the clean room 100.

[0042] In this embodiment, the program (computer program, program product) 52a is provided in a form recorded on a recording medium 98 such as a memory card or an optical disc, and the management device 5 reads the program 52a from the recording medium 98 and stores it in the storage unit 52. However, the program 52a may also be written to the storage unit 52, for example, during the manufacturing stage of the management device 5. Alternatively, for example, the program 52a may be distributed by a remote server device or the like and acquired by the management device 5 via communication. For example, the program 52a may be read by a writing device from the recording medium 98 and written to the storage unit 52 of the management device 5. The program 52a may be provided in a form distributed via a network or in a form recorded on the recording medium 98.

[0043] The safety management DB 52b is a database for collecting and storing safety-related images from among the images captured by the multiple cameras 3 installed in the clean room 100. In the present embodiment, when the management device 5 determines that protective equipment worn by a person working in the clean room 100 is inappropriate, for example, the management device 5 collects information related to this determination and stores it in the safety management DB 52b. The safety management DB 52b stores, in association with the image captured by the camera 3, various related information such as the date, time, and location at which the image was captured, identification information of the camera 3 that captured the image or the information processing device 1 that controls it, the work determined to be performed by the person in the captured image, the protective equipment determined to be worn by the person, and the reason for the determination that the protective equipment was inappropriate.

[0044] The schedule DB 52c stores schedule information for work to be performed in the clean room 100. For example, the schedule DB 52c stores information such as the type of work to be performed in the clean room 100, the scheduled start time of the work, and the scheduled end time of the work, in association with each other. The schedule DB 52c may also store information such as the number of people performing the work, and the identification information or names of the workers. After the work is performed in the clean room 100, the management device 5 may also store the actual start time and end time of the work in the schedule DB 52c.

[0045] The communication unit 53 transmits and receives data between a plurality of information processing devices 1 and a plurality of terminal devices 7 via a wired or wireless network. In the present embodiment, the communication unit 53 transmits information related to work performed in the clean room 100 to the information processing device 1, and receives information related to the judgment result of the suitability of the protective equipment from the information processing device 1. The communication unit 53 also receives information related to work performed in the clean room 100 from the terminal device 7, and transmits a message to the terminal device 7 notifying that the protective equipment is inappropriate. The communication unit 53 may transmit and receive various other information between the information processing device 1 and the terminal device 7. The communication unit 53 also communicates with a camera 3 having a communication function via a wired or wireless network, and can directly acquire captured images from the camera 3.

[0046] The display unit 54 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 51. In the present embodiment, the display unit 54 displays, for example, notification messages and captured images received from the information processing device 1. The management device 5 can also display various information including captured images stored in the safety management DB 52b on the display unit 54. The management device 5 can also display images captured by the camera 3 obtained via the communication unit 53 on the display unit 54 in real time.

[0047] The operation unit 55 accepts user operations and notifies the processing unit 51 of the accepted operations. For example, the operation unit 55 may be input devices such as a mouse and a keyboard, and these input devices may be configured to be detachable from the management device 5. Furthermore, for example, the operation unit 55 accepts user operations via an input device such as a mechanical button or a touch panel provided on the surface of the display unit 54.

[0048] The storage unit 52 may be an external storage device connected to the management device 5. The management device 5 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The management device 5 is not limited to the above configuration, and may not include, for example, the display unit 54 and the operation unit 55.

[0049] In addition, in the management device 5 according to this embodiment, the processing unit 51 reads and executes the program 52a stored in the memory unit 52, thereby realizing an information acquisition unit 51a, a notification processing unit 51b, a schedule information management unit 51c, a sheet reading unit 51d, etc. as software functional units in the processing unit 51.

[0050] The information acquisition unit 51a communicates with a plurality of information processing devices 1 via the communication unit 53, and performs processing to acquire information such as notifications regarding protective equipment and captured image data from the information processing devices 1. The information acquisition unit 51a associates the acquired information such as notifications and captured images with information such as date and time, location, identification information, and the presence or absence of protective measures, and stores the information in the safety management DB 52b.

[0051] The notification processing unit 51b performs processing to notify terminal devices 7 carried by people working in the clean room 100 that the protective equipment worn by the people working in the clean room is inappropriate. When the notification processing unit 51b is notified by the information processing device 1 that the protective equipment is inappropriate, it transmits a message to one or more terminal devices 7 notifying that the protective equipment worn by the people working in the clean room 100 is inappropriate. At this time, the notification processing unit 51b may transmit a message containing information regarding the location of people wearing inappropriate protective equipment, or data of a captured image of a person determined to be wearing inappropriate protective equipment, to the terminal device 7, and notify the worker carrying the terminal device 7 which person is inappropriate. Furthermore, when it is possible to distinguish between terminal devices 7 carried by people determined to be wearing inappropriate protective equipment, the notification processing unit 51b may transmit a message only to this terminal device 7.

[0052] The notification to the terminal device 7 may be sent directly by the information processing device 1. In this case, the notification can be sent to the worker more quickly. Whether the notification to the terminal device 7 is sent from the management device 5 or the information processing device 1 can be determined appropriately from the perspective of the size of the clean room 100 and the optimization of the information system.

[0053] The schedule information management unit 51c performs a process of managing schedule information related to work performed in the clean room 100. The schedule information management unit 51c communicates with the terminal device 7 used by a user such as a worker via the communication unit 53, and acquires information related to work performed in the substrate processing apparatus 101 that has been input by the user via the terminal device 7. Based on the information acquired from the terminal device 7, the schedule information management unit 51c associates information such as the work content, scheduled start time, and scheduled end time for the work performed in the clean room 100 with the information, and stores the information in the schedule DB 52c.

[0054] In this embodiment, a user such as an operator can handwrite information about work performed in the substrate processing apparatus 101 on a predetermined sheet, and the sheet can be photographed by a camera of the terminal device 7 and the photographed image can be transmitted to the management device 5. In this case, the schedule information management unit 51c of the management device 5 can provide the photographed image obtained from the terminal device 7 to a sheet reading unit 51d (described later), and the sheet reading unit 51d can obtain information read from the sheet. In this embodiment, the management device 5 reads characters and the like written on the sheet, but this is not limiting, and the sheet can also be read by the terminal device 7, and the information read by the terminal device 7 can be transmitted to the management device 5.

[0055] Furthermore, the schedule information management unit 51c performs processing to notify each information processing device 1 of work being performed in the clean room 100, based on the information stored in the schedule DB 52c. The schedule information management unit 51c may notify of work at a predetermined interval, for example, once every few seconds or once every few minutes, or may notify when the work content changes, for example, when the work starts or ends, or may notify of work in response to a request from the information processing device 1, for example.

[0056] The sheet reading unit 51d reads characters written on a captured image of a sheet acquired by the schedule information management unit 51c from the terminal device 7. The sheet reading unit 51d reads characters written on a sheet using a so-called OCR function. Reading characters using OCR may be performed, for example, by extracting the characteristics of the written characters and comparing them with a dictionary prepared in advance, or may be performed using a learning model that has undergone machine learning to infer characters from images of handwritten characters. Reading handwritten characters using OCR is an existing technology, so a detailed description will be omitted.

[0057] <Protective Gear Determination Process> Figures 5 and 6 are flowcharts illustrating an example of the protective gear determination process performed by the information processing device 1 according to this embodiment. The captured image acquisition unit 11a of the processing unit 11 of the information processing device 1 according to this embodiment acquires captured images of the clean room 100 captured by the camera 3 by exchanging information with the camera 3 via the interface unit 14 (step S1). The human detection unit 11b of the processing unit 11 inputs the captured image data acquired in step S1 to the human detection model stored in the learning model storage unit 12b (step S2). The human detection unit 11b acquires a human detection result output by the human detection model in response to the input in step S2 (step S3). The human detection unit 11b determines whether a human is captured in the captured image based on the human detection result acquired in step S3 (step S4). If a human is not captured in the captured image (S4: NO), the human detection unit 11b returns the process to step S1.

[0058] If a person is captured in the captured image (S4: YES), the processing unit 11 extracts an image area containing the person from the captured image based on the person detection result of the person detection model (Step S5). Next, the protective equipment detection unit 11c of the processing unit 11 inputs the person image area extracted in Step S5 to the protective equipment detection model stored in the learning model storage unit 12b (Step S6). The protective equipment detection unit 11c acquires the protective equipment detection result output by the protective equipment detection model in response to the input in Step S6 (Step S7).

[0059] Furthermore, the work determination unit 11d of the processing unit 11 inputs the image area of ​​the person extracted in step S5 to the posture estimation model stored in the learning model storage unit 12b (step S8). The work determination unit 11d acquires the posture estimation result output by the posture estimation model in response to the input in step S8 (step S9). Next, the work determination unit 11d inputs the posture estimation result acquired in step S9 to the work determination model stored in the learning model storage unit 12b (step S10). The work determination unit 11d acquires the work determination result output by the work determination model in response to the input in step S10 (step S11). The work determination unit 11d also reads information about the work being performed in the clean room 100 stored in the work information storage unit 12c (step S12).

[0060] The protective equipment determination unit 11e of the processing unit 11 determines whether the protective equipment worn by the person detected in the captured image is appropriate, by referring to a necessary protective equipment table pre-stored in the storage unit 12, based on the protective equipment detection result acquired in step S7, the task determination result acquired in step S11, and / or the task information read out in step S12 (step S13). At this time, the protective equipment determination unit 11e can determine whether the protective equipment is appropriate based on whether the person is wearing all of the protective equipment set as necessary in the necessary protective equipment table for the task being performed by the detected person. If the protective equipment is appropriate (S13: YES), the protective equipment determination unit 11e returns the process to step S1.

[0061] If the protective equipment is inappropriate (S13: NO), the notification processing unit 11f of the processing unit 11 communicates with the management device 5 via the communication unit 13 and notifies the management device 5 that the protective equipment worn by the person in the clean room 100 is inappropriate (step S14). The notification processing unit 11f also transmits the captured image acquired in step S1 to the management device 5 (step S15). The notification processing unit 11f outputs a control signal from the interface unit 14 to the LED lamp 4 to turn on the LED lamp 4 (step S16), and the process returns to step S1.

[0062] 7 is a flowchart showing an example of the procedure of schedule management processing performed by the management device 5 according to this embodiment. In the information processing system according to this embodiment, when performing work in the clean room 100, the worker inputs information such as a work schedule in advance to the terminal device 7. The information input by the terminal device 7 is transmitted to the management device 5. The worker can also input information by handwriting information such as a work schedule on a predetermined sheet and photographing it with the terminal device 7, and the terminal device 7 transmits the photographed image of the sheet to the management device 5.

[0063] The schedule information management unit 51c of the processing unit 51 of the management device 5 according to this embodiment determines whether information regarding work in the clean room 100 has been received from the terminal device 7 that has accepted input from the worker (step S31). If information regarding work has been received (S31: YES), the schedule information management unit 51c determines whether the received information is a photographed image of a sheet (step S32). If the received information is a photographed image of a sheet (S32: YES), the sheet reading unit 51d of the processing unit 51 performs OCR processing on the received photographed image of the sheet to read the information written on the sheet (step S33), and the process proceeds to step S34. If the received information is not a photographed image of a sheet (S32: NO), the processing unit 51 proceeds to step S34. The schedule information management unit 51c stores the information received from the terminal device 7 or the information read from the received photographed image in the schedule DB 52c (step S34), and returns the process to step S31.

[0064] If no information about work has been received from the terminal device 7 (S31: NO), the schedule information management unit 51c determines whether it is time to notify the information processing device 1 (step S35). In this embodiment, the management device 5 notifies each information processing device 1 in the clean room 100 of information about work being performed in the clean room 100 at an appropriate interval, such as once every few seconds or once every few minutes. The schedule information management unit 51c can determine whether it is time to notify based on whether a predetermined interval has elapsed since the previous notification. If it is not time to notify (S35: NO), the schedule information management unit 51c returns the process to step S31.

[0065] If the timing for notification has arrived (S35: YES), the schedule information management unit 51c reads out the schedule information stored in the schedule DB 52c (step S36). The schedule information management unit 51c determines the type of work being performed in the clean room 100 based on the read out schedule information, notifies each information processing device 1 in the clean room 100 of the information on the type of work (step S37), and returns the process to step S31.

[0066] 8 is a flowchart showing an example of the procedure of the protective equipment determination process performed by the management device 5 according to this embodiment. The information acquisition unit 51a of the processing unit 51 of the management device 5 according to this embodiment determines whether or not a notification that the protective equipment is inappropriate has been received from any of the information processing devices 1 via the communication unit 53 (step S51). If no notification has been received (S51: NO), the information acquisition unit 51a waits until a notification is received from any of the information processing devices 1. If a notification has been received (S51: YES), the information acquisition unit 51a stores information included in the notification from the information processing device 1 and information received together with the notification, such as various information such as captured images, in the safety management DB 52b (step S52). Next, the notification processing unit 51b of the processing unit 51 transmits a message via the communication unit 53 to one or more terminal devices 7 notifying them that the protective equipment is inappropriate (step S53), and the process ends.

[0067] <Summary> In the information processing system according to the present embodiment configured as described above, the information processing device 1 acquires images captured by the camera 3 inside the clean room 100, detects protective equipment worn by people captured in the captured images, determines whether the protective equipment is appropriate based on a combination of the detected protective equipment and the work performed by the person, and notifies the management device 5 or people in the clean room 100 of the suitability determination result. As a result, the information processing system according to the present embodiment is expected to prevent, for example, people working without wearing appropriate protective equipment from being exposed to danger, and is expected to realize safety monitoring based on the images captured by the camera 3.

[0068] Furthermore, in the information processing system according to this embodiment, the information processing device 1 acquires schedule information for work to be performed in the clean room 100, and acquires information regarding protective equipment required for the work, for example, from a required protective equipment table. Based on this acquired information, the information processing device 1 determines the appropriateness of protective equipment based on whether a person captured in a photographed image at the time the work is to be performed in the clean room 100 is wearing the protective equipment required for the work. In this way, the information processing system according to this embodiment can determine the work to be performed in the clean room 100 based on the schedule information and determine the appropriateness of protective equipment.

[0069] Furthermore, in the information processing system according to this embodiment, the information processing device 1 determines the work being performed by a person captured in a photographed image, for example, using a posture estimation model and a work estimation model, and determines the suitability of protective equipment based on whether the person is wearing protective equipment required for the determined work. In this way, the information processing system according to this embodiment can determine the suitability of protective equipment for each person in the clean room 100.

[0070] Furthermore, in the information processing system according to this embodiment, for example, the management device 5 or the terminal device 7 reads information from a sheet on which information about work is entered, and stores and manages the read information in the schedule DB 52c. Furthermore, in the information processing system according to this embodiment, the management device 5 accepts input of information about work from a worker or the like via the terminal device 7 and stores the information in the schedule DB 52c. The information processing device 1 obtains the information read from the sheet or the information input via the terminal device 7 from the management device 5, which manages this information. As a result, the information processing system according to this embodiment is expected to facilitate the input of information about work.

[0071] Furthermore, in the information processing system according to this embodiment, when the information processing device 1 determines that the protective equipment worn by a person in the clean room 100 is inappropriate, the management device 5 stores the captured image used for the determination in the safety management DB 52b. As a result, the information processing system according to this embodiment can save, as a captured image, the state inside the clean room 100 when it is determined that the protective equipment is inappropriate, and is expected to contribute to safety measures inside the clean room 100.

[0072] Furthermore, in the information processing system according to this embodiment, a camera that measures the distance (depth) to an object can be used as the camera 3 installed in the clean room 100. By using the distance (depth) information together with the RGB image, the information processing system according to this embodiment is expected to improve the accuracy of processes such as person detection, protective equipment detection, posture estimation, and behavior determination based on the captured image.

[0073] Furthermore, in the information processing system according to this embodiment, for example, if a person in the clean room 100 is performing work using a stepladder and the protective equipment worn by the person is protective glasses, a safety belt, and a helmet, the information processing device 1 determines that the protective equipment is appropriate. For example, if a person in the clean room 100 is performing work involving gas and the protective equipment worn by the person is protective glasses or goggles and a gas mask, the information processing device 1 determines that the protective equipment is appropriate. For example, if a person in the clean room 100 is performing work involving liquid and the protective equipment worn by the person is goggles, a facepiece, an apron, protective gloves, and protective foot covers, the information processing device 1 determines that the protective equipment is appropriate. For example, even if a person captured in a captured image is not performing work, if the person is not wearing protective glasses, the information processing device 1 determines that the protective equipment is inappropriate. As a result, the information processing system according to this embodiment is expected to accurately determine the appropriateness of the protective equipment worn by a person working in the clean room 100.

[0074] In this embodiment, a plurality of cameras 3 and information processing devices 1 are installed inside the clean room 100, but this is not limiting, and at least one camera 3 and one information processing device 1 may be installed. The information processing device 1 may also be installed outside the clean room 100. In this embodiment, one camera 3 is connected to one information processing device 1, but this is not limiting, and a plurality of cameras 3 may be connected to one information processing device 1.

[0075] In addition, in the information processing system of this embodiment, some or all of the processing performed by information processing device 1 may be performed by management device 5, and conversely, some or all of the processing performed by management device 5 may be performed by information processing device 1.

[0076] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0077] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, the claims use a format in which a claim references two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used.

[0078] DESCRIPTION OF SYMBOLS 1 Information processing device (computer) 3 Camera 4 LED lamp 5 Management device (information processing device, computer) 7 Terminal device 11 Processing unit 11a Photographed image acquisition unit 11b Person detection unit 11c Protective equipment detection unit 11d Work determination unit 11e Protective equipment determination unit 11f Notification processing unit 12 Memory unit 12a Program (computer program) 12b Learning model memory unit 12c Work information memory unit 13 Communication unit 14 Interface unit 51 Processing unit 51a Information acquisition unit 51b Notification processing unit 51c Schedule information management unit 51d Sheet reading unit 52 Memory unit 52a Program (computer program) 52b Safety management DB 52c Schedule DB 53 Communication unit 54 Display unit 55 Operation unit 98, 99 Recording medium 101 Substrate processing apparatus

Claims

1. An information processing method comprising: an information processing device that acquires an image captured by a camera inside a room; detects protective equipment worn by a person captured in the acquired image based on the acquired image; judges the suitability of the protective equipment based on a combination of the detected protective equipment and the work performed by the person; and notifies the judgment result.

2. The information processing method of claim 1, further comprising the steps of: acquiring information regarding work to be performed in the room; acquiring information regarding protective equipment required for the work; and, based on the acquired information regarding the work to be performed in the room and the information regarding the necessary protective equipment, determining whether the protective equipment is appropriate based on whether a person captured in the captured image is wearing the necessary protective equipment when the work is being performed.

3. The information processing method according to claim 2, further comprising: determining the work being performed by the person photographed in the captured image; and determining whether the protective equipment required for the determined work is worn or not, and determining whether the protective equipment is appropriate.

4. The information processing method according to claim 2, further comprising the step of acquiring information about work to be performed in the room by reading a sheet on which information is written.

5. The information processing method according to claim 2, further comprising the step of acquiring information about work being performed in the room by accepting an input via a terminal device.

6. The information processing method according to claim 1, further comprising the steps of: if the protective equipment is determined to be inappropriate, storing the captured image used for the determination in a memory unit.

7. The information processing method according to claim 1, further comprising the step of determining that the protective equipment is appropriate if the work involves the use of a stepladder and the protective equipment is protective glasses, a safety belt and a helmet.

8. The information processing method according to claim 1, further comprising the step of determining that the protective equipment is appropriate if the work involves handling gas and the protective equipment is a combination of protective glasses or goggles and a gas mask.

9. The information processing method according to claim 1, further comprising the step of determining that the protective equipment is appropriate if the work involves handling liquid and the protective equipment is goggles, a facepiece, an apron, protective gloves, and protective foot covers.

10. The information processing method according to claim 1, further comprising determining that the protective equipment is inappropriate if the person photographed in the captured image is not wearing protective glasses, even if the person is not performing work.

11. The information processing method according to claim 1, wherein the camera is a camera that measures a distance to an object.

12. A computer program that causes a computer to execute the following processes: acquire images taken by a camera of an interior room; detect protective equipment worn by a person appearing in the acquired images; determine the suitability of the protective equipment based on a combination of the detected protective equipment and the work performed by the person; and notify the user of the result of the determination.

13. An information processing device having a processing unit, which acquires an image of a room captured by a camera, detects protective equipment worn by a person captured in the acquired image based on the acquired image, judges the suitability of the protective equipment based on a combination of the detected protective equipment and the work performed by the person, and notifies the judgment result.

Images