Information processing method, computer program, and information processing device

Abstract

The invention provides an information processing method, a computer program, and an information processing device, which are expected to realize safe monitoring and the like of images captured by a camera. In the information processing method according to the present embodiment, a captured image in which a camera captures an indoor image is acquired, a protective device worn by a person shown in the captured image is detected on the basis of the acquired captured image, and a job is processed on the basis of a combination of the detected protective device and the job performed by the person. Whether the protector is appropriate is determined, and the determination result is notified.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing method, a computer program, and an information processing apparatus. BACKGROUND

[0002] In Patent Literature 1, a monitoring system is proposed in which an authentication controller performs authentication processing based on authentication information received from an authentication terminal through electric field communication, an image recognition server recognizes a person included in a photographing region of an image captured by a camera, and the person authenticated by the authentication controller and the person image-recognized by the image recognition server are collated based on position information of the authentication terminal and coordinate information of the photographing region of the camera.

[0003] Patent Literature 1: Japanese Patent Application Publication No. 2012-8802 SUMMARY

[0004] The present disclosure provides an information processing method, a computer program, and an information processing apparatus that can expect to achieve monitoring and the like based on safety of a captured image by a camera.

[0005] In an information processing method according to one embodiment, a captured image obtained by a camera capturing an indoor space is acquired, a protector worn by a person appearing in the captured image is detected based on the acquired captured image, it is determined whether the protector is appropriate or not based on a combination of the detected protector and a work performed by the person, and the determination result is notified.

[0006] According to the present disclosure, it is possible to expect to achieve monitoring and the like based on safety of a captured image by a camera. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a schematic diagram for explaining an outline of an information processing system according to the present embodiment.

[0008] Figure 2 is a block diagram showing one configuration example of an information processing apparatus according to the present embodiment.

[0009] Figure 3 is a schematic diagram showing one example of a necessary protector table possessed by the information processing apparatus according to the present embodiment.

[0010] Figure 4 is a block diagram showing one configuration example of a management apparatus according to the present embodiment.

[0011] Figure 5 is a flowchart showing one example of steps of protector determination processing performed by the information processing apparatus according to the present embodiment.

[0012] Figure 6is a flowchart showing an example of steps of the protector determination processing by the information processing device according to the present embodiment.

[0013] Figure 7 is a flowchart showing an example of steps of the schedule management processing by the management device according to the present embodiment.

[0014] Figure 8 is a flowchart showing an example of steps of the protector determination processing by the management device according to the present embodiment. DETAILED DESCRIPTION

[0015] An example of an information processing system according to the present embodiment will be described below with reference to the accompanying drawings. Furthermore, the present disclosure is not limited to these examples, and is intended to include the meaning indicated by the claims and equivalent thereto, and all modifications within the scope.

[0016] < SYSTEM OVERVIEW >

[0017] Figure 1 is a schematic diagram for explaining an overview of the information processing system according to the present embodiment. The information processing system according to the present embodiment is, for example, a system that performs safety monitoring in a clean room 100 in which a plurality of substrate processing devices 101 are provided. The information processing system is configured to include a plurality of cameras 3 disposed at appropriate positions in the clean room 100, a plurality of information processing devices 1 that control the respective cameras 3, and a management device 5 that manages information obtained from the plurality of information processing devices 1.

[0018] In the clean room 100, various operations such as an operation of supplying a gas to the substrate processing device 101, an operation of supplying a chemical liquid, and a maintenance operation of the device are performed. When performing these operations, the operator is required to appropriately wear various protective devices such as protective glasses, a safety belt, a helmet, goggles, a gas mask, an apron, protective gloves, or protective shoe covers, in order to ensure the safety of the operator.

[0019] In the information processing system according to the present embodiment, the inside of the clean room 100 is imaged by a plurality of cameras 3 disposed at appropriate positions within the clean room 100, and each information processing device 1 performs a process of determining whether or not the protective equipment of the worker is appropriate based on the captured image captured by each camera 3. The information processing device 1 detects a person within the clean room 100 based on the captured image of the camera 3, and determines which kind of protective equipment the detected person is wearing. In addition, in the information processing system according to the present embodiment, schedule information related to the work performed within the clean room 100 is managed by the management device 5, and information related to the work performed within the clean room 100 at the point in time is transmitted from the management device 5 to the information processing device 1. Each information processing device 1 compares the protective equipment necessary for the work at the point in time and the protective equipment worn by the person appearing in the captured image based on the information from the management device 5, and determines whether or not the protective equipment is appropriate.

[0020] In a case where it is determined that the protective equipment worn by the person performing the work within the clean room 100 is inappropriate, the information processing device 1 notifies the management device 5 of the gist, and performs notification to the worker or the like by light or sound or the like. In this case, the information processing device 1 transmits the captured image in which it is determined that the protective equipment is inappropriate to the management device 5 together with the notification of the gist that the protective equipment is inappropriate. The management device 5 notified of the gist that the protective equipment is inappropriate from the information processing device 1 notifies the terminal device or the like held by the worker within the clean room 100 of the gist that the protective equipment is inappropriate. In addition, the management device 5 acquires the captured image at the time when it is determined that the protective equipment is inappropriate from the information processing device 1, and stores the information such as the date and time and the work content in a database, for example.

[0021] In addition, in the present embodiment, the camera 3 disposed within the clean room 100 is a camera capable of imaging an image of a general visible light region and measuring a distance from the camera 3, a so-called depth camera. Therefore, the data of the captured image obtained by the imaging of the camera 3 includes data of the color of RGB and data indicating the distance from the camera 3. However, the camera 3 can also not measure the distance and only image the image of the visible light region. In addition, a camera that images the visible light region and a camera that measures the distance (which can be a sensor or a LiDAR (Light Detection And Ranging: laser radar) or the like) can be provided individually, and the information processing device 1 acquires the data obtained from the two cameras individually and performs a process. In the following description, for example, in a case where it is described as the captured image of the camera 3, the data of the color of RGB based on the imaging of the visible light region and the data of the distance (depth) from the camera 3 can be included in the data of the captured image.

[0022] In addition, in the information processing system according to the present embodiment, the schedule information of the work in the clean room 100 is managed by the management device 5. The management device 5 notifies each information processing device 1 of the work to be performed in the clean room 100 on the basis of the schedule information and the date and time of the time point and the like. Each information processing device 1, for example, stores in advance a correspondence relationship of the kind of work that can be performed in the clean room 100 and the protective equipment necessary for each work. The information processing device 1 determines the protective equipment necessary in accordance with the kind of work at the time point notified from the management device 5, and determines whether the protective equipment worn by the person performing the work in the clean room 100 is appropriate or not.

[0023] In addition, in the information processing system according to the present embodiment, the management device 5 manages the schedule of the work in the clean room 100, and thus receives the input of information such as work reservation from the worker or the like via the terminal device 7. The terminal device 7 can use, for example, a tablet terminal device or a device such as a smartphone. The terminal device 7, for example, displays an input screen of work reservation or the like on a display section and receives the input of information from the worker, and transmits the received information to the management device 5. In addition, the terminal device 7, for example, can also transmit an image obtained by an attached camera capturing the content of the worker or the like handwriting filled in a form (paper) on which work reservation or the like is filled to the management device 5. The management device 5 can also acquire information of work reservation or the like by reading a character string described in a captured image of the form received from the terminal device 7 by a function of so-called OCR (Optical Character Recognition). The management device 5 generates and stores schedule information of the work performed in the clean room 100 on the basis of the information acquired via the terminal device 7, and notifies each information processing device 1 of information related to the work performed in the clean room 100.

[0024] <Device Structure>

[0025] Figure 2 is a block diagram showing one configuration example of the information processing device 1 according to the present embodiment. The information processing device 1 according to the present embodiment is an information processing device called an edge computer in the field of IoT (Internet of Things), for example. The information processing device 1 is configured to include a processing section 11, a storage section 12, a communication section 13, and an interface section 14, and the like. In the present embodiment, one camera 3 is connected to one information processing device 1, but is not limited thereto, and a plurality of cameras 3 can be connected to one information processing device 1.

[0026] The processing unit 11 is configured to use a computing device such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), or quantum processor, as well as ROM (Read Only Memory) and RAM (Random Access Memory). The processing unit 11 reads and executes the program 12a stored in the storage unit 12 to perform various processes, such as detecting people and protective equipment within the cleanroom 100 based on images captured by the camera 3, and determining whether the protective equipment worn by the detected people is appropriate.

[0027] The storage unit 12 is configured to use a large-capacity storage device, such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11, as well as various data necessary for the processing by the processing unit 11. In this embodiment, the storage unit 12 stores program 12a executed by the processing unit 11. Furthermore, the storage unit 12 includes a learning model storage unit 12b that stores information related to one or more learning models obtained through pre-processed machine learning, and a job information storage unit 12c that stores information related to the work performed within the cleanroom 100.

[0028] In this embodiment, the program (computer program, program product) 12a is provided in the form of a recording medium 99 such as a memory card or optical disc, and the information processing device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may also be written to the storage unit 12 during the manufacturing stage of the information processing device 1, for example. Additionally, the program 12a may be a program distributed by a remote server device or the like, obtained by the information processing device 1 through communication. For example, the program 12a may also be read from the recording medium 99 by a writing device and written to the storage unit 12 of the information processing device 1. The program 12a may also be provided by means of distribution via a network, or it may be provided by means of recording on the recording medium 99.

[0029] The learning model storage unit 12b stores in advance information related to the learning models used in the various processes described above by the information processing apparatus 1 according to this embodiment. The information related to the learning models stored in the learning model storage unit 12b may include, for example, information indicating the structure of the learning model and the values ​​of the internal parameters of the learning model predetermined through machine learning. In this embodiment, the information processing apparatus 1 uses multiple learning models to determine whether a protective device is appropriate; therefore, information related to the multiple learning models that have been learned in advance through machine learning is stored in the learning model storage unit 12b.

[0030] For example, the information processing device 1 stores information related to a person detection model that detects people based on images captured by the camera 3 in the learning model storage unit 12b. The person detection model is a learning model obtained by performing machine learning in advance, using data from the captured image as input and detecting people reflected in the captured image. The person detection model may employ, for example, a learning model with an image structure that processes CNNs. Furthermore, in this embodiment, the data from the captured image input to the person detection model is data from four channels, including RGB data and depth data, but it could also be data from three channels of RGB. The person detection result output by the person detection model can be set as the coordinates of a rectangle, or bounding box, enclosing the region in the captured image where a person is reflected. The person detection model is generated in advance by performing supervised machine learning processing, using learning data (supervised data) that establishes corresponding learning data by constructing a corresponding learning dataset (supervised data) from the image data and positive value data representing the region in the image data where a person is reflected. Additionally, the person detection model can utilize existing learned models such as YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector).

[0031] For example, the information processing device 1 stores information related to a protective device detection model that detects protective devices worn by a person based on a photographed image of that person in the learning model storage unit 12b. The protective device detection model is a learning model obtained by performing machine learning in advance, using data from a photographed image of a person as input and detecting the protective devices worn by that person. Furthermore, in this embodiment, the data input to the protective device detection model is data extracted from the image captured by the camera 3, showing the image region of the person detected by the photographed image. The output of the protective device detection model is, for example, an estimate of whether a specific protective device (such as protective glasses) is being worn. In this embodiment, the information processing device 1 detects multiple types of protective devices, therefore employing the following structure: multiple protective device detection models for each type of protective device, or a single protective device detection model outputting multiple estimates corresponding to each protective device. The protective device detection model is pre-generated by performing so-called supervised machine learning processing, using image data of a person and flags indicating whether the person in the image data is wearing a specific protective device. The protective equipment detection model can employ a learning model that can be referred to as a classification model or a multi-valued classification model, etc.

[0032] For example, the information processing device 1 stores information related to a pose estimation model that estimates the person's pose based on a captured image of a person in the learning model storage unit 12b. The pose estimation model is a learning model obtained by performing machine learning in advance, taking data from a captured image of a person as input and outputting an estimation result of the person's pose. Furthermore, in this embodiment, the data input to the pose estimation model is data extracted from the image captured by the camera 3, representing the image region of the person detected by the captured image detection model. The estimation result output by the pose estimation model is information representing, for example, the head, waist, and joints of the hands and feet of a person using multiple points, and the pose estimation model outputs the coordinate information of these multiple points. The pose estimation model can utilize existing learning models such as OpenPose. Since the pose estimation model is existing technology, detailed descriptions of the learning method, etc., are omitted.

[0033] For example, the information processing device 1 stores information related to a task estimation model that estimates the task being performed by the person based on the estimation results of the person's posture in the learning model storage unit 12b. The task estimation model is a learning model obtained by performing machine learning in advance, which takes as input the estimation results of the person's posture from the posture estimation model, i.e., information such as the joints of the person's body represented by multiple points, and outputs the estimation results of the task being performed by the person. The estimation results output by the task estimation model are, for example, an estimate of whether a specific task (such as supplying gas to the substrate processing device 101) is performed. In this embodiment, the information processing device 1 estimates multiple types of tasks, and therefore adopts the following structure: it has multiple task estimation models for each type of task, or a single task estimation model outputs multiple estimation values ​​corresponding to each task. The task estimation model is generated in advance by performing so-called supervised machine learning processing, which establishes corresponding learning data (supervised data) by using data representing the person's posture and flags indicating whether the person is performing a specific task. The task estimation model can employ a learning model that can be called a classification model or a multi-value classification model, etc.

[0034] The communication unit 13 communicates with the management device 5 via a wired or wireless network. The communication unit 13 sends data provided by the processing unit 11 to the management device 5 and receives data sent from the management device 5, providing it to the processing unit 11. In this embodiment, the communication unit 13 receives information from the management device 5 related to operations performed within the cleanroom 100, and sends the determination results of whether protective equipment is appropriate, as well as data from images captured by the camera 3, to the management device 5.

[0035] The interface unit 14 is connected to devices such as the camera 3 and the LED (Light Emitting Diode) lamp 4 via a communication line or signal line, and transmits and receives data or signals with these devices. In this embodiment, the interface unit 14 acquires the image data output by the camera 3 and sends it to the processing unit 11. Additionally, the interface unit 14 outputs an on / off control signal from the processing unit 11 to the LED lamp 4 to switch the LED lamp 4 on / off. Furthermore, in this embodiment, the camera 3 is connected to the interface unit 14 of the information processing device 1, but it is not limited to this. For example, if the camera 3 has a communication function, the information processing device 1 may be configured to communicate with the camera 3 via a communication unit 13. In this case, the information processing device 1 may also acquire the image data output by the camera 3 from the communication unit 13 via a wired or wireless network. The same applies to the LED lamp 4.

[0036] Furthermore, the storage unit 12 may be an external storage device connected to the information processing device 1. Additionally, the information processing device 1 may be a multi-computer system comprising multiple computers, or a virtual machine constructed virtually by software. Furthermore, the information processing device 1 is not limited to the above-described structure and may also include a display unit such as a liquid crystal display and an operation unit for receiving user input.

[0037] Furthermore, in the information processing apparatus 1 according to this embodiment, the processing unit 11 reads and executes the program 12a stored in the storage unit 12, thereby implementing the image acquisition unit 11a, the human detection unit 11b, the protective device detection unit 11c, the work determination unit 11d, the protective device determination unit 11e, and the notification processing unit 11f as software functional units.

[0038] The image acquisition unit 11a acquires the image data output by the camera 3 via the interface unit 14, thereby processing the acquisition of images obtained by photographing the floor within the cleanroom 100. The camera 3, for example, takes pictures at a frequency of several to tens of times per second and outputs images. The image acquisition unit 11a acquires images at the same frequency as the camera 3 and temporarily stores the acquired images in the storage unit 12. Furthermore, if the camera 3 has a communication function, the image acquisition unit 11a can also communicate with the camera 3 via the communication unit 13 to acquire images from the camera 3.

[0039] The human detection unit 11b performs human detection processing based on the captured images obtained by the image acquisition unit 11a from the camera 3. In this embodiment, the human detection unit 11b uses a human detection model stored in the learning model storage unit 12b to detect people. The human detection unit 11b inputs the captured image data obtained by the image acquisition unit 11a into the human detection model, and thereby obtains the human detection result output by the human detection model. The human detection result includes the coordinate information of the bounding box surrounding the person reflected in the captured image, and the human detection unit 11b detects people by obtaining the information of this bounding box.

[0040] The protective equipment detection unit 11c detects the protective equipment worn by people in the cleanroom 100 based on the images captured by the image acquisition unit 11a from the camera 3 and the detection results of people by the person detection unit 11b. In this embodiment, the protective equipment that the information processing device 1 determines to be appropriate includes, for example, face shields, goggles, face masks, gas masks, protective glasses, helmets, aprons, seat belts, protective gloves, and protective shoe covers, but is not limited to these; any other protective equipment can be used as the object of appropriateness determination. In this embodiment, the protective equipment detection unit 11c uses the protective equipment detection model stored in the learning model storage unit 12b to detect the protective equipment worn by each person. The protective equipment detection unit 11c extracts the image region of the person detected by the person detection unit 11b from the image captured by the image acquisition unit 11a, inputs the extracted image region data into the protective equipment detection model, and obtains the protective equipment detection result output by the protective equipment detection model. Furthermore, if multiple people are detected in the captured image, the protective equipment detection unit 11c performs protective equipment detection on each individual.

[0041] The job determination unit 11d determines the jobs performed within the cleanroom 100 and the jobs being performed by personnel within the cleanroom 100. In the information processing system of this embodiment, the management device 5 manages the schedule information of jobs performed within the cleanroom 100, and periodically sends information related to jobs performed within the cleanroom 100 to each information processing device 1. The job determination unit 11d of the information processing device 1 receives this information from the management device 5 and stores it in the job information storage unit 12c. At the time when determination is required, the job determination unit 11d reads the information stored in the job information storage unit 12c to determine what type of job is being performed within the cleanroom 100.

[0042] Furthermore, the job determination unit 11d uses the pose estimation model and the job determination model stored in the learning model storage unit 12b to individually determine the jobs being performed by people inside the cleanroom 100. First, the job determination unit 11d extracts image regions reflecting people detected by the person detection unit 11b from the image data acquired by the image acquisition unit 11a, and inputs the extracted image region data into the pose estimation model to obtain the pose estimation result output by the pose estimation model. Next, the job determination unit 11d inputs the pose estimation result into the job determination model to obtain the job determination result output by the job determination model. Moreover, when multiple people are detected in the image, the job determination unit 11d determines the job for each person separately.

[0043] In the information processing system of this embodiment, the information processing device 1 performs both job determination based on schedule information managed by the management device 5 and job determination based on images captured by the camera 3. However, it is not limited to this and may be a structure that performs job determination based on only one of these two methods. However, by performing job determination based on both methods and combining the determination results, the job determination unit 11d can determine the job content in greater detail. For example, even if the image captured by the camera 3 indicates that a person in the cleanroom 100 is operating the board processing device 101, the necessary protective equipment may differ depending on whether it is a maintenance operation or a gas replacement operation of the board processing device 101. Even in such cases, by combining the job determination based on schedule information, it is possible to determine the job content that is difficult to determine based solely on the person's posture with high accuracy. In addition, for example, even if the schedule information indicates that a gas replacement operation is in progress, the necessary protective equipment may differ between the person actually performing the operation and the person assisting in the operation. Even in such cases, by combining the job determination based on images captured by the camera 3, it is possible to perform individual job determination for multiple people in the cleanroom 100.

[0044] The protective equipment determination unit 11e determines whether the protective equipment worn by each person is appropriate based on the detection results of the protective equipment detection unit 11c for each person and the work determination results of the work for each person. The information processing device 1 of this embodiment stores, for example, a table in the storage unit 12 showing the types of work performed in the cleanroom 100 and the corresponding protective equipment required for each work. For each person in the cleanroom 100, the protective equipment determination unit 11e, based on the work determined by the work determination unit 11d, refers to the table to obtain information about the necessary protective equipment, compares it with the protective equipment detected by the protective equipment detection unit 11c, and thus determines whether each person is wearing appropriate protective equipment.

[0045] Figure 3This is a schematic diagram illustrating an example of a list of necessary protective equipment included in the information processing apparatus 1 according to this embodiment. The illustrated list of necessary protective equipment stores information corresponding to the "type of work" and the "necessary protective equipment." The "type of work" may include, for example, "ladder work," "gas handling work," and "liquid handling work." For "ladder work," necessary protective equipment includes, for example, "protective glasses," "safety belt," and "helmet." For "gas handling work," necessary protective equipment includes, for example, "protective glasses" (or "goggles") and "gas mask." For "liquid handling work," necessary protective equipment includes, for example, "goggles," "face mask," "apron," "protective gloves," and "protective shoe covers." Furthermore, if the type of work does not fall under any of the above categories such as "ladder work," "gas handling work," and "liquid handling work," then for "other than these types of work," "protective glasses" may be listed as a necessary protective equipment. Furthermore, the contents of the required protection list shown in the illustration are merely an example and are not limited to this. The required protection list may be predetermined by, for example, the designer, administrator, or user of this information processing system.

[0046] For example, if the work determination unit 11d determines that the work being performed by a person in the cleanroom 100 is "liquid handling," the protective equipment determination unit 11e retrieves information from the illustrated necessary protective equipment list that the necessary protective equipment includes "goggles," "face mask," "apron," "protective gloves," and "protective shoe covers." The protective equipment determination unit 11e compares the protective equipment detected by the protective equipment detection unit 11c with these protective equipment stored in the necessary protective equipment list. If the person being determined is wearing all the necessary protective equipment, the protective equipment is deemed appropriate for that person. If the person being determined is not wearing any of the necessary protective equipment, the protective equipment determination unit 11e determines that the protective equipment is inappropriate for that person.

[0047] The notification processing unit 11f processes the determination result of the protective equipment determination unit 11e. In this embodiment, the notification processing unit 11f can perform two types of notifications: notification to the management device 5 and notification based on the illumination of the LED light 4. For example, if the protective equipment determination unit 11e determines that the protective equipment worn by the person in the cleanroom 100 is inappropriate, the notification processing unit 11f sends a message to the management device 5 notifying it of this. At this time, the notification processing unit 11f sends the data of the original captured image indicating that the protective equipment is inappropriate, along with the notification message, to the management device 5. Alternatively, if the protective equipment is determined to be inappropriate, the notification processing unit 11f outputs a control signal from the interface unit 14 to illuminate the LED light 4 and notify the person in the cleanroom 100 of the inappropriate protective equipment. Furthermore, in this embodiment, the information processing device 1 performs both notification to the management device 5 and notification based on the LED light 4, but it is not limited to this and may also be structured to perform only one of the notifications. Furthermore, if the information processing device 1 is configured to communicate with the user's terminal device 7, the notification processing unit 11f can also send notification messages to the terminal device 7.

[0048] Figure 4 This is a block diagram illustrating a structural example of the management device 5 according to this embodiment. The management device 5 according to this embodiment can be implemented by installing a specified application program on a general information processing device such as a personal computer or a server computer. The management device 5 is configured to include a processing unit 51, a storage unit 52, a communication unit 53, a display unit 54, and an operation unit 55. Furthermore, in this embodiment, processing is described as being performed on a single device, but the management device 5 can also be processed by multiple devices distributed among them.

[0049] The processing unit 51 is configured to use a computing device such as a CPU, MPU, GPU or quantum processor, ROM and RAM. The processing unit 51 reads and executes the program 52a stored in the storage unit 52, thereby performing various processes such as managing the work schedule within the cleanroom 100, collecting information from multiple information processing devices 1, and notifying the staff of inappropriate protective equipment.

[0050] The storage unit 52 is configured to use a large-capacity storage device such as a hard disk. The storage unit 52 stores the program 52a executed by the processing unit 51. In addition, the storage unit 52 is provided with a security management database 52b and a schedule database 52c. The security management database 52b stores and accumulates data such as safety-related images captured by multiple cameras 3 installed in the cleanroom 100, and the schedule database 52c stores schedule information related to the work performed in the cleanroom 100.

[0051] In this embodiment, the program (computer program, program product) 52a is provided in the form of a recording medium 98 such as a memory card or optical disc, and the management device 5 reads the program 52a from the recording medium 98 and stores it in the storage unit 52. However, the program 52a may also be written to the storage unit 52, for example, during the manufacturing stage of the management device 5. Alternatively, the program 52a may be a program distributed by a remote server device or the like, obtained by the management device 5 through communication. For example, the program 52a may also be read from the recording medium 98 by a writing device and written to the storage unit 52 of the management device 5. The program 52a may be provided either by distribution via a network or by being recorded on the recording medium 98.

[0052] The safety management DB52b is a database used to collect and store safety-related data from images captured by multiple cameras 3 installed within the cleanroom 100. In this embodiment, for example, if the protective equipment worn by a person working in the cleanroom 100 is deemed inappropriate, the management device 5 collects information related to this determination and stores it in the safety management DB52b. The safety management DB52b, for example, establishes and stores various related information such as the date, time, and location of the captured images, the identification information of the camera 3 or the information processing device 1 controlling the camera 3, the work being performed by the person depicted in the captured image, the protective equipment being worn by that person, and the reason for the inappropriateness determination.

[0053] The schedule DB52c stores schedule information for operations performed within the cleanroom 100. For example, the schedule DB52c stores information such as the type of scheduled operation, the scheduled start time, and the scheduled end time for each operation within the cleanroom 100. Additionally, the schedule DB52c may store information such as the number of people performing the operation, operator identification information, or names. Furthermore, the management device 5 can also store the actual start and end times of each operation performed within the cleanroom 100 in the schedule DB52c.

[0054] The communication unit 53 transmits and receives data with multiple information processing devices 1 and multiple terminal devices 7 via wired or wireless networks. In this embodiment, the communication unit 53 sends information related to operations performed in the cleanroom 100 to the information processing devices 1 and receives information related to the determination of the suitability of protective equipment from the information processing devices 1. Additionally, the communication unit 53 receives information related to operations performed in the cleanroom 100 from the terminal devices 7 and sends messages to the terminal devices 7 notifying them of the inappropriateness of protective equipment. The communication unit 53 can transmit and receive various other types of information with the information processing devices 1 and the terminal devices 7. Furthermore, the communication unit 53 can communicate with a camera 3 equipped with communication capabilities via a wired or wireless network and directly acquire captured images from the camera 3.

[0055] The display unit 54 is configured to use a liquid crystal display or the like, and displays various images and text based on the processing of the processing unit 51. In this embodiment, the display unit 54 displays, for example, notification messages received from the information processing device 1 and captured images. Furthermore, the management device 5 can display various information, including captured images stored in the security management DB 52b, on the display unit 54. Additionally, the management device 5 can also display images captured by the camera 3 obtained via the communication unit 53 on the display unit 54 in real time.

[0056] The operation unit 55 accepts user operations and notifies the processing unit 51 of the received operations. For example, the operation unit 55 may be an input device such as a mouse and keyboard, and these input devices may also be configured to be detachable from the management device 5. Alternatively, the operation unit 55 may accept user operations via input devices such as mechanical buttons or a touch panel provided on the surface of the display unit 54.

[0057] Furthermore, the storage unit 52 may be an external storage device connected to the management device 5. Additionally, the management device 5 may be a multi-computer system comprising multiple computers, or a virtual machine virtualized by software. Furthermore, the management device 5 is not limited to the above-described structure; for example, it may not include a display unit 54 and an operation unit 55.

[0058] Furthermore, in the management device 5 according to this embodiment, the processing unit 51 reads and executes the program 52a stored in the storage unit 52, thereby the information acquisition unit 51a, the notification processing unit 51b, the schedule information management unit 51c, and the form reading unit 51d are implemented by the processing unit 51 as software function units.

[0059] The information acquisition unit 51a communicates with multiple information processing devices 1 via the communication unit 53 to process information such as notifications related to protective devices and captured image data from the information processing devices 1. The information acquisition unit 51a stores information such as date and time, location, identification information, and the presence or absence of protective measures in a corresponding manner with the acquired notifications and captured images in the security management DB52b.

[0060] The notification processing unit 51b processes notifications to the terminal devices 7 held by personnel working in the cleanroom 100 regarding the inappropriateness of protective equipment worn by those working in the cleanroom. Upon receiving notification of inappropriate protective equipment from the information processing device 1, the notification processing unit 51b sends a message to one or more terminal devices 7 notifying them of the inappropriateness of the protective equipment worn by those working in the cleanroom 100. At this time, the notification processing unit 51b may also send information related to the location of the person with inappropriate protective equipment, or data containing an image of the person deemed to have inappropriate protective equipment, to the terminal device 7, notifying the operator holding the terminal device 7 which person is inappropriate. Furthermore, if it is possible to distinguish the terminal device 7 held by the person deemed to have inappropriate protective equipment, the notification processing unit 51b may send a message only to that terminal device 7.

[0061] Furthermore, notifications to terminal device 7 can also be made directly by information processing device 1. In this case, notifications to operators can be made more quickly. It can be appropriately decided, from management device 5 to information processing device 1, to notify terminal device 7 based on the scale of cleanroom 100 and the optimization of the information system.

[0062] The schedule information management unit 51c manages and processes schedule information related to operations performed within the cleanroom 100. The schedule information management unit 51c communicates with a terminal device 7 used by users such as operators via a communication unit 53, acquiring information related to operations received from the user in the board processing apparatus 101 via the terminal device 7. Based on the information acquired from the terminal device 7, the schedule information management unit 51c stores information such as the operation content, scheduled start time, and scheduled end time for operations performed within the cleanroom 100 in a corresponding schedule DB 52c.

[0063] In this embodiment, users such as operators can fill in information related to their work in the substrate processing apparatus 101 into a prescribed form by handwriting, and send the image of the form captured by the camera of the terminal device 7 to the management device 5. In this case, the schedule information management unit 51c of the management device 5 can provide the image captured from the terminal device 7 to the form reading unit 51d (described later), and the form reading unit 51d can retrieve the information read from the form. Furthermore, in this embodiment, the text recorded on the form is read by the management device 5, but it is not limited to this; the form can also be read by the terminal device 7, and the information read by the terminal device 7 can be sent to the management device 5.

[0064] Furthermore, the schedule information management unit 51c, based on the information stored in the schedule DB 52c, processes the notifications to each information processing device 1 regarding the work performed within the cleanroom 100. The schedule information management unit 51c may also notify work at a predetermined interval, such as once every few seconds or minutes; alternatively, it may notify work at regular intervals based on changes in work content, such as the start or end time of the work; or it may notify work based on requests from the information processing device 1.

[0065] The form reading unit 51d reads the text recorded in the captured image of the form obtained by the schedule information management unit 51c from the terminal device 7. The form reading unit 51d reads the text recorded in the form using a function known as OCR. OCR-based text reading can also be performed, for example, by extracting features of the recorded text and comparing them with a pre-prepared dictionary. Alternatively, it can be performed using a learning model obtained through machine learning that infers text from images of handwritten characters. OCR-based handwritten text reading is existing technology, therefore detailed explanation is omitted.

[0066] <Protective Equipment Judgment and Handling>

[0067] Figure 5 as well as Figure 6This is a flowchart illustrating an example of the protective device determination process performed by the information processing apparatus 1 according to this embodiment. The image acquisition unit 11a of the processing unit 11 of the information processing apparatus 1 according to this embodiment acquires an image of the cleanroom 100 captured by the camera 3 by transmitting and receiving information with the camera 3 via the interface unit 14 (step S1). The human detection unit 11b of the processing unit 11 inputs the data of the captured image acquired in step S1 into a human detection model stored in the learning model storage unit 12b (step S2). The human detection unit 11b acquires the human detection result output by the human detection model based on the input in step S2 (step S3). Based on the human detection result acquired in step S3, the human detection unit 11b determines whether a person is reflected in the captured image (step S4). If no person is reflected in the captured image (S4: No), the human detection unit 11b returns the processing to step S1.

[0068] If a person is reflected in the captured image (S4: Yes), the processing unit 11 extracts the image region reflecting the person from the captured image based on the person detection result of the person detection model (step S5). Next, the protective equipment detection unit 11c of the processing unit 11 inputs the image region of the person extracted in step S5 into the protective equipment detection model stored in the learning model storage unit 12b (step S6). The protective equipment detection unit 11c obtains the protective equipment detection result output by the protective equipment detection model based on the input in step S6 (step S7).

[0069] Additionally, the job determination unit 11d of the processing unit 11 inputs the image region of the person extracted in step S5 into the pose estimation model stored in the learning model storage unit 12b (step S8). The job determination unit 11d obtains the pose estimation result output by the pose estimation model based on the input in step S8 (step S9). Next, the job determination unit 11d inputs the pose estimation result obtained in step S9 into the job determination model stored in the learning model storage unit 12b (step S10). The job determination unit 11d obtains the job determination result output by the job determination model based on the input in step S10 (step S11). In addition, the job determination unit 11d reads the information related to the job performed in the cleanroom 100 stored in the job information storage unit 12c (step S12).

[0070] The protective equipment determination unit 11e of the processing unit 11 determines, based on the protective equipment detection result obtained in step S7, the job determination result obtained in step S11, and / or the job information read in step S12, whether the protective equipment worn by the person detected from the captured image is appropriate (step S13), referring to the necessary protective equipment table pre-stored in the storage unit 12. At this time, the protective equipment determination unit 11e can determine whether the protective equipment is appropriate based on whether the person is wearing all the protective equipment set as necessary in the necessary protective equipment table for the job being performed by the detected person. If the protective equipment is appropriate (S13: Yes), the protective equipment determination unit 11e returns the processing to step S1.

[0071] If the protective equipment is inappropriate (S13: No), the notification processing unit 11f communicates with the management device 5 via the communication unit 13 to notify the person in the cleanroom 100 that the protective equipment they are wearing is inappropriate (step S14). Additionally, the notification processing unit 11f sends the captured image obtained in step S1 to the management device 5 (step S15). The notification processing unit 11f outputs a control signal to the LED light 4 from the interface unit 14, thereby illuminating the LED light 4 (step S16), and returns the processing to step S1.

[0072] Figure 7 This is a flowchart illustrating an example of the schedule management process performed by the management device 5 according to this embodiment. In the information processing system according to this embodiment, when working in a cleanroom 100, the operator pre-inputs information such as work reservations on the terminal device 7. This input information received by the terminal device 7 is then sent to the management device 5. Alternatively, the operator can input information by using the terminal device 7 to photograph a form obtained by handwriting the work reservation information onto a prescribed form; the terminal device 7 then sends the photographed image of the form to the management device 5.

[0073] In this embodiment, the schedule information management unit 51c of the processing unit 51 of the management device 5 determines whether it has received work-related information from the terminal device 7 that received input from the operator (step S31). If work-related information is received (S31: Yes), the schedule information management unit 51c determines whether the received information is a photographed image of a form (step S32). If the received information is a photographed image of a form (S32: Yes), the form reading unit 51d of the processing unit 51 performs OCR processing on the received photographed image of the form and reads the information recorded in the form (step S33), causing the processing to proceed to step S34. If the received information is not a photographed image of a form (S32: No), the processing unit 51 causes the processing to proceed to step S34. The schedule information management unit 51c stores the information received from the terminal device 7 or the information read from the received photographed image in the schedule DB 52c (step S34) and returns the processing to step S31.

[0074] If no work-related information is received from the terminal device 7 (S31: No), the schedule information management unit 51c determines whether it is time to notify the information processing device 1 (step S35). In this embodiment, the management device 5 notifies each information processing device 1 in the cleanroom 100 of information related to the work being performed in the cleanroom 100 at an appropriate interval, such as once every few seconds or once every few minutes. The schedule information management unit 51c can determine whether the notification time has been reached based on whether a predetermined period has elapsed since the last notification. If the notification time has not been reached (S35: No), the schedule information management unit 51c returns the processing to step S31.

[0075] When the notification time is reached (S35: Yes), the schedule information management unit 51c reads the schedule information stored in the schedule DB 52c (step S36). Based on the read schedule information, the schedule information management unit 51c determines the type of work performed in the cleanroom 100, and notifies each information processing device 1 in the cleanroom 100 of the type of work and other information (step S37), and returns the processing to step S31.

[0076] Figure 8This is a flowchart illustrating an example of the protective device determination process performed by the management device 5 according to this embodiment. The information acquisition unit 51a of the processing unit 51 of the management device 5 according to this embodiment determines whether a notification indicating that the protective device is unsuitable has been received from any information processing device 1 via the communication unit 53 (step S51). If no notification is received (S51: No), the information acquisition unit 51a remains on standby until a notification is received from any information processing device 1. If a notification is received (S51: Yes), the information acquisition unit 51a stores the information contained in the notification from the information processing device 1, as well as other information received along with the notification, such as captured images, in the security management DB 52b (step S52). Next, the notification processing unit 51b of the processing unit 51 sends a message indicating that the protective device is unsuitable to one or more terminal devices 7 via the communication unit 53 (step S53), ending the processing.

[0077] <Summary>

[0078] In the information processing system of this embodiment with the above structure, the information processing device 1 acquires images captured by the camera 3 inside the cleanroom 100, detects protective equipment worn by a person reflected in the captured images, determines whether the protective equipment is appropriate based on the combination of the detected protective equipment and the work performed by the person, and notifies the management device 5 or the person inside the cleanroom 100 of the appropriateness of the determination result. Therefore, the information processing system of this embodiment can, for example, prevent people working without properly wearing protective equipment from being exposed to danger, and can also achieve safe monitoring based on images captured by the camera 3.

[0079] Furthermore, in the information processing system of this embodiment, the information processing device 1 acquires schedule information of the work performed in the cleanroom 100, such as information related to the protective equipment necessary for the work obtained from a list of necessary protective equipment. Based on this acquired information, the information processing device 1 determines whether the protective equipment is appropriate based on whether the person shown in the time-captured image of the work performed in the cleanroom 100 is wearing the protective equipment necessary for the work. Thus, the information processing system of this embodiment can determine whether the protective equipment is appropriate based on the schedule information and the work performed in the cleanroom 100.

[0080] Furthermore, in the information processing system of this embodiment, the information processing device 1 uses, for example, a posture estimation model and a job estimation model to determine the job being performed by the person shown in the captured image, and determines whether the protective equipment is appropriate based on whether the person is wearing the protective equipment necessary for the determined job. Therefore, the information processing system of this embodiment can determine the appropriateness of protective equipment for each person in the cleanroom 100.

[0081] Furthermore, in the information processing system of this embodiment, for example, the management device 5 or the terminal device 7 reads information from a form containing job-related information and stores the read information in the schedule DB52c for management. Additionally, in the information processing system of this embodiment, the management device 5 receives job-related information input from workers via the terminal device 7 and stores it in the schedule DB52c. The information processing device 1 obtains information read from the form or input via the terminal device 7 from the management device 5, which manages this information. Therefore, the information processing system of this embodiment is expected to facilitate the input of job-related information.

[0082] Furthermore, in the information processing system of this embodiment, if the information processing device 1 determines that the protective equipment worn by a person inside the cleanroom 100 is inappropriate, the management device 5 stores the captured image used for the determination in the safety management DB52b. Therefore, the information processing system of this embodiment can save the situation inside the cleanroom 100 when the protective equipment is determined to be inappropriate as captured images, and is expected to contribute to safety measures inside the cleanroom 100.

[0083] Furthermore, in the information processing system of this embodiment, the camera 3 installed in the cleanroom 100 can be a camera that measures the distance (depth) to the object. By using the distance (depth) information in conjunction with the RGB image, the information processing system of this embodiment is expected to improve the accuracy of processing such as human detection, protective equipment detection, posture estimation, and action determination based on the captured image.

[0084] Furthermore, in the information processing system of this embodiment, for example, if a person in the cleanroom 100 is performing work using a ladder and is wearing protective eyewear, a safety belt, and a helmet, the information processing device 1 determines that the protective equipment is appropriate. Also, for example, if a person in the cleanroom 100 is performing work handling gases and is wearing protective eyewear or goggles and a gas mask, the information processing device 1 determines that the protective equipment is appropriate. Also, for example, if a person in the cleanroom 100 is performing work handling liquids and is wearing goggles, a face shield, an apron, protective gloves, and protective shoe covers, the information processing device 1 determines that the protective equipment is appropriate. Furthermore, for example, even if the person shown in the captured image is not performing any work, if the person is not wearing protective eyewear, the information processing device 1 determines that the protective equipment is inappropriate. Therefore, the information processing system of this embodiment can be expected to determine with high accuracy whether the protective equipment worn by a person working in the cleanroom 100 is appropriate.

[0085] Furthermore, in this embodiment, multiple cameras 3 and information processing device 1 are provided inside the cleanroom 100, but this is not a limitation; at least one camera 3 and information processing device 1 may be provided. Alternatively, the information processing device 1 may be located outside the cleanroom 100. Also, in this embodiment, one camera 3 is connected to one information processing device 1, but this is not a limitation; multiple cameras 3 may be connected to one information processing device 1.

[0086] Furthermore, in the information processing system of this embodiment, part or all of the processing performed by the information processing device 1 may also be performed by the management device 5, and conversely, part or all of the processing performed by the management device 5 may also be performed by the information processing device 1.

[0087] The embodiments disclosed herein should be considered illustrative in all respects and not restrictive. The scope of this disclosure is not as defined above, but is indicated by the claims and is intended to include their equivalent meaning and all modifications within that scope.

[0088] The items described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in all possible combinations, regardless of the form of reference. Moreover, the claims may use a form that refers to two or more other claims (multiple claim form), but are not limited to this. A form that refers to at least one multiple claim (multiple reference multiple claim) may also be used.

[0089] Explanation of reference numerals in the attached figures

[0090] 1…Information processing device (computer); 3…Camera; 4…LED light; 5…Management device (information processing device, computer); 7…Terminal device; 11…Processing unit; 11a…Image acquisition unit; 11b…Person detection unit; 11c…Protective equipment detection unit; 11d…Job determination unit; 11e…Protective equipment determination unit; 11f…Notification processing unit; 12…Storage unit; 12a…Program (computer program); 12b…Learning model storage unit; 12c…Job information storage unit; 13…Communication unit; 14…Interface unit; 51…Processing unit; 51a…Information acquisition unit; 51b…Notification processing unit; 51c…Schedule information management unit; 51d…Form reading unit; 52…Storage unit; 52a…Program (computer program); 52b…Safety management database; 52c…Schedule database; 53…Communication unit; 54…Display unit; 55…Operation unit; 98, 99…Recording medium; 101…Board processing device.

Claims

1. An information processing method, wherein an information processing device processes: Acquire images captured by a camera inside an indoor space; Based on the acquired images, detect the protective gear worn by the person in the images; Based on the combination of the detected protective equipment and the work performed by the person, determine whether the protective equipment is appropriate; and The judgment result will be announced.

2. The information processing method according to claim 1, wherein, To obtain information related to the work performed in the aforementioned indoor area. Obtain information related to the protective equipment necessary for the above operations. Based on the information obtained related to the work performed in the aforementioned indoor space and the information related to the necessary protective equipment, the appropriateness of the protective equipment is determined based on whether the person shown in the aforementioned photographed image is wearing the necessary protective equipment during the aforementioned work.

3. The information processing method according to claim 2, wherein, Determine what work the people depicted in the aforementioned images are currently performing. The suitability of the protective equipment is determined based on whether the necessary protective equipment is worn for the aforementioned work.

4. The information processing method according to claim 2, wherein, By reading the forms containing information, information related to the work being carried out in the aforementioned room can be obtained.

5. The information processing method according to claim 2, wherein, Information related to the work being performed in the aforementioned indoor space is obtained by receiving input via a terminal device.

6. The information processing method according to claim 1, wherein, If the aforementioned protective equipment is deemed inappropriate, the captured images used for the determination will be stored in the storage unit.

7. The information processing method according to claim 1, wherein, If the above-mentioned work involves the use of a ladder and the above-mentioned protective equipment includes protective glasses, safety belts, and helmets, then the above-mentioned protective equipment is deemed appropriate.

8. The information processing method according to claim 1, wherein, If the above-mentioned work involves handling gases and the above-mentioned protective equipment is protective eyewear or goggles and a gas mask, then the above-mentioned protective equipment is deemed appropriate.

9. The information processing method according to claim 1, wherein, If the above-mentioned work involves handling liquids and the above-mentioned protective equipment is goggles, face mask, apron, protective gloves, and protective foot covers, then the above-mentioned protective equipment is deemed appropriate.

10. The information processing method according to claim 1, wherein, Even if the person shown in the above-mentioned images is not performing any work, the protective equipment is deemed inappropriate if the person is not wearing protective glasses.

11. The information processing method according to claim 1, wherein, The camera mentioned above is used to measure the distance to an object.

12. A computer program that causes a computer to perform the following processes: Acquire images captured by a camera inside an indoor space; Based on the acquired images, detect the protective gear worn by the person in the images; Based on the combination of the detected protective equipment and the work performed by the person, determine whether the protective equipment is appropriate; and The judgment result will be announced.

13. An information processing apparatus, comprising a processing unit. The above-mentioned processing department will handle the following: Acquire images captured by a camera inside an indoor space; Based on the acquired images, detect the protective gear worn by the person in the images; Based on the combination of the detected protective equipment and the work performed by the person, determine whether the protective equipment is appropriate; and The judgment result will be announced.

Images