Measurement device, measurement data utilization system, and measurement data acquisition method

Abstract

This invention relates to a measurement device that acquires measurement data obtained by measuring maintenance work by a worker by using a 3D sensor installed in a space in which maintenance work for a semiconductor manufacturing device is performed. The measurement device includes: an object identification unit 223 that identifies an object included in a raw data image acquired from a 3D sensor; a coordinate conversion unit 224 that acquires the position in space of the identified identification object; a work determination unit 225 that determines a work process being performed by a worker, on the basis of the movement of one or more identification objects; a region setting unit 226 that sets, in the raw data image, a region that can be output or cannot be output according to the work process; and a data limiting unit 227 that generates limited data in which a region of the raw data image that the region setting unit has set as a region that cannot be output is made invisible.

Description

Technical Field

[0001] The present invention relates to a measurement device, a measurement data utilization system, and a measurement data acquisition method.

Background Art

[0002] In order to improve the productivity of semiconductor devices, semiconductor manufacturing equipment is required not only to improve its performance but also to improve its operating rate. Since many of the maintenance operations of semiconductor manufacturing equipment involve manual work, the maintenance operations need to be carried out efficiently and appropriately. In order to prevent backtracking in maintenance work, for example, a system that accumulates and analyzes the logs of maintenance work by workers is considered effective. In that case, in order to quantitatively characterize the maintenance work, it is necessary to accumulate logs on how each worker moves with respect to the equipment and parts. For example, the movement of the hand while wiping the part while holding the part, the positional relationship between both hands when attaching the part, the way of carrying the hands when disassembling and attaching large parts, the relationship between the worker working inside the equipment and the worker preparing in the surrounding area, etc.

[0003] Patent Document 1 relates to a technique for restricting an image so that areas other than the area of the object to be monitored are concealed. When remotely supporting work such as maintenance on the object to be monitored, the worker sends an image of the object to be monitored taken by the worker to a terminal device at a remote location, and the instructor confirms the image on the terminal device to support the worker's work. The technique of Patent Document 1 conceals such confidential information that the instructor does not want to see around the object to be monitored and enables remote support.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, distance measurement technology has advanced, and by using 3D sensors, it has become possible to measure the spatial movement of workers relative to equipment at a real-world level.

[0006] While Patent Document 1 exemplifies a substrate processing device as an object to be monitored, maintenance work on such devices is often performed in the user's factory, which is a location with many other companies' devices and confidential information unrelated to the maintenance work. Furthermore, maintenance work frequently involves removing parts from the device being maintained and performing the work in a location separate from the device's installation site. For this reason, simply monitoring only the object to be monitored in a fixed manner, as in Patent Document 1, is insufficient.

[0007] Generally, maintenance work involves frequent movement of workers, requiring the acquisition of measurement data that tracks the movement of parts or workers. However, this process may inadvertently capture sensitive information, posing a risk of data leakage.

[0008] Furthermore, 3D sensors generate a large amount of data, and even in areas where data acquisition is possible, continuing to acquire data when the situation remains unchanged for extended periods is undesirable from a data capacity standpoint. [Means for solving the problem]

[0009] One embodiment of the present invention is a measuring device that acquires measurement data of maintenance work performed by a worker using a first 3D sensor installed in a space where maintenance work is performed on semiconductor manufacturing equipment, and comprises: an object identification unit that identifies objects included in the raw data image acquired from the first 3D sensor; a coordinate transformation unit that acquires the spatial position of the identified object identified by the object identification unit; a work determination unit that determines the work process being performed by the worker based on the movement of one or more identified objects; a region setting unit that sets regions of the raw data image that can be output or not according to the work process determined by the work determination unit; and a data limiting unit that generates limited data that makes regions of the raw data image that the region setting unit has deemed not to be output invisible. [Effects of the Invention]

[0010] This invention provides a measuring device, a measurement data utilization system, and a measurement data acquisition method that enable the acquisition of measurement data with a low risk of information leakage and small data volume. Other challenges and novel features will become apparent from the description and accompanying drawings herein. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram shows the space where workers perform maintenance work. [Figure 2A] This is an example of a hardware configuration for an information processing device. [Figure 2B] This is a functional block diagram of the measurement function of the data processing device. [Figure 3] This is a workflow for maintenance work on gate valves. [Figure 4A] This diagram shows the data that is loaded into the data storage unit. [Figure 4B] This is an example of an ROI (Region of Interest) set in a virtual space. [Figure 4C] This is an example of the data structure for a work process table. [Figure 5] This is an example of the gate valve dismantling process. [Figure 6] This diagram shows a method for reducing ROI based on the gate valve model. [Figure 7] This is an example of the assembly process for the upper chamber. [Figure 8] This is an example of the dismantling process for the upper chamber. [Figure 9] This is an example of a system configuration for utilizing measurement data. [Modes for carrying out the invention]

[0012] This embodiment will be described below with reference to the drawings. [Examples]

[0013] Figure 1 shows the space in which a worker performs maintenance work on semiconductor manufacturing equipment in Example 1. The real space 100 shown on the left side of Figure 1 is a schematic top view showing the space in which the worker performs maintenance work (only the worker is shown as a side view for drawing purposes). The semiconductor manufacturing equipment 40 is the object of maintenance work, and passageways 10 and work areas 15 are provided around it. Note that each passageway and work area is divided into multiple sections, and a reference numeral is used to indicate the passageway or work area of ​​a specific section. In addition, an example of an etcher is shown as the semiconductor manufacturing equipment 40, and the figure shows a base plate 41, a lid 42, and a valve box 43.

[0014] During maintenance operations, for example, operator 30 disassembles the gate valve of semiconductor manufacturing apparatus 40. Specifically, the operator removes gate valve 44 from valve box 43, moves to one of the work areas (work area 15b in this example) according to the situation at that time, and performs maintenance work on gate valve 44. In order to measure the work of such an operator with 3D sensor 21, one or more 3D sensors 21 are fixedly installed in real space 100 so that the space where the operator may work can be observed. When providing a plurality of 3D sensors, it is desirable to install them so that the observation ranges of adjacent 3D sensors overlap. Regarding the 3D sensors as well, when indicating a specific 3D sensor, it shall be indicated with a reference numeral. In this example, 3D sensor 21a is arranged so that the work in work area 15b is included in visual field 23a, and 3D sensor 21b is arranged so that the work on semiconductor manufacturing apparatus 40 is included in visual field 23b. 3D sensor 21 is connected to data processing device 20 via cable 22. Also, in this example, operator 30 is wearing a head-mounted display (HMD) 31. HMD 31 also has a function as a 3D sensor.

[0015] Data processing device 20 receives measurement data (raw data) from 3D sensor 21 or HMD 31, and has a function of limiting the visible area according to the work process performed by the operator for the image of the measurement data. Hereinafter, the measurement data with the visible area limited is referred to as limited data. Note that data processing device 20 does not necessarily need to be arranged within real space 100 and may be arranged outside real space 100.

[0016] The data processing device 20 holds a space model corresponding to the real space 100 to generate limited data. The virtual space 110 shown by the space model is shown on the right side of FIG. 1. In the virtual space 110, the passage 10 corresponds to the passage model 50, the work area 15 corresponds to the work area model 55, the base plate 41, the lid 42, the valve box 43 and the gate valve 44 respectively correspond to the base plate model 81, the lid model 82, the valve box model 83 and the gate valve model 84, and the worker 30 corresponds to the worker model 70. In the example of the virtual space 110 shown in FIG. 1, the passage 10 and the work area 15 are modeled as they are in accordance with the layout, but they may be simplified, or the divisions such as the passage model 50 and the work area model 55 may be omitted. The coordinate reference 51 indicates the origin of the world coordinate system of the virtual space. In order to associate the virtual space 110 with the real space 100, a marker 11 is installed at the position of the real space 100 corresponding to the coordinate reference 51 of the virtual space 110. As the marker 11, for example, a two-dimensional barcode or an easily distinguishable pattern can be displayed on the floor surface, or a tape can be pasted.

[0017] Note that the size of the real space 100 for creating the space model may be determined according to the maintenance work to be measured. In the above example, even if the whole building is not modeled, it is sufficient if the range where the worker may work is modeled as the virtual space 110.

[0018] Figure 2A shows an example of the hardware configuration of the data processing device 20. The data processing device 20 is implemented by an information processing device that primarily includes a processor (CPU) 201, memory 202, storage device 203, input interface (I / F) 204, output I / F 205, communication I / F 206, input / output ports 207, and a bus 208, as shown in Figure 2A. The processor 201 functions as a functional unit (functional block) that provides predetermined functions by executing processing according to a program loaded into the memory 202. The storage device 203 stores data and programs used by the functional unit. For example, a non-volatile storage medium such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) is used for the storage device 203. The input I / F 204 is an interface for connecting input devices 209 such as a keyboard and pointing device, and the output I / F 205 is an interface for connecting a display device 210. The communication I / F 206 enables communication with other information processing devices via a network. The input / output port 207 is connected to the cable 22 of the 3D sensor 21, and measurement data from the 3D sensor 21 is input to it. These are connected to each other via bus 208, enabling communication between them.

[0019] Furthermore, the data processing device 20 does not need to be implemented with a single information processing device; it may be implemented with multiple information processing devices. In addition, some or all of the functions of the data processing device 20 may be implemented as an application on the cloud.

[0020] Figure 2B shows a functional block diagram of the measurement function of the data processing device 20. This function includes an initial setting unit 221, a data acquisition unit 222, an object identification unit 223, a coordinate transformation unit 224, a work determination unit 225, a region setting unit 226, a data limiting unit 227, a data output unit 228, and a data storage unit 229 that stores data necessary for the measurement function. Details of these will be described below, using the maintenance work on the gate valve shown in Figure 3 as an example. This maintenance work includes gate valve dismantling work performed near the semiconductor manufacturing equipment 40 and gate valve cleaning work in the work area. The measurement function of the data processing device 20 in this embodiment can flexibly limit the visible area of ​​the raw data image, so that measurement data of maintenance work performed inside the building where the semiconductor manufacturing equipment 40 is installed can be acquired without capturing confidential information, and can be shared with third parties without the risk of confidential information leakage.

[0021] [Initial setting section 221] The initial setup unit 221 activates the 3D sensor 21. Here, an example is shown in which a LiDAR (Light Detection And Ranging) camera is used as the 3D sensor 21. A LiDAR camera can output depth image data indicating the distance to the target in addition to RGB (color) image data. However, the 3D sensor 21 can be any sensor that can obtain measurement data that can be converted into 3D data (volume data such as point cloud data, mesh, or voxel). In addition to a LiDAR camera, a stereo camera equipped with two CMOS image sensors, a structured light sensor combining a projection pattern light-emitting element and an image sensor, or a sensor device that combines a distance sensor and an RGB camera and adjusts the relationship between pixels may also be used. Furthermore, a sensor that implements a function to estimate the distance information of each pixel from the RGB image using machine learning or the like may also be used.

[0022] Furthermore, the initial setup unit 221 reads and initializes data for executing the measurement function. Figure 4A shows the data read into the data storage unit 229. The CAD data 301 is the CAD data of the equipment that is the target of the maintenance work, in this example, the semiconductor manufacturing equipment 40. The spatial model 302 is a model of the real space in which the worker performs the maintenance work as explained using Figure 1, and the spatial model 302 has regions (hereinafter referred to as ROI (Region of Interest)) that have attribute information regarding whether or not raw data images can be output. Figure 4B shows an example of an ROI set in the virtual space represented by the spatial model 302. The attribute information of the ROI is divided into three categories: "Output Allowed," "Output Not Allowed," and "Output Prohibited." For example, ROI 321 is a region with the "Output Prohibited" attribute, and the output of measurement data is always prohibited, while ROI 322, 323, and 324 are regions with the "Output Allowed" / "Output Not Allowed" attribute, and the output of measurement data may be permitted at the discretion of the data processing device 20. Initially, all ROIs should be set to "Output Disabled" or "Output Prohibited."

[0023] Here, we will explain how to set ROIs for a spatial model. ROIs do not need to be partitioned according to the passage 10 and work area 15 described in Figure 1. For example, in Figure 4B, a part of the work area model 55a is set to "no output" by ROI 321. However, if, for example, a metal surface or mirror is provided on the wall surface of the work area 15a corresponding to the work area model 55a, unintended information may be reflected on the metal surface or mirror and leaked as part of the measurement data. To prevent such a situation, ROI 321 with the "no output" attribute can be set for a part of the work area model 55a corresponding to the wall surface of the work area 15a. Also, although the work area model 55b is shown as a single ROI (ROI 322), it may be divided into multiple ROIs, for example. It can be set according to the actual usage of the real space. For ROIs with the "Output Allowed" or "Output Disabled" attributes, it is sufficient if they are set to cover the entire field of view of the 3D sensor installed in the real world. However, ROIs with the "Output Prohibited" attribute should be set for the entire virtual space, regardless of the placement of the 3D sensor installed in the real world. This is to allow for restriction of measurement data from moving 3D sensors such as the HMD31.

[0024] The work process table 303 contains information about the work process of the maintenance work to be measured. Figure 4C shows an example of the data structure of the work process table 303. ID 331 contains an ID that identifies the work process. Work process name 332 contains the name of the work process. Judgment criteria 333 contains judgment criteria for determining whether the work process is being performed based on measurement data from the 3D sensor, etc. Work area candidate 334 contains candidate work areas in which the work process is performed. There may be one work area candidate or multiple work area candidates. Related ROI 335 contains the ROI corresponding to the work area candidate. Attributes 336 contain the attributes of the related ROI. As mentioned above, in the initial settings by the initial setting unit 221, ROIs with the "Outputable" / "OutputNot Possible" attribute are registered as "OutputNot Possible". Restriction mode 337 contains the mode for restricting the raw data image. There are various ways in which this limitation can occur, but for example, it could involve blurring or blacking out areas of the image that should be invisible.

[0025] [Data acquisition unit 222] The data acquisition unit 222 processes the measurement data (raw data) received from the 3D sensor 21 so that it can be processed by the object recognition unit 223. Specifically, the data acquisition unit 222 matches the field of view of the RGB image data and depth image data obtained from the 3D sensor 21, and acquires camera parameters for identifying the position of objects included in the image data in a world coordinate system with the coordinate reference 51 as the origin. The camera parameters include in-camera parameters and out-camera parameters. In-camera parameters refer to parameters for camera calibration, including lens distortion in the 3D sensor 21 and the positional relationship between the RGB sensor and the distance sensor, while out-camera parameters refer to parameters that indicate the position and orientation in which the 3D sensor 21 is installed in the world coordinate system. If the in-camera parameters are stored in the built-in memory of the 3D sensor 21, the in-camera parameters are acquired from the 3D sensor 21; otherwise, the in-camera parameters are generated in advance by calibration. On the other hand, the external camera parameters may be set to the installation of the 3D sensor 21, or they may be set based on the installation position and orientation estimated using a self-localization algorithm such as SLAM (Simultaneous Localization and Mapping).

[0026] The data processing in the data acquisition unit 222 depends on the data processing method in the subsequent object recognition unit 223. If object recognition in the object recognition unit 223 is 2D processing, it performs processing to match the field of view of the RGB image data and depth image data. On the other hand, if object recognition in the object recognition unit 223 is 3D processing, it converts the RGB image data and depth image data with matched field of view into 3D data (3D point cloud data) of a virtual space using camera parameters.

[0027] [Object Identification Unit 223] The object recognition unit 223 performs object recognition processing using image data or 3D data processed by the data acquisition unit 222 and camera parameters. The object recognition processing may be performed using either 2D or 3D processing, and known methods can be used. The object recognition unit 223 extracts the objects identified by the object recognition processing (identified objects) as 3D data. Identification information is assigned to each extracted identified object.

[0028] [Coordinate transformation unit 224] The coordinate transformation unit 224 uses camera parameters to convert the extracted 3D data of the identified object into a world coordinate system. Through this process, the data processing device 20 can determine the identified object contained in the raw image data acquired by the 3D sensor 21, along with its position in the virtual space.

[0029] [Work judgment unit 225] By continuously performing the above processing on the measurement data from the 3D sensor 21, the data processing device 20 becomes able to grasp the movement of the identified object in the virtual space. Here, the movement of the identified object includes not only the movement of the identified object in the virtual space, but also changes in the shape of the identified object itself. The work determination unit 225 detects a work process if the movement of one or more identified objects in the virtual space matches the determination criterion 333 of the work process table 303.

[0030] Figure 5 shows an example of the gate valve dismantling process. In the real world, the removal of the gate valve 44 from the valve box 43 is represented in the virtual world as the gate valve model 84 moving away from the valve box model 83. For example, the movement of the gate valve model 84 can be determined from the trajectory of the minimum distance between the valve box model 83 and the gate valve model 84, and the removal of the gate valve can be determined. Note that the gate valve model 84 is an identification object extracted from image data from the 3D sensor 21, but since the valve box model 83 is a fixed device, its shape can be determined in advance based on the CAD data 301 without having to extract it as an identification object.

[0031] The data processing device 20 tracks the gate valve that has been removed during gate valve dismantling. If a large number of 3D sensors 21 are placed in the real world, this can be done by tracing the trajectory in the virtual space as the gate valve continues to be identified as an identified object by the object identification unit 223. Alternatively, if the worker 30 is wearing a sensor that allows the worker 30's position to be determined, such as an HMD 31, the gate valve may be tracked based on the position information of the worker 30 performing maintenance.

[0032] The work determination unit 225 traces the movement of the gate valve (identification object) in the virtual space and compares it with the determination criteria 333 in the work process table 303. If the maintenance work includes a series of work processes from dismantling the gate valve to reassembling it into the valve box (see Figure 4C (ID: P01-1~3)), the work determination unit 225 detects that the gate valve model 84 has moved to one of the ROIs registered as the associated ROI 335, and that the gate valve cleaning work has been performed.

[0033] [Region setting unit 226] This section explains the process of setting the output / non-output area for raw data images related to the cleaning operation of a gate valve as an example. The area setting unit 226 sets the areas for which the raw data images acquired by the 3D sensor 21 should be output or not. In this example, the attribute of the ROI to which the gate valve model 84 has moved is switched from the initial state of non-output to output enabled.

[0034] If the raw image data contains an area where an ROI with the output prohibition attribute is visible, that area is always considered a non-output area. Since the 3D sensor 21 is fixed and the area where the ROI with the output prohibition attribute is visible is also fixed within the image data, a non-output area based on the ROI with the output prohibition attribute can be defined in advance.

[0035] As in this example, when an identification object moves to an ROI with the "Output Allowed" / "Output Not Allowed" attribute, the simplest method is to switch the attribute from "Output Not Allowed" to "Output Allowed" at the time of movement, and then switch it back from "Output Allowed" to "Output Not Allowed" when the identification object leaves. In addition, further conditions can be added to the attribute switching of ROIs with the "Output Allowed" / "Output Not Allowed" attribute. For example, the attribute could be switched to "Output Allowed" after a certain period of time (e.g., about 5 seconds) has elapsed since entering the ROI. This eliminates the need to process ROIs that are only temporarily passed through and not for work. Alternatively, the worker's hand could be recognized as an identification object, and the ROI's attribute could be switched from "Output Allowed" to "Output Not Allowed" when it is determined that the worker's hand has left the gate valve.

[0036] Furthermore, in addition to switching ROI attributes, the range of the ROI may also be limited. Depending on the operation in the real world, for example, another worker may be performing a different task in the same work area, and images of this task cannot be output. For this reason, the ROI can be reduced or adjusted. Figure 6 shows a method for reducing the ROI based on the gate valve model 84. In this example, the bounding box 350 of the gate valve model 84 is calculated, and the ROI is changed to an area enlarged by a predetermined number of times (e.g., 1.5 times) based on the center of gravity of the bounding box 350 so that the worker's hands are included in the bounding box. Here, the reduced ROI of ROI 322 is shown as ROI 322'. Alternatively, the worker may be able to adjust the range of the ROI. The ROI can be reduced by displaying the ROI in AR on the HMD 31 worn by the worker 30 and allowing the outputtable range to be specified by hand motion.

[0037] Furthermore, the area setting unit 226 includes an editing screen that allows the user (information management officer) to check the output enabled / disabled settings made by the algorithm of the area setting unit 226 or the judgment of the operator. The screen displays the output enabled / disabled settings and the areas that will be output for visual confirmation, and allows the user to adjust the output areas as needed.

[0038] Furthermore, it is desirable for the region setting unit 226 to output a restricted report that shows the algorithm of the measuring device applied to the output-enabled / disabled settings described above, as well as a log of the editing history. For example, the restricted report includes information about the process of restricting the measurement data. Specifically, it includes information about regions that have been set in advance as data output disabled, information about objects (identified objects) identified as the basis for determining the work process, and information such as the position of the object or the worker holding the object. It is also possible to display a comparison between the image of the raw data and the image of the restricted data.

[0039] [Data Restriction Section 227] The data limiting unit 227 generates limited data in which a portion of the raw image data that has been deemed unoutputtable by the region setting unit 226 is made invisible. According to the limiting mode 337 of the work process table 303, for example, processing such as blacking out, blurring, or deformation (a patch may be prepared in advance, or it may be replaced with a simple shape) is applied to the unoutputtable region and the output prohibited region. Alternatively, since there are cases where the unoutputtable region can be outputtable, blurring may be used, and the output prohibited region may be blacked out. However, if there are many blacked-out areas, spatial recognition may be hindered, in which case blurring or deformation should also be used on the output prohibited region.

[0040] [Data output unit 228] The data restriction unit 227 outputs the restricted data generated by it. [Examples]

[0041] While Example 1 demonstrated measurement using a fixed 3D sensor, Example 2 demonstrates measurement using both a fixed and a movable 3D sensor. The movable 3D sensor is not particularly limited, but here we show an example where an HMD 31 worn by worker 30 is used as the 3D sensor. In the case of a movable 3D sensor, unlike a fixed 3D sensor, it is not possible to pre-determine the ROI in the virtual space.

[0042] Using Figure 7, an example of the assembly process for the upper chamber discussed in Example 2 will be explained. Here, the semiconductor manufacturing apparatus 40 is schematically shown as comprising a main body 401, an upper chamber 402, and a lower chamber 403. As shown in the top view (schematic diagram) of the upper chamber 402 in the lower right of Figure 7, the upper chamber 402 has a sample stage 404 in its center and a sample stage base 405 that supports the sample stage 404. Specifically, maintenance work involves disassembling the semiconductor manufacturing apparatus 40, pulling out the upper chamber 402, and attaching a cover to the sample stage base 405.

[0043] As shown on the left side of Figure 7, the semiconductor manufacturing apparatus 40 is normally integrated into a main body 401, an upper chamber 402, and a lower chamber 403, and in this state, it is included in the ROI 411, which is an ROI measured by a fixed 3D sensor. To perform the maintenance work described above, the main body 401 is raised, while the upper chamber 402 is pulled out to the side. For example, the upper chamber 402 is pulled out into the passage 10c (see Figure 1), and no fixed 3D sensor is installed that includes the passage 10c in its field of view. Therefore, no ROI is set, and measurements are taken using the HMD 31 worn by the worker 30 as a 3D sensor. The process in this case will be explained below. The following explanation will focus on the differences from Example 1.

[0044] The HMD31 acquires RGB image data and depth data as measurement data (raw data), similar to the 3D sensor 21 described in Example 1. The measurement data from the HMD31 is also processed as described in Example 1 by the data acquisition unit 222, object identification unit 223, and coordinate transformation unit 224, so that identified objects are extracted and their positions in the world coordinate system are determined.

[0045] [Work judgment unit 225] The movement of a characteristic identification object is defined in the judgment criterion 333 of the work process table 303, and the assembly work of the upper chamber 402 is determined. For example, the detection condition is that a worker is holding the cover to be installed and the air spray (see Figure 4C (ID: P02-1)). The cover, air spray, and hand are detected, and if the cover and air spray are each less than a predetermined distance from the center of gravity of the hand, it is determined that the work is the assembly work of the upper chamber 402.

[0046] [Region setting unit 226] Based on the judgment of the work judgment unit 225, the region setting unit 226 sets an ROI with output capabilities based on the identification object of the upper chamber 402 for the measurement data of the HMD 31. Similar to Figure 6, the bounding box of the upper chamber model is calculated, and an ROI 412 with output capabilities is set, which is an area enlarged by a predetermined number of times (e.g., 1.5 times) based on the center of gravity of the bounding box, so that the worker's hands are included in the bounding box.

[0047] This allows measurement data to be acquired using a 3D sensor worn by a worker, even in areas where a fixed 3D sensor is not installed. Furthermore, by limiting the output video data to only the area surrounding the work object, unintended information leakage can be prevented. When acquiring measurement data from the HMD31 within an ROI with output capabilities, the entire area of ​​the measurement data image may be used as the ROI. [Examples]

[0048] Example 3 also uses a movable 3D sensor, and here we show an example in which a 3D sensor mounted on a trolley used for transporting large parts is used.

[0049] Using Figure 8, an example of the dismantling process for the upper chamber discussed in Example 3 will be explained. The worker separates the upper chamber 402 from the lower chamber 403 and loads the upper chamber 402 onto the trolley 420.

[0050] The 3D sensor mounted on the trolley 420 acquires RGB image data and depth data as measurement data (raw data), similar to the 3D sensor 21 described in Example 1. The measurement data from the 3D sensor mounted on the trolley 420 is also processed by the data acquisition unit 222, object identification unit 223, and coordinate transformation unit 224 as described in Example 1, to extract the identified object and determine its position in the world coordinate system.

[0051] [Work judgment unit 225] The movement of a characteristic identification object is defined in the judgment criterion 333 of the work process table 303, and the decision to dismantle the upper chamber 402 is made. The first condition is that the 3D sensor 21b detects that the upper chamber 402 and the lower chamber 403 have been separated. It can be determined that they have been separated when the distance between the center of gravity of the bounding box of the upper chamber 402 and the center of gravity of the bounding box of the lower chamber 403 is greater than or equal to a predetermined distance. The second condition is that the 3D sensor mounted on the trolley 420 detects that the upper chamber 402 has been placed on the trolley 420. For example, it can be determined that it has been placed on the trolley when the distance between the top surface of the trolley 420 and the center of gravity of the bounding box of the upper chamber 402 is less than a predetermined distance. When these two detection conditions are met, it is determined that the upper chamber 402 is being dismantled.

[0052] [Region setting unit 226] In this example, the 3D sensor mounted on the trolley 420 is assumed to have its field of view fixed to the top surface of the trolley 420. In this case, based on the judgment of the work judgment unit 225, an ROI 413 with output capabilities is set for the entire field of view of the 3D sensor mounted on the trolley 420. Alternatively, an ROI with output capabilities may be set based on the bounding box of an object mounted on the top surface of the trolley 420.

[0053] This allows measurement data to be acquired using 3D sensors installed on a mobile device, even in areas where fixed 3D sensors are not installed. Furthermore, by limiting the output video data to only the surrounding area related to the work object, unintended information leakage can be prevented. [Examples]

[0054] Figure 9 shows an example configuration of a measurement data utilization system that utilizes measurement data on maintenance work performed on semiconductor manufacturing equipment 40, collected by the measurement device (data processing device 20) described in Examples 1 to 3. This system is operated by the vendor, which is the manufacturer of the semiconductor manufacturing equipment 40, and the user, which is the user of the semiconductor manufacturing equipment 40. In Figure 9, the information processing device at the user site is shown on the left, and the information processing device at the vendor site is shown on the right. The user site and the vendor site are able to communicate via a network. For these information processing devices, devices that can access the raw measurement data of maintenance work performed on semiconductor manufacturing equipment 40 using a 3D sensor are enclosed in a solid black frame, and devices that can access limited data, which is measurement data in which a portion of the area has been concealed by the measurement device 20, are enclosed in a dashed black frame.

[0055] The raw data, restricted data, and restricted reports collected by the measuring device 20 are stored on the data sharing server 501. Users with access privileges to confidential information can view both the raw data and restricted data on their user terminal 502 while referring to the restricted report. Meanwhile, vendor personnel staying at the user site can also view the measurement data from the vendor room server 504 via the authentication server 503, but the measurement data they can access is limited to the restricted data.

[0056] The restricted data can be accessed via the network, and also at the vendor's site via the authentication server 511. For example, the vendor can analyze maintenance work logs based on the restricted data using the analysis server 512 and report the analysis results to the user. This enables remote training of workers, etc.

[0057] The above embodiments and modifications are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment or modification with parts of another embodiment or modification, and it is also possible to add parts of other embodiments or modifications to the configuration of one embodiment or modification. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment or modification with other configurations. [Explanation of Symbols]

[0058] 10: Passageway, 11: Marker, 15: Work area, 21: 3D sensor, 22: Cable, 23: Field of view, 30: Worker, 31: HMD, 40: Semiconductor manufacturing equipment, 41: Base plate, 42: Lid, 43: Valve box, 44: Gate valve, 50: Passageway model, 51: Coordinate reference, 55: Work area model, 70: Worker model, 81: Base plate model, 82: Lid model, 83: Valve box model, 84: Gate valve model, 100: Real space, 110: Virtual space, 201: Processor (CPU), 202: Memory, 203: Storage device, 204: Input interface, 205: Output interface, 206: Communication interface, 207: Input / output port, 208: Bus, 209: Input device, 210: Display device, 221: Initialization unit, 222: Data acquisition 223: Object identification unit, 224: Coordinate transformation unit, 225: Work judgment unit, 226: Area setting unit, 227: Data limiting unit, 228: Data output unit, 229: Data storage unit, 301: CAD data, 302: Spatial model, 303: Work process table, 321, 322, 322', 323, 324: ROI, 331: ID, 332: Work process name, 333: Judgment criteria, 334: Candidate work area, 3 35: Related ROI, 336: Attributes, 337: Limiting characteristics, 350: Bounding box, 401: Main body, 402: Upper chamber, 403: Lower chamber, 404: Sample stage, 405: Sample stage base, 411, 412, 413: ROI, 420: Cart, 501: Data sharing server, 502: User terminal, 503, 511: Authentication server, 504: Vendor room server, 512: Analysis server.

Claims

1. A measuring device that acquires measurement data of maintenance work performed by workers using a first 3D sensor installed in a space where maintenance work is performed on semiconductor manufacturing equipment, An object identification unit that identifies objects included in the raw image data acquired from the first 3D sensor, A coordinate transformation unit that acquires the position of the identified object in the space identified by the object identification unit, A work determination unit that determines the work process being performed by the worker based on the movement of one or more of the aforementioned identification objects, A region setting unit sets regions of the raw data image that can be output or not, according to the work process determined by the work determination unit, A measuring device having a data limiting unit that generates limited data that makes areas of the raw data image that the area setting unit has deemed unoutputtable impossible to see.

2. In claim 1, In the aforementioned space, an ROI is set up that has an attribute regarding whether or not the raw data image can be output. The ROI includes a first ROI with an output prohibition attribute that always prevents the output of the raw data image, and a second ROI that can switch between enabling and disabling the output of the raw data image. The region setting unit is a measuring device that sets, from the raw data image, at least one of the second ROIs determined by the work determination unit as a related ROI in the work process to be outputtable, and sets the other regions to be not outputtable.

3. In claim 2, The region setting unit is a measuring device that switches the output of the second ROI on or off in accordance with the movement of the identification object.

4. In claim 2, The region setting unit is a measuring device capable of further reducing and setting the region of the second ROI that has been set to output from the raw data image.

5. In claim 4, The region setting unit can set the region of the second ROI that is outputtable from the raw data image to include the identification object or to the region instructed by the worker. The range instructed by the worker is determined by a measuring device set by a second 3D sensor worn by the worker.

6. In claim 2, The aforementioned worker was wearing a second 3D sensor, The object identification unit identifies objects included in the raw image data acquired from the second 3D sensor. The region setting unit is a measuring device that sets the range of the raw image data acquired from the second 3D sensor, including the identified object identified by the object identification unit, as an output region.

7. In claim 2, In the aforementioned maintenance work, a trolley is used, and the trolley is equipped with a third 3D sensor whose upper surface is the third ROI. The object identification unit identifies objects included in the raw image data acquired from the third 3D sensor. The region setting unit is a measuring device that, upon detecting that an object has been placed on the trolley, sets the third ROI to be outputtable for the raw image data acquired from the third 3D sensor.

8. In claim 1, The area setting unit is a measuring device that displays the raw data and the limited data in a comparable manner and allows editing of areas of the limited data image that are not visible.

9. In claim 1, The area setting unit is a measuring device that creates a limitation report including information about the process by which an area to be made invisible in the limitation data was set.

10. In claim 1, The data limiting unit is a measuring device that applies blacking out, blurring, or deformation to areas of the raw data image that the area setting unit has deemed unoutputtable.

11. A measuring device according to any one of claims 1 to 10, The measuring device has an information processing device that can be connected to the aforementioned measuring device via a network, The aforementioned information processing device is prohibited from providing the raw data, and the aforementioned limited data is provided to the measurement data utilization system.

12. A method for acquiring measurement data, which uses a measuring device to acquire measurement data obtained by measuring the maintenance work performed by a worker using a 3D sensor installed in a space where maintenance work on semiconductor manufacturing equipment is performed, The measuring device includes an object identification unit, a coordinate transformation unit, a work determination unit, a region setting unit, and a data limiting unit. The object identification unit identifies objects included in the raw image data acquired from the 3D sensor. The coordinate transformation unit obtains the position in space of the identified object identified by the object identification unit, The work determination unit determines the work process being performed by the worker based on the movement of one or more of the identified objects. The area setting unit sets areas of the raw data image that can be output or not, according to the work process determined by the work determination unit. The data limiting unit is a measurement data acquisition method that generates limited data in which the region setting unit makes the region of the raw data image that it has deemed unoutputtable is not visible.

13. In claim 12, In the aforementioned space, an ROI is set up that has an attribute regarding whether or not the raw data image can be output. The ROI includes a first ROI with an output prohibition attribute that always prevents the output of the raw data image, and a second ROI that can switch between enabling and disabling the output of the raw data image. The region setting unit sets at least one of the second ROIs, which is determined by the work determination unit as a related ROI in the work process, to be outputtable from the raw data image, and sets the other regions to not be outputtable.

Images