Image Limiting Method, Image Limiting Program, Information Processing Apparatus, and Support System

Abstract

To provide a technique that restricts an image so that an area other than an area of an object to be monitored in the image is concealed.SOLUTION: An image restriction method comprises the steps of: acquiring image data obtained by photographing a space in which an object to be monitored is arranged by a photographing unit; acquiring photographing position data indicating a photographing position of the photographing unit in the space when the object to be monitored is photographed; acquiring shape information indicating a three-dimensional shape of the object to be monitored; identifying an arrangement position of the object to be monitored in the space; identifying, on the basis of the photographing position data, the shape information, and the arrangement position of the object to be monitored, the shape of the object to be monitored arranged at the arrangement position with the photographing position indicated by the photographing position data as a viewpoint position; and identifying, on the basis of an identified result, an area of the object to be monitored within an image of the acquired image data; masking an area of an image of the acquired image data other than an area of the identified object to be monitored; and outputting masked image data.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present disclosure relates to an image restriction method, an image restriction program, an information processing apparatus, and a support system.

Background Art

[0002] Patent Document 1 selects an image file in which something related to confidential information appears and an image file in which nothing related to confidential information appears. Then, it discloses a technique of excluding an image file in which something related to confidential information appears from the target of disclosure and publicly disclosing only an image in which nothing related to confidential information appears to others.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a technique for restricting an image so that an area other than the area of a surveillance target is concealed.

Means for Solving the Problems

[0005] An image restriction method according to one aspect of the present disclosure includes: a step of acquiring image data obtained by photographing, by a photographing unit, a space in which a monitoring target is arranged; a step of acquiring photographing position data indicating a photographing position of the photographing unit in the space when the monitoring target is photographed; a step of acquiring shape information indicating a three-dimensional shape of the monitoring target; a step of specifying an arrangement position of the monitoring target in the space; based on the photographing position data, the shape information, and the arrangement position of the monitoring target, specifying the shape of the monitoring target arranged at the arrangement position having the photographing position indicated by the photographing position data as a viewpoint position, and based on the specifying result, specifying a region of the monitoring target in the image of the acquired image data; a step of masking a region other than the region of the specified monitoring target in the image of the acquired image data; and a step of outputting the masked image data.

Effect of the Invention

[0006] According to the present disclosure, an image can be restricted so that a region other than the region of the monitoring target is concealed.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

[0008] Hereinafter, embodiments of an image restriction method, an image restriction program, an information processing apparatus, and a support system disclosed in the present application will be described in detail with reference to the drawings. Note that the image restriction method, the image restriction program, the information processing apparatus, and the support system disclosed are not limited by the present embodiment.

[0009] In recent years, there has been a demand for a support system that remotely supports work such as maintenance on a monitoring target. For example, in the support system, an image taken by a worker of a monitoring target is sent to a terminal device at a remote location, and an instructor confirms the image on the terminal device to support the worker's work. However, there is confidential information around the monitoring target that the instructor does not want to show, and it may not be possible to use the captured image as it is.

[0010] Therefore, a technique for restricting an image so that areas other than the area of the monitoring target are concealed is expected.

[0011] [First Embodiment] [Schematic Configuration of Support System 10] FIG. 1 is a diagram showing an example of the configuration of a support system 10 according to the first embodiment. The support system 10 is a system that remotely supports operations such as maintenance on a monitoring target object.

[0012] The support system 10 includes a server device 11, an operator terminal 12, and a terminal device 13. The server device 11, the operator terminal 12, and the terminal device 13 are connected to a network N and are capable of communicating with the network N. As an aspect of such a network N, any type of communication network such as mobile communication such as a mobile phone, the Internet, a LAN (Local Area Network), or a VPN (Virtual Private Network) can be adopted regardless of whether it is wired or wireless.

[0013] The server device 11 is a device that provides a support function for remotely supporting operations on a monitoring target object. The server device 11 is, for example, a computer such as a server computer. The server device 11 is provided, for example, in a data sensor or the like, is assigned a global IP address, and is accessible by the terminal device 13 and the operator terminal 12 via the network N. The server device 11 is accessed from the terminal device 13 and the operator terminal 12, and the support function becomes available by performing account authentication. The server device 11 relays communication between the operator terminal 12 and the terminal device 13 by the support function. For example, the server device 11 designates an account that permits communication from the operator terminal 12. The server device 11 relays communication between the operator terminal 12 and the terminal device 13 of the account for which communication is permitted. In the present embodiment, the case where the server device 11 is a single computer is described as an example, but it may be implemented as a computer system including a plurality of computers.

[0014] The worker terminal 12 is a device operated by a worker who performs work on the object to be monitored. In the present embodiment, the worker terminal 12 is a wearable device that can be worn and used by the worker. For example, the worker terminal 12 is configured as a head-mounted display. The worker terminal 12 is worn on the worker's head. The worker terminal 12 visually displays various information to the worker. The worker terminal 12 has a transmissive display portion in the lens portion so that the external real environment can be visually recognized even when the worker is wearing it. The worker terminal 12 displays various information such as graphics on the display portion, and superimposes and displays the various information on the object visible through the display portion. By superimposing and displaying various information on the visible object, the worker terminal 12 can realize AR (Augmented Reality) and MR (Mixed Reality). AR is a technology that expands and represents reality by the device imparting information to the image of an object existing in the real space. MR is a technology that changes the image of an object existing in the real space by the device, and composites and represents the changed image of the object and the image of an object existing in the virtual space. AR is based on the real world, overlaps the real and the unreal, and expands the real world. Since AR overlays virtual objects on the real world, the virtual objects are displayed only on the entire surface. MR is based on the virtual space, recognizes the space, and fuses the virtual space and the real space. MR can place virtual objects in the real space. The worker terminal 12 according to the present embodiment realizes MR.

[0015] In addition, the worker terminal 12 is provided with a camera, and the camera can capture an image in the front direction of the worker wearing it. The worker terminal 12 transmits the image captured by the camera to the terminal device 13 via the server device 11. Further, the worker terminal 12 displays the image received from the terminal device 13 via the server device 11 on the display unit. As such a worker terminal 12, for example, Hololens 2 (registered trademark) of Microsoft Corporation in the United States can be mentioned. Although the case where the worker terminal 12 is a wearable device has been described as an example, it is not limited thereto. The worker terminal 12 may be a terminal that can be held in the hand, such as a tablet or a notebook PC. In the first embodiment, the worker terminal 12 corresponds to the information processing apparatus of the present disclosure.

[0016] The terminal device 13 is a device operated by an instructor who remotely instructs work on the object to be monitored. The terminal device 13 is, for example, a computer such as a personal computer. An image received from the worker terminal 12 via the server device 11 is displayed on the terminal device 13. The instructor transmits a work instruction from the terminal device 13 by voice or image.

[0017] [Configuration of Server Device 11] Next, the configuration of each device will be described. First, the configuration of the server device 11 will be described. FIG. 2 is a diagram showing an example of the functional configuration of the server device 11 according to the first embodiment. The server device 11 includes a communication I / F (interface) unit 20, a storage unit 21, and a control unit 22. Note that the server device 11 may have other devices that a computer has in addition to the above devices.

[0018] The communication I / F unit 20 is an interface that performs communication control with other devices. The communication I / F unit 20 is connected to the network N and transmits and receives various information to and from the terminal device 13 and the worker terminal 12 via the network N.

[0019] The storage unit 21 is a storage device such as a hard disk, SSD, or optical disk. Note that the storage unit 21 may be a semiconductor memory capable of rewriting data, such as a RAM, flash memory, or NVSRAM.

[0020] The storage unit 21 stores the OS (Operating System) and various programs executed by the control unit 22. Furthermore, the storage unit 21 stores various data used in the programs executed by the control unit 22. For example, the storage unit 21 stores the shape information 21a.

[0021] The shape information 21a is data that stores the three-dimensional shape of the object to be monitored. The details of the shape information 21a will be described later.

[0022] The control unit 22 is a device that controls the server device 11. As the control unit 22, an electronic circuit such as a CPU or MPU, or an integrated circuit such as an ASIC or FPGA can be adopted. The control unit 22 has an internal memory for storing programs and control data that define various processing procedures, and executes various processes based on these. The control unit 22 functions as various processing units when various programs operate. For example, the control unit 22 includes a support control unit 22a and a relay unit 22b.

[0023] The support control unit 22a is a processing unit that performs control related to the support function. When the support control unit 22a receives an access, it causes the access source to input account information and authenticates the account. For example, when the support control unit 22a receives an access from the terminal device 13 and the operator terminal 12, it authenticates the account. When the authentication of a legitimate account is performed, the support control unit 22a provides various operation screens of the support function to the access source and receives operations of the support function from the operation screen. For example, the support control unit 22a designates an account that permits communication from the terminal device 13.

[0024] The relay unit 22b relays communication between the operator terminal 12 and the terminal device 13 of the account for which communication is permitted.

[0025] [Configuration of Operator Terminal 12] Next, the configuration of the operator terminal 12 will be described. FIG. 3 is a diagram showing an example of the functional configuration of the operator terminal 12 according to the first embodiment. As described above, the operator terminal 12 is configured as a head-mounted display. The operator terminal 12 includes a communication I / F unit 30, a display unit 31, a photographing unit 32, a sensor unit 33, an audio input unit 34, an audio output unit 35, a storage unit 36, and a control unit 37. Note that the operator terminal 12 may include other devices in addition to the above devices.

[0026] The communication I / F unit 30 is an interface that performs communication control with other devices. The communication I / F unit 30 is connected to the network N by wireless communication and transmits and receives various information to and from other devices via the network N.

[0027] The display unit 31 is a device that displays various information. The display unit 31 is provided on the operator terminal 12 so as to face the eyes of the user who wears the operator terminal 12. The display unit 31 has transparency so that the user can visually recognize the external real environment even while wearing it. The display unit 31 displays various information under the control of the control unit 37. For example, the display unit 31 displays an image transmitted from the server device 11. In the present embodiment, the display unit 31 corresponds to both eyes, but may correspond to only one eye.

[0028] The photographing unit 32 generates an image by photographing the surroundings of the operator terminal 12 under the control of the control unit 37. The photographing unit 32 includes a camera. For example, the photographing unit 32 includes a camera directed in the front direction of the user wearing the operator terminal 12. The camera may be a 3D camera, a stereo type in which at least two cameras are arranged at a predetermined interval, or a ToF (Time of Flight) type. The photographing unit 32 may include a stereo type 3D camera and a ToF type 3D camera respectively. Further, the photographing unit 32 may include a plurality of cameras that photograph the surroundings of the operator terminal 12. The photographing unit 32 photographs an image with a camera and outputs the image data of the photographed image to the control unit 37.

[0029] The sensor unit 33 includes sensor devices such as an acceleration sensor, a gyro sensor, and a direction sensor, and has a function of sensing information used in the processing in the control unit 37.

[0030] The voice input unit 34 includes a microphone that inputs voice and converts it into an electrical signal, and generates voice data by performing A / D (Analog / Digital) conversion or the like on the electrical signal. The voice input unit 34 outputs the generated voice data to the control unit 37.

[0031] The voice output unit 35 includes a speaker, converts the digital voice signal input from the control unit 37 into an analog voice signal by D / A (Digital / Analog) conversion, and outputs voice corresponding to the analog voice signal from the speaker.

[0032] The storage unit 36 is a storage device that stores various types of information. For example, the storage unit 36 is a semiconductor memory such as a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non Volatile Static Random Access Memory) in which data can be rewritten. Note that the storage unit 36 may be a storage device such as a hard disk, an SSD (Solid State Drive), or an optical disk.

[0033] The storage unit 36 stores control programs executed by the control unit 37 and various programs including the program for image limit processing described later. Further, the storage unit 36 stores various data used in the programs executed by the control unit 37.

[0034] The control unit 37 is a device that controls the operator terminal 12. As the control unit 37, electronic circuits such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit), or integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array) can be adopted. The control unit 37 has an internal memory for storing programs and control data that define various processing procedures, and executes various processes based on these.

[0035] The control unit 37 functions as various processing units when various programs operate. For example, the control unit 37 includes a shooting control unit 37a, an operation reception unit 37b, an image acquisition unit 37c, a shooting position acquisition unit 37d, a shape acquisition unit 37e, a position identification unit 37f, a region identification unit 37g, a mask processing unit 37h, and an output unit 37i. Some or all of the functions of the various processing units may be standard functions of the operator terminal 12, or additional functions added to the standard functions.

[0036] The shooting control unit 37a controls the shooting unit 32 to shoot an image. For example, the shooting control unit 37a causes the shooting unit 32 to shoot an image at a predetermined frame rate. For example, the shooting control unit 37a causes the shooting unit 32 to shoot a video.

[0037] The operation reception unit 37b receives various operations. For example, the operation reception unit 37b receives operations by hand or voice. For example, the operation reception unit 37b displays operation menus and operation buttons on the display unit 31 according to predetermined operations. The operation reception unit 37b recognizes a hand from an image of image data obtained by the imaging unit 32, and receives various operations such as selections for operation menus and operation buttons by hand tracking. For example, after an operation button for designating a position is selected, the operation reception unit 37b receives the designation of a position in space by recognizing the position pointed to by a finger of the hand. Also, for example, when an operation button for instructing image transmission or transmission stop is selected, the operation reception unit 37b receives an instruction for image transmission or transmission stop. Further, the operation reception unit 37b receives various operations by voice through voice recognition of voice data.

[0038] The image acquisition unit 37c acquires image data obtained by photographing the space where the object to be monitored is arranged. For example, the operator wears the operator terminal 12 and walks around the object to be monitored while shooting a video with the imaging unit 32. The image acquisition unit 37c acquires image data (video data) obtained by photographing the object to be monitored from the surroundings by the imaging unit 32.

[0039] The imaging position acquisition unit 37d acquires imaging position data indicating the imaging position of the imaging unit 32 in the space when the object to be monitored is imaged. For example, the imaging position acquisition unit 37d specifies the surrounding shape from the image data obtained by the imaging unit 32 and the information obtained by the sensor unit 33, performs spatial mapping, and acquires spatial information indicating the shape of the space where the object to be monitored is arranged. For example, the imaging position acquisition unit 37d specifies the surrounding shape from the data of the 3D camera obtained by the imaging unit 32, and acquires spatial information indicating the shape of the space by connecting the specified surrounding shapes. The imaging position acquisition unit 37d specifies the position in the space of the spatially mapped spatial information from the specified surrounding shape, and acquires the imaging position data with the specified position as the imaging position of the imaging unit 32.

[0040] The shape acquisition unit 37e acquires shape information indicating the three-dimensional shape of the object to be monitored. For example, the range for acquiring the shape information is specified by the operator. For example, the range is specified so as to surround the object to be monitored. The shape acquisition unit 37e performs spatial mapping within the specified range from the image data obtained by photographing the object to be monitored from the surroundings to recognize the object (the object to be monitored). Then, the shape acquisition unit 37e arranges voxels according to the shape of the recognized object, and acquires shape information indicating the three-dimensional shape of the object to be monitored by the voxels. The shape acquisition unit 37e may be able to correct the three-dimensional shape of the specified object to be monitored according to a correction instruction. For example, the shape acquisition unit 37e displays the voxels arranged according to the shape of the recognized object on the display unit 31 to display the shape of the recognized object. The operation reception unit 37b receives a correction instruction for the displayed voxels by hand or voice. The shape acquisition unit 37e performs a correction to add or delete voxels according to the correction instruction received by the operation reception unit 37b, and acquires shape information indicating the three-dimensional shape of the object to be monitored by the corrected voxels.

[0041] The shape information acquired by the shape acquisition unit 37e can be saved in the server device 11. It is possible to add additional information such as a name, an identification ID of the object to be monitored, and an arrangement space to the shape information. The operation reception unit 37b receives an instruction to save by adding additional information such as a name, an identification ID of the object to be monitored, and an arrangement space to the shape information by hand or voice. When the operation reception unit 37b is instructed to save the shape information, the shape acquisition unit 37e transmits the shape information with the additional information added to the server device 11. The server device 11 associates the received shape information with the account of the worker terminal 12 and stores it in the storage unit 21 as shape information 21a.

[0042] The shape information 21a stored in the server device 11 can be read from the operator terminal 12. The server device 11 notifies the operator terminal 12 of the additional information of the shape information 21a stored in the storage unit 21 in association with the account of the operator terminal 12. The operation reception unit 37b displays the name of the shape information and the additional information of the arrangement space, and receives an instruction to read the shape information. When an instruction to read the shape information is given by the operation reception unit 37b, the shape acquisition unit 37e acquires the shape information 21a for which the reading has been instructed from the server device 11.

[0043] Thereby, once the operator terminal 12 performs spatial mapping to acquire the shape information of the monitoring object and temporarily stores it in the server device 11, the stored shape information can be used.

[0044] The position specifying unit 37f specifies the arrangement position of the monitoring object within the space where the monitoring object is arranged. For example, in the space where the monitoring object is arranged, a reference point serving as a reference position is defined. The reference point may be specified as a mark on the floor or the like of the space. Further, the operation reception unit 37b may receive a specification of a reference point serving as a reference position within the space by hand or voice. When the shape acquisition unit 37e performs spatial mapping to recognize the monitoring object and acquire the shape information, the position specifying unit 37f specifies the arrangement position of the monitoring object within the space of the spatial information with reference to the reference point. On the other hand, when the shape information is acquired from the server device 11, on the operator terminal 12, the operator aligns the shape of the monitoring object indicated by the shape information with the space of the spatial information. The operator terminal 12 displays the shape of the monitoring object indicated by the shape information on the display unit 31, and the operation reception unit 37b receives an alignment operation by hand or voice. The position specifying unit 37f specifies the arrangement position of the monitoring object within the space of the aligned spatial information with reference to the reference point.

[0045] The region specifying unit 37g specifies the region of the object to be monitored within the image of the image data acquired by the image acquisition unit 37c. For example, the region specifying unit 37g specifies the shape of the object to be monitored arranged at the arrangement position with the shooting position indicated by the shooting position data as the viewpoint position, based on the shooting position data acquired by the shooting position acquisition unit 37d, the shape information acquired by the shape acquisition unit 37e, and the arrangement position of the object to be monitored specified by the position specifying unit 37f. For example, the region specifying unit 37g arranges voxels indicating the three-dimensional shape of the object to be monitored based on the shape information at the position that becomes the arrangement position of the object to be monitored in the space of the spatially mapped space information. Then, the region specifying unit 37g specifies the shape seen from the position that becomes the shooting position indicated by the shooting position data within the space of the space information. The region specifying unit 37g specifies the region of the object to be monitored within the image of the image data based on the specified shape. For example, the region specifying unit 37g specifies the region of the specified shape within the image of the image data as the region of the object to be monitored. Alternatively, the region specifying unit 37g specifies the face orientation direction (front direction) from the image data obtained by the imaging unit 32 and the information obtained by the sensor unit 33, and specifies the region of the specified shape corresponding to the face orientation direction (front direction) of the image of the image data as the region of the object to be monitored. Note that the region specifying unit 37g may display an image of the voxel on the display unit 31 corresponding to the voxel arranged in the space of the space information.

[0046] The mask processing unit 37h performs mask processing on a region other than the region of the object to be monitored specified by the region specifying unit 37g in the image of the image data acquired by the image acquisition unit 37c. For example, the mask processing unit 37h generates first image data obtained by performing mask processing to make a region other than the region of the object to be monitored in the image of the image data acquired by the image acquisition unit 37c in an opaque state. In addition, the mask processing unit 37h generates second image data obtained by performing mask processing to make a region other than the region of the object to be monitored in the image of the image data acquired by the image acquisition unit 37c in a semi-transparent state.

[0047] The output unit 37i outputs the image data that has been mask-processed by the mask processing unit 37h. For example, the output unit 37i outputs the image data that has been mask-processed by the mask processing unit 37h under the control of the operation reception unit 37b. For example, the output unit 37i outputs the first image data that has been mask-processed by the mask processing unit 37h to the terminal device 13 of the instructor via the server device 11. Also, the output unit 37i outputs the second image data to the display unit 31.

[0048] [Specific Example] Next, a specific example of remotely assisting the work on the monitoring object using the support system 10 according to the first embodiment will be described. Hereinafter, the case where the monitoring object is the substrate processing apparatus 50 and the work such as maintenance of the substrate processing apparatus 50 is remotely assisted will be described as an example. FIG. 4 is a diagram for explaining remote assistance of work by the support system 10 according to the first embodiment.

[0049] The worker 51 wears the head-mounted display which is the worker terminal 12 on the head. The worker 51 operates the worker terminal 12 to access the server device 11 and perform account authentication.

[0050] The instructor 52 who supports the maintenance operates the terminal device 13 to access the server device 11 and perform account authentication.

[0051] The worker 51 designates an account that permits communication from the worker terminal 12. For example, the worker 51 designates the account of the instructor 52 as the account that permits communication. Thereby, the server device 11 relays the communication between the worker terminal 12 and the terminal device 13. Thereby, for example, the instructor 52 can talk to the worker 51 wearing the worker terminal 12 by connecting a microphone, a speaker, or a headset to the terminal device 13.

[0052] When the operator restricts the image to be transmitted to the terminal device 13 so that areas outside the area of the substrate processing apparatus 50 are concealed, for example, the operator performs the following preparatory work. Note that the preparatory work may be performed at any time as long as it is before receiving remote support.

[0053] The operator 51 moves around the substrate processing apparatus 50 with the operator terminal 12 worn on the head. The operator terminal 12 performs imaging by the imaging unit 32. For example, the imaging control unit 37a causes the imaging unit 32 to capture images at a predetermined frame rate. The image acquisition unit 37c acquires the image data captured by the imaging unit 32.

[0054] The operator terminal 12 acquires imaging position data indicating the imaging position where the substrate processing apparatus 50 is imaged by the imaging unit 32 within the space where the substrate processing apparatus 50 is arranged. For example, the imaging position acquisition unit 37d specifies the surrounding shape from the image data obtained by the imaging unit 32 and the information obtained by the sensor unit 33, performs spatial mapping, and acquires spatial information indicating the shape of the space where the substrate processing apparatus 50 is arranged. The imaging position acquisition unit 37d specifies the position within the spatially mapped spatial information from the specified surrounding shape, and acquires the imaging position data with the specified position as the imaging position of the imaging unit 32.

[0055] The operator terminal 12 acquires shape information indicating the three-dimensional shape of the substrate processing apparatus 50. FIG. 5 is a diagram schematically showing the space where the substrate processing apparatus 50 according to the first embodiment is arranged. In FIG. 5, the substrate processing apparatus 50 to be maintained is arranged in the clean room 60, and devices 61 and 62 such as another substrate processing apparatus are arranged on both sides of the substrate processing apparatus 50. For example, since there is various confidential information in the clean room 60, it is desired to conceal the areas other than the substrate processing apparatus 50 from the instructor 52. For example, the devices 61 and 62 are not desired to be shown to the instructor 52. In such a case, the operator 51 designates a range 63 so as to surround the substrate processing apparatus 50. Further, the operator 51 designates a reference point 64. For example, the operator 51 designates two reference points 64 at a predetermined interval.

[0056] The operator terminal 12 acquires shape information indicating the three-dimensional shape of the substrate processing apparatus 50. For example, the shape acquisition unit 37e performs spatial mapping within the specified range 63 to recognize an object (substrate processing apparatus 50), arranges voxels according to the shape of the recognized object, and acquires shape information indicating the three-dimensional shape of the substrate processing apparatus 50 by the voxels. FIGS. 6 and 7 are diagrams for explaining the flow of acquiring shape information according to the first embodiment. In FIG. 6, the range 63 is specified by four points 65 so as to surround the substrate processing apparatus 50. The range 63 may be specified in any manner. For example, when the operator terminal 12 specifies the range 63 by sequentially specifying the positions of the points 65, the positions of the specified points 65 are sequentially connected by lines, and further, the position of the first-specified point 65 and the position of the last-specified point 65 are connected by a line to form the range 63. FIG. 7 shows a state in which the substrate processing apparatus 50 within the range 63 is recognized and voxels are arranged according to the shape of the substrate processing apparatus 50. As shown in FIG. 7, the shape acquisition unit 37e arranges voxels according to the shape of the substrate processing apparatus 50 and acquires shape information indicating the three-dimensional shape of the substrate processing apparatus 50 by the voxels. The operator terminal 12 stores the acquired shape information.

[0057] Next, the flow of receiving remote assistance for the work on the substrate processing apparatus 50 while restricting the image so that nothing other than the substrate processing apparatus 50 appears will be described.

[0058] The operator terminal 12 is capable of operating to transmit or stop transmitting the image captured by the imaging unit 32. For example, the operation reception unit 37b receives an image transmission instruction by hand or voice. When the operator terminal 12 receives an image transmission instruction, the imaging unit 32 performs imaging. For example, the imaging control unit 37a causes the imaging unit 32 to capture images at a predetermined frame rate. The image acquisition unit 37c acquires the image data captured by the imaging unit 32.

[0059] The operator terminal 12 acquires imaging position data indicating the imaging position at which the substrate processing apparatus 50 is imaged by the imaging unit 32 within the clean room 60. For example, the imaging position acquisition unit 37d specifies the surrounding shape from the image data obtained by the imaging unit 32 and the information obtained by the sensor unit 33, performs spatial mapping, and acquires spatial information indicating the shape of the clean room 60. The imaging position acquisition unit 37d specifies the position within the space (clean room 60) of the spatially mapped spatial information from the specified surrounding shape, and acquires the imaging position data with the specified position as the imaging position of the imaging unit 32.

[0060] The operator terminal 12 specifies the arrangement position of the substrate processing apparatus 50 within the clean room 60. For example, the position specifying unit 37f specifies the arrangement position of the substrate processing apparatus 50 within the space of the spatial information with reference to the reference point 64.

[0061] The operator terminal 12 specifies the area of the substrate processing apparatus 50 within the image of the image data captured by the imaging unit 32. For example, the area specifying unit 37g specifies the shape of the substrate processing apparatus 50 arranged at the arrangement position with the imaging position indicated by the imaging position data as the viewpoint position, based on the imaging position data, the shape information, and the specified arrangement position of the substrate processing apparatus 50. For example, the area specifying unit 37g arranges a voxel indicating the three-dimensional shape of the substrate processing apparatus 50 at the position that becomes the arrangement position of the substrate processing apparatus 50 in the space (clean room 60) of the spatial information, based on the shape information. Then, the area specifying unit 37g specifies the shape seen from the voxel at the position that becomes the imaging position indicated by the imaging position data within the space of the spatial information. The area specifying unit 37g specifies the area of the substrate processing apparatus 50 within the image of the image data based on the specified shape.

[0062] The operator terminal 12 masks the area of the image captured by the imaging unit 32 that is outside the area of the substrate processing apparatus 50. For example, the masking unit 37h generates first image data obtained by masking the area of the image of the image data acquired by the image acquisition unit 37c that is outside the area of the substrate processing apparatus 50 into an opaque state. Further, the masking unit 37h generates second image data obtained by masking the area of the image of the image data acquired by the image acquisition unit 37c that is outside the area of the substrate processing apparatus 50 into a semi-transparent state.

[0063] The operator terminal 12 outputs the masked image data. For example, the output unit 37i outputs the first image data masked by the masking unit 37h to the terminal device 13 of the instructor via the server device 11. Further, the output unit 37i outputs the second image data to the display unit 31.

[0064] The terminal device 13 displays the image received from the operator terminal 12 via the server device 11. For example, the terminal device 13 displays the image of the first image data. The instructor 52 views the image displayed on the terminal device 13 and remotely supports the work. FIG. 8 is a diagram showing an example of the image of the first image data according to the first embodiment. As shown in FIG. 8, the image of the first image data has the area outside the area of the substrate processing apparatus 50 in an opaque state. Thereby, the area outside the area of the substrate processing apparatus 50 can be concealed. For example, surrounding devices such as the devices 61 and 62 can be concealed. Further, the substrate processing apparatus 50 is shown in the image of the first image data. Thereby, since the instructor 52 can visually recognize the substrate processing apparatus 50 from the image of the first image data, the work on the substrate processing apparatus 50 can be smoothly supported.

[0065] The operator terminal 12 displays the image of the second image data on the display unit 31. FIG. 9 is a diagram showing an example of the image of the second image data according to the first embodiment. As shown in FIG. 9, in the image of the ninth image data, the area outside the area of the substrate processing apparatus 50 is in a semi-transparent state, and the surrounding apparatuses 70 and 71 can be seen in a semi-transparent state. Since the operator 51 can see the substrate processing apparatus 50 through the display unit 31, the operator 51 can secure the necessary field of view with respect to the substrate processing apparatus 50 and perform the work safely. Further, since the area outside the area of the substrate processing apparatus 50 can be seen in a semi-transparent state through the display unit 31 by the operator 51, the surroundings can be grasped. For example, the operator 51 can grasp that the apparatuses 70 and 71 are present through the display unit 31.

[0066] Note that the case where the operator terminal 12 displays the image of the second image data on the display unit 31 has been described as an example, but it is not limited thereto. For example, the operator terminal 12 may make the display unit 31 in a transparent state without displaying the image of the second image data on the display unit 31 so that the operator 51 can see the surroundings through the display unit 31. Further, the operator terminal 12 may be configured to be able to switch and display the image of the first image data on the display unit 31 in response to an operation by hand or voice. Thereby, the operator 51 can confirm the image of the first image data displayed on the terminal device 13 of the instructor 52, and can confirm whether the area outside the area of the substrate processing apparatus 50 is in an opaque state and the area outside the area of the substrate processing apparatus 50 in the image is concealed.

[0067] Here, the worker terminal 12 according to the present embodiment identifies the shape of the substrate processing apparatus 50 arranged at the arrangement position with the shooting position as the viewpoint position, and based on the identified shape, identifies the area of the substrate processing apparatus 50 within the image of the image data. Then, the worker terminal 12 performs mask processing on the area other than the area of the substrate processing apparatus 50 in the image of the image data. For this reason, for example, even when the worker 51 moves within the reel room 60 and the shape, size, and surface shown of the substrate processing apparatus 50 change in the image of the image data captured by the imaging unit 32, the area other than the area of the substrate processing apparatus 50 can be masked. Thereby, even if the shape, size, and surface shown of the substrate processing apparatus 50 change in the image, the image can be restricted so that the area other than the area of the substrate processing apparatus 50 is concealed.

[0068] Further, the support system 10 according to the present embodiment generates, in the worker terminal 12, first image data subjected to mask processing that makes the area other than the area of the substrate processing apparatus 50 in an opaque state, and outputs the first image data to the terminal device 13 of the instructing person via the server device 11. Thereby, the first image data is transmitted to the network N. The first image data is subjected to mask processing that makes the area other than the area of the substrate processing apparatus 50 in an opaque state. Thereby, for example, even when the network N is sniffed and the first image data is illegally read, the area other than the area of the substrate processing apparatus 50 in the image can be concealed.

[0069] [Processing Flow] Next, the procedure of the image restriction process executed by the worker terminal 12 according to the first embodiment will be described. FIG. 10 is a flowchart showing the procedure of the image restriction process according to the first embodiment. Note that the flow shown in the flowchart of FIG. 10 shows the flow of transmitting an image when image transmission is instructed by the operation reception unit 37b.

[0070] The image acquisition unit 37c acquires image data obtained by photographing the space in which the object to be monitored is arranged (step S10). For example, the photographing control unit 37a causes the photographing unit 32 to photograph an image at a predetermined frame rate. The image acquisition unit 37c acquires the image data photographed by the photographing unit 32.

[0071] The imaging position acquisition unit 37d acquires imaging position data indicating the imaging position at which the imaging unit 32 has imaged the object to be monitored within the space where the object to be monitored is arranged (step S11). For example, the imaging position acquisition unit 37d specifies the surrounding shape from the image data obtained by the imaging unit 32 and the information obtained by the sensor unit 33, performs spatial mapping, and acquires spatial information indicating the shape of the space where the object to be monitored is arranged. The imaging position acquisition unit 37d specifies the position within the spatially mapped spatial information from the specified surrounding shape, and acquires the imaging position data with the specified position as the imaging position of the imaging unit 32.

[0072] The position specification unit 37f specifies the arrangement position of the object to be monitored within the space where the object to be monitored is arranged (step S12). For example, the position specification unit 37f specifies the arrangement position of the object to be monitored within the space of the spatial information with reference to the reference point 64.

[0073] The region specification unit 37g specifies the region of the object to be monitored within the image of the image data acquired by the image acquisition unit 37c (step S13). For example, the region specification unit 37g arranges a voxel indicating the three-dimensional shape of the object to be monitored based on the shape information at the position that is the arrangement position of the object to be monitored in the space of the spatially mapped spatial information. Then, the region specification unit 37g specifies the shape seen from the voxel at the position that is the imaging position indicated by the imaging position data within the space of the spatial information.

[0074] The mask processing unit 37h performs mask processing on the region other than the region of the object to be monitored specified by the region specification unit 37g in the image of the image data acquired by the image acquisition unit 37c (step S14). The output unit 37i outputs the image data mask-processed by the mask processing unit 37h (step S15).

[0075] The output unit 37i determines whether an instruction to stop transmitting an image has been given (step S16). If the stop of transmission has not been instructed (step S16: No), the process proceeds to step S10 described above. On the other hand, if the stop of transmission has been instructed (step S16: Yes), the process ends.

[0076] As described above, the worker terminal 12 according to the first embodiment includes an image acquisition unit 37c, a shooting position acquisition unit 37d, a shape acquisition unit 37e, a position identification unit 37f, a region identification unit 37g, a mask processing unit 37h, and an output unit 37i. The image acquisition unit 37c acquires image data obtained by the imaging unit 32 capturing the space (clean room 60) in which the object to be monitored (substrate processing apparatus 50) is arranged. The shooting position acquisition unit 37d acquires shooting position data indicating the shooting position of the imaging unit 32 in the space when the object to be monitored is shot. The shape acquisition unit 37e acquires shape information indicating the three-dimensional shape of the object to be monitored. The position identification unit 37f identifies the arrangement position of the object to be monitored in the space. The region identification unit 37g identifies the shape of the object to be monitored arranged at the arrangement position with the shooting position indicated by the shooting position data as the viewpoint position based on the shooting position data acquired by the shooting position acquisition unit 37d, the shape information acquired by the shape acquisition unit 37e, and the arrangement position of the object to be monitored identified by the position identification unit 37f. Based on the identification result, the region identification unit 37g identifies the region of the object to be monitored in the image of the image data acquired by the image acquisition unit 37c. The mask processing unit 37h performs mask processing on the region of the image of the image data acquired by the image acquisition unit 37c other than the region of the object to be monitored identified by the region identification unit 37g. The output unit 37i outputs the image data mask-processed by the mask processing unit 37h. Thereby, the worker terminal 12 can limit the image so that the region other than the region of the object to be monitored is concealed. Further, the worker terminal 12 generates image data (video data) that is mask-processed following the movement of the worker by MR technology. In MR technology, based on a virtual space, the space is recognized and the virtual space and the real space are fused. By generating image data (video data) that is mask-processed following the movement of the worker by the MR technology, the worker terminal 12 can avoid the possibility that confidential information that may be reflected in the region other than the region of the object to be monitored is accidentally sent to the outside (the terminal device 13 of the instructor 52 who gives remote instructions). Further, although the viewpoint moves due to changes in position and the like during the work of the worker, by using MR technology, the worker terminal 12 can easily follow the movement of the viewpoint of the worker and limit the image.

[0077] Further, the shape acquisition unit 37e specifies the three-dimensional shape of the object to be monitored from the image data obtained by photographing the object to be monitored from the surroundings, and corrects the specified three-dimensional shape of the object to be monitored according to a correction instruction to obtain shape information. As a result, the operator terminal 12 can correct the three-dimensional shape of the specified object to be monitored to the correct shape. As a result, the entire object to be monitored can be imaged. Further, the image can be accurately limited so that objects other than the object to be monitored are not imaged.

[0078] Further, the mask processing unit 37h generates first image data obtained by performing mask processing to make the area other than the area of the object to be monitored in the image of the image data opaque, and second image data obtained by performing mask processing to make the area other than the area of the object to be monitored in the image of the image data semi-transparent. The output unit 37i outputs the first image data to the terminal device 13 of the instructor 52 who remotely instructs the work on the object to be monitored. The output unit 37i outputs the second image data to the display unit 31 of the head-mounted display worn by the worker 51 who performs the work on the object to be monitored according to the instruction of the instructor. As a result, the area other than the area of the object to be monitored in the image can be concealed from the instructor 52. Further, since the image can be provided to the worker 51 in a state where the surroundings can be grasped, the worker 51 can secure a necessary field of view and perform the work safely.

[0079] [Second Embodiment] Next, the second embodiment will be described. Since the support system 10, the operator terminal 12, and the terminal device 13 according to the second embodiment have the same configuration as those in the first embodiment, the description of the same parts will be omitted, and mainly the different points will be described.

[0080] FIG. 11 is a diagram showing an example of the functional configuration of the server device 11 according to the second embodiment. Since the server device 11 according to the second embodiment has a configuration partly similar to that of the server device 11 according to the first embodiment shown in FIG. 2, the same reference numerals are given to the same parts and the description thereof is omitted, and mainly the different parts will be described.

[0081] The storage unit 21 further stores the shape information 21b. The shape information 21b is data indicating the three-dimensional shape of the monitoring target for each type of the monitoring target. In the storage unit 21, for each type of the monitoring target, the shape information 21b of the monitoring target of the corresponding type is stored in association with the type. For example, when the monitoring target is the substrate processing apparatus 50, the shape information 21b is data indicating the three-dimensional shape of the substrate processing apparatus 50. In the storage unit 21, for each type such as the model number of the substrate processing apparatus 50, the shape information 21b of the substrate processing apparatus 50 of the corresponding type is stored in association with the type.

[0082] Each shape information 21b stored in the storage unit 21 of the server device 11 can be read from the operator terminal 12. In the operator terminal 12, the shape information corresponding to the type of the monitoring target is acquired from the storage unit 21 of the server device 11. For example, the server device 11 notifies the operator terminal 12 of information on the type of the substrate processing apparatus 50 such as the model number associated with each shape information 21b. The operation reception unit 37b displays the type information and receives an instruction to read the shape information 21b. The operator 51 designates the type and instructs to read the shape information 21b. When a read is instructed by the operation reception unit 37b, the shape acquisition unit 37e acquires the shape information 21b of the type for which the read is instructed from the storage unit 21 of the server device 11.

[0083] Note that the shape acquisition unit 37e may identify the type of the monitoring target from the image of the image data captured by the imaging unit 32, and acquire the shape information 21b of the identified type from the storage unit 21 of the server device 11. For example, when the substrate processing apparatus 50 is provided with identification information such as numbers, characters, marks, or code images indicating the type of the substrate processing apparatus 50 such as the model number, the shape acquisition unit 37e identifies the type of the substrate processing apparatus 50 from the identification information appearing in the image of the image data captured by the imaging unit 32. Then, the shape acquisition unit 37e may acquire the shape information 21b of the identified type from the storage unit 21 of the server device 11.

[0084] As a result, the worker terminal 12 can acquire the shape information 21b corresponding to the shape of the monitoring object without performing spatial mapping by acquiring the shape information 21b from the storage unit 21 of the server device 11. At the worker terminal 12, the worker aligns the shape of the monitoring object indicated by the shape information 21b with the space of the spatial information. As a result, the position specifying unit 37f can specify the arrangement position of the monitoring object in the space of the aligned spatial information with reference to the reference point.

[0085] As described above, the server device 11 according to the second embodiment stores, in the storage unit 21, shape information 21b indicating the three-dimensional shape of the monitoring object for each type of the monitoring object. At the worker terminal 12, the shape acquisition unit 37e acquires the shape information 21b corresponding to the type of the monitoring object photographed by the photographing unit 32 from the storage unit 21 of the server device 11. As a result, the worker terminal 12 can acquire the shape information 21b corresponding to the shape of the monitoring object without acquiring the shape information from the monitoring object.

[0086] The shape information 21b acquired from the storage unit 21 of the server device 11 may be modifiable. For example, the shape acquisition unit 37e arranges voxels according to the shape of the monitoring object indicated by the shape information 21b, and displays the arranged voxels on the display unit 31. The operation reception unit 37b receives a modification instruction for the displayed voxels by hand or voice. The shape acquisition unit 37e may perform a modification of adding or deleting voxels according to the modification instruction received by the operation reception unit 37b, and modify the shape information 21b according to the modification.

[0087] [Third Embodiment] Next, the third embodiment will be described. Since the support system 10, the server device 11, the worker terminal 12, and the terminal device 13 according to the third embodiment have the same configuration as those in the first or second embodiment, the description of the same parts will be omitted, and mainly the different points will be described.

[0088] The worker terminal 12 further receives a designation of a display range of the monitoring object. For example, the operation reception unit 37b receives a designation of a display range of the monitoring object by an operation by hand or voice.

[0089] The mask processing unit 37h performs mask processing on the areas other than the area of the monitoring target object specified by the area specifying unit 37g in the image of the image data acquired by the image acquisition unit 37c and the areas outside the range received by the operation reception unit 37b. For example, the mask processing unit 37h generates first image data obtained by performing mask processing to make the areas other than the area of the monitoring target object in the image of the image data acquired by the image acquisition unit 37c and the areas outside the range received by the operation reception unit 37b in an opaque state. Further, the mask processing unit 37h generates second image data obtained by performing mask processing to make the areas other than the area of the monitoring target object in the image of the image data acquired by the image acquisition unit 37c in a semi-transparent state.

[0090] A specific example will be described. FIG. 12 is an example of an image displayed on the display unit 31 of the worker terminal 12 according to the third embodiment. FIG. 12 is an image of the second image data, and the substrate processing apparatus 50 and the surroundings are visible. The worker designates the display range of the substrate processing apparatus 50 by an operation using hands or voice. In FIG. 12, the display range 80 of the substrate processing apparatus 50 is designated.

[0091] The worker terminal 12 generates first image data obtained by performing mask processing to make the areas other than the area of the substrate processing apparatus 50 in the image of the image data captured by the imaging unit 32 and the areas outside the range 80 in an opaque state. The first image data is transmitted to the terminal device 13 of the instructing person via the server device 11 and is displayed on the terminal device 13. FIG. 13 is a diagram showing an example of an image of the first image data according to the third embodiment. As shown in FIG. 13, in the image of the first image data, the areas other than the area within the range 80 of the substrate processing apparatus 50 are in an opaque state. Thereby, the image is restricted so that unnecessary parts and objects to be concealed in work such as maintenance do not flow out via the server device 11, for example, to the outside of the clean room.

[0092] As described above, in the worker terminal 12 according to the third embodiment, the operation reception unit 37b (reception unit) receives the designation of the display range of the object to be monitored. The mask processing unit 37h performs mask processing on the area outside the range received by the operation reception unit 37b in the image of the image data acquired by the image acquisition unit 37c. Thereby, the worker terminal 12 can limit the image so that unnecessary image areas are not included in the object to be monitored.

[0093] As described above, the embodiments have been described. However, the embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. In fact, the above-described embodiments can be embodied in various forms. Also, the above-described embodiments may be omitted, replaced, or changed in various forms without departing from the scope and spirit of the appended claims.

[0094] For example, in the above embodiment, the case where the worker terminal 12 is configured as a head-mounted display that can be worn on the worker's head has been described as an example, but it is not limited thereto. The worker terminal 12 may be configured to be divided into a plurality of housings. For example, the worker terminal 12 may be configured by a head-mounted device worn on the worker's head and an information processing device that can communicate with the head-mounted device by wire or wirelessly. The head-mounted device is provided with a display unit 31, a photographing unit 32, a sensor unit 33, a voice input unit 34, a voice output unit 35, and the like. The information processing device is provided with a communication I / F unit 30, a storage unit 36, and a control unit 37, and may perform the processing of each embodiment based on various information such as image data obtained from the head-mounted unit. In this case, the information processing device corresponds to the information processing device of the present disclosure. Also, the processing of each embodiment may be performed by the server device 11. In this case, the server device 11 corresponds to the information processing device of the present disclosure.

[0095] Note that the embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. In fact, the above-described embodiments can be embodied in various forms. Also, the above-described embodiments may be omitted, replaced, or changed in various forms without departing from the scope and spirit of the appended patent claims.

Description of Symbols

[0096] 10 Support System 11 Server Device 12 Operator Terminal 13 Terminal Device 20 Communication I / F Section 21 Memory Section 21a, 21b Shape Information 22 Control Section 30 Communication I / F Section 31 Display Section 32 Photographing Section 33 Sensor Section 34 Voice Input Section 35 Voice Output Section 36 Memory Section 37 Control Section 37a Photographing Control Section 37b Operation Reception Section 37c Image Acquisition Section 37d Photographing Position Acquisition Section 37e Shape Acquisition Section 37f Position Identification Section 37g Region Identification Section 37h Mask Processing Section 37i Output Section 50 Substrate Processing Apparatus 51 Operator 52 Instructor 60 Clean Room

Claims

1. A step of acquiring image data obtained by photographing a space in which a monitoring object is arranged by a photographing unit; A step of acquiring photographing position data indicating the photographing position of the photographing unit in the space when the monitoring object is photographed; A step of acquiring shape information indicating the three-dimensional shape of the monitoring object; A step of specifying the arrangement position of the monitoring object in the space; Based on the photographing position data, the shape information, and the arrangement position of the monitoring object, the shape of the monitoring object arranged at the arrangement position with the photographing position indicated by the photographing position data as the viewpoint position is specified, and based on the specific result, the region of the monitoring object in the image of the acquired image data is specified; A step of masking a region other than the region of the specified monitoring object in the image of the acquired image data; A step of outputting the masked image data; An image limiting method having the above.

2. In the step of acquiring the shape information, the three-dimensional shape of the monitoring object is specified from the image data obtained by photographing the monitoring object from the surroundings, and the three-dimensional shape of the specified monitoring object is corrected according to a correction instruction to obtain the shape information. The image limiting method according to Claim 1.

3. In the step of acquiring the shape information, for each type of monitoring object, shape information corresponding to the type of the monitoring object photographed by the photographing unit is acquired from a storage unit that stores shape information indicating the three-dimensional shape of the monitoring object. The image limiting method according to Claim 1.

4. In the masking step, first image data obtained by masking a region other than the region of the monitoring object in the image of the image data to an opaque state and second image data obtained by masking a region other than the region of the monitoring object in the image of the image data to a semi-transparent state are generated. In the outputting step, the first image data is output to a terminal device of an instructor who remotely instructs an operation on the monitoring object, and the second image data is output to a display unit. The image limiting method according to any one of Claims 1 to 3.

5. The first image data and the second image data are image data created by recognizing the real space in which the monitoring object is arranged from the acquired image data and fusing the virtual space and the real space. The image limiting method according to Claim 4.

6. The method further includes a step of receiving a designation of a display range of the monitoring object. The step of performing the mask process masks an area outside the received range in the image of the acquired image data. The image limiting method according to any one of claims 1 to 5.

7. A step of acquiring image data obtained by photographing, by a photographing unit, a space in which a monitoring target is arranged; A step of acquiring photographing position data indicating the photographing position of the photographing unit in the space when the monitoring target is photographed; A step of acquiring shape information indicating the three-dimensional shape of the monitoring target; A step of specifying the arrangement position of the monitoring target in the space; Based on the photographing position data, the shape information, and the arrangement position of the monitoring target, specifying the shape of the monitoring target arranged at the arrangement position having the photographing position indicated by the photographing position data as a viewpoint position, and based on the specification result, specifying the area of the monitoring target in the image of the acquired image data; A step of masking an area outside the area of the specified monitoring target in the image of the acquired image data; A step of outputting the masked image data; An image limiting program for causing a computer to execute the above.

8. An image acquisition unit configured to acquire image data obtained by photographing, by a photographing unit, a space in which a monitoring target is arranged; A photographing position acquisition unit configured to acquire photographing position data indicating the photographing position of the photographing unit in the space when the monitoring target is photographed; A shape acquisition unit configured to acquire shape information indicating the three-dimensional shape of the monitoring target; A position specifying unit configured to specify the arrangement position of the monitoring target in the space; Based on the photographing position data acquired by the photographing position acquisition unit, the shape information acquired by the shape acquisition unit, and the arrangement position of the monitoring target specified by the position specifying unit, specifying the shape of the monitoring target arranged at the arrangement position having the photographing position indicated by the photographing position data as a viewpoint position, and based on the specification result, an area specifying unit configured to specify the area of the monitoring target in the image of the image data acquired by the image acquisition unit; A mask processing unit configured to mask an area outside the area of the monitoring target specified by the area specifying unit in the image of the image data acquired by the image acquisition unit; An output unit configured to output the image data masked by the mask processing unit; An information processing apparatus having the above components.

9. The shape acquisition unit identifies the three-dimensional shape of the object to be monitored from the image data obtained by photographing the object to be monitored from the surroundings, and modifies the identified three-dimensional shape of the object to be monitored according to a modification instruction to obtain the shape information. The information processing apparatus according to claim 8.

10. The shape acquisition unit is configured to obtain, from a storage unit that stores shape information indicating the three-dimensional shape of the object to be monitored for each type of the object to be monitored, shape information corresponding to the type of the object to be monitored photographed by the photographing unit. The information processing apparatus according to claim 8.

11. The mask processing unit is configured to generate first image data obtained by performing mask processing to make a region other than the region of the object to be monitored in the image of the image data in an opaque state, and second image data obtained by performing mask processing to make a region other than the region of the object to be monitored in the image of the image data in a semi-transparent state. The output unit is configured to output the first image data to a terminal device of an instructor who remotely instructs an operation on the object to be monitored, and output the second image data to a display unit. The information processing apparatus according to any one of claims 8 to 10.

12. The first image data and the second image data are image data created by recognizing the real space in which the object to be monitored is arranged from the acquired image data and fusing the virtual space and the real space. The information processing apparatus according to claim 11.

13. The apparatus further includes a reception unit configured to receive a designation of a range for displaying the object to be monitored, The mask processing unit performs mask processing on a region other than the range received by the reception unit in the image of the image data acquired by the image acquisition unit. The information processing apparatus according to any one of claims 8 to 12.

14. It is communicable with a wearable device worn by an operator who works on the object to be monitored, and is configured to control the wearable device. The information processing apparatus according to any one of claims 8 to 13.

15. A support system in which an operator terminal of an operator who works on an object to be monitored communicates with a terminal device of an instructor who remotely instructs an operation on the object to be monitored, The operator terminal includes: an image acquisition unit that acquires image data obtained by photographing the space in which the object to be monitored is arranged by a photographing unit; a photographing position acquisition unit that acquires photographing position data indicating the photographing position of the photographing unit in the space when the object to be monitored is photographed. A shape acquisition unit that acquires shape information indicating the three-dimensional shape of the object to be monitored; A position identification unit that identifies the arrangement position of the object to be monitored in the space; Based on the imaging position data acquired by the imaging position acquisition unit, the shape information acquired by the shape acquisition unit, and the arrangement position of the object to be monitored identified by the position identification unit, the shape of the object to be monitored arranged at the arrangement position with the imaging position indicated by the imaging position data as the viewpoint position is identified, and based on the identification result, a region identification unit that identifies the region of the object to be monitored in the image of the image data acquired by the image acquisition unit; A mask processing unit that performs mask processing on a region other than the region of the object to be monitored identified by the region identification unit in the image of the image data acquired by the image acquisition unit; An output unit that outputs the image data mask-processed by the mask processing unit; It has, The terminal device displays the mask-processed image data. Support system.

16. The support system has a server device that relays communication between the operator terminal and the terminal device. The server device relays the image data output from the operator terminal to the terminal device. The terminal device displays the image data relayed by the server device. The support system according to claim 15.

Images