Maintenance support system, maintenance support method, program, method for generating processed image, and processed image
By using SLAM technology in the maintenance auxiliary system to determine the three-dimensional area of the maintenance object and the non-confidential area, generate and transmit the processing images, the problems of information leakage and limited camera conditions in the prior art are solved, and information security and maintenance operations are realized Improved immediacy and efficiency.
Patent Information
- Application Number
- CN202080004994.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-07-16
- Filing Date
- 2020-03-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-03-19
AI Technical Summary
Existing maintenance assistance systems are prone to leaking confidential information when transmitting fault information, and the camera conditions are restricted, making it difficult to respond to maintenance needs immediately.
Wearable terminals are used to determine the three-dimensional area of maintenance objects and non-confidential areas through SLAM technology, and processed images that do not contain confidential information are generated, and sent to maintenance assistants through the Internet to ensure information security and flexibility in operation.
It effectively prevents the leakage of confidential information other than the maintenance object, reduces the constraints on camera conditions, and improves the immediacy and efficiency of maintenance operations.
Smart Images

Figure CN112655027B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a maintenance support system, a maintenance support method, a program, a method for generating a processed image, and a processed image for supporting maintenance work of an industrial device to be maintained. In particular, the present invention relates to a maintenance support system, a maintenance support method, a program, a method for generating a processed image, and a processed image that can prevent leakage of confidential information other than the industrial device to be maintained and have few restrictions on imaging conditions. Background Art
[0002] In the past, when a malfunction occurred in industrial equipment such as a substrate processing device, the following response was generally adopted: after confirming the condition of the malfunction, the maintenance personnel at the installation site of the industrial equipment to be maintained would communicate the condition of the malfunction to the maintenance assistant at the manufacturing plant side of the maintenance object by telephone, and the maintenance assistant would issue various instructions to the maintenance personnel by telephone.
[0003] If the failure cannot be solved despite the above telephone response, the following response is taken: a skilled worker is dispatched to the installation site of the maintenance object at a timing when the skilled worker at the manufacturing plant is scheduled to be available.
[0004] In the case of the above-mentioned telephone response, since it is a verbal communication, there is a situation that the fault status cannot be well communicated to the maintenance assistant, or the instructions of the maintenance assistant cannot be well communicated to the maintenance operator, so that the fault cannot be solved. As needed, the maintenance operator sometimes sends the camera image of the maintenance object obtained to the maintenance assistant via email, etc., and the maintenance assistant understands the fault status by confirming the camera image, but there is a problem that it takes time to understand the status.
[0005] In addition, in the case of the above-mentioned response of dispatching skilled workers, there is a problem that it is difficult to respond promptly.
[0006] In order to solve the problems mentioned above, the following maintenance assistance system is proposed: the maintenance worker wears a wearable terminal (head-mounted display) equipped with a camera unit and a display, uses the wearable terminal to photograph the maintenance object, and sends the camera image (moving image) to the maintenance assistant side terminal via an electrical communication line such as the Internet.
[0007] However, in the above maintenance support system, when the camera image is sent directly to the maintenance support personnel terminal, the camera image may contain confidential information of the installation site. When the camera image containing confidential information is sent to the maintenance support personnel side, the maintenance support personnel may know the confidential information, which becomes a problem.
[0008] Therefore, for example, a maintenance support system described in Patent Document 1 (an image processing system in Patent Document 1) has been proposed as a maintenance support system capable of preventing leakage of confidential information.
[0009] The system described in Patent Document 1 is an image processing system comprising: an acquisition unit that acquires an original image that captures an original area; an identification unit that identifies one or more identifiers in the original image; a determination unit that determines at least one of a first image portion that captures a first area in the original area and a second image portion that captures a second area after the first area is removed from the original area based on the one or more identifiers identified by the identification unit; and a generation unit that generates a processed image including the first image portion based on the determination result of the determination unit (Claim 1 of Patent Document 1).
[0010] According to the system described in Patent Document 1, it is considered possible to prevent leakage of confidential information or the like by making the first area a maintenance target and making the second area confidential information or the like other than the maintenance target.
[0011] However, in the system described in Patent Document 1, there is a restriction that the original image must contain the identifier. Specifically, Patent Document 1 gives an example of setting the internal area of the printing device 300 (a rectangular area surrounded by four identifiers 30) as a maintenance object, and describes the following: the original area changes in accordance with the movement of the operator 10 wearing the wearable tool 100 equipped with the camera 170, and the original images 171a to 173a that capture different original areas are obtained, but any of the original images 171a to 173a must contain the identifier 30 (Paragraphs 0063 to 0065 of Patent Document 1, Figure 6 , Figure 7 wait).
[0012] Therefore, for example, when one wants to approach the maintenance object and photograph the details of the maintenance object, there is a restriction that the approach can only be made within the range where the identifier is included in the original image.
[0013] In addition, in the system described in Patent Document 1, the first image portion (the portion corresponding to the maintenance object) is two-dimensionally determined based on the identifier, which results in the restriction that the maintenance object with the identifier attached must be photographed from a specific direction facing the identifier.
[0014] Prior art literature
[0015] Patent Literature
[0016] Patent Document 1: Japanese Patent Application Publication No. 2017-211766 Summary of the invention
[0017] Problem that the invention aims to solve
[0018] The present invention is completed in order to solve the problems of the above-mentioned prior art, and its subject is to provide a maintenance assistance system, maintenance assistance method, program, method for generating processed images and processed images that can prevent the leakage of confidential information other than industrial equipment that is the object of maintenance and have fewer restrictions on camera conditions.
[0019] Solutions for solving problems
[0020] In order to solve the above-mentioned problem, the present invention provides a maintenance assistance system, comprising: a wearable terminal, which is worn by a maintenance worker and has a camera unit; a first determination unit, which determines a specified first three-dimensional area including an industrial equipment to be maintained and / or a specified second three-dimensional area other than the industrial equipment to be maintained, based on a specified reference point in a camera image acquired by the camera unit in an initial state; a second determination unit, which determines a mask pixel area other than a valid pixel area corresponding to the first three-dimensional area and / or a mask pixel area corresponding to the second three-dimensional area in a camera image acquired by the camera unit in a moved state in which the wearable terminal has moved; a processed image generation unit, which generates a processed image obtained by setting the mask pixel area determined by the second determination unit to be invisible for the camera image acquired by the camera unit in the moved state; and a communication unit, which sends the processed image generated by the processed image generation unit to a maintenance assistant side terminal operated by the maintenance assistant.
[0021] According to the present invention, a first determination unit determines a first three-dimensional region including industrial equipment to be maintained and / or a second three-dimensional region excluding industrial equipment to be maintained, using a predetermined reference point in a camera image acquired by a camera unit in an initial state as a reference. The "camera unit in an initial state" of the present invention refers to a camera unit in a state where a predetermined reference point is located in the camera image.
[0022] Specifically, for example, when an arbitrary point of a captured image is specified using an AR cursor and the specified position is used as a reference point, the camera unit in the initial state refers to a camera unit in a position where it can capture the point specified using the AR cursor.
[0023] In addition, for example, the camera image is subjected to prescribed image processing by the first determination unit to extract a pixel area in the camera image corresponding to a characteristic part (for example, a corner of a maintenance object, etc.), and when the center of the pixel area is used as a reference point, the camera unit in the initial state refers to a camera unit in a position where it can capture the characteristic part.
[0024] Furthermore, for example, when the reference point is defined by a mark added to a maintenance object or the like, the imaging unit in the initial state refers to the imaging unit in a position where the mark can be imaged.
[0025] In addition, for example, in the case where the first three-dimensional region and the second three-dimensional region are rectangular three-dimensional regions, the first three-dimensional region and the second three-dimensional region are defined by the coordinates of each of the eight vertices of the three-dimensional region represented by a three-dimensional coordinate system based on a reference point.
[0026] In the above case, for example, the coordinates of each vertex based on the reference point are pre-stored in the first determination unit or each vertex is specified using an AR cursor, and the well-known SLAM (Simultaneous Localization and Mapping) technology is used as a technology for constructing a map of the sensor's own position and the surrounding environment in an unknown environment, so that the first determination unit can determine the first three-dimensional area and / or the second three-dimensional area.
[0027] Fig.10 This is an explanatory diagram for explaining the outline of SLAM technology.
[0028] like Fig.10 As shown, in the SLAM technology, the following first to fourth steps are performed.
[0029] (1) First step: First, in two captured images of the object captured at different times t0 and t1, corresponding identical points (feature points) are detected. Fig.10 In the example shown, corners of the imaging object are detected as feature points.
[0030] (2) Step 2: Next, the position of the feature points in the two camera images is compared to calculate the change in the camera unit's posture (parallel / rotational component). In addition, based on the combination of feature points, the position of the feature points in three-dimensional space is determined using the principle of triangulation. This is equivalent to building an environmental map.
[0031] (3) Third step: In two captured images obtained by capturing the object at separate times t1 and t2, the change in position of feature points in three-dimensional space is calculated, and the position of the imaging unit itself is estimated based on the change.
[0032] (4) Fourth step: register and update feature points in three-dimensional space (update the environment map).
[0033] Afterwards, in the SLAM technology, the third step and the fourth step are repeatedly executed.
[0034] As described above, the SLAM technology is a technology that constantly grasps a three-dimensional space by alternately repeating the construction of an environment map (restoration of a three-dimensional shape) and the estimation of the own position of an imaging unit.
[0035] The reference point of the present invention is arbitrarily set in the three-dimensional space after constructing the environment map (restoring the three-dimensional shape, in other words, grasping the three-dimensional space).
[0036] In addition, more specific contents of SLAM technology are well known, such as those recorded in the Information Processing Society Research Report Vol. 2014-CVIM-190 No. 40 "Three-dimensional shape restoration / self-position estimation based on moving camera images (SLAM) and high-density three-dimensional shape restoration", so the detailed description above is omitted in this specification.
[0037] Next, according to the present invention, a second determination unit is used to determine a mask pixel area other than the effective pixel area corresponding to the first three-dimensional area in a camera image acquired by the camera unit in a moving state in which the wearable terminal has moved, and / or to determine a mask pixel area corresponding to the second three-dimensional area. In the case where only the first three-dimensional area is determined by the first determination unit, the mask pixel area other than the effective pixel area corresponding to the first three-dimensional area is determined by the second determination unit. In addition, in the case where only the second three-dimensional area is determined by the first determination unit, the mask pixel area corresponding to the second three-dimensional area is determined by the second determination unit. And, in the case where the first three-dimensional area and the second three-dimensional area are determined by the first determination unit, the mask pixel area other than the effective pixel area corresponding to the first three-dimensional area and the mask pixel area corresponding to the second three-dimensional area are determined by the second determination unit.
[0038] As described above, the first three-dimensional area and / or the second three-dimensional area based on the reference point in the camera image acquired by the camera unit in the initial state is determined by the first determination unit, so that in the moved state where the wearable terminal has moved, the first three-dimensional area and / or the second three-dimensional area based on the reference point in the camera image acquired by the camera unit can be determined by using, for example, SLAM technology. Moreover, when determining the first three-dimensional area and / or the second three-dimensional area in the moved state, the camera image acquired by the camera unit in the moved state may not necessarily contain the reference point.
[0039] Next, according to the present invention, a processed image is generated by making the mask pixel area determined by the second determination unit (the mask pixel area other than the effective pixel area corresponding to the first three-dimensional area and / or the mask pixel area corresponding to the second three-dimensional area) invisible. As the "processed image obtained by making the mask pixel area invisible" of the present invention, for example, an image in which the captured area is the same as the camera image but the mask pixel area in the camera image is blackened, and an image obtained by cutting out only the pixel area other than the mask pixel area from the camera image.
[0040] In the case where the mask pixel area determined by the second determination unit is a mask pixel area other than the effective pixel area corresponding to the first three-dimensional area, the first three-dimensional area includes the maintenance object, and the effective pixel area corresponds to the first three-dimensional area. Therefore, as long as the first three-dimensional area is not set to be excessively large compared to the maintenance object, the processed image obtained by setting the mask pixel area other than the effective pixel area to invisible will not include pixel areas corresponding to confidential information other than the maintenance object.
[0041] In addition, in a case where the mask pixel area determined by the second determination unit is a mask pixel area corresponding to the second three-dimensional area, the second three-dimensional area does not include the maintenance object. Therefore, by presetting the second three-dimensional area to include at least confidential information other than the maintenance object, the processed image obtained by setting the mask pixel area to invisible does not include pixel areas corresponding to confidential information other than the maintenance object.
[0042] Finally, according to the present invention, the processed image generated by the processed image generating unit is sent to the maintenance assistant side terminal operated by the maintenance assistant by using the communication unit. Therefore, at the maintenance assistant side terminal, the maintenance object contained in the first three-dimensional area can be visually identified, and / or confidential information other than the maintenance object contained in the second three-dimensional area cannot be visually identified, thereby preventing the leakage of confidential information other than the maintenance object.
[0043] As described above, according to the present invention, as long as the reference point is included in the camera image acquired by the camera unit in the initial state, the reference point does not need to be included in the camera image acquired by the camera unit in the moving state after the wearable terminal has moved. In addition, according to the present invention, a prescribed first three-dimensional area including the industrial equipment to be maintained and / or a prescribed second three-dimensional area other than the industrial equipment to be maintained is determined, so there is no restriction on the camera direction of the camera unit. Therefore, it is possible to prevent the leakage of confidential information other than the maintenance object, and there are fewer restrictions on the camera conditions.
[0044] As in the present invention, a processed image is generated by setting the mask pixel area other than the effective pixel area corresponding to the specified first three-dimensional area including the maintenance object and / or the mask pixel area corresponding to the specified second three-dimensional area other than the maintenance object to be invisible. The inventor calls the technology for preventing the leakage of confidential information other than the maintenance object contained in the mask pixel area "Virtual Partition" technology.
[0045] In addition, the "industrial equipment" of the present invention includes mining machinery, chemical machinery, environmental equipment, power transmission equipment, containers, commercial washing machines, boiler engines, plastic machinery, wind power and hydraulic machinery, transportation machinery, iron and steel machinery, etc., which are defined as industrial machinery by the Japan Industrial Machinery Industry Association. In addition, from another point of view, the "industrial equipment" of the present invention includes manufacturing devices for manufacturing tangible objects. The substrate processing device is a type of industrial equipment, and is a type of manufacturing device for manufacturing tangible objects in terms of manufacturing processed substrates (for example, substrates after etching treatment, substrates after film forming treatment). IT systems that are electrically connected to electronic devices via servers are not included in manufacturing devices for manufacturing tangible objects, nor are they included in industrial equipment.
[0046] In addition, in the present invention, "the state after the wearable terminal has moved" does not only refer to the state in which the wearable terminal has reached its moving end point (for example, the position of the wearable terminal when the maintenance work is completed), but refers to the state in which the wearable terminal has moved to each position at a predetermined short time interval during the moving process before reaching the moving end point. Therefore, the processed image generation unit generates a processed image in which the mask pixel area is set to invisible for all the camera images acquired by the camera unit in real time (at predetermined short time intervals) before reaching the moving end point.
[0047] In the present invention, the following method can also be used: the wearable terminal is moved to assist maintenance in such a manner that the reference point is included in the camera image acquired by the camera unit in the initial state and the camera image acquired by the camera unit in the moved state always includes the reference point. However, in this method, the same restriction of the camera condition as that of the system described in Patent Document 1 occurs.
[0048] According to the present invention, a method can be adopted in which the reference point does not need to be included in the camera image acquired by the camera unit in the moved state. In other words, maintenance assistance can be performed by removing the restriction of the movement position of the wearable terminal by moving the camera unit from a position where the reference point exists within the field of view of the camera unit to a position where the reference point does not exist.
[0049] In the present invention, for example, the reference point is specified by a mark.
[0050] In the case where the reference point is specified by a mark, for example, the structure is as follows: after the first determination unit grasps the three-dimensional space using the camera image acquired by the camera unit (for example, by using the SLAM technology of the camera image), it determines a specified first three-dimensional area including the industrial equipment to be maintained and / or a specified second three-dimensional area excluding the industrial equipment to be maintained, based on the reference point in the camera image acquired by the camera unit in an initial state.
[0051] Preferably, the second determination unit includes a plurality of second imaging units having different imaging directions, and determines the mask pixel region based on the captured images acquired by the plurality of second imaging units.
[0052] According to the above preferred structure, the second determination unit has a plurality of second camera units with different camera directions, so, for example, it is possible to apply SLAM technology using so-called binocular stereoscopic vision (stereoscopic method) based on the camera images acquired by the plurality of second camera units. In addition, for example, the SLAM technology using monocular vision is applied to each camera image in the camera images acquired by the second camera unit, and they are combined, thereby making it possible to determine the mask pixel area with high accuracy.
[0053] Preferably, the wearable terminal is a head-mounted display having a display, and the head-mounted display is configured to be switchable between a state in which the captured image is displayed on the display and a state in which the processed image is displayed on the display in a recognizable manner.
[0054] According to the above preferred structure, when the camera image is displayed on the display, for example, unlike the case where a processed image obtained by painting the mask pixel area black is displayed, there is no part that blocks the field of view of the maintenance worker, so maintenance work can be performed safely. In addition, the "state of displaying the camera image" of the above preferred structure is the following concept: in addition to including the state of actually displaying the camera image acquired by the camera unit on the display as in an MR head-mounted device (the state of the maintenance worker visually recognizing the represented camera image), it also includes the state of the display being a transmissive type such that the maintenance worker can directly visually recognize the maintenance object through the transmissive display without any display, as in a transmissive AR head-mounted device.
[0055] On the other hand, according to the above-mentioned preferred structure, when the processed image is displayed in a recognizable manner on the display, the maintenance personnel can also recognize the processed image sent to the terminal on the maintenance assistant side. Therefore, the maintenance personnel can also confirm that confidential information other than the maintenance object has been prevented from being leaked, which can give the maintenance personnel a sense of security. In addition, the "state of displaying the processed image in a recognizable manner" in the above-mentioned preferred structure includes not only the state of displaying the same image as the processed image sent to the terminal on the maintenance assistant side (an image in which the captured area is the same as the camera image but the mask pixel area in the camera image is blackened, an image obtained by cutting out only the pixel area other than the mask pixel area from the camera image, etc.), but also the state of displaying the image obtained by covering the boundary line between the mask pixel area and the pixel area other than the mask pixel area in the camera image, and the state of displaying the processed image in a reduced size on a part of the display screen (e.g., a corner).
[0056] Preferably, the first determining unit, the second determining unit, the processed image generating unit, and the communication unit are installed in the wearable terminal.
[0057] According to the above preferred structure, all units that perform a series of actions until a processed image is generated by making the mask pixel area in the camera image obtained by shooting the maintenance object invisible and sending the processed image to the terminal on the maintenance assistant side are installed in the wearable terminal. This is an inside-out structure that uses a sensor installed on the wearable terminal itself to detect the position of the wearable terminal (camera unit). In the case of an outside-in structure that uses an external device independent of the wearable terminal to detect the position of the wearable terminal (for example, a structure that uses a camera unit, a sensor, etc. that are set outside and are separate from the camera unit of the wearable terminal to detect the position of the wearable terminal (camera unit of the wearable terminal), although there is an advantage of reducing the power consumption of the wearable terminal, there are disadvantages such as the restriction on the environmental side that there is no shielding between the external device and the wearable terminal, and the high cost due to the need for high-precision external devices. Therefore, by setting it as the above preferred structure, the advantages of low cost and less environmental restrictions can be obtained.
[0058] Preferably, the present invention further comprises the maintenance support staff side terminal, and the maintenance support staff side terminal is configured to be able to perform data communication bidirectionally with the communication unit.
[0059] According to the above-mentioned preferred structure, the terminal on the maintenance assistant side and the communication unit can perform data communication in both directions, so not only can processed images be sent from the communication unit to the terminal on the maintenance assistant side, but also images for maintenance work or sound data for indicating the content of maintenance work can be sent from the terminal on the maintenance assistant side to the communication unit, etc., and effective maintenance work can be performed using only the maintenance assistance system involved in the present invention.
[0060] In the above-mentioned preferred structure, it is more preferred that the structure is as follows: it also has a server, which is electrically connected to the communication unit and the maintenance assistant side terminal, and stores first identification information as identification information of the maintenance worker and second identification information as identification information of the maintenance assistant. The server determines whether the identification information sent from the communication unit to the server matches the first identification information stored in the server, and determines whether the identification information sent from the maintenance assistant side terminal to the server matches the second identification information stored in the server. When the identification information sent from the communication unit to the server matches the first identification information, and the identification information sent from the maintenance assistant side terminal to the server matches the second identification information, two-way data communication between the communication unit and the maintenance assistant side terminal can be performed via the server.
[0061] According to the above-mentioned more preferred structure, the server is used to determine whether the identification information sent from the communication unit to the server matches the first identification information (identification information of the maintenance worker) stored in the server. In other words, the server is used to determine whether the identification information of the maintenance worker using the maintenance assistance system involved in the present invention is sent to the server. In addition, the server is used to determine whether the identification information sent from the maintenance assistant side terminal to the server matches the second identification information (identification information of the maintenance assistant) stored in the server. In other words, the server is used to determine whether the identification information of the maintenance assistant using the maintenance assistance system involved in the present invention is sent to the server. Then, in the case where the identification information sent from the communication unit to the server matches the first identification information, and the identification information sent from the maintenance assistant side terminal to the server matches the second identification information (in other words, when the identification information of the maintenance worker using the maintenance assistance system involved in the present invention and the identification information of the maintenance assistant are both authenticated when they are sent to the server), two-way data communication between the communication unit and the maintenance assistant side terminal can be performed via the server.
[0062] According to the above-mentioned more preferred structure, two-way data communication between the communication unit and the terminal on the maintenance assistant side can only be carried out when both the identification information of the maintenance worker and the identification information of the maintenance assistant who use the maintenance assistance system involved in the present invention are sent to the server and are authenticated. Therefore, only predetermined maintenance workers and maintenance assistants can use the maintenance assistance system involved in the present invention, and confidential information other than industrial equipment that is the maintenance object can be more reliably prevented from being leaked.
[0063] In addition to the above-mentioned preferred structure or the above-mentioned more preferred structure, a structure is also adopted that can communicate data from the industrial equipment that is the maintenance object (for example, a substrate processing device, etc.) to the terminal on the maintenance assistant side. For example, as long as the processing log data (measurement values and set values related to the processing of the maintenance object) obtained from the maintenance object is sent to the terminal on the maintenance assistant side, the maintenance assistant can refer to the processing log data to indicate the content of the maintenance work, and can perform maintenance work with better results.
[0064] In addition, in order to solve the above-mentioned problem, the present invention also provides a maintenance assistance method, using a wearable terminal equipped with a camera unit worn by a maintenance worker, the maintenance assistance method comprising: a first determination step, using the first determination unit to determine a specified first three-dimensional area including industrial equipment as a maintenance object and / or a specified second three-dimensional area other than the industrial equipment as a maintenance object based on a specified reference point in a camera image acquired by the camera unit in an initial state; a second determination step, using the second determination unit to determine a mask pixel area other than a valid pixel area corresponding to the first three-dimensional area in a camera image acquired by the camera unit in a moved state in which the wearable terminal has moved, and / or to determine a mask pixel area corresponding to the second three-dimensional area; a processed image generation step, using the processed image generation unit to generate a processed image obtained by setting the mask pixel area determined in the second determination step to be invisible for the camera image acquired by the camera unit in the moved state; and a communication step, using the communication unit to send the processed image generated in the processed image generation step to a terminal on the maintenance assistant side.
[0065] In addition, in order to solve the problem, the present invention also provides a program for enabling the first determination step, the second determination step, the processing image generation step and the communication step included in the maintenance assistance method to be executed by the first determination unit, the second determination unit, the processing image generation unit and the communication unit respectively.
[0066] Furthermore, it is possible to provide a computer (CPU) readable storage medium storing the above-mentioned program.
[0067] In addition, in order to solve the above-mentioned problem, the present invention also provides a method for generating a processed image, in a camera image acquired by a camera unit of a wearable terminal worn by a maintenance worker, determining a mask pixel area other than a valid pixel area corresponding to a specified first three-dimensional area including industrial equipment that is a maintenance object, and / or determining a mask pixel area corresponding to a specified second three-dimensional area other than the industrial equipment that is a maintenance object, and generating a processed image in which the mask pixel area is set to invisible.
[0068] In addition, in order to solve the above-mentioned problem, the present invention also provides a method for generating a processed image, which generates a processed image using a camera image acquired by a camera unit of a wearable terminal worn by a maintenance worker, the generation method comprising: a first determination step, using the first determination unit to determine a specified first three-dimensional area including industrial equipment as a maintenance object and / or a specified second three-dimensional area other than the industrial equipment as a maintenance object, based on a specified reference point in the camera image acquired by the camera unit in an initial state; a second determination step, using the second determination unit to determine a mask pixel area other than a valid pixel area corresponding to the first three-dimensional area in the camera image acquired by the camera unit in a moved state in which the wearable terminal has moved, and / or determine a mask pixel area corresponding to the second three-dimensional area; and a processed image generation step, using the processed image generation unit to generate a processed image in which the mask pixel area determined in the second determination step is set invisible for the camera image acquired by the camera unit in the moved state.
[0069] In addition, in order to solve the above-mentioned problem, the present invention also provides a processed image, which is obtained by setting invisible a mask pixel area except for a valid pixel area corresponding to a specified first three-dimensional area including the industrial equipment to be maintained and / or a mask pixel area corresponding to a specified second three-dimensional area except for the industrial equipment to be maintained in a camera image acquired by a camera unit of a wearable terminal worn by a maintenance worker.
[0070] Furthermore, in order to solve the above-mentioned problem, the present invention also provides a processed image generated by the above-mentioned generation method, wherein the processed image is obtained by setting the mask pixel area other than the effective pixel area corresponding to the first three-dimensional area including the industrial equipment to be maintained and / or the mask pixel area corresponding to the second three-dimensional area other than the industrial equipment to be maintained to invisible.
[0071] Effects of the Invention
[0072] According to the present invention, it is possible to prevent leakage of confidential information other than the maintenance target, and to reduce restrictions on imaging conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Figure 1 It is a diagram schematically showing a general configuration of a maintenance support system according to the first embodiment of the present invention.
[0074] Figure 2 Yes Description Figure 1 The diagram is an explanatory diagram of the schematic operation of the maintenance support system shown.
[0075] Figure 3 It is shown by Figure 1 FIG. 4 is a diagram showing an example of a captured image acquired by the imaging unit in an initial state and a processed image generated by the processed image generating unit.
[0076] Figure 4 It is shown Figure 1 FIG. 1 is a diagram showing an example of a captured image acquired by an imaging unit and a processed image generated by a processed image generating unit in a state where the wearable terminal has moved.
[0077] Figure 5 It is an explanatory diagram for explaining a setting example of the first three-dimensional area in the maintenance support system according to the second embodiment of the present invention.
[0078] Figure 6 It is an explanatory diagram for explaining a second three-dimensional area used in the maintenance support system according to the third embodiment of the present invention.
[0079] Figure 7 It is an explanatory diagram for explaining a setting example of the first three-dimensional area in the maintenance support system according to the fourth embodiment of the present invention and a schematic operation of the second specifying unit at that time.
[0080] Figure 8 Is the description Figure 7 This is a top view showing the schematic operation of the second specifying unit when the first three-dimensional area is set as shown.
[0081] Fig. 9 It is shown in Figure 7 FIG. 1 is a diagram showing an example of a processed image generated by the processed image generating unit when the first three-dimensional region is set and the second three-dimensional region is set as shown.
[0082] Fig.10 This is an explanatory diagram for explaining the outline of SLAM technology. DETAILED DESCRIPTION
[0083] Hereinafter, a maintenance support system according to an embodiment of the present invention will be described with reference to the drawings.
[0084] <First Embodiment>
[0085] Figure 1 It is a diagram schematically showing a general configuration of a maintenance support system according to the first embodiment of the present invention. Figure 1 (a) is the overall structure diagram, Figure 1 (b) is a diagram showing a wearable terminal and components mounted on the wearable terminal, Figure 1 (c) shows Figure 1 (b) is a block diagram of the internal structure of the control signal processing unit shown.
[0086] like Figure 1 As shown in (a) of FIG. 1 , the maintenance support system 100 according to the present embodiment includes a wearable terminal 1, a maintenance support staff terminal 2, and a control signal processing unit 3. In addition, the maintenance support system 100 includes a server 6 as a preferred configuration.
[0087] The wearable terminal 1 is a terminal worn by a maintenance worker who is at a site where a maintenance target (industrial equipment such as a substrate processing apparatus) T is installed and performs maintenance work.
[0088] like Figure 1 As shown in (b), the wearable terminal 1 of this embodiment is a glasses-type wearable terminal (head-mounted display), which has a frame 11 that the maintenance worker hangs on the ear. In addition, the wearable terminal 1 has a camera unit 12 in the center of the front side of the frame 11 (a position corresponding to the vicinity of the brow of the maintenance worker when the maintenance worker hangs the frame 11 on the ear). A color (RGB) camera is used as the camera unit 12 of this embodiment. Moreover, the camera direction (line of sight direction) L1 of the camera unit 12 is set to the direction facing the brow of the maintenance worker when the maintenance worker hangs the frame 11 on the ear. In addition, the wearable terminal 1 has a display 13, which is mounted on the front side of the frame 11 and covers the front of the maintenance worker's field of vision when the maintenance worker hangs the frame 11 on the ear. A transmissive display is used as the display 13 of this embodiment. Therefore, when there is no display on the display 13, it can be used in the same way as ordinary glasses.
[0089] Furthermore, the imaging unit 12 can also be used as a hardware portion of a second determination unit 32 described later, as necessary.
[0090] like Figure 1As shown in (b), in the wearable terminal 1 of the present embodiment, a plurality of second camera units 32a and 32b having different camera directions are installed. Specifically, in the present embodiment, a pair of second camera units 32a and 32b are installed at the left and right ends of the front side of the frame 11, and the camera direction (line of sight direction) L2 of the second camera unit 32a and the camera direction (line of sight direction) L3 of the second camera unit 32b are set to different directions. As the second camera units 32a and 32b, for example, a color (RGB) camera can be used in the same manner as the camera unit 12. In addition, as the second camera units 32a and 32b, for example, a wide-angle camera having a wide-angle lens (including an ultra-wide-angle lens and a fisheye lens) can also be used. In addition, the second camera units 32a and 32b can be used as the hardware part of the second determination unit 32 described later.
[0091] like Figure 1 As shown in (b), the wearable terminal 1 of the present embodiment is equipped with a distance image sensor (also called a depth camera, a 3D camera) 32c having a line of sight direction substantially the same as that of the camera unit 12. Specifically, the distance image sensor 32c is installed below the camera unit 12 at the center of the front side of the frame 11. As the distance image sensor 32c, for example, the following distance image sensor is used: the distance to the subject in the field of view is measured by a TOF (Time Of Flight) method, and a distance image is obtained in which the distance to the subject is represented by the concentration value of each pixel. In addition, the distance image sensor 32c can be used as a hardware part of the second determination unit 32 described later.
[0092] In addition, the wearable terminal 1 of the present embodiment is equipped with an inertial sensor 32d. Specifically, the inertial sensor 32d is installed on the side of the frame 11. The inertial sensor 32d includes, for example, a 3-axis acceleration sensor, a 3-axis angular velocity sensor (gyro sensor) and a 3-axis geomagnetic (orientation) sensor. In addition, the inertial sensor 32d can be used as a hardware part of the second determination unit 32 described later.
[0093] Furthermore, a speaker 4 and a microphone 5 are mounted on the side of the frame 11 of the wearable terminal 1 of the present embodiment.
[0094] The wearable terminal 1 having the above-described configuration can be configured by using, for example, “HoloLens” (registered trademark), which is smart glasses manufactured by Microsoft Corporation, and improving it.
[0095] Generally speaking, the maintenance assistant side terminal 2 is a terminal operated by a maintenance assistant at a location away from the installation site of the maintenance object T. The maintenance assistant side terminal 2 is electrically connected to the wearable terminal 1 (specifically, the control signal processing unit 3 installed on the wearable terminal 1) through an electrical communication line N such as the Internet. Specifically, the maintenance assistant side terminal 2 of the present embodiment is electrically connected to the wearable terminal 1 via a server 6 electrically connected to the electrical communication line N. Moreover, the maintenance assistant side terminal 2 is configured to be able to perform data communication bidirectionally with the communication unit 34 described later. As the maintenance assistant side terminal 2, for example, a desktop computer can be used, but it is not limited to this. As long as it can display processed images as described later, various terminals such as portable computers, tablet computers, smart phones, etc. can be used.
[0096] The maintenance assistant terminal 2 of the present embodiment is not an essential structure, but is also electrically connected to the maintenance object T through the electrical communication line N. Specifically, in the present embodiment, for example, a computer (in Figure 1 (a) (not shown), the computer is electrically connected to the maintenance assistant side terminal 2 via a server 6 electrically connected to the electrical communication line N. In addition, the processing log data obtained from the maintenance object T is sequentially sent to the maintenance assistant side terminal 2 and stored in the maintenance assistant side terminal 2. By providing the server 6, it is easy to manage the identification information of the wearable terminal 1 or the maintenance object T connected to the maintenance assistant side terminal 2 and manage the use history.
[0097] The control signal processing unit 3 is installed on the wearable terminal 1 (on the side of the frame 11), and is electrically connected to the various structural elements of the camera unit 12, the display 13, the second camera unit 32a, 32b, the distance image sensor 32c, the inertial sensor 32d, the speaker 4 and the microphone 5, and has the function of controlling each structural element or processing the output signal of each structural element.
[0098] The control signal processing unit 3 is mainly composed of a CPU, a memory such as a ROM or a RAM, and a program stored in the memory for causing the CPU to execute an operation described later.
[0099] like Figure 1As shown in (c), the control signal processing unit 3 includes: a program that causes the CPU to execute the action of the first determination unit 31; a program that causes the CPU to execute the action of the software part 32e of the second determination unit 32, and a program that causes the CPU to execute the action of the processed image generation unit 33. In addition, the control signal processing unit 3 has a communication unit 34. The communication unit 34 is composed of an antenna, a program that causes the antenna to operate, and the like.
[0100] The above-described program may be updated by directly connecting a predetermined computer to the control signal processing unit 3 via a USB cable or the like, or may be directly updated from the server 6 .
[0101] In this embodiment, the first determination unit 31, the second determination unit 32 (the camera unit 12, the second camera units 32a, 32b, at least one of the distance image sensor 32c and the inertial sensor 32d, and the software part 32e), the processed image generation unit 33 and the communication unit 34 are all installed in the wearable terminal 1. This is an inside-out structure in which the position of the wearable terminal 1 (camera unit 12) is detected by a sensor installed in the wearable terminal 1 itself as described above.
[0102] Below, in addition to reference Figure 1 In addition, refer to Figure 2 to Figure 4 The operation (maintenance support method) of the maintenance support system 100 according to the present embodiment having the above-described schematic configuration will be described.
[0103] Initially, when the maintenance support system 100 according to the present embodiment is used, the server 6 authenticates both the maintenance worker's identification information and the maintenance support worker's identification information. Specifically, it determines whether both the maintenance worker's identification information and the maintenance support worker's identification information have been sent to the server 6.
[0104] Before using the maintenance support system 100, the first identification information as the identification information of the maintenance worker and the second identification information as the identification information of the maintenance support worker are pre-stored in the server 6. Specifically, for example, the maintenance support worker uses the keyboard of the maintenance support worker side terminal 2 to input the first identification information (for example, ID number) of the maintenance worker who has decided to use the maintenance support system 100 into the maintenance support worker side terminal 2, and the input first identification information is transmitted to the server 6 through the electrical communication line N and stored in the server 6. Similarly, for example, the maintenance support worker uses the keyboard of the maintenance support worker side terminal 2 to input the second identification information (for example, ID number) of the maintenance worker who has decided to use the maintenance support system 100 into the maintenance support worker side terminal 2, and the input second identification information is transmitted to the server 6 through the electrical communication line N and stored in the server 6. Alternatively, the first identification information of the maintenance worker who may use the maintenance assistance system 100 and the second identification information of the maintenance assistant are stored in the maintenance assistant side terminal 2 in advance, and the maintenance assistant uses the keyboard of the maintenance assistant side terminal 2 to select the first identification information of the maintenance worker who actually uses the maintenance assistance system 100 from now on and the second identification information of the maintenance assistant from the first identification information and the second identification information stored in the maintenance assistant side terminal 2, and the selected first identification information and second identification information are sent to the server 6 via the electrical communication line N and stored in the server 6.
[0105] In the present embodiment, when the first identification information and the second identification information are stored in the server 6 , the utilization start time and utilization time of the maintenance support system 100 by the maintenance worker and the maintenance support worker are also stored in the server 6 .
[0106] At or near the start time of the use of the maintenance support system 100, identification information is transmitted from the communication unit 34 of the wearable terminal 1 to the server 6. The identification information transmitted from the communication unit 34 is, for example, identification information of the wearable terminal 1 pre-stored in the control signal processing unit 3 of the wearable terminal 1. In this case, the wearable terminal 1 worn by the maintenance worker is predetermined, and the identification information of the wearable terminal 1 is transmitted to the server 6 as identification information of the maintenance worker (first identification information). Similarly, at or near the start time of the use of the maintenance support system 100, identification information is transmitted from the maintenance assistant side terminal 2 to the server 6. The identification information transmitted from the maintenance assistant side terminal 2 is, for example, second identification information of the maintenance assistant input by the maintenance assistant using the keyboard of the maintenance assistant side terminal 2. When the maintenance assistant who operates the maintenance assistant side terminal 2 is predetermined, the identification information of the maintenance assistant side terminal 2 pre-stored in the maintenance assistant side terminal 2 may be transmitted to the server 6 as identification information of the maintenance assistant (second identification information).
[0107] The server 6 determines whether the identification information transmitted from the communication unit 34 to the server 6 as described above matches the first identification information stored in the server 6. Similarly, the server 6 determines whether the identification information transmitted from the maintenance assistant side terminal 2 to the server 6 as described above matches the second identification information stored in the server 6. Then, in the case where the identification information transmitted from the communication unit 34 to the server 6 matches the first identification information, and the identification information transmitted from the maintenance assistant side terminal 2 to the server 6 matches the second identification information (in the case where both the identification information of the maintenance worker who uses the maintenance assistance system 100 and the identification information of the maintenance assistant are transmitted to the server 6 and are authenticated), bidirectional data communication between the communication unit 34 and the maintenance assistant side terminal 2 can be performed via the server 6 from the utilization start time until the utilization time has passed.
[0108] In the maintenance support system 100 according to the present embodiment, as described above, bidirectional data communication between the communication unit 34 and the maintenance support staff side terminal 2 is possible only when both the identification information of the maintenance worker and the identification information of the maintenance support staff are authenticated when they are sent to the server 6. Therefore, only the maintenance worker and the maintenance support staff who are predetermined can use the maintenance support system 100, and leakage of confidential information other than the maintenance object T can be more reliably prevented.
[0109] Furthermore, it is also possible to configure such that, at the time point when bidirectional data communication between the communication unit 34 and the maintenance assistant terminal 2 is enabled as described above, voice data can be bidirectionally transmitted and received, and the transmitted and received voice data is stored in the server 6 and / or the maintenance assistant terminal 2. By storing the voice data, a history of maintenance assistance can be reliably recorded.
[0110] In addition, it can also be configured as follows: when at least one of the maintenance worker and the maintenance assistant wants to terminate or interrupt the use of the maintenance assistance system 100 before the use time has passed, the specified instruction information for instructing the termination or interruption of use is sent from the communication unit 34 of the wearable terminal 1 and / or the maintenance assistant side terminal 2 to the server 6, thereby the server 6 terminates or interrupts the two-way data communication between the communication unit 34 and the maintenance assistant side terminal 2.
[0111] Furthermore, in the case where there are multiple wearable terminals 1 and multiple terminals 2 on the maintenance assistant side, a method can be adopted in which all two-way data communications between the terminals 1 and 2 can be performed only when all identification information sent from the terminals 1 and 2 is authenticated in the server 6, or a method can be adopted in which two-way data communications can only be performed between the terminals 1 and 2 that have been authenticated.
[0112] Figure 2 1 is an explanatory diagram for explaining the general operation of the maintenance support system 100 according to the present embodiment. Figure 2 In FIG. 1 , the maintenance worker wearing the wearable terminal 1 is omitted from the illustration. Figure 3 1 is a diagram showing an example of a captured image acquired by the imaging unit 12 in the initial state and a processed image generated by the processed image generating unit 33 . Figure 3 (a) shows an example of a camera image. Figure 3 (b) shows an example of a processed image. Figure 4 1 is a diagram showing an example of a captured image acquired by the imaging unit 12 and a processed image generated by the processed image generating unit 33 in a state where the wearable terminal 1 has moved. Figure 4 (a) shows an example of a camera image. Figure 4 (b) shows an example of a processed image. Figure 3 and Figure 4 In FIG. 1 , for convenience of explanation, the boundary line of the first three-dimensional area TA is shown by a dotted line, but the boundary line does not exist in the actual camera image.
[0113] like Figure 2As shown in FIG. 1 , members T1 to T3 are provided on the front surface of the maintenance object T in the present embodiment. Next to the maintenance object T, a device S related to confidential information is arranged.
[0114] First, the maintenance worker wearing the wearable terminal 1 moves to a position facing the maintenance object T, so that the camera unit 12 of the wearable terminal 1 can capture the maintenance object T. Figure 2 The camera unit 12 of the wearable terminal 1 shown in solid line in the figure captures the maintenance object T to obtain the maintenance object T. Figure 3 The camera image (moving image) shown in (a).
[0115] Next, for example, the maintenance worker uses the AR cursor to designate an arbitrary point of the camera image displayed on the display 13, whereby the first determination unit 31 determines the designated point as the reference point M and stores it in the first determination unit 31. In addition, the first determination unit 31 can also extract a pixel area corresponding to a characteristic portion (e.g., a corner of the maintenance object T) in the camera image by performing a predetermined image processing on the camera image, and determine the center of the pixel area as the reference point M and store it. Furthermore, when a mark is added to the maintenance object T (a mark such as an AR mark, a QR code (registered trademark), or a barcode is added by pasting, printing, or engraving), the first determination unit 31 can also extract a pixel area corresponding to the mark by performing a predetermined image processing on the camera image, and determine the center of the pixel area as the reference point M.
[0116] Moreover, in the present embodiment, the first determination unit 31 determines a predetermined first three-dimensional area (for example, a three-dimensional area in a rectangular parallelepiped shape) TA including the maintenance object T based on the reference point M. The first three-dimensional area TA is a three-dimensional area in the virtual space. Specifically, for example, a three-dimensional coordinate system ( Figure 2 The first determination unit 31 determines the first three-dimensional area TA based on the coordinates of the vertices P1 to P8 of the first three-dimensional area TA represented by a three-dimensional coordinate system having an X-axis, a Y-axis and a Z-axis (as shown). The first determination unit 31 uses, for example, SLAM technology to determine the first three-dimensional area TA based on the coordinates of the vertices P1 to P8.
[0117] In addition, the maintenance worker may use the AR cursor to specify each vertex P1 to P8 of the first three-dimensional area TA with respect to the camera image displayed on the display 13, and the first determination unit 31 may determine the first three-dimensional area TA using, for example, the SLAM technology. In this case, at the moment when the vertices P1 to P8 of the first three-dimensional area TA are specified using the AR cursor, since the vertices P1 to P8 are configured in the three-dimensional coordinate system based on the reference point M, the coordinates of the vertices P1 to P8 are already determined, and as a result, the first three-dimensional area TA can be determined.
[0118] Furthermore, when a mark is added to the maintenance object T, the first determination unit 31 can also calculate the feature quantity of the pixel area corresponding to the mark, determine the identification information of the mark based on the feature quantity, and determine the first three-dimensional area TA using, for example, SLAM technology based on the coordinates of each vertex P1 to P8 stored in the first determination unit 31 in association with the determined identification information. In this way, when the reference point M is specified by the mark, the coordinates of each vertex P1 to P8 have been stored in the first determination unit 31, and the maintenance operator does not need to set them, so there is no need to spend time. In the case where the shape of the maintenance object T is complex, it is particularly effective to set each vertex P1 to P8 (or more than 8 vertices) in detail in advance based on the design drawings of the maintenance object T and associate them with the identification information of the mark.
[0119] In addition, Figure 2 , a case where the wearable terminal 1 worn by the maintenance worker moves within the first three-dimensional area TA is illustrated.
[0120] Next, the maintenance worker wearing the wearable terminal 1 moves, and the wearable terminal 1 also moves. In the state where the wearable terminal 1 moves, the camera unit 12 (in Figure 2 The camera unit 12 of the wearable terminal 1 shown by the dotted line in the figure captures the maintenance object T to obtain the maintenance object T. Figure 4 The captured image shown in (a) does not include the reference point M.
[0121] Moreover, during the above-mentioned movement, the camera unit 12 and the second camera units 32a and 32b are used to continuously acquire camera images. In addition, the distance image sensor 32c is used to continuously acquire distance images. Furthermore, the acceleration, angular velocity, and geomagnetism (orientation) of the camera unit 12 of the wearable terminal 1 are continuously detected by the inertial sensor 32d.
[0122] The second determination unit 32 determines a mask pixel region other than a valid pixel region corresponding to the first three-dimensional area TA in a captured image acquired by the imaging unit 12 in a moved state in which the wearable terminal 1 has moved.
[0123] Specifically, when the second camera units 32a and 32b constitute the second determination unit 32, the software part 32e constituting the second determination unit 32 uses, for example, SLAM technology to construct a map of the wearable terminal 1 (camera unit 12)'s own position and the surrounding environment based on the camera images continuously acquired by the second camera units 32a and 32b during the movement, thereby determining the first three-dimensional area TA in the camera image after movement.
[0124] In addition, when the distance image sensor 32c constitutes the second determination unit 32, the software part 32e constituting the second determination unit 32 determines the first three-dimensional area TA in the camera image in the state after movement based on the distance image continuously acquired by the distance image sensor 32c during the movement, using, for example, SLAM technology.
[0125] The same is true when the camera unit 12 and the inertial sensor 32d constitute the second determination unit 32. In addition, it is also possible to combine any of the second camera units 32a and 32b, the distance image sensor 32c, the camera unit 12, and the inertial sensor 32d to constitute the second determination unit 32. In this case, the second determination unit 32 also determines the first three-dimensional area TA in the camera image in the moved state using, for example, the SLAM technology.
[0126] In addition, the second determination unit 32 of this embodiment does not use GPS when determining the first three-dimensional area TA in the camera image in the moved state (as described later, the same is true when determining the second three-dimensional area SA in the camera image in the moved state). This is because GPS not only cannot be used indoors but also generates a position detection error of at least several meters.
[0127] The second determination unit 32 determines the first three-dimensional area TA in the camera image in the moved state as described above, and determines the mask pixel area excluding the effective pixel area corresponding to the first three-dimensional area TA. Since it is possible to determine which part of the field of view of the camera unit 12 in the moved state the first three-dimensional area TA is captured by geometric operations, in other words, to determine which pixel area in the camera image acquired by the camera unit 12 the first three-dimensional area TA corresponds to, it is possible to determine the effective pixel area corresponding to the first three-dimensional area TA in the camera image, and determine the mask pixel area excluding the effective pixel area.
[0128] Next, the processed image generation unit 33 generates a processed image (moving image) in which the mask pixel area determined by the second determination unit 32 is made invisible for the camera image acquired by the camera unit 12 in the moved state. Figure 4 In the example shown in (b), an image in which the mask pixel area is painted black is generated as a processed image. However, the processed image is not limited to this, and it can also be an image obtained by cutting out only the effective pixel area from the camera image (cutting out only the pixel area other than the mask pixel area). In addition, mosaic processing or fogging can be performed instead of painting the mask pixel area black, and various methods can be used as long as it can prevent the leakage of confidential information.
[0129] Finally, the communication unit 34 transmits the processed image generated by the processed image generation unit 33 to the maintenance assistant terminal 2 via the electrical communication line N. Figure 4 In a state where the device S related to confidential information can be visually identified in the captured image as shown in (a) of FIG. 1 , the maintenance assistant side terminal 2 also performs the same Figure 4 As shown in (b) of FIG. 1 , only the maintenance target T included in the first three-dimensional area TA can be visually recognized, so that confidential information other than the maintenance target T can be prevented from being leaked.
[0130] In addition, the second determination unit 32, the processed image generation unit 33 and the communication unit 34 also Figure 3 The processing image generating unit 33 also generates the following image processing steps for the camera image obtained by the camera unit 12 in the initial state: Figure 3 The processed image shown in (b) is obtained by making the mask pixel area invisible, and the processed image is sent to the maintenance assistant side terminal 2.
[0131] Furthermore, if a malfunction or some problem occurs in the second camera units 32a, 32b, etc. constituting the second determination unit 32, it is impossible to accurately determine the change in the position of the camera unit 12 in the moved state relative to the initial state. Accordingly, it is impossible to accurately determine the effective pixel area corresponding to the first three-dimensional area TA in the camera image acquired by the camera unit 12 in the moved state. As a result, there is a possibility that confidential information other than the maintenance object T is included in the generated processed image.
[0132] Therefore, it is preferable to have the following function: when a failure or some problem occurs in the second camera unit 32a, 32b, etc., the processed image generation unit 33 generates a processed image in which all pixel areas are made invisible, or the communication unit 34 stops sending the processed image. In addition, at this time, it is preferable to display the meaning of the failure or some problem on the display 13, or to notify the meaning of the failure or some problem by sound through the speaker 4, so that the maintenance worker can quickly understand that the failure or some problem occurs in the second camera unit 32a, 32b, etc.
[0133] The display 13 of the present embodiment is configured to switch between a state in which a camera image is displayed on the display 13 (specifically, in the present embodiment, the transmissive display 13 has no display, and the maintenance worker can directly visually recognize the state of the maintenance object T through the display 13 without any display) and a state in which a processed image is displayed in a recognizable manner on the display 13. Specifically, for example, the following method can be adopted: a switching button (not shown) is provided on the frame 11, and by pressing the switching button, the control signal processing unit 3 is controlled to select a certain state.
[0134] In a state where the captured image is displayed on the display 13 (in a state where the maintenance worker can directly visually recognize the maintenance object T), there is no portion blocking the field of vision of the maintenance worker, so the maintenance work can be performed safely.
[0135] On the other hand, when the processed image is displayed in a recognizable manner on the display 13, the processed image sent to the maintenance assistant side terminal 2 can also be recognized by the maintenance personnel. Therefore, the maintenance personnel can also confirm that the confidential information other than the maintenance object T has been prevented from being leaked, which can give the maintenance personnel a sense of security. In addition, as a state in which the processed image is displayed in a recognizable manner, in addition to the state in which an image identical to the processed image sent to the maintenance assistant side terminal 2 (an image in which the captured area is the same as the camera image but the mask pixel area in the camera image is blackened, an image obtained by cutting out only the effective pixel area from the camera image, etc.), it also includes a state in which the processed image is displayed in a recognizable manner. Figure 3 (a) Figure 4 The state of an image obtained by covering the boundary line between the mask pixel area and the effective pixel area (the boundary line of the first three-dimensional area TA) in the captured image as shown in (a), the state of displaying the processed image in a reduced size on a part of the display screen, etc.
[0136] In addition, in the present embodiment, an example is given of determining one first three-dimensional area TA for one maintenance object T, but the present invention is not limited thereto. For example, when there are multiple maintenance objects T, it is sufficient to set, for example, a reference point M for each maintenance object T and determine the first three-dimensional area TA for each maintenance object T. For example, when one maintenance object T is a substrate processing device, and when repairs are performed on a specified operating machine after removing parts of the substrate processing device, it is sufficient to set another maintenance object T as the operating machine and also set the reference point M in the operating machine.
[0137] In addition, in this embodiment, a rectangular parallelepiped-shaped three-dimensional area with eight vertices P1 to P8 is used as an example as the first three-dimensional area TA, but the present invention is not limited thereto. The first three-dimensional area TA may be defined by more than eight points and may be a shape other than a rectangular parallelepiped, depending on the shape of the maintenance object T.
[0138] In the present embodiment, the case where the maintenance support staff terminal 2 is electrically connected to one wearable terminal 1 is described as an example, but the present invention is not limited to this. A configuration where a plurality of wearable terminals 1 are electrically connected to the maintenance support staff terminal 2 may also be adopted.
[0139] In this case, the camera units 12 of multiple wearable terminals 1 can be configured to determine the same first three-dimensional area TA based on the same reference point M, or can be configured to make the reference point M different or the determined first three-dimensional area TA different according to the wearable terminal 1.
[0140] In addition, various modes can be adopted, such as simultaneously displaying the processed images transmitted from the plurality of wearable terminals 1 in parallel on the maintenance support staff-side terminal 2 or being able to switch the processed images to be displayed.
[0141] Furthermore, the number of the wearable terminal 1 may be one or more, and a plurality of maintenance support staff terminals 2 electrically connected to the wearable terminal 1 may be prepared.
[0142] In addition, in this embodiment, if Figure 2 As shown, the area including the maintenance object T and the range in which the wearable terminal 1 moves is set as the first three-dimensional area TA, but the present invention is not limited to this. With respect to the area where it is predetermined that there is no device S related to confidential information, even if there is no maintenance object T or the wearable terminal 1 does not move, the first three-dimensional area TA can be further expanded to include the area. In addition, the first three-dimensional area TA is not limited to a single area formed by joining, and multiple separated areas can also be set as the first three-dimensional area TA.
[0143] <Second Embodiment>
[0144] The maintenance support system according to the second embodiment has the same Figure 1 The maintenance support system 100 according to the first embodiment described below has the same schematic configuration as that of the maintenance support system 100 according to the first embodiment.
[0145] However, the maintenance support system according to the second embodiment is different from the maintenance support system 100 according to the first embodiment in the setting of the first three-dimensional area TA. This point will be described below.
[0146] Figure 5 It is an explanatory diagram for explaining a setting example of the first three-dimensional area TA in the maintenance support system according to the second embodiment.
[0147] like Figure 5 As shown, in this embodiment, multiple (in Figure 5 In the example shown, 8 virtual partition walls W1 to W8 constitute the first three-dimensional area TA. Figure 5 In the example shown, the area surrounded by the eight partition walls W1 to W8 is set as the first three-dimensional area TA. For example, the maintenance worker can specify the partition walls W1 to W8 using the AR cursor with respect to the camera image displayed on the display 13. Thus, the first determination unit 31 determines the area surrounded by the partition walls W1 to W8 as the first three-dimensional area TA.
[0148] Preferably, the maintenance operator can freely specify Figure 5 The position, size, and number of the partition walls shown in the figure. According to this preferred structure, even if the shape and size of the maintenance object T are changed, the position, size, and number of the partition walls can be freely changed accordingly. Therefore, in addition to excellent versatility, it is also easy for the second determination unit 32 to determine the mask pixel area. In addition, for example, in the case where the maintenance object T includes not only the substrate processing device body but also accessories (for example, pumps, coolers, etc.) that are slightly away from the substrate processing device body, the maintenance object T can be easily changed. Figure 2 Compared with the first three-dimensional area TA of such a fixed size, the first three-dimensional area TA can be configured by appropriately arranging the partition walls (easy to change the layout).
[0149] In addition, as long as the device S related to confidential information cannot be visually recognized by the maintenance assistant side terminal 2 (that is, becomes invisible in the processed image), a certain gap may be provided between the partition walls. Figure 5, there are illustrated examples of several gaps between partition walls W1 and W2, between partition walls W2 and W3, between partition walls W5 and W6, and between partition walls W6 and W7. However, in order to reliably prevent the device S related to confidential information from being visually identified by the maintenance assistant side terminal 2, it is more preferred that the partition walls have overlapping portions when viewed from the imaging direction (line of sight) L1 of the imaging unit 12 of the wearable terminal 1 within the first three-dimensional area TA.
[0150] In addition, Figure 5 In the example shown, no partition walls are provided on the ceiling or floor of the location where the maintenance object T is located. However, partition walls may be freely provided that the device S related to confidential information around the maintenance object T cannot be visually identified by the terminal 2 on the maintenance assistant side.
[0151] <Third Embodiment>
[0152] The maintenance support system according to the third embodiment also has the same Figure 1 The maintenance support system 100 according to the first embodiment described below has the same schematic configuration as that of the maintenance support system 100 according to the first embodiment.
[0153] However, the maintenance support system according to the third embodiment is different from the maintenance support system 100 according to the first embodiment in that the second three-dimensional area SA is used instead of the first three-dimensional area TA. This point will be described below.
[0154] In the maintenance support system according to the third embodiment, the first determination unit 31 determines a predetermined second three-dimensional area (for example, a second three-dimensional area) other than the maintenance object T based on the reference point M. Figure 6 As shown, a second three-dimensional area SA including the device S related to confidential information is determined, and the second determination unit 32 determines the mask pixel area corresponding to the second three-dimensional area SA. The second three-dimensional area SA is a three-dimensional area in the virtual space. For the case of determining the second three-dimensional area SA, the above-mentioned contents described for the case of determining the first three-dimensional area TA can also be applied in the same manner.
[0155] The first determination unit 31 may determine the second three-dimensional area SA instead of determining the first three-dimensional area TA, or may determine both the first three-dimensional area TA and the second three-dimensional area SA instead of determining the first three-dimensional area TA. In the case where the first determination unit 31 determines the second three-dimensional area SA instead of determining the first three-dimensional area TA, the second determination unit 32 determines the mask pixel area corresponding to the second three-dimensional area SA. In the case where the first determination unit 31 determines both the first three-dimensional area TA and the second three-dimensional area SA, the second determination unit 32 determines the mask pixel area other than the effective pixel area corresponding to the first three-dimensional area TA, and determines the mask pixel area corresponding to the second three-dimensional area SA. In addition, in the case of determining both parties, the mask pixel area other than the effective pixel area corresponding to the first three-dimensional area TA and the mask pixel area corresponding to the second three-dimensional area SA may sometimes overlap. In this case, as long as the pixel areas determined as mask pixel areas in at least one party are all mask pixel areas.
[0156] <Fourth Embodiment>
[0157] The maintenance support system according to the fourth embodiment also has the same Figure 1 The maintenance support system 100 according to the first embodiment described below has the same schematic configuration as that of the maintenance support system 100 according to the first embodiment.
[0158] However, the maintenance support system according to the fourth embodiment is different from the maintenance support system 100 according to the first embodiment and the maintenance support system according to the second embodiment in the setting of the first three-dimensional area TA. This point will be described below. In addition, in the following description, a case where there are two devices (devices S1 and S2) related to confidential information is exemplified.
[0159] Figure 7 It is an explanatory diagram for explaining a setting example of the first three-dimensional area TA in the maintenance support system according to the fourth embodiment and a schematic operation of the second specifying unit 32 at that time. Figure 7 (a) is a stereoscopic diagram showing a setting example of the first three-dimensional area TA. Figure 7 (b) is a top view illustrating the general operation of the second determination unit 32, showing that the wearable terminal 1 is located at the front surface ( Figure 7 The center of the surface (a) on which the components T1 to T3 are provided) is directly opposite to the ( Figure 7 The wearable terminal 1 is in a state where the imaging direction L1 of the imaging unit 12 is orthogonal to the front surface of the maintenance object T as shown in (a).
[0160] Figure 8 Is the description Figure 7This is a plan view showing a schematic operation of the second specifying unit 32 when the first three-dimensional area TA is set as shown. Figure 8 (a) shows that the wearable terminal 1 is located from Figure 7 The state shown in (b) is the state of the position after being translated toward the device S2 side. Figure 8 (b) shows that the wearable terminal 1 is located from Figure 7 The state shown in (b) is a state in which the device is rotated and moved toward the device S2 side (moved counterclockwise around the maintenance object T). Figure 8 (c) shows that the wearable terminal 1 is Figure 7 From the state shown in (b), the direction is changed to the left on the spot.
[0161] In addition, Figure 7 (b) Figure 8 In the figure, the wearable terminal 1 is illustrated as a rectangle for convenience.
[0162] In the aforementioned Figure 2 In the example shown, an area including the maintenance object T and the range in which the wearable terminal 1 moves is set as the first three-dimensional area TA, but in Figure 7 In the example shown in (a), the first three-dimensional area TA is set to match the maintenance target T, and the wearable terminal 1 moves outside the first three-dimensional area TA.
[0163] like Figure 7 As shown in (b), when the wearable terminal 1 is located at a position directly opposite to the center of the front surface of the maintenance object T, when the maintenance object T (first three-dimensional area TA) is photographed by the camera unit of the wearable terminal 1, the subject within the range of the viewing angle θ of the camera unit of the wearable terminal 1 is imaged and acquired as a camera image. In other words, Figure 7 The subject outside the imaging area OA shown in (b) does not constitute a captured image. The imaging area OA is outside the range of the angle of view θ of the imaging unit of the wearable terminal 1 .
[0164] The second determination unit 32 determines the mask pixel area other than the effective pixel area corresponding to the first three-dimensional area TA in the acquired camera image. Figure 7 A pixel region obtained by imaging the subject in the mask region MA shown in (b) or a pixel region obtained by imaging the subject through the mask region MA is determined as a mask pixel region.
[0165] From the perspective of the camera unit of the wearable terminal 1, the subject in the blind spot area DA located behind the first three-dimensional area TA (maintenance object T) is blocked by the maintenance object T and is not imaged, so it does not constitute a camera image. As a result, it has substantially the same effect as determining the pixel area other than the effective pixel area corresponding to the first three-dimensional area TA as the mask pixel area.
[0166] In addition, when the first three-dimensional area TA (maintenance object T) is photographed by the camera unit of the wearable terminal 1, the area between the wearable terminal 1 and the first three-dimensional area TA and within the range of the viewing angle θ of the camera unit is originally an area other than the first three-dimensional area TA, but the second determination unit 32 determines the area as an exception area EA located in front of the first three-dimensional area TA. Then, the second determination unit 32 determines the pixel area obtained by imaging through the exception area EA in the camera image as a valid pixel area. The processed image generation unit 33 does not set the pixel area obtained by imaging through the exception area EA (that is, the pixel area corresponding to the front surface of the maintenance object T) as an effective pixel area, because the pixel area obtained by imaging through the exception area EA is a valid pixel area. That is, processing is performed to give priority to visually identifying the first three-dimensional area TA (maintenance object T).
[0167] like Figure 8 As shown in (a), the wearable terminal 1 is Figure 7 In the case where the maintenance worker wearing the wearable terminal 1 moves to the right side while looking at the maintenance object T from the front, Figure 7 Compared with the state shown in (b), the position of the exception area EA changes significantly. Figure 8 In the example shown in (a), the device S2 is located in the mask area MA, so the second determination unit 32 determines the pixel area obtained by imaging the device S2 in the camera image as the mask pixel area, and the processed image generation unit 33 sets the mask pixel area to be invisible, so that the device S2 is not captured in the processed image. However, depending on the position of the wearable terminal 1, the device S2 sometimes enters the exception area EA. In this case, the aforementioned second three-dimensional area SA can also be set separately.
[0168] like Figure 8 As shown in (b), the wearable terminal 1 is Figure 7 When the maintenance worker wearing the wearable terminal 1 moves counterclockwise around the maintenance object T, the maintenance worker also rotates toward the device S2 from the state shown in (b) of FIG. Figure 8Similarly to the case (a), the devices S2 and S1 may enter the exception area EA. In this case, as described above, it is possible to deal with it by setting the second three-dimensional area SA separately.
[0169] like Figure 8 As shown in (c), the wearable terminal 1 is Figure 7 In the case where the direction is changed to the left on the spot from the state shown in (b) (in the case where the maintenance worker wearing the wearable terminal 1 changes the line of sight (head direction) to the left on the spot), as long as Figure 7 As shown in (b), the devices S1 and S2 are originally located in the blind spot area DA, and remain located in the blind spot area DA even if the line of sight direction is changed.
[0170] Above, as referenced Figure 7 and Figure 8 In the maintenance support system of the fourth embodiment described above, when the first three-dimensional area TA is set in a manner matching the maintenance object T and the wearable terminal 1 moves outside the first three-dimensional area TA, it is necessary to set a second three-dimensional area SA in addition to prepare for the possibility that the device S (S1, S2) related to confidential information is located in the exception area EA, but unlike the aforementioned Figure 2 Similarly to the illustrated case, only the first three-dimensional area TA (maintenance target T) can be visually recognized in the maintenance support staff-side terminal 2, so that confidential information other than the maintenance target T can be prevented from being leaked.
[0171] Fig. 9 It is shown in Figure 7 FIG. 2 is a diagram showing an example of a processed image generated by the processed image generating unit 33 when the first three-dimensional area TA is set and the second three-dimensional area SA is set as shown. Fig. 9 (a) is a diagram for explaining an imaging direction (line of sight direction) L1 of the imaging unit 12 of the wearable terminal 1. Fig. 9 (b) is shown in Fig. 9 An example of a processed image generated when the camera direction L1 is shown in (a). Fig. 9 In (a), the wearable terminal 1 is omitted.
[0172] like Fig. 9 As shown in (b), the camera image acquired by the camera unit 12 of the wearable terminal 1 worn by the maintenance worker is obtained by removing the first three-dimensional area TA (in Fig. 9 In the example shown, the mask pixel area outside the effective pixel area corresponding to the maintenance object T=first three-dimensional area TA) and the mask pixel area corresponding to the maintenance object T=first three-dimensional area TA Fig. 9The mask pixel areas corresponding to the second three-dimensional areas SA (not shown) set by the devices S1 and S2 shown in (a) are set to be invisible (in Fig. 9 In the example shown, the processed image is obtained (blackened).
[0173] In addition, in terms of layout, when the device S related to the confidential information is not near the maintenance object T (that is, there is no device S related to the confidential information between the area where the operator wearing the wearable terminal 1 moves and the maintenance object T), there is no problem in setting only the first three-dimensional area TA. When the device S related to the confidential information is near the maintenance object T, the second three-dimensional area SA needs to be set.
[0174] Description of Reference Numerals
[0175] 1: wearable terminal; 2: terminal on the maintenance assistant side; 3: control signal processing unit; 12: camera unit; 13: display; 31: first determination unit; 32: second determination unit; 32a, 32b: second camera unit; 32c: distance image sensor; 32d: inertial sensor; 33: processed image generation unit; 34: communication unit; 100: maintenance assistance system; M: reference point; N: electrical communication line; S, S1, S2: devices related to confidential information; T: maintenance object; TA: first three-dimensional area; SA: second three-dimensional area; W1, W2, W3, W4, W5, W6, W7, W8: partition wall; θ: viewing angle; DA: blind spot area; OA: outside the camera area; EA: exception area; MA: mask area.
Claims
1. A maintenance assistance system, characterized in that: have: A wearable terminal, which is worn by maintenance workers and has a camera unit; a first determining unit that determines a predetermined first three-dimensional area including the industrial equipment to be maintained and / or a predetermined second three-dimensional area excluding the industrial equipment to be maintained, based on a predetermined reference point in the camera image acquired by the camera unit in an initial state; a second determining unit, which determines, in a camera image acquired by the camera unit in a moving state in which the wearable terminal has moved, a mask pixel area other than a valid pixel area corresponding to the first three-dimensional area, and / or determines a mask pixel area corresponding to the second three-dimensional area; a processed image generating unit configured to generate a processed image obtained by setting the mask pixel area determined by the second determining unit to be invisible, with respect to the camera image acquired by the camera unit in the moved state; as well as a communication unit that transmits the processed image generated by the processed image generating unit to a maintenance assistant side terminal operated by a maintenance assistant, Even when the wearable terminal moves to a position where the reference point does not exist within the field of view of the imaging unit in the moved state, the second determining unit determines the mask pixel area.
2. The maintenance assistance system according to claim 1, characterized in that: The reference points are defined by markings.
3. The maintenance support system according to claim 1 or 2, characterized in that: The second determination unit includes a plurality of second imaging units having different imaging directions, and determines the mask pixel region based on the captured images acquired by the plurality of second imaging units.
4. The maintenance support system according to claim 1 or 2, characterized in that: The wearable terminal is a head mounted display having a display, The head mounted display is configured to be switchable between a state in which the captured image is displayed on the display and a state in which the processed image is displayed on the display in a recognizable manner.
5. The maintenance support system according to claim 1 or 2, characterized in that: The first determining unit, the second determining unit, the processed image generating unit, and the communication unit are installed in the wearable terminal.
6. The maintenance support system according to claim 1 or 2, characterized in that: It also includes the maintenance assistant side terminal, The maintenance assistant-side terminal is configured to be capable of bidirectional data communication with the communication unit.
7. The maintenance support system according to claim 6, characterized in that: A server is further provided, the server being electrically connected to the communication unit and the maintenance assistant side terminal, and storing first identification information as identification information of the maintenance worker and second identification information as identification information of the maintenance assistant, The server determines whether the identification information sent from the communication unit to the server matches the first identification information stored in the server, and determines whether the identification information sent from the maintenance assistant side terminal to the server matches the second identification information stored in the server. When the identification information sent from the communication unit to the server matches the first identification information, and the identification information sent from the maintenance assistant side terminal to the server matches the second identification information, two-way data communication can be performed between the communication unit and the maintenance assistant side terminal via the server.
8. A maintenance assistance method, using a wearable terminal equipped with a camera unit and worn by a maintenance worker, the maintenance assistance method comprising: A first determination step, using a first determination unit to determine a first three-dimensional region including the industrial equipment to be maintained and / or a second three-dimensional region excluding the industrial equipment to be maintained, based on a predetermined reference point in the camera image acquired by the camera unit in an initial state; A second determination step, using a second determination unit to determine a mask pixel area other than a valid pixel area corresponding to the first three-dimensional area in a camera image acquired by the camera unit in a moving state in which the wearable terminal has moved, and / or to determine a mask pixel area corresponding to the second three-dimensional area; a processed image generating step, using a processed image generating unit to generate a processed image obtained by setting the mask pixel area determined in the second determining step to be invisible for the camera image acquired by the camera unit in the moved state; as well as a communication step of transmitting the processed image generated in the processed image generating step to a maintenance assistant side terminal by using a communication unit, Here, in the second determination step, even when the wearable terminal moves to a position where the reference point does not exist within the field of view of the camera unit in the moved state, the second determination unit determines the mask pixel area.
9. A computer-readable storage medium storing a program for causing the first determination step, the second determination step, the processed image generation step and the communication step included in the maintenance assistance method according to claim 8 to be executed by the first determination unit, the second determination unit, the processed image generation unit and the communication unit respectively.
10. A method for generating a processed image, characterized in that: The method is a method of generating a processed image using the maintenance support system according to any one of claims 1 to 7, In the method, in a camera image acquired by the camera unit of the wearable terminal worn by a maintenance worker, a mask pixel area excluding a valid pixel area corresponding to a specified first three-dimensional area including the industrial equipment to be maintained is determined, and / or a mask pixel area corresponding to a specified second three-dimensional area excluding the industrial equipment to be maintained is determined, and a processed image is generated in which the mask pixel area is set to be invisible.
Citation Information
Patent Citations
Image processing system, image processing program, and image processing method
JP2017211766A
Self-position detector and method and device for inspection in reactor using it
JP2005315709A
Image processing apparatus, method, and program
JP2012054891A
Display control method, display control program, and information processing apparatus
JP2016162079A
System and method for remotely supporting it operation work
JP2018036812A
