Maintenance support system, maintenance support method and procedure
By using wearable terminals in the maintenance support system to determine the three-dimensional area of the maintenance object and generate processing images, the problems of confidential information leakage and limited camera conditions are solved, and safe and efficient troubleshooting is achieved.
Patent Information
- Application Number
- CN201980094425.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-03-22
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2039-03-22
AI Technical Summary
When passing camera images, existing maintenance support systems are prone to leak confidential information, and the camera conditions are limited, making it difficult to deal with faults immediately.
Using a wearable terminal, the reference point is obtained through the camera unit in the initial state, the three-dimensional area of the maintenance object is determined, and the camera unit position is adjusted in the moving state, to generate a processed image of the effective pixel area without confidential information, and send it to Maintenance supporter.
It effectively prevents the leakage of confidential information other than the maintenance object, reduces the limitation of camera conditions, and improves the efficiency and security of fault handling.
Smart Images

Figure CN113614792B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a maintenance support system, a maintenance support method and a program for supporting maintenance work of a maintenance object such as industrial equipment, etc. In particular, the present invention relates to a maintenance support system, a maintenance support method and a program that can prevent leakage of confidential information other than the maintenance object and have few restrictions on imaging conditions. Background Art
[0002] In the past, generally speaking, when an industrial equipment such as a substrate processing device malfunctioned, the following response was taken: after confirming the condition of the malfunction, the maintenance operator at the installation site of the industrial equipment to be maintained informed the condition of the malfunction by telephone to the maintenance support person on the manufacturer side of the maintenance object, and the maintenance support person gave various instructions to the maintenance operator by telephone.
[0003] If the failure cannot be solved even by the above-mentioned telephone response, the following response is performed: when the skilled workers on the manufacturer's side are available, the skilled workers are dispatched to the installation site of the maintenance object.
[0004] In the case of the above-mentioned telephone response, since the communication is conducted verbally, sometimes the fault status cannot be well conveyed to the maintenance supporter, or the instructions of the maintenance supporter cannot be well conveyed to the maintenance operator, so the fault cannot be solved. As needed, sometimes the maintenance operator sends the acquired camera image of the maintenance object to the maintenance supporter via email, etc., and the maintenance supporter understands the fault status by confirming the camera image, but there is a problem that it takes time to understand the status.
[0005] In addition, when a skilled worker is dispatched to deal with the problem as described above, there is a problem that it is difficult to deal with the problem immediately.
[0006] In order to solve the above-mentioned problems, a maintenance support system is proposed: a maintenance worker wears a wearable terminal equipped with a camera unit and a head-mounted display, uses the wearable terminal to photograph the maintenance object, and sends the camera image (moving image) to the maintenance supporter's side terminal via telecommunication lines such as the Internet.
[0007] However, there is a concern that in the above maintenance support system, when the camera image is sent to the maintenance supporter side terminal as it is, the camera image includes confidential information of the installation site. There is a problem that when the camera image including confidential information is sent to the maintenance supporter side, the maintenance supporter may learn the confidential information.
[0008] Therefore, as a maintenance support system capable of preventing leakage of confidential information, for example, a maintenance support system described in Patent Document 1 (an image processing system in Patent Document 1) has been proposed.
[0009] The system described in Patent Document 1 is an image processing system (Invention 1 of Patent Document 1), comprising: an acquisition unit, which acquires an original image capturing an original area; an identification unit, which identifies one or more identifiers in the original image; a determination unit, which determines at least one of a first image portion capturing a first area in the original area and a second image portion capturing a second area obtained by removing the first area from the original area based on the one or more identifiers identified by the identification unit; and a generation unit, which generates a processed image including the first image portion based on the determination result of the determination unit.
[0010] According to the system described in Patent Document 1, it is considered possible to prevent leakage of confidential information or the like by setting the first area as a maintenance target and setting the second area as confidential information or the like other than the maintenance target.
[0011] However, in the system described in Patent Document 1, there is a limitation that the original image must include the identifier. Specifically, Patent Document 1 describes the following: an example is given in which the internal area of the printing device 300 (a rectangular area surrounded by four identifiers 30) is set as a maintenance object, the original area changes according to the movement of the operator 10 wearing the wearable tool 100 equipped with the camera 170, and the original images 171a to 173a capturing different original areas are obtained, but any of the original images 171a to 173a must include the identifier 30 (paragraphs 0063 to 0065, Figures 6 and 7 of Patent Document 1, etc.).
[0012] Therefore, for example, even if one wants to approach the maintenance object to photograph a detailed portion of the maintenance object, there is a limitation that one can only approach within the range where the identifier is included in the original image.
[0013] Furthermore, in the system described in Patent Document 1, the first image portion (the portion corresponding to the maintenance object) is two-dimensionally determined based on the identifier, which results in a limitation that the maintenance object with the identifier attached must be photographed from a specific direction facing the identifier.
[0014] Prior art literature
[0015] Patent Literature
[0016] Patent Document 1: Japanese Patent Application Publication No. 2017-211766 Summary of the invention
[0017] Problem that the invention aims to solve
[0018] The present invention has been made to solve the above-mentioned problems of the prior art, and an object thereof is to provide a maintenance support system, a maintenance support method, and a program that can prevent leakage of confidential information other than the maintenance target and have few restrictions on imaging conditions.
[0019] Solutions for solving problems
[0020] In order to solve the above-mentioned problems, the present invention provides a maintenance support system, characterized in that it comprises: a wearable terminal, which is equipped with a camera unit and is worn by a maintenance operator; a first determination unit, which determines a specified three-dimensional area including the maintenance object based on a specified reference point in a camera image acquired by photographing the maintenance object by the camera unit in an initial state; a second determination unit, which determines the position of the camera unit in the initial state relative to the reference point; a third determination unit, which determines the change in position of the camera unit from the initial state in a moved state after the wearable terminal moves; and a fourth determination unit, which determines the position of the camera unit in the initial state relative to the reference point based on the position determined by the second determination unit and the change in position of the camera unit from the initial state determined by the third determination unit. The processing unit comprises a processing unit for generating a processed image by setting a mask pixel area other than the effective pixel area determined by the fourth determining unit as invisible for the camera image acquired by the camera unit in the moved state; and a communication unit for transmitting the processed image generated by the processed image generation unit to a maintenance supporter side terminal.
[0021] According to the present invention, the first determination unit determines a specified three-dimensional area including the maintenance object based on a specified reference point in a camera image acquired by photographing the maintenance object by a camera unit in an initial state. The "camera unit in the initial state" of the present invention means a camera unit in a state where a specified reference point is located in the camera image. Specifically, for example, in the case where a reference point is specified by a mark attached to the maintenance object, the camera unit in the initial state means a camera unit located at a position where the mark can be photographed and recognized.
[0022] Next, according to the present invention, the second determination unit determines the position of the camera unit in the initial state relative to the reference point. That is, the distance and direction of the camera unit in the initial state relative to the reference point are determined. Specifically, for example, in the case where the reference point is specified by a mark attached to the maintenance object, the position from which the camera unit in the initial state photographed the mark is determined based on the reference point specified by the mark.
[0023] Next, according to the present invention, the third determining unit determines the change in position of the camera unit from the initial state in the moved state after the wearable terminal moves. When the maintenance worker moves, the wearable terminal worn by the maintenance worker also moves, and the camera unit of the wearable terminal also moves. At this time, the change in position of the camera unit from the initial state in the moved state is determined by the third determining unit.
[0024] Next, according to the present invention, the fourth determination unit first determines the position of the reference point based on the position of the camera unit in the initial state determined by the second determination unit relative to the reference point and the change in position of the camera unit in the moved state from the initial state determined by the third determination unit. Specifically, when the position of the camera unit in the initial state determined by the second determination unit relative to the reference point is set as a vector A with the reference point as the starting point, and the change in position of the camera unit in the moved state determined by the third determination unit from the initial state is set as a vector B with the camera unit in the initial state as the starting point, the position of the camera unit in the moved state based on the reference point is represented by a composite vector of vector A and vector B. Therefore, the position of the reference point based on the camera unit in the moved state is determined as the inverse vector C of the above composite vector.
[0025] Then, the fourth determination unit determines the effective pixel area corresponding to the three-dimensional area in the camera image acquired by the camera unit in the moved state based on the determined position of the reference point (i.e., the above-mentioned inverse vector C based on the camera unit in the moved state) and the three-dimensional area determined by the first determination unit. Specifically, first, the three-dimensional area based on the camera unit in the moved state is determined based on the determined position of the reference point (inverse vector C) and the three-dimensional area determined by the first determination unit (the three-dimensional area based on the reference point). In other words, the three-dimensional area represented by the three-dimensional coordinate system based on the reference point becomes represented by the three-dimensional coordinate system based on the camera unit in the moved state. Since it is possible to determine which part of the field of view of the camera unit in the moved state the three-dimensional area represented by the three-dimensional coordinate system based on the camera unit can be mapped by geometric calculation, in other words, it is possible to determine which pixel area in the camera image acquired by the camera unit corresponds to, so it is possible to determine the effective pixel area corresponding to the three-dimensional area in the camera image. Furthermore, the reference point does not need to be included in the captured image acquired by the image capturing unit in the moved state.
[0026] Next, according to the present invention, the processed image generation unit generates a processed image obtained by making the mask pixel area other than the effective pixel area determined by the fourth determination unit invisible. As the "processed image obtained by making the mask pixel area invisible" in the present invention, for example, an image in which the projected area is the same as the area of the camera image but the mask pixel area in the camera image is blacked out, or an image in which only the effective pixel area is cut out from the camera image. Since the three-dimensional area includes the maintenance object and the effective pixel area corresponds to the three-dimensional area, as long as the three-dimensional area is not set to be excessively larger than the maintenance object, the processed image obtained by making the mask pixel area other than the effective pixel area invisible does not include pixel areas corresponding to confidential information other than the maintenance object.
[0027] Finally, according to the present invention, the communication unit transmits the processed image generated by the processed image generation unit to the maintenance supporter side terminal. Therefore, in the maintenance supporter side terminal, only the maintenance object included in the three-dimensional area corresponding to the effective pixel area is visually confirmed, thereby preventing the leakage of confidential information other than the maintenance object.
[0028] As described above, according to the present invention, the reference point can be included in the camera image acquired by the camera unit in the initial state, and it is not necessary to include the reference point in the camera image acquired by the camera unit in the moved state after the wearable terminal moves. In addition, according to the present invention, a prescribed three-dimensional area including the maintenance object is determined, so the shooting direction of the camera unit is not limited. Therefore, it is possible to prevent the leakage of confidential information other than the maintenance object and there are fewer restrictions on the camera conditions.
[0029] In addition, in the present invention, "the state after the wearable terminal moves" refers to a concept that also includes the case where the movement amount is 0 (i.e., the case where the camera unit is in the initial state). Therefore, in the processed image generation unit, for the camera image acquired by the camera unit in the initial state, a processed image is also generated in which the mask pixel area is set to invisible.
[0030] In the present invention, preferably, the reference point does not need to be included in the captured image acquired by the imaging unit in the moved state.
[0031] Preferably, the reference point is defined by a mark attached to the maintenance object.
[0032] Specifically, for example, the center point of the pixel area corresponding to the mark in the camera image is set as the reference point. As the mark, an AR mark, a QR code (registered trademark), a barcode, etc. can be exemplified. In addition, as a method of attaching the mark to the maintenance object, in addition to pasting, printing or engraving can also be exemplified.
[0033] Preferably, the second determination unit determines the position of the imaging unit in the initial state relative to the reference point based on the size, position and shape of a pixel area corresponding to the mark in the imaging image acquired by the imaging unit in the initial state.
[0034] If the distance of the imaging unit in the initial state relative to the mark changes, the size of the pixel area corresponding to the mark in the camera image acquired by the imaging unit changes. In addition, if the direction of the imaging unit in the initial state relative to the mark (the direction of the straight line connecting the mark with the center of the imaging element group of the imaging unit) changes, the position and shape of the pixel area corresponding to the mark in the camera image acquired by the imaging unit changes. Therefore, in the above-mentioned preferred structure, for example, if the correspondence between the size of the pixel area corresponding to the mark and the distance of the imaging unit (standard curve) and the correspondence between the feature quantity related to the position and shape of the pixel area corresponding to the mark and the direction of the imaging unit (standard curve) are investigated in advance and stored in the second determination unit, the position (distance and direction) of the imaging unit relative to the reference point in the initial state can be determined based on these correspondences and the size, position and shape of the pixel area corresponding to the mark in the camera image.
[0035] In addition, as feature quantities related to the shape of the pixel area corresponding to the mark, various feature quantities can be used that reflect different projection methods of the mark according to the direction of the camera unit and can be calculated through well-known image processing, such as the aspect ratio of the pixel area corresponding to the mark (when the mark is rectangular), the roundness (when the mark is circular), etc.
[0036] Preferably, the second determination unit has a distance image sensor, which is installed on the wearable terminal and has a line of sight direction that is approximately the same as the line of sight direction of the camera unit. The second determination unit determines the position of the camera unit relative to the reference point in the initial state based on the grayscale value, position and shape of the pixel area corresponding to the mark in the distance image acquired by the distance image sensor.
[0037] A distance image sensor (also called a depth camera or a 3D camera) is a sensor that measures the distance to a subject in its field of view using a TOF (Time Of Flight) method, for example, and acquires a distance image in which the distance to the subject is represented by the grayscale value of each pixel.
[0038] If the distance of the camera unit in the initial state relative to the mark changes, the distance of the distance image sensor relative to the mark also changes. If the distance of the distance image sensor relative to the mark changes, the grayscale value of the pixel area corresponding to the mark in the distance image acquired by the distance image sensor changes. In addition, if the direction of the camera unit in the initial state relative to the mark changes, the direction of the distance image sensor relative to the mark also changes. If the direction of the distance image sensor relative to the mark changes, the position and shape of the pixel area corresponding to the mark in the distance image acquired by the distance image sensor change. Therefore, in the above-mentioned preferred structure, for example, if the correspondence between the grayscale value of the pixel area corresponding to the mark in the distance image and the distance of the camera unit (standard curve) and the correspondence between the feature quantity related to the position and shape of the pixel area corresponding to the mark in the distance image and the direction of the camera unit (standard curve) are investigated in advance and stored in the second determination unit, the position (distance and direction) of the camera unit relative to the reference point in the initial state can be determined based on these correspondences and the grayscale value, position and shape of the pixel area corresponding to the mark in the distance image.
[0039] Furthermore, in order to determine the position of the imaging unit in the initial state relative to the reference point defined by the marker, an AR development library “ARToolKit” manufactured by M SOFT can also be used as the second determining unit.
[0040] Specifically, a parallelogram (including a square and a rectangle) marker is used as a marker. Then, the camera image that captures the marker is used to create a quadrangular pyramid surface including the four sides of the marker. Next, the intersection vector of the opposite surface of the quadrangular pyramid surface is obtained through "ARToolKit", and the normal direction with respect to the marker surface is calculated based on the outer product of the intersection vector, thereby determining the position of the camera unit in the initial state relative to the reference point specified by the marker.
[0041] Preferably, the third determination unit includes an inertial sensor installed in the wearable terminal.
[0042] The inertial sensor is a sensor composed of, for example, a three-axis acceleration sensor and a three-axis angular velocity sensor (gyro sensor).
[0043] Therefore, in the above-mentioned preferred structure, the acceleration and angular velocity of the wearable terminal can be detected by the inertial sensor. In other words, the acceleration and angular velocity of the camera unit of the wearable terminal can be detected. The third determination unit can determine the change in position of the camera unit in the moved state from the initial state by using the detection result of the inertial sensor.
[0044] Preferably, the wearable terminal has a head-mounted display, and the head-mounted display is configured to be switchable between the following two states: a state in which the camera image is displayed in the head-mounted display; and a state in which the processed image is displayed in a recognizable manner in the head-mounted display.
[0045] According to the above-mentioned preferred structure, when the camera image is displayed on the head-mounted display, for example, unlike the case of displaying a processed image obtained by painting the mask pixel area black, there is no part that blocks the field of view of the maintenance worker, so maintenance work can be performed safely. In addition, the "state of displaying the camera image" in the above-mentioned preferred structure refers to the following concept: in addition to the state of actually displaying the camera image acquired by the camera unit on the head-mounted display as in an MR head-mounted device (the state in which the maintenance worker visually confirms the displayed camera image), it also includes the state in which the head-mounted display is transmissive and the maintenance worker can directly visually confirm the state of the maintenance object through the transmissive head-mounted display that displays nothing, such as a transmissive AR head-mounted device.
[0046] On the other hand, according to the above-mentioned preferred structure, when the processed image is displayed in a recognizable manner in the head-mounted display, the processed image sent to the maintenance supporter side terminal can also be recognized by the maintenance operator. Therefore, the maintenance operator can also confirm that the leakage of confidential information other than the maintenance object is prevented, which can give the maintenance operator a sense of security. In addition, in the "state in which the processed image is displayed in a recognizable manner" in the above-mentioned preferred structure, in addition to the state of displaying the same image as the processed image sent to the maintenance supporter side terminal (the imaged area is the same as the area of the camera image but the mask pixel area in the camera image is blackened, or the image is obtained by cutting out only the effective pixel area from the camera image, etc.), it also includes the state of displaying the image obtained by superimposing the boundary line of the mask pixel area and the effective area on the camera image, and the state of reducing the processed image to display it on a part of the display screen (for example, a corner), etc.
[0047] Preferably, the first determining unit, the second determining unit, the third determining unit, the fourth determining unit, the processed image generating unit and the communication unit are installed in the wearable terminal.
[0048] According to the above-mentioned preferred structure, all units for executing a series of actions until the following action are installed in the wearable terminal, and the action is: generating a processed image obtained by setting the mask pixel area in the camera image obtained by shooting the maintenance object by the camera unit to be invisible, and sending the processed image to the maintenance supporter side terminal. This is an inside-out structure in which the position of the wearable terminal (camera unit) is detected by the sensor (second determination unit and third determination unit) installed on the wearable terminal itself. In the case of an outside-in structure in which an external device independent of the wearable terminal is used to detect the position of the wearable terminal (for example, a structure in which a camera unit set outside and different from the camera unit of the wearable terminal is used to detect the position of the wearable terminal (camera unit of the wearable terminal), although there is an advantage of reducing the power consumption required by the wearable terminal, there are environmental restrictions such as the absence of a shielding object between the external device and the wearable terminal, and the need for a high-precision external device, resulting in high costs. Therefore, by adopting the above-mentioned preferred structure, advantages of low cost and few environmental restrictions can be obtained.
[0049] Preferably, the present invention includes the maintenance supporter-side terminal, and the maintenance supporter-side terminal is configured to be capable of bidirectional data communication with the communication unit.
[0050] According to the above-mentioned preferred structure, the maintenance supporter side terminal and the communication unit can perform data communication in both directions. Therefore, not only can the processed image be sent from the communication unit to the maintenance supporter side terminal, but vice versa, images for maintenance work can be sent from the maintenance supporter side terminal to the communication unit, and sound data for indicating the content of the maintenance work can be sent, etc., so that effective maintenance work can be performed using only the maintenance support system involved in the present invention.
[0051] In addition to the above-mentioned preferred structure, a structure is also provided that is capable of performing data communication from a maintenance object (e.g., industrial equipment such as a substrate processing device) to a terminal on the maintenance support side. For example, if processing log data (measurement values and setting values related to the processing of the maintenance object) obtained in the maintenance object is sent to the terminal on the maintenance support side, the maintenance supporter can indicate the content of the maintenance work while referring to the processing log data, thereby being able to perform the maintenance work more efficiently.
[0052] Preferably, the maintenance object of the present invention is industrial equipment such as a substrate processing device.
[0053] In addition, in order to solve the above-mentioned problem, the present invention also provides a maintenance support method, which is a maintenance support method using a wearable terminal equipped with a camera unit and worn by a maintenance operator, and is characterized in that it includes: a first determination step, in which the first determination unit determines a specified three-dimensional area including the maintenance object based on a specified reference point in a camera image acquired by photographing the maintenance object by the camera unit in an initial state; a second determination step, in which the second determination unit determines the position of the camera unit in the initial state relative to the reference point; a third determination step, in which the third determination unit determines the change in position of the camera unit from the initial state in a moved state after the wearable terminal moves; and a fourth determination step, in which the fourth determination unit determines the change in position of the camera unit from the initial state based on the position of the camera unit in the initial state determined in the second determination step. For the position of the reference point and the change in the position of the camera unit in the moved state determined in the third determination step from the initial state, the position of the reference point is determined with the camera unit in the moved state as a reference, and based on the determined position of the reference point and the three-dimensional area determined in the first determination step, the effective pixel area corresponding to the three-dimensional area in the camera image acquired by the camera unit in the moved state is determined; a processed image generation step, in which the processed image generation unit generates a processed image by setting the mask pixel area other than the effective pixel area determined in the fourth determination step to be invisible for the camera image acquired by the camera unit in the moved state; and a communication step, in which the communication unit sends the processed image generated in the processed image generation step to a maintenance supporter side terminal.
[0054] In addition, in order to solve the above-mentioned problem, the present invention also provides a program, which is used to enable the first determination unit, the second determination unit, the third determination unit, the fourth determination unit, the processed image generation unit and the communication unit to respectively execute the first determination process, the second determination process, the third determination process, the fourth determination process, the processed image generation process and the communication process included in the maintenance support method.
[0055] Furthermore, it is also possible to provide a computer (CPU) readable storage medium storing the above-mentioned program.
[0056] Effects of the Invention
[0057] According to the present invention, it is possible to obtain an effect that leakage of confidential information other than the maintenance target is prevented and there are few restrictions on imaging conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 It is a diagram schematically showing a general configuration of a maintenance support system according to an embodiment of the present invention.
[0059] Figure 2 Yes Description Figure 1 The diagram is an explanatory diagram of the schematic operation of the maintenance support system shown.
[0060] Figure 3 It is shown by Figure 1 FIG. 1 is a diagram showing an example of a captured image acquired by the imaging unit in the initial state and a processed image generated by the processed image generating unit.
[0061] Figure 4 It is shown by Figure 1 FIG. 1 is a diagram showing an example of a captured image acquired by an imaging unit in a moved state after the wearable terminal has moved and a processed image generated by a processed image generating unit in the moved state. DETAILED DESCRIPTION
[0062] A maintenance support system according to an embodiment of the present invention will be described below with reference to the drawings.
[0063] Figure 1 It is a diagram schematically showing a general configuration of a maintenance support system according to the present embodiment. Figure 1 (a) is the overall structure diagram, Figure 1 (b) is a diagram showing a wearable terminal and components mounted on the wearable terminal, Figure 1 (c) shows Figure 1 (b) is a block diagram of the internal structure of the control signal processing unit shown.
[0064] like Figure 1 As shown in FIG. 2 (a), the maintenance support system 100 according to the present embodiment includes a wearable terminal 1 , a maintenance supporter-side terminal 2 , and a control signal processing unit 3 .
[0065] The wearable terminal 1 is a terminal that is located at an installation site of a maintenance target (industrial equipment such as a substrate processing apparatus) and is worn by a maintenance worker who performs maintenance work.
[0066] like Figure 1As shown in (b), the wearable terminal 1 of this embodiment is set as a glasses-type wearable terminal, and is provided with a frame 11 hung on the ear of the maintenance worker. In addition, the wearable terminal 1 is provided with a camera unit 12 in the center of the front side of the frame 11 (equivalent to the position near the brow of the maintenance worker when the maintenance worker hangs the frame 11 on the ear). As the camera unit 12 of this embodiment, a color (RGB) camera is used. Moreover, the line of sight direction of the camera unit 12 is set to be roughly parallel to the line of sight direction of the maintenance worker when the maintenance worker hangs the frame 11 on the ear. In addition, the wearable terminal 1 is provided with a head-mounted display 13, which is mounted on the front side of the frame 11 and covers the front of the maintenance worker's field of vision when the maintenance worker hangs the frame 11 on the ear. As the head-mounted display 13 of this embodiment, a transmissive head-mounted display is used. Therefore, when the head-mounted display 13 does not display any content, it can be used in the same way as ordinary glasses.
[0067] like Figure 1 As shown in (b), the wearable terminal 1 of this embodiment is equipped with a distance image sensor 32a, and the distance image sensor 32a has a line of sight direction that is substantially the same as the line of sight direction of the camera unit 12. Specifically, the distance image sensor 32a is installed at the center of the front side of the frame 11 and below the camera unit 12. As the distance image sensor 32a, for example, a TOF distance image sensor is used. In addition, the distance image sensor 32a constitutes the hardware part of the second determination unit 32 described later.
[0068] In addition, the wearable terminal 1 of this embodiment is equipped with an inertial sensor 33a. Specifically, the inertial sensor 33a is installed on the side of the frame 11. The inertial sensor 33a is composed of a three-axis acceleration sensor and a three-axis angular velocity sensor. In addition, the inertial sensor 33a constitutes the hardware part of the third determination unit 33 described later.
[0069] Furthermore, a speaker 4 and a microphone 5 are mounted on the side of the frame 11 of the wearable terminal 1 of the present embodiment.
[0070] For example, the wearable terminal 1 having the above-described structure can be configured by using and improving smart glasses “HoloLens” manufactured by Microsoft Corporation.
[0071] The maintenance supporter-side terminal 2 is electrically connected to the wearable terminal 1 (specifically, the control signal processing unit 3 installed on the wearable terminal 1) through a telecommunication line N such as the Internet. The maintenance supporter-side terminal 2 is configured to be able to perform data communication bidirectionally with a communication unit 36 described later. As the maintenance supporter-side terminal 2, for example, a desktop computer can be used, but it is not limited thereto, and various terminals such as a laptop computer, a tablet computer, a smartphone, etc. can be used as long as it can display a processed image as described later.
[0072] In addition, although it is not an essential configuration, the maintenance supporter-side terminal 2 of the present embodiment is also electrically connected to the maintenance object T through the telecommunication line N. Specifically, for example, a computer (in the computer system) that collects processing log data of the maintenance object T (measured values and set values related to the processing of the maintenance object T) is provided. Figure 1 (a) (not shown), the computer is electrically connected to the maintenance supporter terminal 2 via a telecommunication line N. Furthermore, the processing log data obtained in the maintenance target T is sequentially transmitted to the maintenance supporter terminal 2 and stored in the maintenance supporter terminal 2.
[0073] In addition, although it is not an essential configuration, the maintenance supporter-side terminal 2 of the present embodiment is electrically connected to the wearable terminal 1 and / or the maintenance object T via the server 6 electrically connected to the telecommunication line N. By providing the server 6, it becomes easy to manage the ID (identification information) of the wearable terminal 1 and the maintenance object T connected to the maintenance supporter-side terminal 2 and to manage the usage history.
[0074] The control signal processing unit 3 is installed on the wearable terminal 1 (on the side of the frame 11), and is electrically connected to the camera unit 12, the head-mounted display 13, the distance image sensor 32a, the inertial sensor 33a, the speaker 4 and the microphone 5. It has the function of controlling each component and processing the output signal of each component.
[0075] The control signal processing unit 3 is mainly composed of a CPU, memories such as a ROM and a RAM, and programs stored in the memories for causing the CPU to execute operations described later.
[0076] like Figure 1As shown in (c), the control signal processing unit 3 includes: a program for causing the CPU to execute the operation of the first determination unit 31, a program for causing the CPU to execute the operation of the software part 32b of the second determination unit 32, a program for causing the CPU to execute the operation of the software part 33b of the third determination unit 33, a program for causing the CPU to execute the operation of the fourth determination unit 34, and a program for causing the CPU to execute the operation of the processed image generation unit 35. In addition, the control signal processing unit 3 includes a communication unit 36. The communication unit 36 is composed of an antenna and a program for causing the antenna to operate.
[0077] In addition, the program update as described above may be performed by directly connecting a predetermined computer to the control signal processing unit 3 via a USB cable or the like, or may be directly updated from the server 6 .
[0078] In the present embodiment, the first determination unit 31, the second determination unit 32 (distance image sensor 32a, software part 32b), the third determination unit 33 (inertial sensor 33a, software part 33b), the fourth determination unit 34, the processed image generation unit 35 and the communication unit 36 are all installed in the wearable terminal 1. As described later, this is an inside-out structure in which the position of the wearable terminal 1 (camera unit 12) is detected by a sensor installed on the wearable terminal 1 itself. Specifically, since it is an inside-out structure in which the position of the wearable terminal 1 (camera unit 12) in the initial state is determined by the second determination unit 32 and the change of the position of the wearable terminal 1 (camera unit 12) is determined by the third determination unit 33, the advantages of low cost and few environmental restrictions can be obtained.
[0079] Below, in addition to reference Figure 1 In addition, refer to Figure 2 to Figure 4 The operation (maintenance support method) of the maintenance support system 100 according to the present embodiment having the above-described schematic configuration will be described.
[0080] Figure 2 1 is an explanatory diagram for explaining the general operation of the maintenance support system 100 according to the present embodiment. Figure 2 In the figure, the maintenance worker wearing the wearable terminal 1 is omitted. Figure 3 1 is a diagram showing an example of a captured image acquired by the imaging unit 12 in the initial state and a processed image generated by the processed image generating unit 35 . Figure 3 (a) shows an example of a camera image. Figure 3 (b) shows an example of a processed image. Figure 41 is a diagram showing an example of a captured image acquired by the imaging unit 12 in a moved state after the wearable terminal 1 has moved and a processed image generated by the processed image generating unit 35 in a moved state. Figure 4 (a) shows an example of a camera image. Figure 4 (b) shows an example of a processed image. Figure 3 (a) and Figure 3 In (b), the boundary line of the three-dimensional area TA is illustrated by a dotted line, but the boundary line does not exist in the actual camera image.
[0081] like Figure 2 As shown, a mark M for defining a reference point is attached to the front surface of the maintenance object T in this embodiment. Members T1 to T3 are provided on the front surface of the maintenance object T. Next to the maintenance object T, a device S related to confidential information is arranged.
[0082] First, the maintenance worker wearing the wearable terminal 1 moves to a position facing the mark M so that the camera unit 12 of the wearable terminal 1 can capture the mark M. Then, the camera unit 12 in the initial state (in Figure 2 The camera unit 12 of the wearable terminal 1 shown by the solid line in the figure captures the maintenance object T to obtain the maintenance object T. Figure 3 The camera image (moving image) shown in (a).
[0083] Next, the first determination unit 31 extracts the pixel area corresponding to the marker M by performing a predetermined image processing on the acquired camera image, thereby determining a reference point (e.g., the center point of the pixel area) based on the pixel area. Then, the first determination unit 31 determines a predetermined three-dimensional area (e.g., a three-dimensional area in a rectangular shape) TA including the maintenance object T based on the reference point. Specifically, for example, in the first determination unit 31, a three-dimensional coordinate system (having Figure 2 The coordinates of each vertex P1 to P8 of the three-dimensional area TA represented by the three-dimensional coordinate system of the X1 axis, the Y1 axis, and the Z1 axis shown in the figure are stored in association with the identification information represented by the marker M. Then, the first determination unit 31 calculates the feature amount of the pixel area corresponding to the marker M, determines the identification information of the marker M based on the feature amount, and determines the three-dimensional area TA represented by the three-dimensional coordinate system based on the reference point based on the coordinates of each vertex P1 to P8 stored in association with the determined identification information.
[0084] In addition, the following structure can be adopted: the first determination unit 31 directly calculates the coordinates of each vertex P1 to P8 of the three-dimensional area TA represented by the three-dimensional coordinate system (three-dimensional coordinate system having the X1 axis, the Y1 axis, and the Z1 axis) based on the reference point by calculating the feature amount of the pixel area corresponding to the marker M. In other words, it can also be configured so that the marker M has a pattern or the like corresponding to the coordinates of each vertex P1 to P8, and the coordinates of each vertex P1 to P8 can be calculated by calculating the feature amount of the predetermined pattern or the like.
[0085] In addition, in the present embodiment, a method is described in which a reference point is specified using a marker M in order to enable high-precision determination, and a three-dimensional area TA is determined based on the reference point, but the present invention is not limited to this. For example, the following method can also be adopted: an AR cursor is used to click on any point of the camera image displayed in the head-mounted display 13, and the clicked position is set as the reference point. In addition, for example, the following method can also be adopted: each vertex P1 to P8 of the three-dimensional area TA is specified by clicking on the camera image displayed in the head-mounted display 13 using an AR cursor, and one of these vertices is set as the reference point. Furthermore, the following method can also be adopted: the first determination unit 31 performs a prescribed image processing on the camera image, thereby extracting a pixel area corresponding to a characteristic part (corner, etc.) of the maintenance object T, and the center of the pixel area is set as the reference point.
[0086] Next, the second determination unit 32 determines the position of the camera unit 12 in the initial state relative to the reference point. In other words, the second determination unit 32 determines (in the presence of Figure 2 The second determining unit 32 determines the distance and direction from which the camera unit 12 in the initial state photographed the marker M. That is, the second determining unit 32 determines the coordinate system starting from the reference point and ending at the camera unit 12. Figure 2 Vector A shown.
[0087] Specifically, a distance image including a pixel area corresponding to the marker M is acquired by the distance image sensor 32a constituting the second determination unit 32. Then, the software part 32b constituting the second determination unit 32 extracts the pixel area corresponding to the marker M by performing a predetermined image processing on the acquired distance image, and determines the position (vector A) of the camera unit 12 relative to the reference point in the initial state based on the grayscale value, position, and shape of the pixel area. More specifically, the correspondence relationship (standard curve) between the grayscale value of the pixel area corresponding to the marker M in the distance image and the distance of the camera unit 12 and the correspondence relationship (standard curve) between the feature quantity related to the position and shape of the pixel area corresponding to the marker M in the distance image and the direction of the camera unit 12 are pre-stored in the memory constituting the second determination unit 32, and the software part 32b constituting the second determination unit 32 determines the position of the camera unit 12 relative to the reference point in the initial state based on these correspondence relationships and the grayscale value, position, and shape of the pixel area corresponding to the marker M in the distance image.
[0088] In addition, in the present embodiment, the distance image sensor 32a is used as a component of the second determination unit 32 to determine the position (vector A) of the camera unit 12 relative to the reference point in the initial state, but the present invention is not limited thereto. For example, the second determination unit 32 may use a camera image acquired by the camera unit 12.
[0089] Specifically, the second determination unit 32 (the software part 32b constituting the second determination unit 32) can also determine the position (vector A) of the camera unit 12 in the initial state relative to the reference point based on the size, position and shape of the pixel area corresponding to the marker M in the camera image acquired by the camera unit 12 in the initial state. More specifically, if the correspondence relationship (standard curve) between the size of the pixel area corresponding to the marker M and the distance to the camera unit 12 and the correspondence relationship (standard curve) between the feature quantity related to the position and shape of the pixel area corresponding to the marker M and the direction of the camera unit 12 are pre-stored in the memory constituting the second determination unit 32, the software part 32b constituting the second determination unit 32 can determine the position of the camera unit 12 in the initial state relative to the reference point based on these correspondence relationships and the size, position and shape of the pixel area corresponding to the marker M in the camera image.
[0090] Next, the maintenance worker wearing the wearable terminal 1 moves, and the wearable terminal 1 also moves. The camera unit 12 (in Figure 2 The camera unit 12 of the wearable terminal 1 shown by the dotted line in the figure captures the maintenance object T to obtain the maintenance object T. Figure 4The captured image shown in (a) of FIG. Pixel regions corresponding to the marker M are not included in this captured image.
[0091] Then, when the wearable terminal 1 moves, the third determining unit 33 determines the change in position of the camera unit 12 in the moved state from the initial state. That is, the third determining unit 33 determines a path starting from the camera unit 12 in the initial state and ending at the camera unit 12 in the moved state. Figure 2 Vector B shown.
[0092] Specifically, the acceleration and angular velocity of the camera unit 12 of the wearable terminal 1 are detected by the inertial sensor 3 constituting the third determination unit 33. Then, the software part 33b constituting the third determination unit 33 uses the detected acceleration and angular velocity of the camera unit 12 to perform operations such as time integration, thereby determining the change in position of the camera unit 12 in the moved state from the initial state.
[0093] Next, the fourth determination unit 34 first determines the position of the reference point with respect to the camera unit 12 in the moved state, based on the position (vector A) of the camera unit 12 in the initial state determined by the second determination unit 32 relative to the reference point, and the change in position (vector B) of the camera unit 12 in the moved state from the initial state determined by the third determination unit 33. Specifically, the position of the camera unit 12 in the moved state with respect to the reference point is represented by a composite vector of vector A and vector B. Therefore, the position of the reference point with respect to the camera unit 12 in the moved state is determined as Figure 2 The inverse vector C of the above-mentioned combined vector is shown (a vector having the image pickup unit 12 after the movement as a starting point and the reference point as an end point).
[0094] After determining the position of the reference point (inverse vector C) based on the image pickup unit 12 in the moved state as described above, the fourth determination unit 34 determines the image pickup image (refer to the image pickup unit 12 in the moved state) acquired by the image pickup unit 12 based on the determined position of the reference point (inverse vector C) and the three-dimensional area TA determined by the first determination unit 31. Figure 4 The effective pixel area corresponding to the three-dimensional area TA in (a)).
[0095] Specifically, the fourth determining unit 34 first determines the position of the reference point (the inverse vector C) and the three-dimensional area TA determined by the first determining unit 31 (the three-dimensional area based on the reference point, i.e., the three-dimensional area with the reference point as the reference ... Figure 2 The three-dimensional area TA (represented by the three-dimensional coordinate system of the X1 axis, Y1 axis and Z1 axis shown in FIG. 1 ) is determined based on the image pickup unit 12 in the moved state. Figure 2In other words, the three-dimensional area TA represented by the three-dimensional coordinate system (X1, Y1, Z1) based on the reference point becomes represented by the three-dimensional coordinate system (X2, Y2, Z2) based on the camera unit 12 in the moved state. Since it is possible to determine by geometric calculation which part of the field of view of the camera unit 12 in the moved state the three-dimensional area TA represented by the three-dimensional coordinate system (X2, Y2, Z2) based on the camera unit 12 is projected, in other words, which pixel area in the camera image acquired by the camera unit 12 corresponds to, it is possible to determine the effective pixel area in the camera image corresponding to the three-dimensional area TA.
[0096] Next, the processed image generation unit 35 generates a processed image (moving image) by making invisible the mask pixel area other than the effective pixel area determined by the fourth determination unit 34, with respect to the camera image acquired by the camera unit 12 in the moved state. Figure 4 In the example shown in (b), as a processed image, an image in which the mask pixel area is painted black is generated. However, the processed image is not limited to this, and it can also be an image obtained by cutting out only the effective pixel area from the camera image. In addition, instead of painting the mask pixel area black, various methods can be used, such as mosaic processing or fogging, as long as it can prevent the leakage of confidential information.
[0097] Finally, the communication unit 36 transmits the processed image generated by the processed image generation unit 35 to the maintenance supporter-side terminal 2 via the telecommunication line N. Figure 4 In a state where the device S related to the confidential information can be visually confirmed in the camera image as shown in (a) of FIG. Figure 4 As shown in (b) of FIG. 1 , only the maintenance target T included in the three-dimensional area TA corresponding to the effective pixel area is visually confirmed in the maintenance supporter-side terminal 2 , so that leakage of confidential information other than the maintenance target T can be prevented.
[0098] In addition, the third determination unit 33, the fourth determination unit 34, the processed image generation unit 35 and the communication unit 36 are also directed to Figure 3 Thus, in the processed image generating unit 35, for the camera image acquired by the camera unit in the initial state, a mask pixel area is also generated as shown in FIG. Figure 3 The processed image shown in (b) is produced and sent to the maintenance supporter-side terminal 2.
[0099] Furthermore, if the inertial sensor 33a or the like constituting the third determination unit 33 fails, the change in position of the camera unit 12 after the move from the initial state cannot be correctly determined. As a result, the effective pixel area corresponding to the three-dimensional area TA in the camera image acquired by the camera unit 12 after the move cannot be correctly determined. As a result, confidential information other than the maintenance target T may be included in the generated processed image.
[0100] Therefore, preferably, when the inertial sensor 33a or the like fails, the processed image generation unit 35 has a function of generating a processed image in which all pixel regions are made invisible, or the communication unit 36 has a function of stopping the transmission of the processed image. In addition, at this time, in order to enable the maintenance operator to quickly understand that the inertial sensor 33a or the like fails, it is preferable to display the fact that the failure has occurred on the head mounted display 13, or to notify the fact that the failure has occurred by sound on the speaker 4.
[0101] In addition, generally, a small MEMS sensor or the like is used as the inertial sensor 33a or the like, and the detection value of the sensor (in the case of the inertial sensor 33a, acceleration and angular velocity) is converted into an electrical signal and then output. For example, in the case where a short circuit or other fault occurs in the circuit provided in the inertial sensor 33a or the like, the electrical signal output from the sensor does not change even if the wearable terminal 1 moves. Therefore, for example, a structure is adopted in which the amount of change per unit time of the output electrical signal is compared with a predetermined threshold value, and if the amount of change does not exceed the threshold value, it is determined that a fault has occurred in the inertial sensor 33a or the like, so that the fault of the inertial sensor 33a or the like can be automatically detected.
[0102] The head mounted display 13 of the present embodiment is configured to be switchable between the following two states: a state in which a camera image is displayed on the head mounted display 13 (specifically, in the present embodiment, nothing is displayed on the transmissive head mounted display 13, so that the maintenance worker can directly visually confirm the state of the maintenance object T through the head mounted display 13 which displays nothing); and a state in which a processed image is displayed in a recognizable manner on the head mounted display 13. Specifically, for example, the following method can be adopted: a switching button (not shown) is provided on the frame 11, and by pressing the switching button, the control signal processing unit 3 is controlled to select a certain state.
[0103] In a state where the captured image is displayed on the head mounted display 13 (a state where the maintenance worker can directly visually confirm the maintenance object T), since there is no portion blocking the field of vision of the maintenance worker, maintenance work can be performed safely.
[0104] On the other hand, when the processed image is displayed in a recognizable manner on the head mounted display 13, the maintenance operator can also recognize the processed image sent to the maintenance supporter side terminal 2. Therefore, the maintenance operator can also confirm that the leakage of confidential information other than the maintenance object is prevented, which can give the maintenance operator a sense of security. In addition, as a state in which the processed image is displayed in a recognizable manner, in addition to the state in which the same image as the processed image sent to the maintenance supporter side terminal 2 is displayed (the displayed area is the same as the area of the camera image but the mask pixel area in the camera image is blackened, or the image is obtained by cutting out only the effective pixel area from the camera image, etc.), it also includes the following state. Figure 3 (a) and Figure 4 As shown in (a), a state in which an image is displayed by superimposing the boundary line between the mask pixel area and the effective pixel area (the boundary line of the three-dimensional area TA) on the captured image, and a state in which the processed image is reduced and displayed on a part of the display screen.
[0105] In addition, in this embodiment, a case where a three-dimensional area TA is determined for one maintenance object T is cited as an example for explanation, but the present invention is not limited thereto. For example, when there are multiple maintenance objects T, a mark M is assigned to each maintenance object T, and a three-dimensional area is determined for each maintenance object T. For example, when one maintenance object T is a substrate processing device and a component of the substrate processing device is removed and repairs are performed on a specified workbench, another maintenance object T is set on the workbench and a mark M is also assigned to the workbench.
[0106] In addition, in this embodiment, a rectangular parallelepiped-shaped three-dimensional area having eight vertices P1 to P8 is used as an example for explanation as the three-dimensional area TA, but the present invention is not limited thereto. A three-dimensional area TA having a shape other than a rectangular parallelepiped and defined by more than eight points can also be set according to the shape of the maintenance object T.
[0107] In the present embodiment, a case where one wearable terminal 1 is electrically connected to the maintenance supporter terminal 2 is described as an example, but the present invention is not limited thereto. A configuration where a plurality of wearable terminals 1 are electrically connected to the maintenance supporter terminal 2 may be adopted.
[0108] In this case, the camera units 12 of multiple wearable terminals 1 can be used to capture the same mark M to determine the same three-dimensional area TA, or the following structure can be used: the mark M to be captured is set to a different mark for each wearable terminal 1, thereby setting the determined three-dimensional area TA to a different three-dimensional area.
[0109] In addition, the maintenance supporter-side terminal 2 can adopt various methods such as arranging the processed images transmitted from the plurality of wearable terminals 1 and displaying them simultaneously or switching between the processed images to be displayed.
[0110] Furthermore, regardless of whether there is one wearable terminal 1 or a plurality of wearable terminals 1 , a plurality of maintenance supporter-side terminals 2 electrically connected to the wearable terminals 1 can be prepared.
[0111] Furthermore, in this embodiment, if Figure 2 As shown, the area close to the maintenance object T is set as the three-dimensional area TA, but the present invention is not limited to this. Even if there is no maintenance object T in the area, the three-dimensional area TA can be expanded for the area where it is predetermined that there is no device S related to confidential information to include the area. In addition, the three-dimensional area TA is not limited to one connected area, and multiple separate areas can also be set as the three-dimensional area TA.
[0112] According to the maintenance support system 100 according to the present embodiment described above, for example, the following maintenance support can be performed.
[0113] (1) The maintenance supporter visually confirms the processed image sent to the maintenance supporter-side terminal 2 to determine the numbers of the components constituting the maintenance object T. Alternatively, the maintenance operator and the maintenance supporter communicate with each other using the speaker 4 and the microphone 5 of the wearable terminal 1 while visually confirming the processed image to determine the numbers of the components constituting the maintenance object T. Alternatively, based on the sound data of the above-mentioned conversation, the numbers of the components constituting the maintenance object T are automatically determined in the maintenance supporter-side terminal 2 (for example, automatic recognition is performed using AI technology). Alternatively, in the maintenance supporter-side terminal 2, based on the image data of the processed image sent, the numbers of the components constituting the maintenance object T are automatically determined, for example, using AI technology. Preferably, when performing the above-mentioned number determination, the identification information of the maintenance object T is represented in the mark, and the number (for example, the part number and the serial number) included in the bill of materials (BOM) associated with the identification information of the maintenance object T is determined.
[0114] Then, the inventory management system connected to the maintenance supporter side terminal 2 is manually or automatically accessed to confirm the inventory and delivery date of the parts corresponding to the determined number, and the information is sent from the maintenance supporter side terminal 2 to the communication unit 36 installed on the wearable terminal 1, and the information is displayed on the head-mounted display 13.
[0115] In the past, for example, the following operations were performed: the part number and serial number of the component constituting the maintenance object T were determined only through a conversation between the maintenance operator and the maintenance supporter using a speaker and a microphone, the part number and serial number were handwritten, the inventory management system was manually accessed, and the part number and serial number were input through a keyboard to confirm the inventory and delivery date of the corresponding component. In the past, this operation may cause human errors when listening to or dictating the part number and serial number of the component, and human errors when inputting the number into the inventory management system. In addition, since it is manual, maintenance support takes time.
[0116] However, according to the maintenance support system 100 involved in this embodiment, especially when a structure is adopted to automatically determine the part number and serial number of the component and automatically access the inventory management system, the advantages of avoiding the above-mentioned human errors and shortening the time required for maintenance support can be obtained.
[0117] (2) In the above (1), when replacement of a component is scheduled, the maintenance supporter-side terminal 2 automatically creates a work report in which the part number and serial number before replacement are recorded.
[0118] By adopting the above structure, the user's business and production efficiency (added value other than maintenance support) can also be improved. In addition, there is also the advantage of obtaining log data of replacement period parts, so as to be able to provide suggestions on when to replace next time (preventive maintenance), etc.
[0119] Furthermore, in the present embodiment, a mode in which the processed image is transmitted to the maintenance supporter-side terminal 2 is described. However, the captured image may be transmitted directly to the maintenance supporter-side terminal 2 as necessary.
[0120] Description of Reference Numerals
[0121] 1: wearable terminal; 2: maintenance supporter side terminal; 3: control signal processing unit; 12: camera unit; 13: head-mounted display; 31: first determination unit; 32: second determination unit; 32a: distance image sensor; 33: third determination unit; 33a: inertial sensor; 34: fourth determination unit; 35: processed image generation unit; 36: communication unit; 100: maintenance support system; M: mark; N: telecommunication line; S: device related to confidential information; T: maintenance object; TA: three-dimensional area.
Claims
1. A maintenance support system, characterized in that: have: A wearable terminal having a camera unit and worn by a maintenance worker; a first determining unit, which determines a specified three-dimensional area including the maintenance object based on a specified reference point in a camera image acquired by the camera unit in an initial state shooting the maintenance object; a second determining unit, which determines a position of the camera unit in the initial state relative to the reference point; a third determining unit, which determines a change in position of the camera unit from the initial state in a post-movement state after the wearable terminal moves; a fourth determining unit that determines the position of the reference point with the camera unit in the moved state as a reference based on the position of the camera unit in the initial state determined by the second determining unit relative to the reference point and the change in the position of the camera unit in the moved state from the initial state determined by the third determining unit, and determines a valid pixel area corresponding to the three-dimensional area in the camera image acquired by the camera unit in the moved state based on the determined position of the reference point and the three-dimensional area determined by the first determining unit; a processed image generating unit for generating a processed image obtained by setting a mask pixel region other than the effective pixel region determined by the fourth determining unit to be invisible, with respect to the camera image acquired by the camera unit in the moved state; as well as a communication unit configured to transmit the processed image generated by the processed image generating unit to a maintenance supporter-side terminal, The reference point is defined by a mark attached to the maintenance object.
2. The maintenance support system according to claim 1, characterized in that: The reference point does not need to be included in the captured image acquired by the imaging unit in the moved state.
3. The maintenance support system according to claim 1 or 2, characterized in that: The second determination unit determines the position of the imaging unit in the initial state relative to the reference point based on the size, position, and shape of a pixel region corresponding to the marker in the captured image acquired by the imaging unit in the initial state.
4. The maintenance support system according to claim 1 or 2, characterized in that: The second determination unit includes a distance image sensor, which is installed on the wearable terminal and has a line of sight direction that is substantially the same as a line of sight direction of the camera unit. The second determination unit determines the position of the camera unit relative to the reference point in the initial state based on the grayscale value, position, and shape of a pixel area corresponding to the mark in the distance image acquired by the distance image sensor.
5. The maintenance support system according to claim 1 or 2, characterized in that: The third determination unit includes an inertial sensor mounted on the wearable terminal.
6. The maintenance support system according to claim 1 or 2, characterized in that: The wearable terminal has a head mounted display. The head mounted display is configured to be switchable between two states: a state in which the captured image is displayed on the head mounted display; and a state in which the processed image is displayed on the head mounted display in a recognizable manner.
7. The maintenance support system according to claim 1 or 2, characterized in that: The first determining unit, the second determining unit, the third determining unit, the fourth determining unit, the processed image generating unit, and the communication unit are installed in the wearable terminal.
8. The maintenance support system according to claim 1 or 2, characterized in that: The maintenance support system includes the maintenance supporter-side terminal. The maintenance supporter-side terminal is configured to be capable of bidirectional data communication with the communication unit.
9. The maintenance support system according to claim 1 or 2, characterized in that: The maintenance object is industrial equipment.
10. The maintenance support system according to claim 1 or 2, characterized in that: When the third determination unit fails, the processed image generation unit generates a processed image in which all pixel regions in the captured image are made invisible, or the communication unit stops transmitting the processed image.
11. A maintenance support method using a wearable terminal having an imaging unit and worn by a maintenance worker, characterized in that: include: a first determining step, in which a first determining unit determines a predetermined three-dimensional area including the maintenance object based on a predetermined reference point in a camera image acquired by the camera unit in an initial state photographing the maintenance object; a second determining step, in which a second determining unit determines a position of the imaging unit in the initial state relative to the reference point; a third determining step, in which a third determining unit determines a change in position of the camera unit from the initial state in a post-movement state after the wearable terminal moves; a fourth determining step, in which a fourth determining unit determines the position of the reference point with respect to the camera unit in the moved state as a reference based on the position of the camera unit in the initial state determined in the second determining step relative to the reference point and the change in position of the camera unit in the moved state determined in the third determining step from the initial state, and determines a valid pixel region corresponding to the three-dimensional region in a camera image acquired by the camera unit in the moved state based on the determined position of the reference point and the three-dimensional region determined in the first determining step; a processed image generating step, in which a processed image generating unit generates a processed image obtained by making invisible a mask pixel region other than the effective pixel region determined in the fourth determining step, with respect to the camera image acquired by the camera unit in the moved state; as well as a communication step in which a communication unit transmits the processed image generated in the processed image generating step to a maintenance supporter-side terminal, The reference point is defined by a mark attached to the maintenance object.
12. A computer-readable storage medium storing a program, wherein the program is used to cause the first determination unit, the second determination unit, the third determination unit, the fourth determination unit, the processed image generation unit, and the communication unit to respectively execute the first determination process, the second determination process, the third determination process, the fourth determination process, the processed image generation process, and the communication process included in the maintenance support method according to claim 11.
