Information processing system and program

Abstract

To provide information for assisting a user in attaching a part removed from a device to its original position without viewing a moving image.SOLUTION: A work support system 10 comprises: an image analysis processing unit 12 that, when a worker wearing AR glasses and performing maintenance work removes a part from a device and stores it in a storage area, analyzes the image captured by an image capture unit 11 to record removed part information for each part, including the time of removal, the removal position (installation position) of the device, the storage position, and a feature amount; and a display control processing unit 13 that, when the worker returns the part removed from the device to its original position in the reverse order of removal, refers to the removed part information, presents the next part to be installed from among the parts in the storage area, and then presents the installation position in the device for the part to be installed.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing system and a program.

Background Art

[0002] In equipment maintenance and the like, various techniques have been proposed to assist in the work of reinstalling parts removed from the equipment.

[0003] For example, in Patent Document 1, a completed drawing of a product created by a 3D CAD system is imported into animation application software, and the animation application software is used to reverse-play and present a video from the completed product on the 3D CAD until it is disassembled into each part in the reverse order of the actual assembly work, thereby proposing a technique for assisting in the work of reinstalling parts into the equipment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the prior art, it is necessary to view a moving image that is reverse-played when reinstalling parts into the equipment. For this reason, it takes time for the operator to reinstall the parts into the equipment.

[0006] An object of the present invention is to provide information for assisting in attaching a part removed from equipment to its original position in the equipment without viewing a moving image.

Means for Solving the Problems

[0007] The information processing system according to the present invention includes a processor. When the processor acquires a captured image of a device and detects that a component has been removed from the device by analyzing the captured image of the device, the processor creates component position information for the component by associating component information that identifies the component with mounting position information that indicates a position that is both the removal position and the mounting position of the component, and when a mounting target component to be mounted on the device is identified, controls to display the mounting position of the mounting target component on the device with reference to the component position information. This is the gist of the invention.

[0008] Further, the processor is characterized in that it uses storage position information that identifies the storage position of the component removed from the device as the component information, and creates the component position information by associating the position where the component is removed with the position where the component is stored.

[0009] Further, when the mounting target component is specified by the user, the processor identifies the mounting position of the mounting target component on the device by comparing the position information of the mounting target component with the storage position information of the component included in the component position information. This is the gist of the invention.

[0010] Further, when the removal position of any component removed from the device is specified by the user, the processor compares the position information of the removal position with the mounting position information of the component included in the component position information, identifies the component corresponding to the any component among the stored components as the mounting target component, and controls to display the storage position of the mounting target component instead of the mounting position of the mounting target component on the device. This is the gist of the invention.

[0011] Further, when the component information is a captured image of the component, the processor identifies the mounting position of the mounting target component on the device by comparing the captured image of the mounting target component with the captured image of the component included in the component position information. This is the gist of the invention.

[0012] Further, the processor includes time information for specifying the time when the component was removed in the component position information, and presents the attachment procedure of the component to the device by reversing the order in which the components were removed with reference to the component position information.

[0013] Further, when the processor detects completion of attachment of the component to be attached to the device by analyzing a photographed image of the device, the processor presents, as an attachment procedure to the device, the component to be next attached to the device.

[0014] The program according to the present invention causes a computer to implement a function of acquiring a photographed image of a device, a function of creating component position information of a component by associating component information for specifying the component and attachment position information indicating a position that is both the removal position and the attachment position of the component when it is detected that the component has been removed from the device by analyzing the photographed image of the device, and a function of controlling to display the attachment position of the component to be attached to the device with reference to the component position information when the component to be attached to the device is specified.

Advantages of the Invention

[0015] According to the invention described in claim 1, it is possible to provide information for assisting in attaching a component removed from a device to the original position of the device without viewing a moving image.

[0016] According to the invention described in claim 2, it is possible to associate the positions of a component before and after removal.

[0017] According to the invention described in claim 3, it is possible to specify the attachment position of a component to be attached to a device by using the position information of the removed component.

[0018] According to the invention described in claim 4, among the parts removed from the device and stored, it is possible to present the part to be attached to the removal position designated by the user, that is, the attachment position.

[0019] According to the invention described in claim 5, using the photographed image of the part, it is possible to specify the attachment position of the part to be attached to the device.

[0020] According to the invention described in claim 6, the parts can be attached to the device in the reverse order of removal.

[0021] According to the invention described in claim 7, when attaching the parts to the device in the reverse order of removal, it is possible to present the next part to be attached.

[0022] According to the invention described in claim 8, it is possible to provide information to assist in attaching the parts removed from the device to their original positions on the device without viewing a moving image.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0024] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0025] Embodiment 1. FIG. 1 schematically shows a configuration diagram of a work support system in the present embodiment and a basic processing flow of the removal and attachment procedures of components. In FIG. 1, an AR (Augmented Reality) glass 2 constituting the whole or main part of the work support system, a device 6 to which the removal and attachment of a component 4 are targeted, and a storage area 8 serving as a place to store the component 4 removed from the device 6 are shown.

[0026] The work support system 10 in the present embodiment is a form of the information processing system according to the present invention and is used to support maintenance work and the like of a device 6 installed in a facility such as a building. A person (hereinafter, "worker") who performs the maintenance work of the device 6 removes a plurality of components 4 assembled to the device 6 in the maintenance work and places the removed components 4 on the storage area 8 once. Then, the work of returning the removed component 4 to the original position of the device 6 is performed.

[0027] Note that the work to which the work support system 10 in the present embodiment can be applied does not need to be limited to maintenance work as long as it includes the work of returning the component 4 removed from the device 6 to its original state. In the present embodiment, an Internet-connected device is assumed as the device 6, and a LAN cable connected to the LAN port 6a of the Internet-connected device is assumed as the component 4 for explanation.

[0028] The AR glasses 2 are wearable terminal devices shaped like glasses, goggles, etc. that are worn on the human head to cover the eyes. It is a device that can superimpose and display digital content on the real space seen through the AR glasses 2, or display videos and screens. In this embodiment, existing AR glasses 2 can be used in terms of hardware. That is, the AR glasses 2 are equipped with a computer and also have a wireless communication function. More specifically, the AR glasses 2 include a CPU, ROM, RAM, storage, a display means such as a display, a communication means for wireless communication with external devices, a photographing means such as a camera capable of photographing moving images and still images, and a sound input means such as a microphone.

[0029] Figure 2 is a block configuration diagram showing the work support system 10 in this embodiment. The work support system 10 in this embodiment has a photographing unit 11, an image analysis processing unit 12, a display control processing unit 13, a display unit 14, an input processing unit 15, a control unit 16, and a storage unit 17. Note that components not used in the description of this embodiment are omitted from Figure 2. Also, unless otherwise specified, all the components 11 to 17 shown in Figure 2 are described as being mounted on the AR glasses 2.

[0030] The AR glasses 2 are worn and used by the operator. The photographing unit 11 is realized by the photographing means and photographs a predetermined range in the direction the operator is facing. The image analysis processing unit 12 acquires the photographed image photographed by the photographing unit 11 when the component 4 is removed from the device 6 and performs image analysis. The image analysis processing unit 12 has a point cloud data extraction unit 121, a removed component extraction unit 122, and a recording unit 123.

[0031] The point cloud data extraction unit 121 extracts the appearance of the images of the components 4 and devices 6 included in the captured image as point cloud data. That is, the point cloud data extraction unit 121 detects the object being photographed by extracting a set of points (i.e., "point cloud data") composed of the coordinates and color information of the points in the three-dimensional space from the captured image. The removable component extraction unit 122 extracts the component 4 from among the objects extracted by the point cloud data extraction unit 121. In particular, since the component 4 is removed from the device 6 by the operator, the removable component extraction unit 122 detects the movement of the component 4 to be removed. The recording unit 123 records information regarding the component 4 removed from the device 6 by the operator in the storage unit 17. More specifically, the recording unit 123 registers information (hereinafter, "removable component information") regarding the component 4 removed from the device 6 in the component list table. However, the data configuration and the like of the removable component information will be described later.

[0032] When the component 4 is attached to the device 6, the display control processing unit 13 performs display control for displaying on the display unit 14 information for assisting in attaching the component 4 removed from the device 6 to the original position of the device 6. The display control processing unit 13 includes an attachment component presentation unit 131 and an attachment position presentation unit 132. The attachment component presentation unit 131 presents to the operator the component (hereinafter, also referred to as "attachment target component") 4 to be attached to the device 6 next. The attachment position presentation unit 132 presents to the operator the position of the device 6 (i.e., the LAN port 6a) to which the attachment target component 4 is to be attached.

[0033] The display unit 14 displays the attachment position of the component 4 to the device 6 under the control of the display control processing unit 13. The input processing unit 15 receives information input by the user to complement the information recorded by the recording unit 123. The control unit 16 controls the operations of the respective components 11 to 15. The storage unit 17 stores the aforementioned component list table.

[0034] Each component 11 to 16 in the work support system 10 is realized by the cooperative operation of a computer included in the work support system 10 and a program that operates on a CPU installed in the computer. Further, the storage unit 17 is realized by a storage included in the work support system 10. Alternatively, a RAM or an external storage means may be used via a network.

[0035] In addition, the program used in the present embodiment can be provided not only by communication means but also by being stored in a computer-readable recording medium such as a USB memory. The program provided from the communication means or the recording medium is installed in the computer, and various processes are realized by the CPU of the computer sequentially executing the program.

[0036] Also, the "system" in the present embodiment is, as an example, configured by a single device built into the AR glasses 2, but may also be configured by combining a plurality of devices. For example, at least one of each processing unit 12, 13, 15 or the storage unit 17 may be realized by an information processing device different from the AR glasses 2, such as a server computer. Further, the imaging unit 11 may be realized by a camera different from the AR glasses 2. Also, the information received by the input processing unit 15 may be information input from another microphone not built into the AR glasses 2 or input means such as a mouse or keyboard provided in an information processing device different from the AR glasses 2.

[0037] The work support system 10 in the present embodiment is characterized in that when a worker performs a maintenance operation of removing the component 4 from the device 6 and then attaching the removed component 4 to its original position, information for assisting the operation of attaching the component 4 to its original position is provided to the worker. Hereinafter, the operation in the present embodiment will be described. First, the component removal process performed when the worker removes the component 4 from the device 6 will be described using the flowchart shown in FIG. 3. Note that the worker performs the maintenance operation while wearing the AR glasses 2.

[0038] The trigger for starting this process does not need to be particularly limited. For example, it is started in response to a predetermined operation such as an operator operating a start button provided by the work support system 10.

[0039] When this process is started, the control unit 16 activates the imaging unit 11 and the image analysis processing unit 12. Then, the imaging unit 11 starts imaging (step S101). The start of imaging can also be said to be the start of acquiring point cloud data. When the imaging unit 11 starts imaging, it continuously images until an instruction to end the imaging is given. Here, the imaging range by the imaging unit 11 will be described.

[0040] The imaging unit 11 sets a predetermined range in the direction that the AR glasses 2 are facing as the imaging range. In other words, a predetermined range in the direction that the operator is looking becomes the imaging area. Since the operator works while looking at the part 4 or the device 6 that is the object of the work, when removing the part 4 from the device 6, the removal position together with the part 4 is at least included in the imaging range. Also, as will be described later, since the operator will place the removed part 4 in the predetermined storage area 8, at this time, the position where it is placed in the storage area 8 together with the part 4 (hereinafter, "storage position") is at least included in the imaging range. Similarly, when attaching the part 4 to the device 6, the storage position of the part 4 that is the object of attachment and the attachment position of the part 4 that the operator has taken in hand to the device 6 are at least included in the imaging range. In short, in the present embodiment, information necessary for attaching the removed part 4 to the original position of the device 6 is acquired from the captured image. However, when the operator wearing the AR glasses 2 directs the line of sight in the desired direction during work, the imaging unit 11 images a predetermined range in which information necessary for attaching the part 4, that is, removed part information, can be acquired.

[0041] When imaging is started, the image analysis processing unit 12 initializes n to 1 (step S101). n is a variable indicating the order of removal of the part 4. It is also the number (hereinafter, "part number") associated with the part 4 in the order of removal.

[0042] When the operator is instructed to finish removing the component 4 (Y in step S103), the control unit 16 causes the imaging unit 11 to finish imaging (step S109), and then finishes the component removal process. However, until the operator instructs the end of the process (N in step S103), the following process is repeated.

[0043] That is, the image analysis processing unit 12 acquires a captured image of the device 6 from the imaging unit 11. As described above, since the operator is looking at the device 6 through the AR glasses 2, the image analysis processing unit 12 can acquire a captured image of the device 6. The point cloud data extraction unit 121 extracts point cloud data by analyzing the captured image. The point cloud data extraction unit 121 extracts the appearance of the device 6 main body as point cloud data. When the component 4 is removed from the device 6, a change occurs in the portion corresponding to the removed component 4, that is, the point cloud data representing the LAN port 6a. When a change appears in this point cloud data (Y in step S104), the removed component extraction unit 122 detects the movement of the component 4. If no change appears in this point cloud data (N in step S104), the process is repeated until it appears.

[0044] When the movement of the component 4 is detected, the recording unit 123 creates removal component information regarding the component 4 whose movement has been detected, that is, the component 4 removed from the device 6, and records it in the storage unit 17 (step S105). The recording unit 123 records the content of the movement of the component 4 removed in this way. Note that the first detection of the movement of the component 4 is the moment when the component 4 is removed from the LAN port 6a. Also, in the present embodiment, the movement of the operator's hand is ignored.

[0045] By the way, as the direction the operator faces changes, changes will continue to appear in the point cloud data extracted by the point cloud data extraction unit 121. That is, focusing on the movement of the removed component 4, the component 4 continues to move and does not stop until it is placed in the storage area 8 (N in step S106). Therefore, the recording unit 123 records the content of the movement of the component 4 in the storage unit 17 while the movement of the component 4 is being detected (step S105).

[0046] When the component 4 is placed in the storage area 8, the movement of the component 4 stops and it enters a stationary state. In this way, when it is detected from the point cloud data extracted by the point cloud data extraction unit 121 that the movement of the component 4 has stopped (Y in step S106), the recording unit 123 records the non-moving state, that is, the end state of the movement of the component 4, in the storage unit 17 (step S107). In this way, when it is confirmed that the removal of one component 4 from the device 6 and its placement in the storage area 8 are completed, the control unit 16 increments n by 1 (step S108) and proceeds to step S103 for recording the next component 4. Then, when all the components 4 to be removed have been removed, as described above, the control unit 16 ends the process in response to an end instruction from the operator. Here, the data configuration of the removed component information will be described with reference to FIG. 4.

[0047] FIG. 4 is a diagram showing an example of the data configuration of the component list table in the present embodiment. In the component list table, a list of removed component information of the component 4 removed from the device 6 is registered. The removed component information includes items such as "component number", "time information", "component position information", "feature quantity", and "memo". The "component number" is the identification information of the component represented by n described above (that is, the component number), and is also information indicating the order in which the component is removed from the device 6. In the "time information", the time when the data is recorded in the removed component information is set. "(Start)" is the time when the movement of the component 4 is detected, in other words, the time when it is removed from the device 6. "(End)" is the time when the movement of the component 4 is no longer detected and the stop is confirmed, in other words, the time when the component 4 is placed in the storage area 8. Note that in step S105, the time between the start and the end is also recorded, but in the present embodiment, since it is information not used for the attachment of the component 4, it may or may not be recorded. If it is recorded, it may be deleted at a desired timing, for example, when the end time is recorded.

[0048] "Component position information" is information that identifies the position of the component 4. "Component position information" is represented by three-dimensional coordinate data indicating point cloud data. For example, it may be represented by the center point of the component 4. "(Removal position)" is information indicating the position of the device 6 from which the component 4 has been removed, that is, the position of the LAN port 6a from which the component 4 has been removed. Here, since it is information recorded when the component 4 is removed, it is called the removal position, but it is also attachment position information indicating the attachment position of the removed component 4. "(Storage position)" is storage position information indicating the position within the storage area 8 on which the component 4 is placed. In the case of this embodiment, the storage position of the component 4 corresponds to the component information that identifies the component. That is, each removed component 4 is identified by the storage position information of the component 4 placed in the storage area 8. In this embodiment, by using the storage position information as component information, component position information is created by associating the position where the component 4 is removed with the position where it is stored. "Component position information", like "time information", will also record the position transition between the removal position and the storage position of the component 4, but in this embodiment, since it is information not used for attaching the component 4, it may or may not be recorded. If it is recorded, it may be deleted at a desired timing such as when the stored position is recorded.

[0049] "Feature quantity" is information indicating the features of the component 4. In this embodiment, the component 4 is photographed from the time it is removed from the device 6 until it is placed in the storage area 8, and the recording unit 123 extracts the feature quantity each time during that period and records it in the storage unit 17. However, it may be possible to leave only any one of the feature quantities. Depending on the type of the component 4 to be removed, while the component 4 is moving, the operator's hand cannot extract the entire image, so there is a possibility that the exact feature quantity of the component 4 cannot be extracted. Therefore, it is considered appropriate to leave the feature quantity extracted when the component 4 is in the state of being placed in the storage area 8.

[0050] As described above, for the removable part information, each item value is automatically written by the recording unit 123. However, "Memo" is an item for the operator to add messages or the like to the part 4 as needed, in addition to the information recorded by the recording unit 123. The operator registers, for example, handling precautions, it's about time to replace, the relationship with other parts 4, or information for correcting the content recorded by the recording unit 123. Of course, the "Memo" may register not only character information but also images such as photos.

[0051] The data registered in the "Memo" is input by the input processing unit 15 and registered in the removable part information. For example, the operator may input it by voice via the microphone provided in the AR glasses 2. The input voice may be registered as audio data or converted into character information and registered. Also, character information input by the operator from an input means of an information processing device different from the AR glasses 2, such as a keyboard, may be received and registered. The means for recording in the "Memo" does not need to be particularly limited.

[0052] As described above, the operator removes all the removable parts 4, places them in the storage area 8 for storage. After that, the operator performs an operation to return the part 4 to the original position of the device 6. The part mounting process performed during the operation of returning the part 4 to its original position will be described using the flowchart shown in FIG. 5.

[0053] The trigger for starting this process does not need to be particularly limited. For example, it is started in response to a predetermined operation such as the operator operating the start button provided by the work support system 10.

[0054] When this process is started, the control unit 16 activates the imaging unit 11 and the display control processing unit 13. Then, the imaging unit 11 starts imaging (step S201). The imaging in this process is not for recording as in the part removal process, but to confirm the work content such as the movement of the operator's hand, and to superimpose digital content on the real space and display it, so as to provide information for supporting the attachment to the original position of the device 6.

[0055] Subsequently, the display control processing unit 13 reads and acquires all the removable part information from the part list table stored in the storage unit 17 (step S202). Then, the display control processing unit 13 acquires one piece of removable part information corresponding to an unprocessed part 4 that is not the target of the processing described later from among the removable part information. When there are no unprocessed parts 4 (N in step S203), the control unit 16 causes the imaging unit 11 to end the imaging (step S210), and then ends the part mounting process. When there are unprocessed parts 4 (Y in step S203), the following processing is repeated.

[0056] That is, when the display control processing unit 13 acquires the removable part information of the unprocessed part 4 from the part list table, it specifies the removable part information corresponding to the part 4 with the largest part number n among the unprocessed parts 4 (step S204). That is, when the display control processing unit 13 repeats the processing of steps S203 to S210, it specifies the removable part information corresponding to the parts 4 with part numbers n, n - 1, n - 2, ···, 2, 1, which decrease one by one.

[0057] By referring to the "(storage position)" of the removable part information of the part 4 to be processed, the position where the part 4 to be mounted in the current installation work by the operator is placed, that is, the coordinate position in the three-dimensional space can be specified. Therefore, when the imaging unit 11 is imaging the range including the storage area 8, in other words, when the operator is looking at the storage area 8, and when the storage area 8 is displayed on the display unit 14, the attachment part presentation unit 131 superimposes and displays information that allows the part 4 to be attached to be visible on the captured image by the imaging unit 11 (that is, the actual three-dimensional space). In this way, the attachment part presentation unit 131 presents the part 4 to be attached this time to the operator (step S205).

[0058] The display indicating the component 4 to be attached means to display it in a distinguishable manner from the components 4 that are not to be attached among the components 4 placed in the storage area 8. Also, as a distinguishable display of the component 4, the component 4 itself to be attached may be changed, or some information may be added to the component 4 to be attached for display. Specifically, for example, the component 4 to be attached may be given movement (so-called animation display), the shape, or the display form may be changed such as changing the brightness or color, a symbol such as an arrow may be added, or it may be indicated by sound.

[0059] The presentation of the component 4 to be attached is repeated until the operator recognizes the component 4 to be attached (N in step S206). Whether the operator has recognized it or not may be determined, for example, by analyzing the captured image by the imaging unit 11 to detect that the operator has picked up or pointed at the component 4 to be attached. In addition, when it is detected that the operator has picked up a component 4 that is not to be attached, a warning sound may be output, or an animation such as a message or vibration may be displayed to notify the operator.

[0060] When the operator recognizes and picks up the component 4 to be attached, subsequently, the attachment position presentation unit 132 presents the correct attachment position on the device 6, that is, the LAN port 6a which is the position where the component 4 was removed, to the operator (step S207). By referring to the “(removal position)” of the removal component information of the component 4 to be attached, the attachment position of the component 4 to be attached in the current attachment operation by the operator, that is, the coordinate position in the three-dimensional space can be specified. Therefore, when the range including the device 6 is being imaged by the imaging unit 11, in other words, when the range including the attachment position is being displayed on the display unit 14 because the operator is looking at the attachment position of the device 6, the attachment position presentation unit 132 superimposes and displays information on the captured image by the imaging unit 11 (that is, the actual three-dimensional space) so that the LAN port 6a which is the attachment position of the component 4 to be attached can be seen.

[0061] The display indicating the mounting position of the component 4 to be mounted means that it is displayed so as to be distinguishable from the LAN ports 6a that are not the mounting positions among the LAN ports 6a provided in the device 6. Further, as the distinguishable display of the LAN ports 6a, the LAN ports 6a themselves that are the mounting positions may be changed, or some information may be added to the LAN ports 6a for display. Specifically, for example, the display form may be changed such as changing the brightness or color of the LAN ports 6a that are the mounting positions, a symbol such as an arrow may be added, or it may be indicated by sound. Further, when both the component 4 to be mounted and the LAN ports 6a that are the mounting positions are displayed on the display unit 14, the mounting position presentation unit 132 may display an animation of an arrow or the like that can imagine the movement from the component 4 to the LAN ports 6a.

[0062] The presentation of the mounting position is repeatedly performed until the operator can confirm that the component 4 to be mounted is mounted at the mounting position (N in step S208). Whether the operator has mounted the component 4 at the correct mounting position, that is, whether the component 4 can be returned to its original position, can be detected by analyzing the captured image by the imaging unit 11, for example, and the mounting position can be confirmed.

[0063] Incidentally, when presenting the component 4 to be mounted, it is determined whether the operator has correctly recognized the component 4 to be mounted, but at the mounting position, it is not determined whether the operator has correctly recognized the mounting position. This is because whether the operator has correctly recognized the mounting position can be confirmed together by confirming that the operator has mounted the component 4 to be mounted at the correct mounting position.

[0064] When it is confirmed that the mounting of the component 4 is completed (Y in step S208), in order to present the component 4 to be mounted on the device 6 next as a mounting procedure for the device 6, the part number of the component 4 to be processed is decreased by one (step S209). Then, the process is shifted to step S203.

[0065] According to the present embodiment, as described above, the component 4 removed from the device 6 can be attached in the reverse order from when it was removed.

[0066] Incidentally, in the above description, it was described on the premise that the removed component 4 is returned to its original position. However, in some cases, an exchange such as replacing the component 4 with a new one may be performed. In this case, since the characteristic amounts of the old and new components 4 are considered to be the same, the component 4 can be easily exchanged by placing the new component 4 at the storage position of the component 4 to be exchanged.

[0067] Also, although the component 4 to be attached next was specified only by the component number (n), after the specification, the characteristic amount obtained from the photographed image of the component 4 placed may be compared with the characteristic amount included in the removed component information to verify the component 4 to be attached.

[0068] Incidentally, in the flowchart shown in FIG. 4, the components 4 are attached in the reverse order from the order in which they were removed. However, it is conceivable that the operator may want to attach the components 4 in an arbitrary order different from the order in which they were removed. Therefore, in the present embodiment, it is made possible to handle such a case.

[0069] For example, assume that the operator designates the component 4 to be attached by picking up the component 4 to be attached from among the components 4 placed in the storage area 8. That is, in the flowchart shown in FIG. 4, the component 4 to be attached was automatically specified based on the component number (n) (steps S204 to S207), but here, the operator designates the component 4 to be attached.

[0070] Incidentally, when the operator specifies the component 4 to be installed, they are naturally looking at the component 4 to be installed. Therefore, the storage position of the component 4 to be installed is included in the captured image by the imaging unit 11. That is, the mounting position presentation unit 132 can identify the storage position of the component 4 to be installed by analyzing the captured image by the imaging unit 11. Accordingly, the mounting position presentation unit 132 compares the storage position of the component 4 to be installed identified from the captured image with the storage position recorded in the "(storage position)" of each removal component information. Then, the mounting position presentation unit 132 identifies the component 4 corresponding to the removal component information with which the storage positions match as the component 4 specified by the operator as the component 4 to be installed.

[0071] Similar to the processing after the component 4 to be installed is identified as described above (step S207), the mounting position presentation unit 132 refers to the removal component information of the component 4 to be installed, obtains the "(removal position)" corresponding to the identified component 4, thereby identifying the mounting position of the component 4 to be installed, and presents the mounting position of the component 4 to be installed.

[0072] By the way, in the above description, when the component 4 to be installed is identified, the mounting position corresponding to the component 4 to be installed is presented to the operator. However, referring to the component position information in the removal component information, when the operator specifies the mounting position of the component 4, the storage position of the component 4 to be mounted at the specified mounting position is identified. That is, in the operation of mounting the component 4, conversely to the above, that is, when the mounting position is specified, the storage position of the component 4 to be mounted can be identified.

[0073] For example, assume that an operator designates the removal position (i.e., the "attachment position") of any of the parts 4 to be removed by pointing at it with a finger. At this time, since the operator is naturally looking at the attachment position on the device 6, the captured image by the imaging unit 11 includes the attachment position where the part 4 is to be attached. Therefore, the attachment part presentation unit 131 can identify the attachment position where the part 4 is to be attached by analyzing the captured image by the imaging unit 11. Therefore, the attachment part presentation unit 131 collates the attachment position identified from the captured image with the removal position recorded in the "(removal position)" of each removal part information. Then, the attachment part presentation unit 131 identifies the part 4 corresponding to the removal part information where the removal position (i.e., the attachment position) matches as the part 4 removed from the removal position designated by the operator, that is, the attachment target part 4. And the attachment part presentation unit 131 presents to the operator the storage position of the part 4 to be attached at the position designated by the operator as the attachment position in the manner described in step S205. Note that presenting the storage position is the same as presenting the part 4.

[0074] As described above, in the present embodiment, when the attachment position of the part 4 to the device 6 is designated by the operator, instead of the attachment position of the attachment target part 4 to the device 6, it is possible to control to display the storage position of the attachment target part 4.

[0075] Embodiment 2. In the above Embodiment 1, the LAN cable was described as an example of the part 4. Since LAN cables have the same appearance in terms of specifications, the characteristic amounts of any of the parts 4 are the same or almost the same with only differences such as symbols indicating color or model numbers. When dealing with parts 4 having the same appearance like LAN cables, that is, when the characteristic amounts that can be extracted from the captured image of the part 4 are the same, it is suitable to identify each part 4 by the storage position of the part 4 as described in Embodiment 1. However, when removing parts 4 having completely different appearances, the captured image of the part 4 can be used as part information for identifying the part.

[0076] FIG. 6 is a diagram showing an example of the data configuration of the parts list table in the present embodiment. As is clear from comparison with the parts list table shown in FIG. 5, the items constituting the part position information are different, and "image" is added. More specifically, in the present embodiment, "image" is used as part information instead of "(storage position)" as part information in Embodiment 1. The "image" is an image of the part 4. The "image" is a comparison target with the captured image of the part 4 by the imaging unit 11 in the part mounting process, and is information used to identify the part 4.

[0077] The system configuration in the present embodiment may be the same as that in Embodiment 1. The part removal process and the part mounting process in the present embodiment may basically be the same as those in Embodiment 1. However, since image data is handled instead of the storage position of the part 4, the process of handling the image data of the part 4 is slightly different.

[0078] That is, when the part information is the captured image of the part 4, in the part removal process, the recording unit 123 extracts the image of the part 4 instead of extracting the storage position of the part 4 from the captured image by the imaging unit 11 and records it in "image", which is different from Embodiment 1.

[0079] However, as described above, the image data recorded in "image" is collated with the captured image of the part 4 in the part mounting process described later and is used when determining whether the part 4 matches or not. Therefore, in order to improve the determination accuracy of whether the part 4 matches or not, the image data recorded in "image" may be generated by separately capturing the part 4 when the part 4 is in a stationary state placed at the storage position, for example. Also, for example, a plurality of image data captured from a plurality of directions such as from above, the right side, and the left side may be recorded in "image".

[0080] Also, in the component mounting process, instead of identifying the storage position of the component 4 to be mounted from the captured image by the imaging unit 11, collating the storage position with "(storage position)", and identifying the mounting position of the component 4 to be mounted, the mounting component presentation unit 131 extracts the captured image of the component 4 to be mounted stored in the storage area 8 from the captured image by the imaging unit 11 and collates it with the "image", that is, the captured images of the respective components included in each removed component information, to identify the mounting position of the component 4 to be mounted. This is different from the first embodiment. Other aspects may be the same as those of the first embodiment, so the description is omitted.

[0081] According to the present embodiment, similar to the first embodiment, the mounting position of the component 4 to be mounted can be presented to the operator, and the storage position of the component 4 to be mounted can be presented to the operator in response to the designation of the mounting position of the component 4. In the present embodiment, since the image data of the component 4 is used, for example, when replacing the component, it is not necessary to place it at the storage position of the component 4 before replacement.

[0082] Note that the removed component information described in each of the above embodiments may be used in appropriate combinations. That is, both the storage position and the image are recorded in the removed component information. For example, when the device is a multifunction machine, in maintenance work, many components such as screws and toner cartridges of various colors are removed and attached from the multifunction machine. In the case of the same type of components such as screws, it is preferable to use the storage position described in the first embodiment as the component information for identifying the component. Also, in the case of different types of components with different shapes such as screws and toner cartridges, it is preferable to use the image described in the second embodiment as the component information for identifying the component.

[0083] In the above-described embodiment, the processor refers to a processor in a broad sense, including a general-purpose processor (e.g., CPU: Central Processing Unit, etc.) and a dedicated processor (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0084] Also, the operation of the processor in the above-described embodiment may be achieved not only by one processor but also by a plurality of physically separated processors cooperating with each other. Also, the order of each operation of the processor is not limited to the order described in each of the above embodiments and may be changed as appropriate.

[0085] (Supplementary Note) (((1))) Comprising a processor, The processor, Acquires a captured image of the device, When it is detected that a component has been removed from the device by analyzing the captured image of the device, component information for identifying the component, and attachment position information indicating a position that is both the removal position and the attachment position of the component are associated with each other to create component position information for the component, When an attachment target component to be attached to the device is identified, it controls to display the attachment position of the attachment target component to the device with reference to the component position information. An information processing system characterized by the above. (((2))) The information processing system according to (((1))), wherein the processor uses storage position information for specifying a storage position of a component removed from the device as the component information, and creates the component position information by associating the position where the component is removed with the position where the component is stored. (((3))) When the processor is specified by the user, the mounting position of the mounting target component on the device is identified by collating the position information of the mounting target component with the storage position information of the components included in the component position information. The information processing system according to ((2)). (((4))) The processor is When the removal position of any component to be removed from the device is specified by the user, the position information of the removal position is collated with the mounting position information of the components included in the component position information, and the component corresponding to any of the components being stored is identified as the mounting target component, Instead of the mounting position of the mounting target component on the device, it controls to display the storage position of the mounting target component, The information processing system according to ((2)). (((5))) When the component information is a photographed image of the component, the processor collates the photographed image of the mounting target component with the photographed images of the components included in the component position information, and the mounting position of the mounting target component on the device is identified. The information processing system according to any one of ((1)) to ((4)). (((6))) The processor is Including time information for identifying the time when the component was removed in the component position information, By reversing the order in which the components were removed with reference to the component position information, the mounting procedure of the components on the device is presented, The information processing system according to any one of ((1)) to ((5)). (((7))) When the processor detects the completion of the mounting of the mounting target component on the device by analyzing the photographed image of the device, as the mounting procedure on the device, it presents the component to be mounted on the device next. The information processing system according to ((6)). (((8))) The computer is caused to acquire a photographed image of the device, when it is detected that a component has been removed from the device by analyzing the photographed image of the device, create component position information of the component by associating component information for identifying the component with attachment position information indicating a position that is both the removal position and the attachment position of the component, when an attachment target component to be attached to the device is identified, control to display the attachment position of the attachment target component to the device with reference to the component position information, A program for realizing the above.

[0086] (((1))) According to the invention described above, it is possible to provide information for assisting in attaching a component removed from a device to the original position of the device without viewing a moving image. (((2))) According to the invention described above, it is possible to associate the positions before and after removal of the component. (((3))) According to the invention described above, it is possible to specify the attachment position of an attachment target component to a device by using the position information of the removed component. (((4))) According to the invention described above, it is possible to present a component to be attached to a specified removal position, that is, an attachment position, among the components removed from the device and stored. (((5))) According to the invention described above, it is possible to specify the attachment position of an attachment target component to a device by using a photographed image of the component. (((6))) According to the invention described above, it is possible to attach the components to the device in the reverse order of removal. (((7))) According to the invention described above, when attaching the components to the device in the reverse order of removal, it is possible to present the next component to be attached. (((8))) According to the invention described above, it is possible to provide information for assisting in attaching a component removed from a device to the original position of the device without viewing a moving image.

Explanation of Signs

[0087] 2 AR glass, 4 parts, 6 devices, 6a LAN port, 8 storage area, 10 work support system, 11 imaging unit, 12 image analysis processing unit, 13 display control processing unit, 14 display unit, 15 input processing unit, 16 control unit, 17 memory unit, 121 point cloud data extraction unit, 122 removable part extraction unit, 123 recording unit, 131 mounting part presentation unit, 132 mounting position presentation unit.

Claims

1. Comprising a processor, The processor, Acquires a captured image of the device, When it detects that a component has been removed from the device by analyzing the captured image of the device, it creates component position information for the component by associating component information that identifies the component with mounting position information that indicates the position that is both the removal position and the mounting position of the component, When a mounting target component to be mounted on the device is identified, it controls to display the mounting position of the mounting target component on the device with reference to the component position information. An information processing system characterized by the above.

2. The processor creates the component position information by associating the storage position information that identifies the storage position of the component removed from the device as the component information with the position where the component is removed and the position where it is stored, according to the information processing system described in Claim 1.

3. When the mounting target component is specified by the user, the processor identifies the mounting position of the mounting target component on the device by comparing the position information of the mounting target component with the storage position information of the components included in the component position information, according to the information processing system described in Claim 2.

4. The processor, When the removal position of any component removed from the device is specified by the user, it identifies the component corresponding to the any component among the stored components as the mounting target component by comparing the position information of the removal position with the mounting position information of the components included in the component position information, Controls to display the storage position of the mounting target component instead of the mounting position of the mounting target component on the device. An information processing system characterized by the above, according to Claim 2.

5. When the component information is a captured image of the component, the processor identifies the mounting position of the mounting target component on the device by comparing the captured image of the mounting target component with the captured images of the components included in the component position information, according to the information processing system described in Claim 1.

6. The processor, Includes time information that identifies the time when the component was removed in the component position information, Presents the mounting procedure of the component on the device by reversing the order in which the components were removed with reference to the component position information. An information processing system characterized by the above, according to Claim 1.

7. The information processing system according to claim 6, wherein when the processor detects completion of attachment of the attachment target component to the device by analyzing a captured image of the device, as an attachment procedure to the device, the processor presents a component to be attached to the device next.

8. A program for causing a computer to acquire a captured image of a device, when it is detected that a component has been removed from the device by analyzing the captured image of the device, create component position information of the component by associating component information for identifying the component with attachment position information indicating a position that is both the removal position and the attachment position of the component, when an attachment target component to be attached to the device is identified, control to display an attachment position of the attachment target component to the device with reference to the component position information. ​

Images