Terminal, system, terminal control method, and program

Abstract

To provide a terminal which contributes to the prevention of mistakes by an operator operating in a data center or the like.SOLUTION: A terminal comprises detection means, determination means and generation means. The detection means detects a first AR (Augmented Reality) marker included in video data. The detection means receives AR marker information related to a second AR marker from the outside, and determines whether the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information. When the first and second AR markers match, the generation means generates a virtual object, and overlays the virtual object generated at the position of the detected first AR marker in the image data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a terminal, a system, a terminal control method, and a program. [Background technology]

[0002] There is technology that utilizes AR (Augmented Reality) markers.

[0003] For example, Patent Document 1 describes a method for determining a display pattern of a virtual object based on a combination of multiple different AR markers detected simultaneously, and adding AR content based on a user's focus on the virtual object. The information display device in Patent Document 1 includes a data transceiver, a data analysis unit, an object creation unit, and an augmented reality management unit. The data transceiver acquires a real image including multiple AR markers. The data analysis unit determines tag information, types, and numbers of AR markers in the real image and determines a first display pattern for a first virtual object based on the types and numbers of AR markers. The object creation unit acquires first additional information for the first virtual object from a predetermined database based on the tag information of the AR markers, and creates a first virtual object based on the first additional information. The augmented reality management unit creates and outputs a first augmented reality image in which the created first virtual object is superimposed on the real image according to the determined first display pattern. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-043752 Summary of the Invention [Problem to be solved by the invention]

[0005] In data centers and other locations, in order to manage a large number of servers and network devices, physical tags are attached to the devices themselves or cables, and the tags are inscribed with the configuration information and corresponding identifiers of each device. By tagging in this way, when performing maintenance work, workers can match the configuration information with the tag identifier, identify the target of maintenance, and then perform the work.

[0006] Meanwhile, there are often operators in remote locations separate from the workers working on-site at data centers, etc. The operators instruct the workers on which equipment to operate, either in advance or while the workers are performing their work. Based on the instructions, the workers refer to the tags attached to the equipment, identify the equipment to be operated, and perform maintenance work, etc. In this case, it takes a lot of time for the workers to correctly find the tag of the equipment to be operated from among the large number of tags. In addition, work errors can occur when the workers misread the tag identifiers.

[0007] A primary object of the present invention is to provide a terminal, a system, a terminal control method, and a program that contribute to preventing work errors by workers working in data centers and the like. [Means for solving the problem]

[0008] According to a first aspect of the present invention, there is provided a terminal including: a detection means for detecting a first AR (Augmented Reality) marker included in video data; a determination means for receiving AR marker information regarding a second AR marker from an external device, and determining whether the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information; and a generation means for generating a virtual object and superimposing the generated virtual object on the position of the detected first AR marker in the video data if the first and second AR markers match.

[0009] According to a second aspect of the present invention, there is provided a system including: an operator terminal used by an operator in a remote location away from a data center where a plurality of devices, each of which is assigned an AR (Augmented Reality) marker, is installed; and a worker terminal used by a worker performing work at the data center, wherein the operator terminal transmits AR marker information regarding AR markers assigned to devices to be operated among the plurality of devices installed in the data center to the worker terminal, the worker terminal detects AR markers included in video data obtained by photographing at least one of the plurality of devices, and determines whether the detected AR marker matches the AR marker assigned to the device to be operated based on information obtained from the detected AR marker and the AR marker information, and if the detected AR marker matches the AR marker assigned to the device to be operated, generates a virtual object and superimposes the generated virtual object at the position of the detected AR marker in the video data.

[0010] According to a third aspect of the present invention, there is provided a method for controlling a terminal, in which a first AR (Augmented Reality) marker included in video data is detected in the terminal, AR marker information regarding a second AR marker is received from an external device, whether or not the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information is determined, and if the first and second AR markers match, a virtual object is generated and the generated virtual object is superimposed on the position of the detected first AR marker in the video data.

[0011] According to a fourth aspect of the present invention, there is provided a program for causing a computer mounted on a terminal to execute the following processes: detecting a first AR (Augmented Reality) marker included in video data; receiving AR marker information regarding a second AR marker from outside, and determining whether the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information; and, if the first and second AR markers match, generating a virtual object and superimposing the generated virtual object at the position of the detected first AR marker in the video data. [Effects of the Invention]

[0012] According to each aspect of the present invention, a terminal, a system, a terminal control method, and a program are provided that contribute to preventing work errors by workers working in a data center or the like. Note that the effects of the present invention are not limited to those described above. The present invention may achieve other effects instead of or in addition to the effects described above. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram for explaining an outline of an embodiment. [Figure 2] FIG. 2 is a flowchart for explaining an outline of the operation of one embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a schematic configuration of a remote operation support system according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of table information according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a diagram illustrating an example of a processing configuration of an operator terminal according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram for explaining the operation of the remote operation support system according to the embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating an example of a processing configuration of the worker terminal according to the embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating an example of a display on the worker terminal according to the embodiment of the present disclosure. [Figure 9] FIG. 9 is a sequence diagram illustrating an example of the operation of the operator terminal and the configuration management server according to the embodiment of the present disclosure. [Figure 10] FIG. 10 is a flowchart illustrating an example of the operation of the supporter terminal according to the embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating an example of a schematic configuration of a remote operation support system according to an embodiment of the present disclosure. [Figure 12] FIG. 12 is a diagram for explaining the operation of the remote operation support system according to the embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram illustrating an example of a schematic configuration of a remote operation support system according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram for explaining the operation of the remote operation support system according to the embodiment of the present disclosure. [Figure 15] 15A and 15B are diagrams illustrating an example of a display on the worker terminal according to an embodiment of the present disclosure. [Figure 16] FIG. 16 is a diagram illustrating an example of a schematic configuration of a remote operation support system according to an embodiment of the present disclosure. [Figure 17] FIG. 17 is a diagram illustrating an example of table information according to an embodiment of the present disclosure. [Figure 18] FIG. 18 is a diagram for explaining the operation of the remote operation support system according to an embodiment of the present disclosure. [Figure 19] FIG. 19 is a diagram illustrating an example of a schematic configuration of a remote operation support system according to an embodiment of the present disclosure. [Figure 20] FIG. 20 is a diagram illustrating an example of table information according to an embodiment of the present disclosure. [Figure 21] FIG. 21 is a diagram illustrating an example of table information according to an embodiment of the present disclosure. [Figure 22] FIG. 22 is a diagram for explaining the operation of the remote operation support system according to an embodiment of the present disclosure. [Figure 23]FIG. 23 is a diagram illustrating an example of a hardware configuration of a worker terminal according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] First, an overview of one embodiment will be described. Note that the reference numerals in the drawings are added to each element for convenience as an example to facilitate understanding, and the description of this overview is not intended to be limiting in any way. Furthermore, unless otherwise specified, the blocks shown in each drawing represent functional units, not hardware units. Connection lines between blocks in each drawing include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of main signals (data) and do not exclude bidirectionality. Note that in this specification and drawings, elements that can be similarly described may be assigned the same reference numerals to avoid redundant explanation.

[0015] A terminal 100 according to one embodiment includes a detection unit 101, a determination unit 102, and a generation unit 103 (see FIG. 1). The detection unit 101 detects a first augmented reality (AR) marker included in video data (step S1 in FIG. 2). The determination unit 102 receives AR marker information related to a second AR marker from an external device, and determines whether the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information (step S2). If the first and second AR markers match, the generation unit 103 generates a virtual object and superimposes the generated virtual object at the position of the detected first AR marker in the video data (step S3).

[0016] As mentioned above, physical tags are sometimes attached to each device to manage the large number of servers and network devices installed in data centers and other facilities. During maintenance work, workers refer to the tags to identify the device to be operated and then perform the work. However, it is difficult for workers to quickly find the tag to be operated from among the large number of tags.

[0017] There is a technology that uses AR markers to efficiently identify the device to be operated from among a large number of devices. With existing AR technology, when multiple AR markers are detected by a device, a virtual object is displayed for each detected AR marker. However, when AR markers are installed to manage a large number of devices in a densely packed area, such as in a data center, displaying a virtual object for each AR marker can reduce visibility for workers and hinder their work.

[0018] In this regard, Patent Document 1 discloses a technique in which, when a worker terminal detects multiple AR markers, it displays a single large virtual object instead of displaying a virtual object for each AR marker. Displaying a single large virtual object improves worker visibility. Patent Document 1 also discloses a technique in which a worker terminal detects the worker's line of sight and changes the position and type of a displayed virtual object according to the position of the worker's line of sight.

[0019] However, in the case of the technology disclosed in Patent Document 1, the virtual objects displayed on the worker terminal change without the intervention of a person such as an operator or approver, which may lead to operational errors due to misunderstandings by the worker.

[0020] Therefore, in the remote operation support system disclosed herein, a unique AR marker is assigned to a tag attached to each device. The terminal 100 acquires AR marker information related to the AR marker assigned to the device to be worked on from the operator, and determines whether the AR marker detected from the video data matches the AR marker designated by the operator. If the two AR markers match, the terminal 100 displays a virtual object corresponding only to the AR marker corresponding to the device designated by the operator, even if multiple devices (multiple AR markers) appear in the video data. As a result, the above-mentioned problem (occurrence of misrecognition by the worker) is resolved. In other words, the terminal 100 can prevent a decrease in visibility when multiple AR markers are detected, and can display a virtual object on the device designated by the operator.

[0021] Specific embodiments will be described in more detail below with reference to the drawings.

[0022] [First embodiment] The first embodiment will be described in more detail with reference to the drawings.

[0023] [System Configuration] 3 is a diagram showing a schematic configuration of a remote work support system according to an embodiment of the present disclosure. In the first embodiment, the configuration, operation, etc. of the remote work support system will be described assuming a case in which the system includes an operator who operates and manages equipment in a data center at a remote location and a worker who performs work on-site (at the data center) under instructions from the operator.

[0024] The remote operation support system includes an operator terminal 10, an operator terminal 20, and a configuration management server 30.

[0025] The operator terminal 10 is a terminal used by an operator in a remote location. Examples of the operator terminal 10 include a personal computer, a notebook personal computer, and a tablet.

[0026] The worker terminal 20 is a terminal used by a worker who performs maintenance work, inspection work, etc. on equipment in a data center. For example, a device such as a smartphone or a tablet is exemplified as the worker terminal 20. Alternatively, the worker terminal 20 may be an HMD (Head Mounted Display) or the like.

[0027] The configuration management server 30 is a server that manages the configuration information of devices installed in a data center.

[0028] Each device installed in the data center (such as a server 41, a network switch 42, and a LAN (Local Area Network) cable 43) is assigned a unique AR (Augmented Reality) marker 50 in advance. The remote work support system includes a group of AR markers assigned to each device in the data center.

[0029] The AR marker functions as a marker (landmark) to realize "augmented reality," which allows digital content to be overlaid on real space.

[0030] The configuration management server 30 manages information about each AR marker 50 (hereinafter referred to as AR marker information) in association with the configuration information of each device.

[0031] Examples of the AR marker information include information read from an image of the AR marker 50 itself, a barcode, a two-dimensional barcode, or other code. For example, the AR marker information may be a character string for identifying each device. Alternatively, the AR marker information may be an AR marker ID.

[0032] The configuration information stored in the configuration management server 30 includes an ID assigned to each device, a mounting position (the position of a rack used in a data center or a position within a rack), cable connection port information, and the like.

[0033] As shown in FIG. 4, the configuration management server 30 stores the AR marker information and the configuration information in association with each other.

[0034] [Device configuration] 5 is a diagram illustrating an example of the internal configuration of the operator terminal 10 according to the embodiment of the present disclosure. Referring to FIG. 5, the operator terminal 10 includes an input unit 111, an output unit 112, and a data transmission / reception unit 113.

[0035] The input unit 111 is a means for accepting input operations from the operator. The output unit 112 is a means for outputting data (information) to the operator. The output unit 112 provides information to the operator by displaying the data on an LCD monitor or the like.

[0036] The data transmission / reception unit 113 is a means for transmitting data to an external device (for example, the configuration management server 30) and receiving data from the external device. The data transmission / reception unit 113 transmits and receives data to and from the external device using a wired or wireless communication means.

[0037] The operator operates the operator terminal 10 to request the configuration management server 30 to provide AR marker information of the device to be worked on, using configuration information such as a server ID (see FIG. 6). For example, the operator terminal 10 transmits the server ID to the configuration management server 30.

[0038] If the configuration management server 30 stores the configuration information (for example, a server ID) received from the operator terminal 10, the configuration management server 30 transmits, to the operator terminal 10, AR marker information associated with the received configuration information.

[0039] If the received configuration information is not stored, the configuration management server 30 sends an error message to the operator terminal 10.

[0040] The operator terminal 10 that has acquired the AR marker information transmits the AR marker information to the worker terminal 20.

[0041] 7 is a diagram illustrating an example of the internal configuration of the worker terminal 20 according to the embodiment of the present disclosure. Referring to FIG. 7, the worker terminal 20 includes a virtual object display unit 201, a data transmission / reception unit 202, a sensor unit 203, an input unit 204, and an output unit 205.

[0042] Furthermore, the virtual object display unit 201 includes a marker detection unit 211 , a marker determination unit 212 , and an object generation unit 213 .

[0043] The virtual object display unit 201 is a means for displaying a virtual object. A virtual object is an object (for example, a figure, a symbol, or a character) that does not exist in the real world and is created virtually.

[0044] The marker detection unit 211 is a unit that detects a first AR marker included in the video data. The marker detection unit 211 is included in the virtual object display unit 201, and detects the AR marker 50 included in the video data received from the sensor unit 203. The marker detection unit 211 detects the AR marker 50 from the video data using an existing AR marker detection algorithm or the like.

[0045] The marker determination unit 212 is a means for receiving AR marker information related to the second AR marker from the outside, and determining whether the first and second AR markers match based on the AR marker information and information obtained from the first AR marker detected from the video data. The marker determination unit 212 is included in the virtual object display unit 201. The marker determination unit 212 determines whether the AR marker 50 detected by the marker detection unit 211 matches the AR marker 50 corresponding to the AR marker information received from the operator terminal 10 (the AR marker 50 attached to the device designated by the operator as the operation target).

[0046] The object generation unit 213 is a means for generating a virtual object when the first and second AR markers match, and superimposing the generated virtual object at the position of the detected first AR marker in the video data. The object generation unit 213 is included in the virtual object display unit 201. The object generation unit 213 generates a virtual object corresponding to the AR marker 50 attached to the device designated by the operator as the operation target. The object generation unit 213 generates composite video data in which the virtual object is superimposed at the position of the target AR marker (the AR marker 50 attached to the device designated as the operation target) in the video data.

[0047] The data transmitter / receiver 202 is a means for transmitting and receiving data to and from the operator terminal 10 .

[0048] The sensor unit 203 is a means for acquiring video data including the AR marker 50. The sensor unit 203 controls a camera device or the like to acquire video data (image data) in which the AR marker 50 appears.

[0049] The input unit 204 is a means for receiving input operations from the operator. The output unit 205 is a means for displaying the processing results of the AR application (virtual object display unit 201) and video data.

[0050] When the data transmitter / receiver 202 receives the AR marker information transmitted from the operator terminal 10 , the data transmitter / receiver 202 stores the AR marker information in the marker determination unit 212 of the virtual object display unit 201 .

[0051] Thereafter, in response to an operation by the worker, the worker terminal 20 starts capturing images of the periphery of the device using the sensor unit 203. The sensor unit 203 acquires image data of the periphery of the device. The image data acquired by capturing images is transferred to the marker detection unit 211 of the virtual object display unit 201 as needed.

[0052] When the marker detection unit 211 detects the AR marker 50 attached to the device in the video data, it passes the detected AR marker 50 to the marker determination unit 212 .

[0053] The marker determination unit 212 determines whether or not the two AR markers match based on information obtained from the detected AR marker 50 and the AR marker information received from the operator terminal 10 (stored AR marker information).

[0054] That is, the marker determination unit 212 determines whether the AR marker 50 detected from the video data matches the AR marker 50 attached to the device designated by the administrator as the operation target. The marker determination unit 212 determines whether the AR marker 50 detected from the video data is the AR marker 50 attached to the device designated by the administrator as the operation target.

[0055] If the AR marker information is an image of the AR marker itself, the marker determination unit 212 determines whether the image of the AR marker 50 detected from the video data matches the AR marker information. If the graphic of the AR marker 50 shown in the video data and the graphic of the AR marker information are the same, the marker determination unit 212 determines that the two AR markers match.

[0056] When the AR marker information is a character string or the like, the marker determination unit 212 decodes the AR marker 50 detected from the video data to extract the character string, and compares the extracted character string with the AR marker information (character string).

[0057] If the two AR markers match, the marker determination unit 212 notifies the object generation unit 213 of this fact and information about the AR marker 50 (target AR marker) attached to the device designated by the operator as the operation target.

[0058] For example, the marker determination unit 212 notifies the object generation unit 213 of the coordinate information of the AR marker 50 (image coordinates of the target AR marker).

[0059] When the object generation unit 213 receives a notification that the two AR markers match, the object generation unit 213 generates a virtual object. For example, the object generation unit 213 generates a virtual object (for example, a rectangular shape surrounded by a dotted line) that clearly indicates the device to be operated.

[0060] The object generation unit 213 superimposes the generated virtual object on the video data at a position corresponding to the target AR marker. The object generation unit 213 outputs the video data with the virtual object superimposed thereon from the output unit 205.

[0061] The worker checks the video data output via the output unit 205. For example, the worker terminal 20 displays the composite video data (video data on which the virtual object 51 is superimposed) as shown in FIG.

[0062] If the two AR markers (the detected AR marker 50 and the AR marker 50 attached to the device designated by the operator as the target for operation) do not match, the virtual object is not generated and the peripheral device continues to be photographed (acquired video data).

[0063] Note that even if multiple AR markers 50 are detected in the video data, the above process is executed for each AR marker 50. As a result, virtual objects are displayed only for the AR markers 50 attached to the equipment designated by the operator as the operation target. By such operation of the operator terminal 20, unnecessary virtual objects are not displayed, and the virtual object of the operation target (equipment to be maintained, etc.) intended by the operator is presented to the operator.

[0064] <Explanation of operation> Next, the operation of the remote operation support system according to the embodiment of the present disclosure will be described.

[0065] First, the operations of the operator terminal 10 and the configuration management server 30 will be described with reference to FIG.

[0066] The operator terminal 10 receives input of configuration information to be transmitted to the configuration management server 30 from the operator (step S01).

[0067] The operator terminal 10 transmits the acquired configuration information to the configuration management server 30 (step S02).

[0068] The configuration management server 30 determines whether the device corresponding to the received configuration information is managed by the configuration management server 30 itself. That is, the configuration management server 30 determines whether the received configuration information is stored (step S03).

[0069] If the device corresponding to the configuration information received from the operator terminal 10 is being managed (step S03, Yes branch), the configuration management server 30 transmits AR marker information linked to the configuration information of the corresponding device to the operator terminal 10 (step S04).

[0070] The operator terminal 10 transmits the AR marker information received from the configuration management server 30 to the worker terminal 20 (step S05).

[0071] If the equipment corresponding to the configuration information received from the operator terminal 10 is not being managed (step S03, No branch), the configuration management server 30 sends an error message to the operator terminal 10 indicating that the received configuration information does not exist in the configuration management server 30 (step S06).

[0072] The operator terminal 10 notifies the operator of the error message received from the configuration management server 30 (step S07).

[0073] Next, the operation of the worker terminal 20 will be described with reference to FIG.

[0074] The worker terminal 20 receives the AR marker information from the operator terminal 10 (step S11).

[0075] The worker terminal 20 receives a response from the worker as to whether or not to start photographing (step S12).

[0076] If the image capturing is not to be started (step S12, No branch), the worker terminal 20 ends the operation.

[0077] When start of shooting is selected (step S12, Yes branch), the worker terminal 20 acquires video data using the sensor unit 203 (step S13).

[0078] The marker detection unit 211 of the worker terminal 20 attempts to detect the AR marker 50 from the acquired video data. The marker detection unit 211 determines whether or not the AR marker 50 exists in the video data (step S14).

[0079] If the AR marker 50 is not detected (step S14, No branch), the worker terminal 20 returns to step S12.

[0080] If an AR marker 50 is detected (step S14, Yes branch), the worker terminal 20 determines whether the detected AR marker 50 matches the AR marker 50 corresponding to the AR marker information received from the operator terminal 10 (step S15).

[0081] If the two AR markers do not match (step S15, No branch), the worker terminal 20 returns to step S12.

[0082] If the two AR markers match (step S15, Yes branch), the worker terminal 20 generates a virtual object for indicating the detected AR marker 50 (the device to which the AR marker 50 is attached) (step S16).

[0083] Furthermore, the object generating unit 213 superimposes the generated virtual object on the position of the target AR marker 50 in the video data, and generates composite video data in which the virtual object is combined (step S17).

[0084] The output unit 205 of the worker terminal 20 outputs the generated composite video data (step S18). The output unit 205 displays the composite video data on a liquid crystal monitor or the like.

[0085] After that, the worker terminal 20 returns to step S12.

[0086] As described above, the remote operation support system according to the first embodiment includes an operator terminal 10 used by an operator in a remote location away from a data center where multiple devices, each with an AR marker 50, are installed. The remote operation system also includes a worker terminal 20 used by a worker performing work at the data center. The operator terminal 10 transmits AR marker information about the AR marker 50 attached to a device to be operated among multiple devices installed in the data center to the worker terminal 20. The worker terminal 20 detects the AR marker 50 included in video data obtained by capturing an image of at least one device among the multiple devices. The worker terminal 20 determines whether the detected AR marker matches the AR marker attached to the device to be operated based on information obtained from the detected AR marker and the AR marker information. If the detected AR marker matches the AR marker attached to the device to be operated, the worker terminal 20 generates an object and superimposes the generated virtual object at the position of the detected AR marker in the video data.

[0087] The remote operation support system also includes a configuration management server 30 that stores the configuration information of each of a plurality of devices installed in the data center in association with the AR marker information. The operator terminal 10 transmits the configuration information of the device to be operated to the configuration management server 30, thereby acquiring AR marker information related to the AR marker attached to the device to be operated.

[0088] As described above, in the remote operation support system according to the first embodiment, an AR marker that matches an AR marker specified by an operator is identified from video data containing multiple AR markers. The operator terminal 20 generates a virtual object for only the identified AR marker and superimposes the generated virtual object on the video data. As a result, if a failure occurs in a device installed in a data center, the operator at the remote location can accurately notify the worker at the data center of the device to be operated without having to visit the data center in person. The worker can then perform maintenance work on the notified device. In other words, the remote operation support system accurately conveys instructions from the operator at the remote location to the worker on-site, allowing the worker to perform the work without making a mistake in the operation of the device.

[0089] [Second embodiment] Next, the second embodiment will be described in detail with reference to the drawings.

[0090] The following description will focus on the differences between the first and second embodiments.

[0091] The remote operation support system may include a monitoring server 60 that monitors the status of devices installed in a data center (see FIG. 11). The monitoring server 60 monitors the status of each device installed in the data center. The monitoring server 60 sends a PING command or the like to each device, and determines that a failure has occurred in a device that does not respond.

[0092] The monitoring server 60 transmits device information (e.g., the name of the device) of the device in which the failure occurred to the configuration management server 30 (see FIG. 12). By transmitting the device information to the configuration management server 30, the monitoring server 60 receives AR marker information linked to the device information (configuration information) from the configuration management server 30.

[0093] The monitoring server 60 transmits a failure occurrence notification including the received AR marker information to the operator terminal 10. The operator terminal 10 notifies (warns) the operator that a failure has occurred in a device within the data center.

[0094] In this way, the AR marker information of the device that is the target of repair or the like may be automatically provided to the operator (operator terminal 10) without the operator requesting the configuration management server 30 to provide the AR marker information.

[0095] As described above, the remote operation support system according to the second embodiment includes a monitoring server 60 that monitors the status of each of a plurality of devices installed in a data center. The monitoring server 60 transmits configuration information (device information) of a device in which a failure has occurred among the plurality of devices to the configuration management server 30, thereby acquiring AR marker information related to the AR marker attached to the device in which the failure has occurred. The monitoring server 60 transmits the acquired AR marker information to the operator terminal 10. In this way, the monitoring server 60 detects a failure in a device installed in a data center or the like. The operator can obtain the AR marker information of the device in which the failure has occurred via the configuration management server 30. As a result, the operator can quickly deal with the failure.

[0096] [Third embodiment] Next, the third embodiment will be described in detail with reference to the drawings.

[0097] The following description will focus on the differences between the first to third embodiments.

[0098] The remote support work system may include an approver terminal 70 operated by a third party who acts as an approver (see FIG. 13).

[0099] The approver terminal 70 acquires the AR marker information from the operator terminal 10 (see FIG. 14). The approver terminal 70 also acquires the video data from the worker terminal 20.

[0100] The approver terminal 70 has the same functions as the virtual object display unit 201 of the worker terminal 20. Similar to the worker terminal 20, if an AR marker corresponding to the AR marker information received from the operator terminal 10 appears in the video data, the approver terminal 70 superimposes a virtual object at the position of the AR marker.

[0101] The approver checks the video data (composite video data) on which the virtual object is superimposed, and determines whether the equipment indicated by the virtual object is the correct target for maintenance or the like.

[0102] If the approver determines that the device is a valid target for maintenance, the approver inputs this information into the approver terminal 70. After obtaining the approver's approval, the approver terminal 70 transmits the AR marker information received from the operator terminal 10 to the worker terminal 20.

[0103] The worker terminal 20 that has received the AR marker information processes the received AR marker information in the same manner as in the first embodiment.

[0104] In this way, the composite image data displayed on the worker terminal 20 from the AR marker information and the image data is displayed in advance on the approver terminal 70. The approver may check the composite image data output from the approver terminal 70 and perform approval work for the equipment to be maintained or the like before the worker performs the work.

[0105] When the target device for maintenance or the like is approved by the approver, a virtual object is displayed on the worker terminal 20. That is, before approval by the approver, a screen such as that shown in Fig. 15A is displayed on the worker terminal 20, and after approval by the approver, a screen such as that shown in Fig. 15B is displayed on the worker terminal 20.

[0106] The operator terminal 10 and the approver terminal 70 may be the same terminal. That is, the operator may also serve as the approver.

[0107] As described above, the remote work support system according to the third embodiment includes the approver terminal 70 used by the approver. The worker terminal 20 transmits video data obtained by capturing an image of at least one device among a plurality of devices to the approver terminal 70. The operator terminal 10 transmits AR marker information of a device to be operated among a plurality of devices installed in a data center to the approver terminal 70. The approver terminal 70 detects an AR marker included in the video data received from the worker terminal 20. The approver terminal 70 determines whether the detected AR marker matches the AR marker attached to the device to be operated based on information obtained from the detected AR marker and the AR marker information. If the detected AR marker matches the AR marker attached to the device to be operated, the approver terminal 70 generates an object and superimposes the generated virtual object at the position of the detected AR marker in the video data. If the approver approves an operation on a device on which a virtual object is superimposed in the video data, the approver terminal 70 transmits AR marker information of the device to be operated to the worker terminal 20. This operation of the remote operation system allows the approver to check whether the equipment being maintained is correct. As a result, the possibility of work (operation) being performed on the wrong equipment is reduced, and maintenance can be performed more reliably.

[0108] [Fourth embodiment] Next, the fourth embodiment will be described in detail with reference to the drawings.

[0109] The following description will focus on the differences between the first to fourth embodiments.

[0110] The remote operation support system may include a task management server 80 that manages the order of operations and the like as one piece of task information by linking a plurality of pieces of AR marker information to a task ID (see FIG. 16).

[0111] The operator registers in advance the order in which the devices are to undergo maintenance, etc., as well as the AR marker information of each device (one or more pieces of AR marker information) in the task management server 80. For example, task information such as that shown in FIG. 17 is registered in the task management server 80.

[0112] Operators and workers share task IDs for work to be done within the data center before starting work.

[0113] Before starting a task, the worker inputs a task ID that has been shared in advance into the worker terminal 20. The worker terminal 20 transmits the task ID to the task management server 80 (see FIG. 18).

[0114] The task management server 80 transmits to the worker terminal 20 at least one piece of AR marker information associated with the task ID.

[0115] The worker terminal 20 superimposes a virtual object on the video data using at least one piece of acquired AR marker information. The worker terminal 20 composites the virtual object on the equipment being worked on.

[0116] The worker terminal 20 generates virtual objects in the order of the AR marker information acquired from the task management server 80 and superimposes the video data.

[0117] As a result, even if the operator and the worker are not working at the same time, the worker can execute the work instructed by the operator by transmitting task information to the task management server 80 and acquiring AR marker information.

[0118] As described above, in the remote operation support system according to the fourth embodiment, the task management server 80 manages the operations to be performed by the workers (the order of the devices to be worked on and the corresponding AR marker information). As a result, even if the timing of the operator and the worker to perform the work differs, the worker can perform the work intended by the operator.

[0119] [Fifth embodiment] Next, the fifth embodiment will be described in detail with reference to the drawings.

[0120] The following description will focus on the differences between the first to fifth embodiments.

[0121] The remote operation support system may include an authentication server 90 and a user management server 91 in addition to the task management server 80 (see FIG. 19).

[0122] The authentication server 90 issues an authentication token. The authentication server 90 performs authentication when the worker starts up the worker terminal 20. The authentication server 90 stores user information and authentication information, which will be described later, in association with each other.

[0123] For example, when a worker uses the worker terminal 20, the worker terminal 20 acquires authentication information (for example, a combination of an ID and a password, or biometric information). The worker terminal 20 transmits the acquired authentication information to the authentication server 90.

[0124] The authentication server 90 authenticates the user of the worker terminal 20 using pre-registered authentication information and authentication information acquired from the worker terminal 20. The authentication server 90 transmits the authentication result (authentication success, authentication failure) to the worker terminal 20.

[0125] When the authentication success is received, the worker terminal 20 accepts the worker's operation. In this way, the authentication server 90 identifies the worker who has legitimate authority for equipment maintenance and the like.

[0126] The user management server 91 manages the user information of the workers. For example, the user management server 91 manages and stores the user information as shown in Fig. 20. As shown in Fig. 20, the user information includes the worker's user ID, name, place of work, work responsibilities, etc.

[0127] The operator acquires the worker's user information (e.g., user ID) from the user management server 91. For example, the operator inputs the worker's name, etc. into the operator terminal 10. The operator terminal 10 transmits the acquired name to the user management server 91. The user management server 91 transmits the user ID corresponding to the acquired name to the operator terminal 10. The operator terminal 10 displays the acquired user ID.

[0128] The administrator registers the acquired user information (e.g., user ID) together with the task information including the AR marker information in the task management server 80. For example, the task management server 80 stores task information such as that shown in Fig. 21. As shown in Fig. 21, the task management server 80 stores the task ID, user ID, and AR marker information in association with each other.

[0129] When the authentication server 90 authenticates the worker using the worker terminal 20, it transmits user information (e.g., user ID) of the successfully authenticated worker to the task management server 80 (see FIG. 22 ). The task management server 80 identifies task information corresponding to the acquired user information (user ID) and transmits AR marker information included in the identified task information to the worker terminal 20.

[0130] The worker terminal 20 processes the acquired AR marker information in the same manner as in the first embodiment. Furthermore, each of multiple workers who have performed login authentication can acquire individual task information (AR marker information). That is, each worker can perform maintenance and inspection work at a different data center or a different work location within a data center.

[0131] As described above, the remote operation support system according to the fifth embodiment includes an authentication server 90 and a task management server 80. The authentication server 90 associates and stores the authentication information and user information of each of a plurality of workers, and authenticates the person to be authenticated who intends to use the worker terminal 20. The task management server 80 stores task information, which is information about work on a device to be operated among a plurality of devices installed in a data center, including user information of the worker operating the device to be operated and AR marker information of the device to be operated. The worker terminal 20 transmits the authentication information of the person to be authenticated to the authentication server 90. The authentication server 90 authenticates the person to be authenticated based on the received authentication information, and if the authentication of the person to be authenticated is successful, transmits user information of the person to be successfully authenticated to the task management server 80. The task management server 80 transmits AR marker information corresponding to the received user information to the authentication server 90. The authentication server 90 transmits the received AR marker information to the worker terminal 20 used by the person to be successfully authenticated. In this way, the task management server 80 transmits information (task information, AR marker information) about the work to be performed by a worker who has been successfully authenticated, based on the user information of the worker, to the worker terminal 20. As a result, even when multiple workers are working at the same time, each worker is reliably notified of the work content assigned to that worker.

[0132] Next, the hardware of each device constituting the information processing system will be described. Fig. 23 is a diagram showing an example of the hardware configuration of the worker terminal 20.

[0133] The worker terminal 20 can be configured by an information processing device (so-called computer), and has the configuration exemplified in Fig. 23. For example, the worker terminal 20 includes a processor 311, a memory 312, an input / output interface 313, and a communication interface 314. The components such as the processor 311 are connected by an internal bus or the like, and are configured to be able to communicate with each other.

[0134] However, the configuration shown in Fig. 23 is not intended to limit the hardware configuration of the worker terminal 20. The worker terminal 20 may include hardware not shown, and may not include the input / output interface 313 as necessary. Furthermore, the number of processors 311 and the like included in the worker terminal 20 is not intended to be limited to the example shown in Fig. 23, and for example, the worker terminal 20 may include multiple processors 311.

[0135] The processor 311 is a programmable device such as a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP). Alternatively, the processor 311 may be a device such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The processor 311 executes various programs including an operating system (OS).

[0136] The memory 312 is a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), etc. The memory 312 stores an OS program, application programs, and various data.

[0137] The input / output interface 313 is an interface for a display device and an input device (not shown). The display device is, for example, a liquid crystal display. The input device is, for example, a device that accepts user operations such as a keyboard or a mouse.

[0138] The communication interface 314 is a circuit, module, etc. that communicates with other devices. For example, the communication interface 314 includes a network interface card (NIC).

[0139] The functions of the worker terminal 20 are realized by various processing modules. The processing modules are realized, for example, by the processor 311 executing a program stored in the memory 312. The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transitory medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. That is, the present invention can also be embodied as a computer program product. The program can be downloaded via a network or updated using a storage medium storing the program. The processing modules can also be realized by a semiconductor chip.

[0140] The operator terminal 10, configuration management server 30, etc. can also be configured using information processing devices, just like the worker terminal 20, and their basic hardware configurations are no different from those of the worker terminal 20, so a description thereof will be omitted.

[0141] The worker terminal 20, which is an information processing device, is equipped with a computer, and the computer executes a program to realize the functions of the worker terminal 20. The worker terminal 20 also executes a control method for the worker terminal 20 by the program.

[0142] [Variations] The configuration, operation, etc. of the information processing system described in the above embodiment are merely examples and are not intended to limit the configuration, etc. of the system.

[0143] The configuration management server 30 may transmit the AR marker information corresponding to the configuration information received from the operator terminal 10 directly to the worker terminal 20 instead of the operator terminal 10 .

[0144] The worker terminal 20 may transmit the acquired video data to the virtual object display unit 201 in real time, or may transmit the video data to the virtual object display unit 201 after the image capturing is completed.

[0145] All or part of the AR marker information transmitted from the operator terminal 10 to the worker terminal 20 may be presented to the worker in advance. All or part of the AR marker information may be displayed on the worker terminal 20 or another terminal, allowing the worker to understand the AR marker information before the work actually begins. By understanding the AR marker information transmitted in advance, the worker can accurately understand the work target instructed by the worker terminal 20. In other words, the worker can understand in advance the AR marker designated as the work target, allowing the worker to efficiently perform work with high precision.

[0146] In the flow charts (flowcharts, sequence diagrams) used in the above explanation, multiple steps (processes) are described in order, but the execution order of the steps executed in the embodiments is not limited to the order described. In the embodiments, the order of the illustrated steps can be changed to the extent that the content is not affected, such as by executing each process in parallel.

[0147] The above-described embodiments have been described in detail to facilitate understanding of the present disclosure, and it is not intended that all of the above-described configurations are required. Furthermore, when multiple embodiments are described, each embodiment may be used alone or in combination. For example, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of one embodiment with another configuration.

[0148] From the above explanation, it is clear that the present invention has industrial applicability, and the present invention can be suitably applied to information processing systems that perform maintenance management of data centers and the like.

[0149] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0150] [Appendix 1] a detection means for detecting a first augmented reality (AR) marker included in the video data; a determination means for receiving AR marker information relating to a second AR marker from an external device, and determining whether or not the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information; a generation means for generating a virtual object when the first and second AR markers match, and superimposing the generated virtual object on the position of the detected first AR marker in the video data; A terminal comprising: [Appendix 2] The terminal according to claim 1, wherein the determination means receives the AR marker information from an operator terminal used by an operator in a remote location away from a data center. [Appendix 3] An operator terminal used by an operator in a remote location away from the data center, which is equipped with multiple devices each with an AR (Augmented Reality) marker; a worker terminal used by a worker performing work at the data center; Including, the operator terminal transmits, to the worker terminal, AR marker information relating to an AR marker attached to a device to be operated among a plurality of devices installed in the data center; The worker terminal includes: Detecting an AR marker included in video data obtained by capturing an image of at least one of the plurality of devices; determining whether the detected AR marker matches the AR marker attached to the device to be operated, based on information obtained from the detected AR marker and the AR marker information; If the detected AR marker matches the AR marker attached to the device to be operated, the system generates a virtual object and superimposes the generated virtual object at the position of the detected AR marker in the video data. [Appendix 4] The data center further includes a configuration management server that stores configuration information of each of a plurality of devices installed in the data center in association with the AR marker information, The system described in Appendix 3, wherein the operator terminal acquires the AR marker information regarding the AR marker attached to the device to be operated by transmitting the configuration information of the device to be operated to the configuration management server. [Appendix 5] The data center further includes a monitoring server that monitors the status of each of the plurality of devices installed in the data center; The system described in Appendix 4, wherein the monitoring server acquires the AR marker information regarding the AR marker attached to the failed device by transmitting the configuration information of the failed device among the plurality of devices to the configuration management server, and transmits the acquired AR marker information to the operator terminal. [Appendix 6] an authentication server that stores the authentication information and user information of each of the plurality of workers in association with each other, and authenticates an authenticatee who attempts to use the worker terminal; a task management server that stores task information related to work on a device to be operated among a plurality of devices installed in the data center, the task information including the user information of the worker operating the device to be operated and the AR marker information of the device to be operated; further comprising the worker terminal transmits the authentication information of the person to be authenticated to the authentication server; the authentication server authenticates the person to be authenticated based on the received authentication information, and when the authentication of the person to be authenticated is successful, transmits the user information of the person to be authenticated who has been successfully authenticated to the task management server; the task management server transmits the AR marker information corresponding to the received user information to the authentication server; The system according to any one of appendixes 3 to 5, wherein the authentication server transmits the received AR marker information to the worker terminal used by the person to be authenticated who has been successfully authenticated. [Appendix 7] further including an approver terminal used by the approver; the worker terminal transmits video data obtained by photographing at least one of the plurality of devices to the approver terminal; the operator terminal transmits the AR marker information of a device to be operated among a plurality of devices installed in the data center to the approver terminal; The approver terminal Detecting an AR marker included in the video data received from the worker terminal; determining whether the detected AR marker matches the AR marker attached to the device to be operated, based on information obtained from the detected AR marker and the AR marker information; If the detected AR marker matches the AR marker attached to the device to be operated, a virtual object is generated and the generated virtual object is superimposed on the position of the detected AR marker in the video data; 6. The system according to claim 3, wherein, when the approver approves an operation on a device on which the virtual object is superimposed in the video data, the AR marker information of the device to be operated is transmitted to the worker terminal. [Appendix 8] The system according to any one of appendixes 3 to 5, wherein the worker terminal presents all or part of the AR marker information to the worker. [Appendix 9] On the device, Detecting a first augmented reality (AR) marker included in the video data; receiving AR marker information relating to a second AR marker from an external device, and determining whether the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information; a terminal control method for generating a virtual object when the first and second AR markers match, and superimposing the generated virtual object at a position of the detected first AR marker in the video data; [Appendix 10] The computer installed in the device A process of detecting a first augmented reality (AR) marker included in the video data; receiving AR marker information relating to a second AR marker from an external device, and determining whether the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information; generating a virtual object when the first and second AR markers match, and superimposing the generated virtual object on the position of the detected first AR marker in the video data; A program to execute.

[0151] Furthermore, part or all of the configurations described in Appendix 2, which are dependent on the above-mentioned Appendix 1, may also be dependent on Appendix 3, Appendix 9, and Appendix 10 in the same dependent relationship as Appendix 2. Furthermore, not limited to Appendix 1, Appendix 3, Appendix 9, and Appendix 10, part or all of the configurations described as appendices may be made dependent on various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0152] The disclosures of the above-cited prior art documents are incorporated herein by reference. Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments. Those skilled in the art will understand that these embodiments are merely illustrative and that various modifications are possible without departing from the scope and spirit of the present invention. In other words, the present invention naturally includes various modifications and alterations that may be made by those skilled in the art in accordance with the entire disclosure, including the claims, and the technical concepts thereof. [Explanation of symbols]

[0153] 10 Operator terminal 20 Worker terminal 30 Configuration Management Server 41 Server 42 Network Switch 43 Cable 50 AR Markers 51 Virtual Objects 60 Monitoring Server 70 Approver terminal 80 Task Management Server 90 Authentication Server 91 User Management Server 100 devices 101 Detection means 102 Judgment means 103 Generation means 111 Input section 112 Output section 113 Data transmission and reception unit 201 Virtual object display section 202 Data transmission and reception unit 203 Sensor section 204 Input section 205 Output section 211 Marker detection unit 212 Marker Judgment Section 213 Object Generation Unit 311 processor 312 memory 313 Input / Output Interface 314 Communication Interface

Claims

1. a detection means for detecting a first augmented reality (AR) marker included in the video data; a determination means for receiving AR marker information relating to a second AR marker from an external device, and determining whether or not the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information; a generation means for generating a virtual object when the first and second AR markers match, and superimposing the generated virtual object at a position of the detected first AR marker in the video data; A terminal comprising:

2. The terminal according to claim 1 , wherein the determining means receives the AR marker information from an operator terminal used by an operator in a remote location away from the data center.

3. an operator terminal used by an operator in a remote location away from the data center, the operator terminal having a plurality of devices, each of which has an AR (Augmented Reality) marker attached thereto; a worker terminal used by a worker performing work at the data center; Including, the operator terminal transmits, to the worker terminal, AR marker information relating to an AR marker attached to a device to be operated among a plurality of devices installed in the data center; The worker terminal includes: Detecting an AR marker included in video data obtained by capturing an image of at least one of the plurality of devices; determining whether the detected AR marker matches the AR marker attached to the device to be operated, based on information obtained from the detected AR marker and the AR marker information; If the detected AR marker matches the AR marker attached to the device to be operated, the system generates a virtual object and superimposes the generated virtual object at the position of the detected AR marker in the video data.

4. a configuration management server that stores configuration information of each of a plurality of devices installed in the data center in association with the AR marker information; The system described in claim 3, wherein the operator terminal acquires the AR marker information regarding the AR marker attached to the device to be operated by transmitting the configuration information of the device to be operated to the configuration management server.

5. The data center further includes a monitoring server that monitors the status of each of the plurality of devices installed in the data center; The system described in claim 4, wherein the monitoring server acquires the AR marker information regarding the AR marker attached to the failed device by transmitting the configuration information of the failed device among the plurality of devices to the configuration management server, and transmits the acquired AR marker information to the operator terminal.

6. an authentication server that stores the authentication information and user information of each of the plurality of workers in association with each other, and authenticates an authenticatee who attempts to use the worker terminal; a task management server that stores task information related to work on a device that is an operation target among a plurality of devices installed in the data center, the task information including the user information of the worker that operates the device that is the operation target and the AR marker information of the device that is the operation target; further comprising the worker terminal transmits the authentication information of the person to be authenticated to the authentication server; the authentication server authenticates the person to be authenticated based on the received authentication information, and when the authentication of the person to be authenticated is successful, transmits the user information of the person to be authenticated who has been successfully authenticated to the task management server; the task management server transmits the AR marker information corresponding to the received user information to the authentication server; The system according to claim 3 , wherein the authentication server transmits the received AR marker information to the worker terminal used by the person who has been successfully authenticated.

7. further including an approver terminal used by the approver; the worker terminal transmits video data obtained by photographing at least one of the plurality of devices to the approver terminal; the operator terminal transmits the AR marker information of a device to be operated among a plurality of devices installed in the data center to the approver terminal; The approver terminal Detecting an AR marker included in the video data received from the worker terminal; determining whether the detected AR marker matches the AR marker attached to the device to be operated, based on information obtained from the detected AR marker and the AR marker information; If the detected AR marker matches the AR marker attached to the device to be operated, a virtual object is generated and the generated virtual object is superimposed on the position of the detected AR marker in the video data; 6. The system according to claim 3, wherein, when the approver approves an operation on a device on which the virtual object is superimposed in the video data, the AR marker information of the device to be operated is transmitted to the worker terminal.

8. The system according to claim 3 , wherein the worker terminal presents all or part of the AR marker information to the worker.

9. On the device, Detecting a first augmented reality (AR) marker included in the video data; receiving AR marker information relating to a second AR marker from an external device, and determining whether the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information; a terminal control method for generating a virtual object when the first and second AR markers match, and superimposing the generated virtual object at a position of the detected first AR marker in the video data;

10. The computer installed in the device A process of detecting a first augmented reality (AR) marker included in the video data; a process of receiving AR marker information relating to a second AR marker from an external device, and determining whether or not the first and second AR markers match based on information obtained from the detected first AR marker and the AR marker information; a process of generating a virtual object and superimposing the generated virtual object at a position of the detected first AR marker in the video data when the first and second AR markers match; A program to execute.

Images