Work monitoring system

Abstract

To enable safety management measures at the same level as those in a face-to-face manner to be taken for a worker who carries out a task and an operation on an information processing system, even from a remote site.SOLUTION: A work monitoring system includes: a worker virtual terminal 42; a monitoring person virtual terminal 43; a worker HMD 21 which is connected to the worker virtual terminal 42 and which is worn by a worker 2; a monitoring person HMD 31 which is connected to the monitoring person virtual terminal 43 and which is worn by a monitoring person 3; and a monitoring unit 41 that enables the worker virtual terminal 42 and the monitoring person virtual terminal 43 to share a screen therebetween. The worker virtual terminal 42 projects, on the virtual space of the worker HMD 21, a worker screen 23 displayed on the worker virtual terminal 42. The monitoring unit 41 shares the worker screen 23 between the worker virtual terminal 42 and the monitoring person virtual terminal 43. The monitoring person virtual terminal 43 projects the shared worker screen 23 on the virtual space of the monitoring person HMD 31, and permits the worker 2 to carry out a predetermined work on the worker virtual terminal 42 when a predetermined condition is satisfied.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to security technology in an information processing system, and particularly to a technology effective when applied to a work monitoring system that monitors operations and actions of an operator on an information processing system.

Background Art

[0002] There are cases where operations and work of an information processing terminal such as a PC (Personal Computer) by an operator must be performed under security control where information leakage, inflow, and other illegal acts are prohibited or severely restricted. For example, when accessing a PC or the like to perform work (production work) on an information processing system during actual operation for program release, operation work, emergency response, etc., or when performing work in an environment where important personal information such as a My Number can be referred to, such as operator work at a call center, it may apply to entrance examinations or qualification examinations where questions are presented and answers are input using a PC or the like.

[0003] When an operator performs operations on an information processing terminal in such an environment, it is necessary to take security management measures so that illegal acts are not committed. In this regard, conventionally, for example, a person who monitors and supervises (monitor) has directly monitored and supervised face-to-face by sitting nearby or making rounds for one or a plurality of operators. On the other hand, technologies for reducing the load of monitoring and supervision and achieving unmanned and automated operations have been studied.

[0004] For example, Japanese Patent No. 6910674 (Patent Document 1) describes that in an online examination system that monitors the actions of examinees using a video camera, an illegal monitoring member colored with fluorescence is attached to a predetermined part of the examinee, and by monitoring the movement of the member based on the image of the video camera, overlooking illegal acts of the examinee due to dead angles of the video camera is suppressed.

[0005] Furthermore, Japanese Patent Publication No. 2016-177475 (Patent Document 2) describes a monitoring system that detects the position of the examinee's gaze by having each examinee wear a device such as glasses-type terminal that can detect the position of their pupils, and determines whether or not cheating has occurred based on whether or not the gaze position is outside a predetermined range of the question paper, answer sheet, etc. for a certain period of time or longer.

[0006] Furthermore, Japanese Patent Publication No. 2009-237801 (Patent Document 3) describes how questions transmitted from a central device can be displayed on a terminal device such as a head-mounted display (HMD) worn on the examinee's head, ensuring that only the wearer can see the questions, preventing the wearer from seeing information other than that displayed on the terminal device, thereby preventing cheating, and eliminating the need to install surveillance cameras or other equipment or to deploy examination supervisors. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Patent No. 6910674 [Patent Document 2] Japanese Patent Publication No. 2016-177475 [Patent Document 3] Japanese Patent Publication No. 2009-237801 [Overview of the project] [Problems that the invention aims to solve]

[0008] In recent years, with the increasing prevalence of remote work / teleworking, there has been a growing need to remotely monitor and supervise terminal operations during actual work or testing using PCs, etc., where safety management measures are required.

[0009] In contrast, conventional technologies, such as those mentioned above, can automate and unmanned monitoring and supervision functions, thus eliminating the need for remote monitors. However, while these unmanned and automated systems can streamline monitoring and supervision, they make it difficult to implement the same level of physical and technical security measures as traditional face-to-face monitoring and supervision, leaving the risk of information leaks, intrusions, and other fraudulent activities.

[0010] For example, while the technology described in Patent Document 3 can prevent the inflow of information, it cannot ascertain what kind of terminal operations the worker is performing, and therefore cannot effectively prevent information leaks or other fraudulent activities. Furthermore, while the technologies described in Patent Documents 1 and 2 can ascertain the worker's actions to a certain extent, they cannot accurately ascertain what kind of terminal operations the worker is actually performing, and therefore cannot be said to be equivalent to the level of safety management measures achieved with face-to-face monitoring and supervision.

[0011] Therefore, the object of the present invention is to provide a work monitoring system that enables the implementation of safety management measures equivalent to those provided in person, even from a remote location, for workers performing tasks or operations on an information processing system.

[0012] The aforementioned and other objectives and novel features of the present invention will become apparent from the description herein and the accompanying drawings. [Means for solving the problem]

[0013] A brief overview of some of the representative inventions disclosed in this application is as follows:

[0014] A typical embodiment of the present invention is a work monitoring system for monitoring worker operations on an information processing system, comprising: a worker virtual terminal operated by the worker; a monitor virtual terminal used by a monitor; a worker head-mounted display connected to the worker virtual terminal and worn by the worker; a monitor head-mounted display connected to the monitor virtual terminal and worn by the monitor; and a monitoring unit that enables screen sharing between the worker virtual terminal and the monitor virtual terminal.

[0015] The worker virtual terminal projects the worker screen displayed on the worker virtual terminal into the virtual space of the worker head-mounted display, the monitoring unit shares the worker screen between the worker virtual terminal and the monitoring virtual terminal, the monitoring virtual terminal projects the shared worker screen into the virtual space of the monitoring head-mounted display, and, if predetermined conditions are met, permits the worker to perform predetermined tasks on the worker virtual terminal. [Effects of the Invention]

[0016] The effects obtained by some of the representative inventions disclosed in this application can be briefly explained as follows.

[0017] In other words, according to a typical embodiment of the present invention, it becomes possible to implement safety management measures equivalent to those taken in person, even from a remote location, for workers performing tasks or operations on an information processing system. [Brief explanation of the drawing]

[0018] [Figure 1] This figure outlines an example configuration of a work monitoring system, which is one embodiment of the present invention. [Figure 2] This diagram outlines an example of a workflow for monitoring work and operations, which is one embodiment of the present invention. [Modes for carrying out the invention]

[0019] Hereinafter, embodiments of the present invention will be described in detail based on the drawings. In all the drawings for explaining the embodiments, the same parts are basically denoted by the same reference numerals, and repeated explanations thereof are omitted. Further, hereinafter, for the sake of easy understanding of the features of the present invention, the description will be made in comparison with the prior art.

[0020] <Overview> A work monitoring system according to an embodiment of the present invention has a configuration in which a monitor remotely monitors and supervises an operator who performs work or operations on an information processing system. On the virtual space of the HMD worn by the operator, the screen of the information processing system operated by the operator is projected, and the screen is shared between the virtual space of the HMD worn by the monitor at a remote location.

[0021] Thereby, on the operator side, it is possible to prevent the inflow of information and the outflow of information due to peeking or the like, and on the monitor side, while preventing the outflow of information, it is possible to grasp the situation of the terminal operation by the operator in real time and perform monitoring and supervision at the same level as in face-to-face communication. In addition, since the operator can be made to perform work or operations within the same virtual space as the monitor, it is possible to make the operator perform work or operations with a sense of tension and concentration as if the monitor were nearby.

[0022] <System Configuration> FIG. 1 is a diagram schematically showing a configuration example of a work monitoring system according to an embodiment of the present invention. The work monitoring system 1 includes an operator HMD 21 which is an HMD worn on the head of an operator 2 in an operator environment, an external camera 24 which captures the situation of the work and actions of the operator 2 as a video, a monitor HMD 31 which is an HMD worn on the head of a monitor 3 in a monitor environment, an operator virtual terminal 42 and a monitor virtual terminal 43 which are respectively operated and used by the operator 2 and the monitor 3 in a management environment constructed on a cloud computing environment, and further includes components such as a monitoring unit 41, an HMD management unit 44, and an authentication unit 45 which are configured as software on a single or individual virtual server.

[0023] The worker HMD21 and the supervisor HMD31 are HMDs that provide the wearer with a worker virtual space 22 and a supervisor virtual space 32, respectively, using VR (Virtual Reality) and MR (Mixed Reality) technologies. For example, publicly available devices such as Microsoft HoloLens 2 (registered trademark) can be used as appropriate.

[0024] These devices connect to the worker virtual terminal 42 and the monitor virtual terminal 43 of the management environment, described later, via wireless communication such as Wi-Fi (registered trademark). They can display the screens shown on these virtual terminals in a virtual space, and users can perform tasks and operations on these virtual terminals using keyboards, mice, or software keyboards connected via Bluetooth (registered trademark), etc. In particular, worker 2 can access and perform tasks on systems and data such as the production system 5, which require a high level of security, through operations on the worker virtual terminal 42.

[0025] In this embodiment, the information processing terminals operated and used by worker 2 and supervisor 3 via worker HMD21 and supervisor HMD31 are not physical terminals, but virtual terminals in a management environment built using cloud computing services such as Microsoft Azure (registered trademark), namely worker virtual terminal 42 and supervisor virtual terminal 43. By configuring the system so that no physical terminals are placed in the worker environment or supervisor environment (for example, the homes of worker 2 and supervisor 3), it is possible to avoid situations such as the loss of these terminals or information leakage from the display screen, thereby reducing security risks. In this case, the worker HMD21 and supervisor HMD31 themselves do not hold business information, but operate as thin clients to the worker virtual terminal 42 and supervisor virtual terminal 43.

[0026] Furthermore, the worker environment may be configured to include an external camera 24, such as a webcam or a smartphone camera, which captures video of the worker's work, operations, appearance, and actions, and sends the data to the management environment described later via the worker virtual terminal 42 for recording. The supervisor 3 can check this video footage in real time on the supervisor virtual terminal 43 to confirm the worker's status, or they can later understand the worker's situation at the time of any incident.

[0027] The monitoring unit 41 of the management environment has communication functions that enable screen sharing and calls between the worker virtual terminal 42 and the monitoring virtual terminal 43, so that the monitor 3 can monitor and review the work and operations performed by worker 2 on the worker virtual terminal 42 via the monitor HMD 31 (monitoring virtual terminal 43). For example, this can be implemented using Jitsi®, an open-source online conferencing system. By sharing the screen between the worker virtual terminal 42 and the monitoring virtual terminal 43, screen sharing can also be performed between the worker HMD 21 and the monitor 31. In other words, the worker screen 23 that worker 2 is operating and displaying on the worker HMD 21 (worker virtual terminal 42) can be displayed and shared in the monitoring virtual space 32 of the monitor HMD 31.

[0028] The HMD management unit 44 has the function of protecting worker HMDs 21 and supervisor HMDs 31 by controlling them so that worker 2, supervisor 3, or others cannot install applications or enable vulnerable functions without authorization. For example, it can be configured using a device management tool such as Microsoft Intune®. The authentication unit 45 has the function of managing user information of worker 2 and supervisor 3 and performing authentication processing.

[0029] Operator 2 can access the production system 5 through operations on the operator virtual terminal 42, but access may be controlled in the monitoring unit 41 (or operator virtual terminal 42 or monitor virtual terminal 43) so that access is only possible when monitor 3 is able to monitor via monitor HMD 31 and has granted access. In addition, during monitoring, the contents of the operator screen 23 may be captured and recorded, and logs of operations performed by operator 2 may be recorded.

[0030] When supervisor 3 grants worker 2 access to the production system 5, conditions may include, for example, that the supervisor HMD 31 tracks the gaze of supervisor 3, who is wearing it, and that it is within a designated area (conversely, if it is outside the designated area for a certain period of time or longer, permission will be denied or revoked), and that screen sharing and communication by the monitoring unit 41 have been established, in which case supervisor 3 is ready to perform safety management measures such as monitoring and supervision. If the worker HMD 21 or supervisor HMD 31 has a function to detect when they are worn, the condition may also include that they are worn by worker 2 or supervisor 3. In addition, the condition may include that supervisor 3 visually confirms the status of worker 2 as captured by the external camera 24.

[0031] Worker 2 will wear the worker HMD 21 while working, and their field of view will be restricted to the worker virtual space 22, thus preventing the influx of information from the outside and any fraudulent activities using that information. In addition, supervisor 3 will also wear the supervisor HMD 31 to monitor, and since there are no physical terminals in either the worker environment or the supervisor environment, it is possible to prevent information leakage due to eavesdropping by third parties or loss of physical terminals. Furthermore, supervisor 3 can monitor and supervise worker 2's work and operations on the worker virtual terminal 41 in real time via screen sharing, enabling monitoring and supervision at a level equivalent to face-to-face interaction.

[0032] If supervisor 3 determines that worker 2 has committed fraudulent activity, for example, supervisor 3 can press a button (not shown) displayed on supervisor virtual space 32 to display a predetermined warning on worker virtual space 22 (worker screen 23). The monitoring unit 41 may also be configured to restrict operation, such as stopping, locking, terminating, or turning off access to the production system 5 (or operation of worker virtual terminal 42). The above warnings and operation restrictions may also be implemented not only in cases of fraudulent activity, but also in cases where normal monitoring and supervision from the monitoring environment becomes impossible, such as when worker 2 or supervisor 3 removes the worker HMD 21 or supervisor HMD 31, or when communication between worker virtual terminal 42 and supervisor virtual terminal 43 is interrupted.

[0033] Furthermore, if supervisor 3 determines that worker 2 has engaged in fraudulent activity, and restricts access to the production system 5 or the operation of the worker's virtual terminal 42, and then confirms that worker 2 has ceased such fraudulent activity, supervisor 3 may, for example, press a button (not shown) displayed on the supervisor's virtual space 32 to lift the restrictions on the worker's virtual terminal 42 and allow worker 2 to resume work.

[0034] <Processing flow> Figure 2 is a diagram illustrating an example of the workflow for monitoring work and operations, which is one embodiment of the present invention. Here, an example of the workflow for initiating monitoring and supervision by monitor 3 via monitor virtual terminal 43 in the monitor environment, and for accessing the production system 5 by worker 2 via worker virtual terminal 42 in the worker environment, is outlined.

[0035] In the worker environment, worker 2 activates the worker HMD 21, connecting to the HMD management unit 44 via the network, and the HMD management unit 44 begins controlling the worker HMD 21 (S01). Then, worker 2 puts on the worker HMD 21 (S02). At this time, the worker HMD 21 may have functions such as iris authentication to limit which worker 2 can wear the worker HMD 21. Subsequently, worker 2 performs user authentication via the authentication unit 45 by operating the worker HMD 21 in the worker virtual space 22 (S03).

[0036] Once user authentication is complete, the worker's HMD21 may switch the WiFi network connection to the management environment to a VPN (Virtual Private Network) connection (S04) to create a secure network. At this time, access to the VPN connection may be restricted to only worker HMD21s that have been previously registered with the HMD management unit 44, etc.

[0037] Once the VPN connection is established, worker 2 logs in to the worker virtual terminal 42 in the management environment by operating on the worker virtual space 22 on the worker HMD 21 (S05). At this time, for example, it may be possible to check whether worker 2 is located in a predetermined location (building, room, area, etc.) by checking whether the IP address assigned to worker HMD 21 is within a predetermined range (S06), and only if worker 2 is located in the predetermined location may login and connection to the worker virtual terminal 42 be permitted (or login may be permitted but access to the production system 5 may be denied).

[0038] On the other hand, in the monitoring environment, monitor 3 activates the monitoring terminal 31 and puts it on (S12). Similar to the worker terminal 21, the monitoring HMD 31 may be controlled by the HMD management unit 44. Subsequently, monitor 3 performs user authentication via the authentication unit 45 by operating the monitoring virtual space 32 on the monitoring HMD 31 (S13). Once user authentication is complete, the monitoring HMD 31 may switch the WiFi network connection to the management environment to a VPN connection (S14) to create a secure network. At this time, VPN connection may be restricted to only monitoring HMD 31s that have been previously registered in the HMD management unit 44, etc.

[0039] Once the VPN connection is established, monitor 3 logs in to the monitor virtual terminal 43 in the management environment by operating on the monitor virtual space 32 on the monitor HMD 31 (S15). Similar to the worker terminal 21, it may be possible to check whether monitor 3 is located at a predetermined location by checking whether the IP address assigned to monitor HMD 31 is within a predetermined range, and to allow login only if monitor 3 is located at the predetermined location.

[0040] Furthermore, the above series of processes in the worker environment and the monitor environment can be performed asynchronously and in parallel in their respective environments. Once the login from the worker HMD21 to the worker virtual terminal 42 and the login from the monitor HMD31 to the monitor virtual terminal 43 are complete, the desktop screen (or a specific window, etc.) of the worker virtual terminal 42 can be displayed in the worker virtual space 22 of the worker HMD21, and the desktop screen (or a specific window, etc.) of the monitor virtual terminal 43 can be displayed in the monitor virtual space 32 of the monitor HMD31.

[0041] In this state, screen sharing is performed between the worker virtual terminal 42 and the monitor virtual terminal 43 by the operation of worker 2 and monitor 3, or automatically by the functions of the monitoring unit 41 (S07, S17). That is, in this embodiment, the worker screen 23 on the worker virtual terminal 42, which is displayed in the worker virtual space 22, is also displayed in the monitor virtual space 32 to be shared. The functions of the monitoring unit 41 may also be used to establish a voice call session in addition to screen sharing between the worker virtual terminal 42 and the monitor virtual terminal 43.

[0042] Subsequently, the supervisor 3, in the supervisor virtual terminal 43 (supervisor virtual space 32), refers to the video footage captured by the external camera 24 in the worker environment (for example, transmitted and recorded from the external camera 24 via the network to the worker virtual terminal 42, etc.) to confirm the status of worker 2, such as whether the person present is worker 2 himself or whether there are other people present (S18). If there are no problems with worker 2's status, the supervisor 3 grants worker 2 (worker virtual terminal 42) permission to work by performing a predetermined operation on the supervisor virtual space 32 of the supervisor HMD 31 (S19). As a result, worker 2 can access the production system 5 by launching a predetermined application on the worker virtual terminal 42, etc. (S20).

[0043] As mentioned above, when supervisor 3 grants worker 2 access to the production system 5, it may also be possible to set conditions such as the supervisor HMD 31 tracking the gaze of supervisor 3 and ensuring that the gaze is within a predetermined area (conversely, if it remains outside the predetermined area for a certain period of time or longer, permission will be denied or revoked), or that screen sharing and voice communication by the monitoring unit 41 have been established, in which case supervisor 3 may be prepared to perform safety management measures such as monitoring and supervision.

[0044] From this point onward, worker 2 can access the production system 5 by operating the worker virtual terminal 42 in the worker virtual space 22 and perform the desired tasks. Meanwhile, supervisor 3 can monitor and supervise in real time whether worker 2 is leaking information or engaging in any other fraudulent activities by viewing the worker screen 23 shared in the supervisor virtual space 32 on the supervisor HMD 31.

[0045] As described above, according to one embodiment of the present invention, the work monitoring system prevents information from flowing in or leaking due to eavesdropping by third parties on the worker 2 side, while also preventing information leakage on the supervisor 3 side, allowing for real-time monitoring and supervision of the worker's terminal operations, equivalent to face-to-face monitoring. Furthermore, since the worker 2 can operate the terminal in the same virtual space as the supervisor 3, it is possible to make the worker 2 operate the terminal with a sense of tension and concentration as if the supervisor 3 were nearby.

[0046] The present invention has been described in detail above based on embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from its essence.

[0047] For example, in the above embodiment, a 1:1 configuration was described in which one supervisor 3 monitors and supervises one worker 2. However, it is also possible to have a 1:N configuration in which one supervisor 3 monitors and supervises multiple workers 2 simultaneously. In this case, for example, the worker screens 23 of the N workers 2 may be displayed as tiles on the supervisor virtual space 32 of the supervisor HMD 31.

[0048] Furthermore, the control system may be configured to change depending on whether worker 2 or supervisor 3 is located in a designated location, such as a dedicated workspace within the company. For example, worker 2 or supervisor 3 who is located in the designated location may perform work or monitoring without wearing the worker HMD 21 or supervisor HMD 31. That is, if worker 2 is located in the designated location and supervisor 3 is not, only supervisor 3 needs to wear the supervisor HMD 31 to monitor. Conversely, if worker 2 is not located in the designated location and supervisor 3 is, only worker 2 needs to wear the worker HMD 21 to perform work. In this case, the person not wearing the HMD accesses a virtual terminal using, for example, a PC.

[0049] Furthermore, in the above embodiment, the display screen on the worker virtual terminal 42 or worker HMD 21 can be captured and recorded, and operation logs can be recorded. However, if the supervisor 3 restricts operation due to fraudulent activity by worker 2, these recorded data and log data may be moved to a separate area and saved. Conversely, the system may be configured to delete this data as appropriate when there are no operation restrictions and the system is functioning normally.

[0050] Furthermore, although the above embodiment was described as a configuration in which the supervisor 3 visually monitors all operations and tasks of the worker 2 in the supervisor's virtual space 32, the configuration is not limited to this, and it is also possible to combine it with, for example, a mechanism for unmanned and automated monitoring as in conventional technology, so that the two complement each other.

[0051] The above embodiments are described in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those comprising all the configurations described. Furthermore, it is possible to add, delete, or replace some of the configurations in the above embodiments with other configurations.

[0052] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. Alternatively, each of the above configurations, functions, and means may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs, or other recording devices, or in recording media such as IC cards, SD cards, or DVDs.

[0053] Furthermore, in the diagrams above, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines that would be present in the actual implementation. In reality, it can be assumed that almost all components are interconnected. [Industrial applicability]

[0054] This invention can be used in a work monitoring system that monitors operations and actions performed by workers on an information processing system. [Explanation of Symbols]

[0055] 1...Work monitoring system, 2...Worker, 3...Supervisor, 5...Production system 21...Worker HMD, 22...Worker virtual space, 23...Worker screen, 24...External camera, 31...Observer HMD, 32...Observer virtual space, 41...Monitoring Department, 42...Worker Virtual Terminal, 43...Monitoring Virtual Terminal, 44...HMD Management Department, 45...Authentication Department

Claims

1. A work monitoring system that monitors the operations of an operator on an information processing system, A virtual terminal operated by the worker, The virtual terminal used by the monitor, A worker head-mounted display, which is connected to the aforementioned worker virtual terminal and worn by the worker, A monitor head-mounted display worn by the monitor is connected to the aforementioned monitor virtual terminal, It includes a monitoring unit that enables screen sharing between the operator virtual terminal and the monitor virtual terminal, The aforementioned worker virtual terminal projects the worker screen displayed on the worker virtual terminal into the virtual space of the worker head-mounted display. The monitoring unit shares the worker's screen between the worker's virtual terminal and the monitoring unit's virtual terminal, and enables communication between the worker's virtual terminal and the monitoring unit's virtual terminal. A work monitoring system in which the virtual monitor terminal projects the shared worker screen into the virtual space of the monitor head-mounted display, and permits the worker to perform a predetermined task on the virtual monitor terminal when the conditions are met, including that the monitor head-mounted display is worn by the monitor, screen sharing and communication of the worker screen by the monitoring unit are established, and the monitor's line of sight detected by the monitor head-mounted display is within a predetermined area corresponding to the worker screen projected into the virtual space of the monitor head-mounted display.

2. In the work monitoring system described in claim 1, A work monitoring system that permits connection to the worker virtual terminal only if the IP address assigned to the worker head-mounted display falls within a predetermined range.

3. In the work monitoring system described in claim 1, further, The system has a camera that photographs the worker's condition, A work monitoring system in which the video of the worker captured by the camera can be viewed by the monitor via the monitor's virtual terminal.

Images