Work monitoring system

Abstract

To enable safety management measures at the same level as those in a face-to-face manner to be taken for a worker who carries out a task and an operation on an information processing system, even from a remote site.SOLUTION: A work monitoring system includes: a worker virtual terminal 42; a monitoring person virtual terminal 43; a worker HMD 21 which is connected to the worker virtual terminal 42 and which is worn by a worker 2; a monitoring person HMD 31 which is connected to the monitoring person virtual terminal 43 and which is worn by a monitoring person 3; and a monitoring unit 41 that enables the worker virtual terminal 42 and the monitoring person virtual terminal 43 to share a screen therebetween. The worker virtual terminal 42 projects, on the virtual space of the worker HMD 21, a worker screen 23 displayed on the worker virtual terminal 42. The monitoring unit 41 shares the worker screen 23 between the worker virtual terminal 42 and the monitoring person virtual terminal 43. The monitoring person virtual terminal 43 projects the shared worker screen 23 on the virtual space of the monitoring person HMD 31, and permits the worker 2 to carry out a predetermined work on the worker virtual terminal 42 when a predetermined condition is satisfied.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a security technology for an information processing system, and more particularly to a technology that is effective when applied to a work monitoring system that monitors the operations and actions of workers on the information processing system. [Background technology]

[0002] There are cases where the operation and work of information processing terminals such as personal computers (PCs) by workers must be carried out under security controls that prohibit or strictly restrict the leakage or inflow of information and other unauthorized acts. For example, this may apply to cases where a worker accesses an information processing system in production from a PC to perform work (production work) such as program release, operational work, or emergency response, or to work in an environment where important personal information such as My Number can be accessed, such as operator work at a call center, or to entrance exams or qualification exams where questions are presented and answers are entered using a PC.

[0003] When workers operate information processing terminals in such an environment, safety management measures must be taken to prevent unauthorized actions. In the past, this was done by a supervisor (a person in charge of monitoring and supervision) sitting nearby or making rounds to directly monitor and supervise one or more workers. In response to this, technologies have been developed to reduce the burden of monitoring and supervision and to automate the process.

[0004] For example, Patent Publication No. 6910674 (Patent Document 1) describes an online examination system that uses a video camera to monitor the behavior of examinees, in which fluorescently colored cheating monitoring components are attached to specific parts of the examinee's body, and the movement of these components is monitored using images from the video camera, thereby preventing examinee cheating from being overlooked due to blind spots in the video camera, etc.

[0005] Furthermore, Japanese Patent Publication No. 2016-177475 (Patent Document 2) describes a monitoring system that detects the gaze position of each examinee by having them wear a device such as an eyeglass-type terminal that can detect pupil position, and determines whether or not cheating has occurred based on whether the gaze position is outside a specified range on the question paper, answer sheet, etc. for a certain period of time.

[0006] Furthermore, Japanese Patent Application Laid-Open Publication No. 2009-237801 (Patent Document 3) describes how by displaying questions sent from a central device on a terminal device such as a head mounted display (HMD) worn on the examinee's head, only the wearer can see the questions, and the wearer is prevented from seeing any information other than that displayed on the terminal device, thereby preventing the occurrence of cheating, and also eliminating the need to install equipment such as surveillance cameras or deploy test supervisors. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6910674 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-177475 [Patent Document 3] Japanese Patent Application Laid-Open No. 2009-237801 Summary of the Invention [Problem to be solved by the invention]

[0008] In recent years, with the spread of remote work / telework, there are an increasing number of situations where remote monitoring and supervision of terminal operations, such as actual work that requires safety management measures and tests using PCs, is required.

[0009] In contrast, with the conventional technologies described above, for example, it is possible to unmann and automate the monitoring and supervision functions, thereby eliminating the need for a supervisor to monitor and supervise from a remote location. However, while these unmanned and automated systems can reduce the workload of monitoring and supervision, it is difficult to implement the same level of physical and technical safety control measures as in the case of traditional face-to-face monitoring and supervision, and the risk of information leaks, inflows, and other fraudulent activities remains.

[0010] For example, the technology described in Patent Document 3 makes it possible to prevent the inflow of information, but it is not possible to grasp what kind of terminal operations the worker is performing, and it is not possible to effectively prevent the leakage of information or other fraudulent acts. Also, the technology described in Patent Documents 1 and 2 makes it possible to grasp the actions of the worker to a certain extent, but it is not possible to grasp with high accuracy what kind of terminal operations the worker is actually performing, and it is difficult to say that these are safety management measures at the same level as face-to-face monitoring and supervision.

[0011] Therefore, an object of the present invention is to provide a work monitoring system that can take safety management measures for workers who perform work or operations on an information processing system from a remote location at the same level as in person.

[0012] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0013] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0014] A work monitoring system according to a representative embodiment of the present invention is a work monitoring system that monitors the operations of workers on an information processing system, and includes a worker virtual terminal operated by the worker, a supervisor virtual terminal used by a supervisor, a worker head-mounted display connected to the worker virtual terminal and worn by the worker, a supervisor head-mounted display connected to the supervisor virtual terminal and worn by the supervisor, and a monitoring unit that enables screen sharing between the worker virtual terminal and the supervisor virtual terminal.

[0015] The worker virtual terminal projects a worker screen displayed on the worker virtual terminal into the virtual space of the worker head-mounted display, the monitoring unit shares the worker screen between the worker virtual terminal and the monitor virtual terminal, and the monitor virtual terminal projects the shared worker screen into the virtual space of the monitor head-mounted display and allows the worker to perform a specified task on the worker virtual terminal if specified conditions are met. [Effects of the Invention]

[0016] The effects obtained by the representative inventions disclosed in this application will be briefly explained as follows.

[0017] That is, according to the representative embodiment of the present invention, it is possible to take safety management measures for workers who perform tasks or operations on an information processing system from a remote location at the same level as in person. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a diagram showing an overview of an example of the configuration of a work monitoring system according to an embodiment of the present invention; [Figure 2] FIG. 1 is a diagram showing an outline of an example of a flow of a work or operation monitoring process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all drawings for explaining the embodiments, the same parts are generally designated by the same reference numerals, and repeated explanations thereof will be omitted. In addition, in the following, the features of the present invention will be described in comparison with the prior art in order to make them easier to understand.

[0020] <Summary> A work monitoring system that is one embodiment of the present invention is configured so that a monitor from a remote location actually monitors and supervises workers who perform work or operations on an information processing system.The screen of the information processing system operated by the worker is projected into a virtual space on an HMD worn by the worker, and the screen is shared with the virtual space on an HMD worn by the monitor in a remote location.

[0021] This prevents information from leaking from the worker's side due to inflow or peeping, while also preventing information from leaking from the supervisor's side, allowing them to grasp the status of the worker's terminal operation in real time and perform monitoring and supervision at the same level as face-to-face.In addition, because the worker can perform tasks and operations in the same virtual space as the supervisor, it is possible for the worker to perform tasks and operations with the same tension and concentration as if the supervisor were nearby.

[0022] <System configuration> 1 is a diagram showing an overview of an example configuration of a work monitoring system according to one embodiment of the present invention. The work monitoring system 1 includes a worker HMD 21, which is an HMD worn on the head of a worker 2 in the worker environment, an external camera 24 that captures video of the work and behavior of the worker 2, a monitor HMD 31, which is an HMD worn on the head of a monitor 3 in the monitor environment, a worker virtual terminal 42 and a monitor virtual terminal 43 operated and used by the worker 2 and the monitor 3, respectively, in a management environment built on a cloud computing environment, and further includes a monitoring unit 41, an HMD management unit 44, and an authentication unit 45, which are configured as software on individual or separate virtual servers.

[0023] The worker HMD 21 and the observer HMD 31 are HMDs that provide the wearer with a worker virtual space 22 and an observer virtual space 32, respectively, using VR (Virtual Reality) and MR (Mixed Reality) technologies, and can be made from commonly available HMDs such as Microsoft HoloLens 2 (registered trademark) as appropriate.

[0024] These are connected to a worker virtual terminal 42 and a supervisor virtual terminal 43 in the managed environment (described later) via wireless communication such as Wi-Fi (registered trademark), and the screens displayed on these virtual terminals can be displayed in the virtual space, and work and operations can be performed on these virtual terminals using a keyboard and mouse connected via Bluetooth (registered trademark) or a software keyboard. In particular, worker 2 can access systems and data such as production system 5, which require high security, through operations on worker virtual terminal 42 to perform work.

[0025] In this embodiment, the information processing terminals operated and used by the worker 2 and the monitor 3 via the worker HMD 21 and the monitor HMD 31 are not physical terminals, but rather worker virtual terminal 42 and monitor virtual terminal 43, which are virtual terminals in a management environment established using a cloud computing service such as Microsoft Azure (registered trademark). In this way, by configuring the environment without physical terminals in the worker environment or the monitor environment (which may be, for example, the home of worker 2 or monitor 3), it is possible to avoid situations such as loss of these terminals or information leakage from the display screen, thereby reducing security risks. In this case, the worker HMD 21 and the monitor HMD 31 themselves do not store business information, and operate like thin clients to the worker virtual terminal 42 and the monitor virtual terminal 43.

[0026] An external camera 24, such as a webcam or a smartphone camera, may be installed in the worker environment to capture video of the work, operations, appearance, behavior, and other conditions of worker 2, and send the data to a management environment (described below) via worker virtual terminal 42 for recording. The monitor 3 may check the video image on monitor virtual terminal 43 in real time to confirm the condition of worker 2, or, if some kind of event occurs, the condition of worker 2 at the time may be grasped later.

[0027] The monitoring unit 41 of the managed environment has a communication function that enables screen sharing and calls between the worker virtual terminal 42 and the monitor virtual terminal 43 so that the monitor 3 can monitor and review the work and operations of the worker 2 on the worker virtual terminal 42 via the monitor HMD 31 (monitor virtual terminal 43). This function can be implemented, for example, using Jitsi (registered trademark), an online conference system based on open source software. Screen sharing between the worker virtual terminal 42 and the monitor virtual terminal 43 also enables screen sharing between the worker HMD 21 and the monitor 31. In other words, the worker screen 23 that the worker 2 is operating and displaying on the worker HMD 21 (worker virtual terminal 42) can be displayed and shared in the monitor virtual space 32 of the monitor HMD 31.

[0028] The HMD management unit 44 has a function to protect the worker HMD 21 and the monitor HMD 31 by controlling them so that the worker 2, the monitor 3, and others cannot install applications or enable functions with vulnerabilities without permission. For example, this can be configured using a device management tool such as Microsoft Intune (registered trademark). The authentication unit 45 has a function to manage user information of the worker 2 and the monitor 3 and perform authentication processing.

[0029] The worker 2 can access the production system 5 through operations on the worker virtual terminal 42, but the monitoring unit 41 (or the worker virtual terminal 42 or the monitor virtual terminal 43) may control access so that access is only possible when the monitor 3 is in a state where monitoring is possible via the monitor HMD 31 and has permitted access. Furthermore, during monitoring, the contents displayed on the worker screen 23 may be captured and recorded, and a log of operations by the worker 2 may be recorded.

[0030] When the monitor 3 permits the worker 2 to access the production system 5, the monitor 3 may be required to ensure that the monitor 3 is ready to take safety control measures such as monitoring and supervision, for example, by tracking the line of sight of the monitor 3 wearing the monitor HMD 31 and ensuring that the monitor 3 is within a predetermined area (conversely, if the monitor 3 is outside the predetermined area for a certain period of time or more, the access is denied or cancelled), or by ensuring that screen sharing and communication are established via the monitoring unit 41. If the worker HMD 21 or the monitor HMD 31 has a function that can detect that the worker 2 is wearing the monitor HMD 31, the condition may include that the monitor 3 is wearing the monitor HMD 31. Another condition may include that the monitor 3 visually confirms the status of the worker 2 captured by the external camera 24.

[0031] Worker 2 wears worker HMD 21 while working, and his field of vision is limited to worker virtual space 22, preventing the inflow of information from outside and any fraudulent activity that could be exploited. Furthermore, supervisor 3 also wears supervisor HMD 31 to monitor, and since neither the worker nor the supervisor environment has a physical terminal, it is possible to prevent information leakage due to third parties peeking or loss of the physical terminal. Furthermore, supervisor 3 can grasp the status of the work and operations being performed by worker 2 on worker virtual terminal 41 in real time via screen sharing, enabling monitoring and supervision at the same level as face-to-face supervision.

[0032] If the supervisor 3 determines that the worker 2 has engaged in fraudulent activity, for example, the supervisor 3 can display a predetermined warning in the worker virtual space 22 (worker screen 23) by, for example, pressing a button (not shown) displayed in the supervisor virtual space 32. The monitoring unit 41 may use its functions to restrict operation, such as stopping, locking, terminating, or turning off the screen of access to the production system 5 (or the operation of the worker virtual terminal 42). The above-described warnings and operation restrictions may be implemented not only when fraudulent activity has occurred, but also when, for example, the worker 2 or the supervisor 3 removes the worker HMD 21 or the supervisor HMD 31, or communication between the worker virtual terminal 42 and the supervisor virtual terminal 43 is cut off, making normal monitoring and supervision from the supervisor environment impossible.

[0033] Furthermore, if the monitor 3 determines that the worker 2 has engaged in fraudulent activity, etc., and restricts access to the production system 5 and the operation of the worker virtual terminal 42, and then confirms that the worker 2 has ceased to engage in fraudulent activity, the monitor 3 may, for example, press a button (not shown) displayed on the monitor virtual space 32 to lift the operation restriction on the worker virtual terminal 42 and allow access to the production system 5 again so that the worker 2 can resume work.

[0034] <Processing flow> 2 is a diagram outlining an example of the flow of work and operation monitoring processing according to one embodiment of the present invention. This diagram outlines an example of the flow of processing in which a monitor 3 in a monitor environment starts monitoring and supervision via a monitor virtual terminal 43, and a worker 2 in a worker environment accesses the production system 5 via a worker virtual terminal 42.

[0035] In the worker environment, the worker 2 starts up the worker HMD 21, thereby connecting to the HMD management unit 44 via the network, and the HMD management unit 44 starts controlling the worker HMD 21 (S01). Then, the worker 2 wears the worker HMD 21 (S02). At this time, a function such as iris authentication provided in the worker HMD 21 may be used to limit the workers 2 who can wear the worker HMD 21 to specific individuals. Thereafter, the worker 2 performs user authentication via the authentication unit 45 by operating the worker HMD 21 in the worker virtual space 22 (S03).

[0036] When user authentication is complete, the worker HMD 21 may switch the WiFi network connection with the management environment to a VPN (Virtual Private Network) connection (S04) to create a secure network. At this time, the worker HMD 21 may be restricted so that only worker HMDs 21 that have been registered in advance in the HMD management unit 44 or the like are able to connect to the VPN.

[0037] Once the VPN connection is established, the worker 2 logs in to the worker virtual terminal 42 of the managed environment by operating the worker virtual space 22 of the worker HMD 21 (S05). At this time, for example, it may be confirmed whether the worker 2 is located in a predetermined location (building, room, area, etc.) by checking whether the IP address assigned to the worker HMD 21 is within a predetermined range (S06), and login and connection to the worker virtual terminal 42 may be permitted only if the worker 2 is located in the predetermined location (alternatively, login may be permitted but access to the production system 5 may be prohibited).

[0038] Meanwhile, in the monitor environment, the monitor 3 also starts up the monitor terminal 31 and wears it (S12). As with the worker terminal 21, the monitor HMD 31 may be controlled by the HMD management unit 44. Thereafter, the monitor 3 performs user authentication via the authentication unit 45 by operating the monitor HMD 31 in the monitor virtual space 32 (S13). Once user authentication is complete, the monitor HMD 31 may switch the WiFi network connection with the managed environment to a VPN connection (S14) to create a secure network. At this time, the VPN connection may be restricted to only monitor HMDs 31 that have been registered in advance in the HMD management unit 44, etc.

[0039] Once the VPN connection is established, the observer 3 logs in to the observer virtual terminal 43 of the managed environment by operating the observer virtual space 32 of the observer HMD 31 (S15). As with the worker terminal 21, it may be possible to check whether the observer 3 is located at a predetermined location by checking whether the IP address assigned to the observer HMD 31 is within a predetermined range, and to permit login only if the observer 3 is located at the predetermined location.

[0040] The above series of processes in the worker environment and the observer environment can be performed asynchronously in parallel in each environment. When login to the worker virtual terminal 42 from the worker HMD 21 and login to the observer virtual terminal 43 from the observer HMD 31 are completed, the desktop screen (or a specific window, etc.) of the worker virtual terminal 42 can be displayed in the worker virtual space 22 of the worker HMD 21, and the desktop screen (or a specific window, etc.) of the observer virtual terminal 43 can be displayed in the observer virtual space 32 of the observer HMD 31.

[0041] In this state, screen sharing is performed between the worker virtual terminal 42 and the monitor virtual terminal 43 by the function of the monitoring unit 41, either automatically or by operation of the worker 2 and the monitor 3 (S07, S17). That is, in this embodiment, the worker screen 23 on the worker virtual terminal 42 displayed in the worker virtual space 22 is also displayed in the monitor virtual space 32, thereby being shared. The function of the monitoring unit 41 may be configured to establish not only screen sharing but also a voice call session between the worker virtual terminal 42 and the monitor virtual terminal 43.

[0042] Thereafter, the monitor 3, on the monitor virtual terminal 43 (monitor virtual space 32), refers to the video captured by the external camera 24 in the worker's environment (for example, transmitted and recorded from the external camera 24 to the worker virtual terminal 42, etc. via a network) to check the status of the worker 2, such as whether the person present there is the worker 2 himself, and whether there are any other people present (S18). If there are no problems with the status of the worker 2, the monitor 3 gives permission for the worker 2 (worker virtual terminal 42) to work by performing a predetermined operation in the monitor virtual space 32 of the monitor HMD 31 (S19). This allows the worker 2 to access the production system 5 by, for example, starting a predetermined application on the worker virtual terminal 42 (S20).

[0043] As mentioned above, when the supervisor 3 permits the worker 2 to access the production system 5, the supervisor 3 may, for example, use the supervisor HMD 31 to track the line of sight of the wearing supervisor 3 and ensure that the line of sight is within a predetermined area (conversely, if the line of sight is outside the predetermined area for a certain period of time or more, the access will be denied or cancelled), or the monitoring unit 41 has established screen sharing and voice communication, etc., and other conditions may be met to ensure that the supervisor 3 is ready to take safety control measures such as monitoring and supervision.

[0044] Thereafter, the worker 2 can access the production system 5 and perform the desired work by operating the worker virtual terminal 42 in the worker virtual space 22. Meanwhile, the monitor 3 can monitor and supervise in real time whether or not the worker 2 is leaking information or engaging in other fraudulent activities, while looking at the worker screen 23 shared in the monitor virtual space 32 of the monitor HMD 31.

[0045] As described above, the work monitoring system according to one embodiment of the present invention prevents the inflow of information on the worker 2 side and the outflow of information due to peeping by a third party, and also enables the monitor 3 side to grasp the status of the worker's terminal operation in real time while preventing the outflow of information, thereby enabling monitoring and supervision at the same level as face-to-face contact. Furthermore, since the worker 2 can operate the terminal in the same virtual space as the monitor 3, the worker 2 can operate the terminal with the same sense of tension and concentration as if the monitor 3 were nearby.

[0046] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention.

[0047] For example, in the above embodiment, a 1:1 configuration has been described in which one supervisor 3 monitors and supervises one worker 2, but a 1:N configuration is also possible in which one supervisor 3 simultaneously monitors and supervises multiple workers 2. In this case, for example, the worker screens 23 of the N workers 2 may be displayed in tiles on the monitor virtual space 32 of the monitor HMD 31.

[0048] Furthermore, the control may be changed depending on whether the worker 2 or the monitor 3 is located in a predetermined location, such as a dedicated workroom within the company. For example, the worker 2 or the monitor 3 who is located in the predetermined location may perform work or monitoring without wearing the worker HMD 21 or the monitor HMD 31. That is, when the worker 2 is located in the predetermined location and the monitor 3 is not, only the monitor 3 may wear the monitor HMD 31 to monitor. Conversely, when the worker 2 is not located in the predetermined location and the monitor 3 is located in the predetermined location, only the worker 2 may wear the worker HMD 21 to perform work. In this case, the person not wearing the HMD may access a virtual terminal using, for example, a PC or the like.

[0049] In the above embodiment, the display screen of the worker virtual terminal 42 or the worker HMD 21 can be captured and recorded, and an operation log can be recorded, but if the supervisor 3 imposes an operation restriction on the worker 2 due to misconduct or the like by the worker 2, the recorded data and log data may be saved in a separate area. Conversely, the data may be deleted as appropriate when there are no operation restrictions and the system is in a normal state.

[0050] Furthermore, in the above embodiment, the configuration has been described in which the supervisor 3 visually monitors all of the operations and work of the worker 2 in the supervisor virtual space 32, but this is not limited to such a configuration, and it is also possible to combine it with an unmanned / automated monitoring mechanism as in conventional technology, so that the two complement each other.

[0051] The above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, some of the configurations of the above-described embodiments may be added to, deleted from, or replaced with other configurations.

[0052] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a storage device such as a memory, hard disk, or SSD, or in a storage medium such as an IC card, SD card, or DVD.

[0053] In addition, in the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. In reality, it can be assumed that almost all components are interconnected. [Industrial Applicability]

[0054] The present invention can be used in a work monitoring system that monitors operations and actions of workers on an information processing system. [Explanation of symbols]

[0055] 1...Work monitoring system, 2...Worker, 3...Monitor, 5...Production system, 21... Worker HMD, 22... Worker virtual space, 23... Worker screen, 24... External camera, 31...Observer HMD, 32...Observer virtual space, 41...Monitoring unit, 42...Worker virtual terminal, 43...Observer virtual terminal, 44...HMD management unit, 45...Authentication unit

Claims

1. A work monitoring system that monitors an operation of a worker on an information processing system, a worker virtual terminal operated by a worker; a monitor virtual terminal used by the monitor; a worker head mounted display connected to the worker virtual terminal and worn by the worker; an observer head mounted display connected to the observer virtual terminal and worn by the observer; a monitoring unit that enables screen sharing between the worker virtual terminal and the monitor virtual terminal, the worker virtual terminal projects a worker screen displayed on the worker virtual terminal into a virtual space of the worker head mounted display; the monitoring unit shares the worker screen between the worker virtual terminal and the monitor virtual terminal; A work monitoring system in which the monitor virtual terminal projects the shared worker screen into the virtual space of the monitor head-mounted display, and allows the worker to perform specified work on the worker virtual terminal if specified conditions are met.

2. The work monitoring system according to claim 1, A work monitoring system that allows connection to the worker virtual terminal only when the IP address assigned to the worker head mounted display is within a predetermined range.

3. The work monitoring system according to claim 1, further comprising: a camera for capturing images of the worker's condition; A work monitoring system in which the video of the worker captured by the camera can be viewed by the monitor via the monitor virtual terminal.

4. The work monitoring system according to claim 1, the observer head mounted display tracks the line of sight of the observer wearing the observer head mounted display, A work monitoring system in which the monitor virtual terminal does not permit the worker to perform the specified work if the monitor's line of sight is outside a specified range for more than a certain period of time.

Images