Work record verification device, work management system, work record verification method, and program

Abstract

To provide a work record verification device capable of accurately verifying a work record of a person to be managed.SOLUTION: The work record verification device comprises: an image acquisition unit which acquires frames constituting moving image data in which a management object floor where a person to be managed is working is captured; an identification unit which identifies the person to be managed who corresponds to a person included in a partial image extracted from the acquired frames and a behavior of the person included in the partial image; a work data acquisition unit which acquires work data including a working start time and a working end time declared by the identified person to be managed; a verification unit which verifies a working state of the person to be managed in accordance with difference between actual work hours specified in accordance with the identified behavior of the person to be managed and declared work hours specified by the declared work data and generates verification information including a verification result relating to the working state of the person to be managed; and an output unit which outputs the generated verification information including the verification result relating to the working state of the person to be managed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a work performance verification device, a work management system, a work performance verification method, and a program. [Background technology]

[0002] As working styles have diversified due to the introduction of flextime systems and other factors, it has become more difficult to accurately manage overtime and long working hours. For example, under a flextime system, workers are free to decide their own start and finish times and break times within a predetermined range of total working hours. When managing work hours under a working style such as a flextime system, working hours are sometimes recorded through self-reporting. In this type of work management, for example, hours actually worked may be recorded as break time. On the other hand, hours not actually worked may be recorded as working hours. As such, it is necessary to manage workers' working styles based not only on the reports of the workers who are the subject of management, but also on their actual behavior.

[0003] Patent Document 1 discloses a determination system aimed at accurately grasping working hours. The system of Patent Document 1 detects operations on an information processing device used by a user for work. The system of Patent Document 1 extracts, from the detected operations, operations from outside via an input unit, types of operations corresponding to a preset user. The system of Patent Document 1 acquires the extracted external operations, types of operations corresponding to a preset user, at predetermined time intervals. When a no-operation time indicating that no operation has been performed within a predetermined time is set among multiple operations, the system of Patent Document 1 calculates the overtime hours of the user of the information processing device by using the time corresponding to the last no-operation time as the end of work. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-031874 Summary of the Invention [Problem to be solved by the invention]

[0005] The method of Patent Document 1 manages a user's working hours based on the user's operations via an information processing device. Therefore, the method of Patent Document 1 cannot manage a user's working hours in a situation where no operations are performed via the information processing device. In other words, the method of Patent Document 1 cannot accurately trace the actions of workers performing their work, and therefore cannot accurately verify the worker's working performance.

[0006] An object of the present disclosure is to provide a work performance verification device, a work management system, a work performance verification method, and a program that can accurately verify the work performance of a person to be managed. [Means for solving the problem]

[0007] A work performance verification device according to one aspect of the present disclosure includes an image acquisition unit that acquires frames constituting video data captured on a managed floor where a managed person works; an identification unit that identifies the managed person corresponding to a person included in a partial image extracted from the acquired frames and the behavior of the person included in the partial image; a work data acquisition unit that acquires work data including start and end times declared by the identified managed person; a verification unit that verifies the work status of the managed person based on the difference between the actual work hours determined in accordance with the behavior of the identified managed person and the declared work hours determined by the declared work data, and generates verification information including the verification results regarding the work status of the managed person; and an output unit that outputs the generated verification information including the verification results regarding the work status of the managed person.

[0008] In one aspect of the work performance verification method of the present disclosure, a computer acquires frames constituting video data captured on a managed floor where a managed person works, identifies the managed person corresponding to a person included in a partial image extracted from the acquired frames and the behavior of the person included in the partial image, acquires work data including the start and end times declared by the identified managed person, verifies the work status of the managed person based on the difference between the actual work hours determined in accordance with the behavior of the identified managed person and the declared work hours determined by the declared work data, generates verification information including the verification results regarding the work status of the managed person, and outputs the generated verification information including the verification results regarding the work status of the managed person.

[0009] A program of one embodiment of the present disclosure causes a computer to perform the following processes: acquiring frames constituting video data captured on a managed floor where a managed person works; identifying the managed person corresponding to a person included in a partial image extracted from the acquired frames and the behavior of the person included in the partial image; acquiring work data including start and end times declared by the identified managed person; verifying the work status of the managed person during the work hours specified by the declared work data in accordance with the behavior of the identified managed person, and generating verification information including the verification results regarding the work status of the managed person; and outputting the generated verification information including the verification results regarding the work status of the managed person. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide a work performance verification device, a work management system, a work performance verification method, and a program that can accurately verify the work performance of a person to be managed. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a conceptual diagram illustrating an example of an installation environment of a work management system according to the present disclosure. [Figure 2] FIG. 2 is a conceptual diagram illustrating an example of a detailed configuration of a work management device according to the present disclosure. [Figure 3] 1 is a table illustrating an example of a management table in which declaration data according to the present disclosure is compiled. [Figure 4] 1 is a conceptual diagram illustrating an example of a detailed configuration of a work performance verification device according to the present disclosure. [Figure 5] 1 is a conceptual diagram showing a bird's-eye view of a floor to be managed from the viewpoint of a surveillance camera in the present disclosure. [Figure 6] 1 is a conceptual diagram illustrating an example of frames constituting video data captured by a surveillance camera according to the present disclosure. [Figure 7] 10 is a conceptual diagram illustrating an example of a partial image extracted by the work performance verification device according to the present disclosure. [Figure 8] 10 illustrates an example of a face recognition log summarizing the results of face recognition performed by the work performance verification device according to the present disclosure. [Figure 9] 10 illustrates an example of a behavior detection log that summarizes the results of behavior detection by the work performance verification device according to the present disclosure. [Figure 10] 10 illustrates an example of a verification log summarizing verification results obtained by the work performance verification device according to the present disclosure. [Figure 11] 10 is a conceptual diagram illustrating an example of a partial image extracted by the work performance verification device according to the present disclosure. [Figure 12] 10 illustrates an example of a verification log summarizing verification results obtained by the work performance verification device according to the present disclosure. [Figure 13] 10 is a conceptual diagram showing an example in which a verification result by the work performance verification device according to the present disclosure is displayed on the screen of a terminal device used by a manager who manages managed persons. FIG. [Figure 14] 10 is a conceptual diagram showing an example in which verification information related to the verification result by the work performance verification device in the present disclosure is displayed on the screen of a terminal device used by a manager who manages managed persons. FIG. [Figure 15] 10 is a conceptual diagram showing an example in which an evidence image of a verification result by a work performance verification device according to the present disclosure is displayed on the screen of a terminal device used by a manager who manages managed persons. FIG. [Figure 16]10 is a conceptual diagram illustrating an example of a partial image extracted by the work performance verification device according to the present disclosure. [Figure 17] 10 illustrates an example of a verification log summarizing verification results obtained by the work performance verification device according to the present disclosure. [Figure 18] 10 is a conceptual diagram showing an example in which verification information related to the verification result by the work performance verification device in the present disclosure is displayed on the screen of a terminal device used by an administrator who manages managed persons. FIG. [Figure 19] 10 is a conceptual diagram illustrating an example of a partial image extracted by the work performance verification device according to the present disclosure. [Figure 20] 10 illustrates an example of a verification log summarizing verification results obtained by the work performance verification device according to the present disclosure. [Figure 21] 10 is a conceptual diagram showing an example in which verification information related to the verification result by the work performance verification device in the present disclosure is displayed on the screen of a terminal device used by an administrator who manages managed persons. FIG. [Figure 22] 1 is a conceptual diagram illustrating an example of frames constituting video data captured by a surveillance camera according to the present disclosure. [Figure 23] 10 illustrates an example of a behavior detection log that summarizes the results of behavior detection by the work performance verification device according to the present disclosure. [Figure 24] 10 is a conceptual diagram showing an example in which verification information related to the verification result by the work performance verification device in the present disclosure is displayed on the screen of a terminal device used by an administrator who manages managed persons. FIG. [Figure 25] 10 is a conceptual diagram showing an example in which verification information related to the verification result by the work performance verification device according to the present disclosure is displayed on the screen of a terminal device used by a person to be managed. FIG. [Figure 26] 10 is an example of a work record table in which work identified by actions detected by the work performance verification device in application example 1 is recorded. [Figure 27] 10 is a conceptual diagram showing an example in which information regarding the proportion of work identified by behavior detected by the work performance verification device according to the present disclosure is displayed on the screen of a terminal device. FIG. [Figure 28]10 is an example of a work record table in which work situations identified by actions detected by a work performance verification device in application example 2 are recorded. [Figure 29] 10 is a conceptual diagram showing an example in which information about a work situation identified by a behavior detected by a work performance verification device according to the present disclosure is displayed on a screen of a terminal device. [Figure 30] 10 is a flowchart illustrating an example of the operation of the work management device according to the present disclosure. [Figure 31] 10 is a flowchart illustrating an example of the operation of the work performance verification device according to the present disclosure. [Figure 32] 10 is a flowchart illustrating an example of a behavior identification process performed by the work performance verification device according to the present disclosure. [Figure 33] 1 is a conceptual diagram illustrating an example of an installation environment of a work management system according to the present disclosure. [Figure 34] 1 is a block diagram illustrating an example of a configuration of a detection device according to the present disclosure. [Figure 35] 1 is a block diagram illustrating an example of the configuration of a work performance verification device according to the present disclosure. [Figure 36] 1 is a block diagram illustrating an example of the configuration of a work performance verification device according to the present disclosure. [Figure 37] FIG. 1 is a conceptual diagram for explaining an example of motion detection and behavior detection in the present disclosure. [Figure 38] 10 illustrates an example of a behavior detection log that summarizes the results of behavior detection by the work performance verification device according to the present disclosure. [Figure 39] 1 is a block diagram illustrating an example of the configuration of a work performance verification device according to the present disclosure. [Figure 40] 10 is a table illustrating an example of a terminal operation log acquired by a work performance verification device according to the present disclosure. [Figure 41] 1 is a conceptual diagram illustrating an example of the configuration of a work performance verification device according to the present disclosure. [Figure 42] 10 is a flowchart illustrating an example of the operation of the work performance verification device according to the present disclosure. [Figure 43]FIG. 2 is a block diagram illustrating an example of a hardware configuration for executing control and processing in the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Below, embodiments for implementing the present disclosure will be described using the drawings. In this disclosure, the drawings used in the description of each embodiment relate to one or more embodiments. Furthermore, elements included in each drawing may apply to one or more embodiments. The embodiments described below are limited in a way that is technically preferable for implementing the present disclosure, but this does not limit the scope of the disclosure to the following. In all drawings used to describe the following embodiments, similar parts are designated by the same reference numerals unless otherwise stated. In the following embodiments, repeated description of similar configurations and operations may be omitted. The direction of arrows in the drawings is an example and does not limit the direction in which signals, data, etc. propagate.

[0013] (First embodiment) First, a work management system according to a first embodiment will be described with reference to the drawings. The work management system according to this embodiment manages the working status of workers (subjects to management) using not only reported data including working hours reported by the workers (subjects to management) but also video data captured on the floor where the subject works (subject to management floor).

[0014] The following is an example of managing the work status of a managed person using video data captured by a surveillance camera installed on a floor to be managed. The managed floor may include not only the floor where the managed person primarily stays during work hours, but also any location where the managed person engaged in work may be filmed. For example, the managed floor may include the entrance and exit gates and corridors of the workplace, conference rooms, elevators, staircases, cafeterias, rest areas, etc. The work management system of this embodiment may also be configured to manage the work status of the managed person using video data captured by a surveillance camera that captures the periphery of the workplace.

[0015] (composition) FIG. 1 is a conceptual diagram showing an example of an installation environment for a work management system according to the present disclosure. The work management system according to this embodiment manages the work status of individuals on a floor to be managed. On the floor to be managed, terminal devices 100 are placed, each of which is used by a plurality of individuals to be managed. For example, the terminal device 100 is realized by an information processing device such as a desktop personal computer (PC) or a notebook PC. For example, the terminal device 100 may be a terminal for connecting to a virtual environment established on a cloud or server.

[0016] The work management system in this embodiment includes a work management device 150 and a work performance verification device 10. The work management system in this embodiment may also include a surveillance camera 160. The work management system in this embodiment is also connected to a terminal device 100 used by an employee to be managed.

[0017] The surveillance camera 160 is installed on a floor to be managed where the person to be managed works. The surveillance camera 160 is a network camera that captures video of the floor to be managed. The surveillance camera 160 is connected to the work performance verification device 10 via a network NW such as the Internet or an intranet. The surveillance camera 160 has a video capture function and a communication function, and may be in any form as long as it can capture video with enough accuracy to perform facial recognition of a person. For example, the surveillance camera 160 is installed on the ceiling or wall of the floor to be managed. For example, the surveillance camera 160 may be installed on a mobile object such as a robot or drone that can move around the floor to be managed. For example, the surveillance camera 160 may be a camera implemented in a terminal device used by the person to be managed.

[0018] The surveillance camera 160 captures images of the floor to be managed and generates video data. There are no particular limitations on the frame rate or number of pixels of the video data generated by the surveillance camera 160. The surveillance camera 160 transmits the captured video data of the floor to be managed to the work performance verification device 10 via the network NW. For example, the surveillance camera 160 may be configured to transmit to the work performance verification device 10 frames captured at a predetermined capture time out of multiple frames that make up the video data of the floor to be managed. With such a configuration, the communication volume of the data transmitted from the surveillance camera 160 can be reduced.

[0019] The work management device 150 is connected to the work performance verification device 10 via a network NW. The work management device 150 is also connected to a terminal device 100 used by the person to be managed. The work management device 150 may be connected to an entrance / exit management system (not shown) located at the entrance / exit of the company or the like where the person to be managed works. For example, the work management device 150 is implemented in a management terminal owned by the company or the like where the person to be managed works. For example, the work management device 150 may be implemented in a server located in a data center or in the cloud. The work management device 150 may be a dedicated information processing device or may be software. For example, the work management device 150 may be realized by general-purpose work management software that manages the work status of the person to be managed.

[0020] The work management device 150 acquires the reporting data entered by the managed person. The reporting data includes the working hours for each workday entered by the managed person through input operations on a terminal device. The working hours include the start time and end time entered by the managed person. For example, the working hours may include break times entered by the managed person. For example, the reporting data may include working hours acquired using an employee ID card or time card. The work management device 150 uses the acquired reporting data to generate work data that includes at least the start time and end time for each workday of the managed person. The work data may also include break times. The work management device 150 transmits the generated work data to the work performance verification device 10 via a network NW such as the Internet or an intranet.

[0021] The work performance verification device 10 is connected to a work management device 150 and a surveillance camera 160 via a network NW. For example, the work performance verification device 10 is implemented on a server located in a data center or in the cloud. For example, the work performance verification device 10 may be implemented on a management terminal owned by the company where the person to be managed works. The work performance verification device 10 may be a dedicated information processing device or software.

[0022] The work performance verification device 10 acquires video data of the managed floor captured by the surveillance camera 160. The work performance verification device 10 may be configured to acquire frames that make up the video data. The work performance verification device 10 also acquires work data generated by the work management device 150. The work performance verification device 10 verifies the work performance of the managed person for each work day using the video data and work data for each work day. The work performance verification device 10 manages the work performance of the managed person based on the difference between the work hours identified using the video data and the work hours included in the work data. For example, the work performance verification device 10 manages the work performance of the managed person using the following procedure.

[0023] The work performance verification device 10 extracts frames captured at a verification time from video data. The work performance verification device 10 detects people from frames at each verification time using object detection technology. For example, the work performance verification device 10 detects people from frames at each verification time using object detection AI (Artificial Intelligence). The object detection AI is a trained model that detects people included in image data based on input image data. For example, the object detection AI is a trained model constructed using a machine learning technique. The work performance verification device 10 extracts a partial area including a person from a frame at the verification time. The work performance verification device 10 extracts a partial image included in the extracted partial area. The work performance verification device 10 assigns a unique identifier (image identifier) ​​to each extracted partial image.

[0024] The work performance verification device 10 uses face recognition technology to identify people included in partial images to which image identifiers have been assigned. For example, the work performance verification device 10 uses a face recognition engine to identify people included in partial images to which image identifiers have been assigned. For example, the face recognition engine is an engine that uses face recognition technology to authenticate employees of a company or the like where a person to be managed works. For example, the face recognition engine is an engine configured to authenticate a person to be managed by comparing a face detected from a partial image with face data registered in advance. The work performance verification device 10 associates a person (person to be managed) identified in a partial image to which an image identifier has been assigned with the image identifier.

[0025] Furthermore, the work performance verification device 10 uses behavior detection technology to detect the behavior of a person included in a partial image to which an image identifier has been assigned. For example, the work performance verification device 10 uses behavior detection AI to detect the behavior of a person included in a partial image to which an image identifier has been assigned. The behavior detection AI is a trained model that detects the behavior of a person included in image data in response to input image data. For example, the behavior detection AI is a trained model constructed using a machine learning technique. For example, the behavior detection AI detects the behavior of a person included in an image based on a combination of the person included in the image and an object included in the image. The behavior detected by the behavior detection AI can be set arbitrarily. For example, if a terminal device 100 and a person are included in a partial image, the behavior detection AI detects that the person is "operating a PC." For example, if a telephone and a person are included in a partial image, the behavior detection AI detects that the person is "on the phone." The work performance verification device 10 associates the behavior detected in a partial image to which an image identifier has been assigned with the image identifier.

[0026] The work performance verification device 10 associates the managed person identified for each partial image assigned an image identifier with the behavior detected in the partial image, thereby identifying the behavior of the managed person. The work performance verification device 10 identifies the behavior of the managed person at the capture time of the frame from which the partial image was extracted. For example, if the behavior of the managed person at a certain capture time is work-related, such as "operating a PC" or "on the phone," the work performance verification device 10 determines that the managed person was engaged in work at that capture time. For example, if the behavior of the managed person at a certain capture time is not work-related, the work performance verification device 10 determines that the managed person was not engaged in work at that capture time. In this way, the work performance verification device 10 determines whether the managed person was engaged in work based on the behavior of the managed person. The work performance verification device 10 identifies the actual working hours of the managed person for each work day, including the first and last working times of the managed person.

[0027] The work performance verification device 10 determines that the work status of the managed person is normal if the difference between the actual work hours determined using the video data and the work hours included in the work data is within a reference time range. The work performance verification device 10 determines that the work status of the managed person is abnormal if the difference between the actual work hours determined using the video data and the work hours included in the work data exceeds the reference time. The criteria for determining whether the work status of the managed person is normal or abnormal can be set by the administrator. For example, if work hours are managed in 15-minute increments, the work performance verification device 10 is configured to determine that the work status of the managed person is abnormal if the difference between the actual work hours determined using the video data and the work hours included in the work data is 15 minutes or more.

[0028] The work performance verification device 10 outputs verification information including the verification result of the work status of the managed person. The output destination of the verification information is set arbitrarily. For example, the work performance verification device 10 transmits the verification information to a terminal device (not shown) used by an administrator who manages the work status of the managed person. The verification information transmitted to the terminal device is displayed on the screen of the terminal device in response to an operation by the administrator. For example, the administrator is the supervisor of the managed person or a person in the human resources department of the company where the managed person works. For example, the work performance verification device 10 may be configured to record the verification information of the managed person managed by the administrator in a database or server accessible from the terminal device used by the administrator. For example, the work performance verification device 10 may be configured to output verification information including the verification result of the managed person's work status to the managed person.

[0029] Next, the detailed configurations of the work management device 150 and the work performance verification device 10 will be described with reference to the drawings. In the following, the work status of the managed person is managed using reporting data including working hours reported by the managed person of work management and video data captured on the managed floor. In the following, an example is given in which the work status of the managed person is managed using video data captured by a surveillance camera 160 installed on the managed floor.

[0030] [Work management device] 2 is a conceptual diagram showing an example of the detailed configuration of the work management device 150 according to the present disclosure. The work management device 150 includes a working time acquisition unit 151, a recording unit 153, a work data generation unit 155, and a work data output unit 157.

[0031] The working time acquisition unit 151 functions as a communication interface. The working time acquisition unit 151 is connected to the terminal devices 100 used by the managed persons so that data communication is possible via a network such as the Internet. In the example of FIG. 2, the working time acquisition unit 151 is connected to the terminal devices 100 used by multiple managed persons. Terminal device 100A is used by managed person A. Terminal device 100B is used by managed person B. Terminal device 100C is used by managed person C. Terminal device 100D is used by managed person D.

[0032] The working time acquisition unit 151 acquires the reporting data transmitted from the terminal device 100 used by the person to be managed. The working time acquisition unit 151 may also be connected to an entrance / exit management system (not shown) located at the entrance / exit of the company where the person to be managed works. For example, the reporting data may include working hours acquired using an employee ID card or a time card. In this case, the working time acquisition unit 151 acquires the working hours of the person to be managed for each workday via the entrance / exit management system. The reporting data includes the working hours of the person to be managed for each workday. The working hours include the start time and end time of each workday entered by the person to be managed. The working hours may also include the break time of each workday entered by the person to be managed. The working time acquisition unit 151 stores the acquired reporting data in the recording unit 153.

[0033] The recording unit 153 includes a storage area in which the reporting data for each of a plurality of managed persons is stored. For example, the recording unit 153 is realized by a hard disk drive. The recording unit 153 may be configured as a database. The recording unit 153 stores the reporting data for each managed person. The reporting data stored in the recording unit 153 is used to generate work data for each working day of the managed person.

[0034] FIG. 3 is a table showing an example of a management table in which reporting data according to the present disclosure is compiled. The work management table 154 includes reporting data in which working hours, which are a combination of the start time and end time for each managed person, are associated with the managed person. For example, the work management table 154 is compiled by workday. Working hours include at least the start time and end time. The start time and end time may include the year, month, and day indicating the work day. Working hours may also include break times. In the work management table 154, a personal identifier uniquely assigned to each managed person is associated with the name of the managed person. For example, the personal identifier is an employee number. The personal identifier is not limited to an employee number as long as it can uniquely identify the managed person. Furthermore, the work management table 154 may include both the name and personal identifier of the managed person, or may include either the name or the personal identifier of the managed person.

[0035] The work data generation unit 155 acquires the reporting data of the managed persons from the recording unit 153. The work data generation unit 155 acquires the reporting data for the verification date for the managed persons. The work data generation unit 155 uses the acquired reporting data to generate work data that includes at least the start time and end time for each verification date (work day) of the managed persons. The work data may also include break times. The timing of generating the work data is set arbitrarily. For example, the work data generation unit 155 generates work data for multiple managed persons by batch processing at a preset date and time. For example, the work data generation unit 155 may be configured to generate work data for managed persons whose work statuses are managed by an administrator, in response to a request from the administrator.

[0036] The work data output unit 157 functions as a communications interface. The work data output unit 157 may be configured as a common communications interface with the working time acquisition unit 151. The work data output unit 157 is connected to the work performance verification device 10 via a network such as the Internet so that data can be communicated with the work performance verification device 10. The work data generated by the work data generation unit 155 is sent to the work performance verification device 10.

[0037] [Work performance verification device] 4 is a conceptual diagram showing an example of the detailed configuration of a work performance verification device according to the present disclosure. The work performance verification device 10 includes an image acquisition unit 11, an object detection unit 12, a face authentication unit 13, a behavior detection unit 14, a work data acquisition unit 16, a memory unit 17, a verification unit 18, and an output unit 19. The object detection unit 12, the face authentication unit 13, and the behavior detection unit 14 constitute an identification unit 15. The work performance verification device 10 also includes an object detection AI 120, a face authentication engine 130, and a behavior detection AI 140. The object detection AI 120, the face authentication engine 130, and the behavior detection AI 140 may be configured external to the work performance verification device 10. In this case, the work performance verification device 10 accesses the object detection AI 120, the face authentication engine 130, and the behavior detection AI 140 via an interface such as an API (Application Programming Interface).

[0038] The image acquisition unit 11 functions as a communication interface. The image acquisition unit 11 is connected to the surveillance camera 160 via the network NW so that data communication is possible. The image acquisition unit 11 acquires video data of the floor under management from the surveillance camera 160. The image acquisition unit 11 may be configured to acquire frames to be verified that constitute the video data. The frames to be verified are frames captured at the time to be verified during working hours. The image acquisition unit 11 sends the acquired frames to be verified to the object detection unit 12.

[0039] The object detection unit 12 acquires a verification target frame from the image acquisition unit 11. The object detection unit 12 detects a person from the verification target frame using an object detection technique. The object detection unit 12 detects a person from the verification target frame using an object detection AI 120. The object detection AI is a trained model that detects a person included in image data in response to input image data.

[0040] Fig. 5 is a conceptual diagram showing a bird's-eye view of a floor to be managed from the viewpoint of a surveillance camera in the present disclosure. A plurality of managed persons are staying on the managed floor. In the example of Fig. 5, a terminal device 100 is placed in front of each managed person. The managed persons working in the front row of the managed floor are indicated by letters (A, B, C, D) in the upper left corner to indicate their names.

[0041] FIG. 6 is a conceptual diagram showing an example of frames constituting video data captured by a surveillance camera in the present disclosure. The upper left corner of frame 115 indicates the time of capture (17:00). The frame 115 may or may not display the time of capture. The time of capture may be recorded as metadata and linked to frame 115. Note that, although the letters (A, B, C, D) indicating the names of the managed persons working in the front row of the managed floor are indicated in the upper left, the letters indicating the names of the managed persons are not actually displayed in frame 115. Frame 115 is a frame to be verified. Frame 115, which is a frame to be verified, is input to object detection AI 120.

[0042] For example, the object detection AI 120 is a trained model constructed using a machine learning technique. For example, the object detection AI 120 is a model trained using a dataset in which each frame (image data) constituting video data is an explanatory variable and the person included in each frame is an objective variable (label) as training data. For example, the object detection AI 120 is a trained model trained using a convolutional neural network (CNN) technique. For example, the object detection AI 120 is a trained model trained using a principal component analysis (PCA) technique. For example, the object detection AI 120 is a trained model trained using a variational autoencoder (VAE). For example, the object detection AI 120 is a trained model trained using a conditional generative adversarial network (GAN) technique. The above techniques are merely examples and do not limit the techniques for training the object detection AI 120.

[0043] The object detection unit 12 extracts a partial region including a person from the verification target frame. For example, the partial region is a rectangular region surrounding the person. For example, the partial region may be an oval, a circle, or a polygon other than a rectangle. The partial region may have any shape as long as it surrounds the periphery of the person. The object detection unit 12 extracts partial images included in the extracted partial region. The object detection unit 12 assigns a unique image identifier to each extracted partial image. The partial images extracted by the object detection unit 12 are sent to the face authentication unit 13 and the behavior detection unit 14. When detecting behavior in a collaborative task, the object detection unit 12 may be configured to extract partial images that include multiple people. The collaborative task to be detected is a task in which two or more people cooperate, such as transporting luggage. The collaborative task to be detected by the object detection unit 12 may be set in advance.

[0044] FIG. 7 is a conceptual diagram illustrating an example of a partial image extracted by the work performance verification device according to the present disclosure. Each of the multiple partial images 125 is assigned an image identifier for uniquely identifying the individual partial image 125. In the example of FIG. 7, the image identifier is shown in the upper right corner of the partial image 125. Note that the letters A, B, C, and D indicating the names of the managed individuals working in the front row of the managed floor are shown in the upper left, but the letters indicating the names of the managed individuals are not actually displayed in the frame 115. When the frame 115 to be verified is input to the object detection AI 120, multiple partial images 125 are extracted. The multiple partial images 125 are input to the face recognition engine 130 and the behavior detection AI 140.

[0045] The face authentication unit 13 acquires a partial image to which an image identifier has been assigned. The face authentication unit 13 uses face authentication technology to identify a person included in the partial image to which the image identifier has been assigned. The work performance verification device 10 uses the face authentication engine 130 to identify a person included in the partial image to which the image identifier has been assigned. For example, the face authentication engine 130 is an engine that uses face authentication technology to authenticate employees of a company or the like where a managed person works. For example, the face authentication engine 130 is an engine configured to authenticate a managed person by comparing a face detected from a partial image with pre-registered face data. The face authentication unit 13 associates an identifier (name or personal identifier) ​​representing a managed person identified in a partial image to which the image identifier has been assigned with the image identifier. The face authentication unit 13 sends data (authentication data) in which the identifier (name or personal identifier) ​​representing a managed person identified using the partial image is associated with the image identifier of the partial image to the verification unit 18.

[0046] The face authentication unit 13 may be configured to identify the facial expression of a person included in a partial image using a machine learning model that detects facial expressions. For example, if facial expressions can be detected, the behavior of the managed person can be determined in more detail. For example, if the managed person has a stern expression, the managed person may be in a difficult situation. For example, if the managed person has a grinning expression, the managed person may be engaged in an activity other than work. For example, if the managed person has a happy expression, something good may have happened to the managed person. Furthermore, if facial expressions can be detected, it becomes possible to determine the level of concentration and the work content of the managed person.

[0047] FIG. 8 shows an example of a face recognition log that summarizes the results of face recognition by the work performance verification device according to the present disclosure. The face recognition log 135 includes the names of the persons to be managed that are associated with image identifiers. The face recognition log 135 may include the personal identifiers of the persons to be managed instead of the names of the persons to be managed. In other words, the face recognition log 135 only needs to include an identifier (name or personal identifier) ​​that represents the persons to be managed. The face recognition log 135 also includes the detection time for each person to be managed that is associated with an image identifier.

[0048] The behavior detection unit 14 acquires a partial image to which an image identifier has been assigned. The behavior detection unit 14 uses behavior detection technology to detect the behavior of a person included in the partial image to which an image identifier has been assigned. The behavior detection unit 14 uses behavior detection AI 140 to detect the behavior of a person included in the partial image to which an image identifier has been assigned. The behavior detection AI 140 is a trained model that detects the behavior of a person included in image data in response to input image data. The behavior detection unit 14 associates the behavior detected in the partial image to which an image identifier has been assigned with the image identifier. The behavior detection unit 14 sends data (behavior data) in which the behavior identified from the partial image is associated with the image identifier of the partial image to the verification unit 18.

[0049] 9 shows an example of a behavior detection log that summarizes the results of behavior detection by the work performance verification device of the present disclosure. The behavior detection log 145 includes behaviors (operating a PC) associated with image identifiers for each partial image extracted from a frame captured at the detection time (17:00). The behavior detection log 145 also includes the detection time for each behavior associated with the image identifier.

[0050] For example, the behavior detection AI 140 is a trained model constructed using a machine learning technique. For example, the behavior detection AI 140 is a model trained using a dataset in which each frame (image data) constituting video data is used as an explanatory variable and the behavior of a person included in each frame is used as a target variable (label), as training data. For example, the behavior detection AI 140 is a trained model trained using a CNN technique. For example, the behavior detection AI 140 is a trained model trained using a PCA technique. For example, the behavior detection AI 140 is a trained model trained using a VAE technique. For example, the behavior detection AI 140 is a trained model trained using a conditional GAN ​​technique. The above techniques are merely examples and are not intended to limit the training techniques for the behavior detection AI 140. For example, the behavior detection AI 140 detects the behavior of a person included in an image based on a combination of the person and an object included in the image.

[0051] The behaviors of the managed individuals working in the managed office vary widely. For example, they may operate a PC, answer phone calls, hold meetings, or work on paper documents. Therefore, it is preferable that the behaviors detected by the behavior detection AI 140 be set arbitrarily. For example, if a partial image extracted from a verification target frame constituting video data captured on the managed floor includes a terminal device 100 and a person, the behavior detection AI 140 detects that the person is "operating a PC." For example, if a partial image extracted from a verification target frame constituting video data captured on the managed floor includes a telephone and a person, the behavior detection AI 140 detects that the person is "on the phone." Furthermore, the behaviors that the managed individuals may engage in during work hours vary depending on their occupation. For example, various occupations, such as software engineering, customer service, consulting, design, field work, accounting, human resources, legal affairs, research, sales, editing, public relations, planning, and reception, may each engage in different behaviors. Therefore, it is preferable that the behaviors to be detected be set according to their occupation.

[0052] The object detection unit 12, face authentication unit 13, and behavior detection unit 14 may be integrated into a classification unit 15. In this case, the classification unit 15 detects a person from the verification target frame using object detection technology. The classification unit 15 extracts a partial region including a person from the verification target frame. The identification unit 15 uses face recognition technology to identify people included in partial images to which image identifiers have been assigned. The identification unit 15 uses behavior detection technology to detect the behavior of people included in partial images to which image identifiers have been assigned. The identification unit 15 associates the managed person identified for each partial image with the behavior detected in that partial image, and identifies the behavior of the managed person at the verification target time when the verification target frame from which the partial image was extracted was captured. In other words, in this case, it is the identification unit 15, not the verification unit 18 described below, that identifies the behavior of the managed person at the verification target time when the verification target frame from which the partial image was extracted was captured.

[0053] The work data acquisition unit 16 functions as a communications interface. The work data acquisition unit 16 may be configured as a communications interface shared with the image acquisition unit 11. The work data acquisition unit 16 is connected to the work management device 150 via the network NW so that data can be communicated with. The work data acquisition unit 16 acquires work data of multiple managed persons from the work management device 150. The work data acquisition unit 16 stores the acquired work data in the storage unit 17. The work data acquisition unit 16 may be configured to acquire entry / exit data of managed persons from an entry / exit system that manages the entry and exit of managed persons. In this case, the entry time included in the entry / exit data corresponds to the start time of work, and the exit time included in the entry / exit data corresponds to the end time of work.

[0054] The storage unit 17 includes a storage area in which work data for each of a plurality of managed persons is stored. For example, the storage unit 17 is realized by a hard disk drive. The storage unit 17 may also be configured as a database. The storage unit 17 stores work data for each managed person. The work data for each managed person stored in the storage unit 17 is used to verify the work performance of each managed person on each work day.

[0055] The verification unit 18 acquires from the face authentication unit 13 authentication data in which an identifier (name or personal identifier) ​​representing the managed person identified using the partial image is associated with the image identifier of the partial image. The verification unit 18 also acquires behavior data in which behavior identified from the partial image is associated with the image identifier of the partial image. The verification unit 18 associates the managed person identified for each partial image with the behavior detected in the partial image, and identifies the behavior of the managed person at the verification target time when the verification target frame from which the partial image was extracted was captured. For example, if the managed person was performing work-related behavior such as "operating a PC" or "on the phone" at a certain verification target time, the verification unit 18 determines that the managed person was engaged in work at that verification target time. For example, if the managed person was performing non-work-related behavior at a certain verification target time, the verification unit 18 determines that the managed person was not engaged in work at that verification target time. In this way, the verification unit 18 determines whether the managed person was engaged in work at the verification target time based on the behavior of the managed person. The verification unit 18 identifies the actual working hours including the first working time and the last working time of the person to be managed for each working day.

[0056] The verification unit 18 acquires work data for each workday related to the managed person from the storage unit 17. The verification unit 18 compares the actual work time determined using the video data with the reported work time included in the work data to verify the work performance of the managed person on that workday. The actual work time is the time from the first detection time (first detection time) to the last detection time (final detection time) of the managed person. The reported work time is the time from the start time to the end time included in the work data. The verification unit 18 determines that the work status of the managed person is normal if the difference between the first detection time included in the actual work time and the start time included in the reported work time is within a reference time range. On the other hand, the verification unit 18 determines that there is an abnormality in the work status of the managed person if the difference between the first detection time included in the actual work time and the start time included in the reported work time exceeds the reference time. Furthermore, the verification unit 18 determines that the work status of the managed person is normal if the difference between the final detection time included in the actual work time and the end time included in the reported work time is within a reference time range. On the other hand, if the difference between the last detection time included in the actual working time and the end time included in the reported working time exceeds a reference time, the verification unit 18 determines that there is an abnormality in the working status of the managed person. The criteria for determining whether the working status of the managed person is normal or abnormal can be set by the administrator. For example, if working hours are managed in 15-minute increments, the work performance verification device 10 can be configured to determine that there is an abnormality in the working status of the managed person if the actual working time identified using video data differs by 15 minutes or more from the working time included in the work data.

[0057] FIG. 10 shows an example of a verification log that summarizes the verification results obtained by the work performance verification device according to the present disclosure. The verification log 180 includes the verification target time and the verification results for each managed person. The verification target time includes the reported working time and actual working time for each managed person. The reported working time includes the start time and end time. The actual working time includes the first detection time and the last detection time. In the example of FIG. 10, the deviation between the actual working time and the reported working time for all managed people is within the reference time range, so the verification result is normal (OK).

[0058] The output unit 19 functions as a communication interface. The output unit 19 may be configured as a common communication interface with the image acquisition unit 11 and the work data acquisition unit 16. The output unit 19 outputs verification information including the verification result of the work status of the managed person. The output destination of the verification information is set arbitrarily. For example, the output unit 19 transmits the verification information to a terminal device (not shown) used by an administrator who manages the work status of the managed person. The verification information transmitted to the terminal device is displayed on the screen of the terminal device in response to an operation by the administrator. For example, the administrator is the supervisor of the managed person or a person in the human resources department of the company where the managed person works. For example, the output unit 19 may be configured to record the verification information of the managed person managed by the administrator in a database or server accessible from the terminal device used by the administrator. For example, the output unit 19 may be configured to output the verification information including the verification result of the work status of the managed person to the managed person.

[0059] [Verification example] Next, an example of verification of work performance by the work performance verification device 10 in this embodiment will be described with reference to the drawings. Below, an example in which the verification result is abnormal (NG) will be given. In the following verification example, it is assumed that the reference time used to determine the discrepancy between the reported working hours and the actual working hours is set to 30 minutes.

[0060] <Verification example 1> Verification example 1 is an example in which the end time included in the reported working hours is later than the reference time than the last detected time included in the actual working hours. In other words, this verification example is an example in which a managed employee may have worked unpaid overtime. In this verification example, the reference time set for the discrepancy between actual working hours and reported working hours is 30 minutes.

[0061] FIG. 11 is a conceptual diagram illustrating an example of a partial image extracted by the work performance verification device according to the present disclosure. The multiple partial images 125 are a group of images output from the object detection AI 120 in response to input of a verification target frame captured at 5:45 PM. Each of the multiple partial images 125 is assigned an image identifier for uniquely identifying the individual partial image 125. In the example of FIG. 11, the image identifier is indicated in the upper right corner of the partial image 125. The managed person working in the front row of the managed floor is indicated in the upper left with an alphabet (A, D) indicating the name of the managed person. The alphabet and image identifier indicating the name of the managed person do not need to be displayed in the partial image 125, but may be displayed in the partial image 125. In this verification example, it is assumed that managed person B and managed person C have returned home at 5:45 PM. The multiple partial images 125 are input to the face recognition engine 130 and the behavior detection AI 140.

[0062] FIG. 12 shows an example of a verification log summarizing the verification results obtained by the work performance verification device according to the present disclosure. The verification log 181 includes the verification target time and the verification result for each managed individual. The verification target time includes the actual working time and reported working time for each managed individual. The actual working time includes the start time and end time. The reported working time includes the first detection time and the last detection time. In the example of FIG. 12, the verification result is normal (OK) for managed individual A, managed individual B, and managed individual C because the difference between the actual working time and the reported working time is within the reference time range. On the other hand, for managed individual D, the end time included in the reported working time is later than the reference time by more than the last detection time included in the actual working time. In other words, for managed individual D, the difference between the actual working time and the reported working time exceeds the reference time range, so the verification result is abnormal (NG). In this verification example, there is a possibility that managed individual D has performed unpaid overtime.

[0063] FIG. 13 is a conceptual diagram showing an example in which the verification result by the work performance verification device according to the present disclosure is displayed on the screen of a terminal device used by an administrator who manages the managed persons. The screen of the terminal device 105S used by the administrator S displays the verification result regarding the work performance of the managed person D on the work day to be verified (Friday, March 1, 2024). On the work day to be verified, the managed person D declared his finishing time as 5:00 PM. However, the last detected time of the managed person D was 5:45 PM. In this case, on the work day to be verified, the managed person D may have worked 45 minutes or more beyond the declared finishing time. In other words, this is an example in which the managed person D may have worked unpaid overtime on the work day to be verified. By viewing the screen of the terminal device 105S, the administrator S can recognize that the managed person D may have worked unpaid overtime on the work day to be verified.

[0064] FIG. 14 is a conceptual diagram illustrating an example of verification information related to the verification result obtained by the work performance verification device according to the present disclosure displayed on the screen of a terminal device used by an administrator who manages the managed persons. The screen of the terminal device 105S used by the administrator S displays verification information related to the verification result regarding the work performance of the managed person D on the workday to be verified (Friday, March 1, 2024). The screen of the terminal device 105S displays verification information related to the verification result, stating, "The last detection time of Mr. D on March 1st is different from the declared end time of work." The screen of the terminal device 105S also displays the declared end time of work, "2024 / 3 / 1_17:00," and the last detection time, "2024 / 3 / 1_17:45," side by side. Furthermore, the screen of the terminal device 105S displays action recommendation information urging the administrator S to take action, stating, "Please check the actual working hours of Mr. D." By viewing the screen of the terminal device 105S, the administrator S is motivated to check the actual working hours of the managed person D.

[0065] FIG. 15 is a conceptual diagram showing an example in which an evidence image of the verification result by the work performance verification device according to the present disclosure is displayed on the screen of a terminal device used by an administrator who manages the managed persons. A partial image extracted from a frame captured at the verification target time (5:45 PM) is displayed on the screen of the terminal device 105S used by the administrator S. The screen of the terminal device 105S clearly indicates the managed persons whose actual working conditions are abnormal. The screen of the terminal device 105S also displays verification information related to the verification result that "D deviation exists." By viewing the screen of the terminal device 105S, the administrator S can confirm that the managed person D was on the managed floor at a time that deviated from the end of work time included in the reported work data.

[0066] <Verification example 2> Verification example 2 is an example in which the end time included in the reported working hours is earlier than the last detection time included in the actual working hours by more than the reference time. In other words, this verification example is an example in which the person under management may have reported excessive overtime hours. In this verification example, the reference time set for the discrepancy between actual working hours and reported working hours is 30 minutes.

[0067] FIG. 16 is a conceptual diagram illustrating an example of a partial image extracted by the work performance verification device according to the present disclosure. The multiple partial images 125 are a group of images output from the object detection AI 120 in response to input of a verification target frame captured at 5:15 PM. Each of the multiple partial images 125 is assigned an image identifier for uniquely identifying the individual partial image 125. In the example of FIG. 16, the image identifier is indicated in the upper right corner of the partial image 125. The names of the managed individuals working in the front row of the managed floor are indicated in the upper left by letters (A, C, D). Note that the letters indicating the names of the managed individuals and the image identifiers do not need to be displayed in the partial image 125, but may be displayed in the partial image 125. In this verification example, managed individuals A, C, and D are detected on the managed floor at 5:15 PM. On the other hand, managed individual B is not detected on the managed floor at 5:15 PM. The plurality of partial images 125 are input to a face recognition engine 130 and a behavior detection AI 140 .

[0068] FIG. 17 shows an example of a verification log summarizing the verification results obtained by the work performance verification device according to the present disclosure. The verification log 182 includes the verification target time and the verification result for each managed individual. The verification target time includes the actual working time and reported working time for each managed individual. The actual working time includes the start time and end time. The reported working time includes the first detection time and the last detection time. In the example of FIG. 17, the verification result is normal (OK) for managed individual A, managed individual C, and managed individual D because the difference between the actual working time and the reported working time is within the reference time range. On the other hand, for managed individual B, the end time included in the reported working time is earlier than the reference time by more than the last detection time included in the actual working time. In other words, for managed individual B, the difference between the actual working time and the reported working time exceeds the reference time range, so the verification result is abnormal (NG). In this verification example, managed individual B may have excessively reported overtime hours.

[0069] FIG. 18 is a conceptual diagram illustrating an example of verification information related to the verification result obtained by the work performance verification device according to the present disclosure displayed on the screen of a terminal device used by an administrator who manages the managed persons. The screen of the terminal device 105S used by the administrator S displays verification information related to the verification result regarding the work performance of the managed person B on the work day being verified (March 8, 2024). The screen of the terminal device 105S displays verification information related to the verification result, stating, "The last detection time of B on March 8 deviates from the declared end time of work." The screen of the terminal device 105S also displays the declared end time of work, "2024 / 3 / 8_17:15," and the last detection time, "2024 / 3 / 8_16:30," side by side. Furthermore, the screen of the terminal device 105S displays action recommendation information urging the administrator S to take action, stating, "Please ask B to check his / her actual work status." By viewing the screen of the terminal device 105S, the administrator S is motivated to check the work status of the managed person B. In this verification example, similarly to verification example 1, the verification result and information including the evidence image of the verification result may be configured to be displayed on the screen of the terminal device 105S.

[0070] <Verification example 3> Verification example 3 is an example in which the first detection time included in the actual working hours is earlier than the start time included in the reported working hours by more than the reference time. In other words, this verification example is an example in which the managed employee may have come in early and worked unpaid overtime. In this verification example, the reference time set for the discrepancy between the actual working hours and the reported working hours is 30 minutes.

[0071] FIG. 19 is a conceptual diagram illustrating an example of partial images extracted by the work performance verification device according to the present disclosure. The multiple partial images 125 are a group of images output from the object detection AI 120 in response to input of a verification target frame captured at 7:30 p.m. Each of the multiple partial images 125 is assigned an image identifier for uniquely identifying the individual partial image 125. In the example of FIG. 19, the image identifier is indicated in the upper right corner of the partial image 125. The alphabet (C) indicating the name of the managed person working in the front row of the managed floor is indicated in the upper left. Note that the alphabet indicating the name of the managed person and the image identifier do not need to be displayed in the partial image 125, but may be displayed in the partial image 125. In this verification example, managed person C is detected on the managed floor at 7:30 p.m. Meanwhile, managed person A, managed person B, and managed person D are not detected on the managed floor at 7:30 p.m. The multiple partial images 125 are input to the face recognition engine 130 and the behavior detection AI 140.

[0072] FIG. 20 shows an example of a verification log summarizing the verification results obtained by the work performance verification device according to the present disclosure. The verification log 183 includes the verification target time and the verification result for each managed individual. The verification target time includes the actual working time and reported working time for each managed individual. The actual working time includes the start time and end time of work. The reported working time includes the first detection time and the last detection time. In the example of FIG. 20, the verification result is normal (OK) for managed individual A, managed individual B, and managed individual D because the difference between the actual working time and the reported working time is within the reference time range. On the other hand, for managed individual C, the first detection time included in the actual working time is earlier than the start time included in the reported working time by more than the reference time. In other words, for managed individual C, the difference between the actual working time and the reported working time exceeds the reference time range, so the verification result is abnormal (NG). In this verification example, there is a possibility that managed individual C came in early and performed unpaid overtime.

[0073] FIG. 21 is a conceptual diagram illustrating an example in which verification information related to the verification result obtained by the work performance verification device according to the present disclosure is displayed on the screen of a terminal device used by an administrator who manages the managed persons. The screen of the terminal device 105S used by the administrator S displays verification information related to the verification result regarding the work performance of the managed person C on the work day to be verified (March 15, 2024). The screen of the terminal device 105S displays verification information related to the verification result, stating, "Mr. C's first detection time on March 15 deviates from the declared start time." The screen of the terminal device 105S also displays the declared end time "2024 / 3 / 15_8:15" and the first detection time "2024 / 3 / 15_7:30" side by side. Furthermore, the screen of the terminal device 105S displays action recommendation information urging the administrator S to take action, stating, "Please ask Mr. C to check his actual work status." By viewing the screen of the terminal device 105S, the administrator S is motivated to check the actual work status of the managed person C. In this verification example, similarly to verification example 1, the verification result and information including the evidence image of the verification result may be configured to be displayed on the screen of the terminal device 105S.

[0074] <Verification example 4> Verification example 4 is an example in which a person under management engaged in an activity that may have been considered a break during working hours. This verification example shows two people under management standing and talking for a long period of time. This verification example shows that the people under management may not have declared their break times.

[0075] FIG. 22 is a conceptual diagram showing an example of frames constituting video data captured by a surveillance camera according to the present disclosure. The shooting time (14:00) is indicated in the upper left corner of frame 116. The shooting time may or may not be displayed in frame 116. The shooting time may be recorded as metadata and may be linked to frame 116. Frame 116 also shows letters (B, P) indicating the names of the managed persons. The letters indicating the names of the managed persons are not displayed in frame 116. In the example of FIG. 22, managed persons B and P are having a casual conversation. Frame 116 is a frame to be verified. Frame 116, which is the frame to be verified, is input to the object detection AI 120. In this verification example, it is assumed that 1 is assigned to the image identifier of the partial image including managed person B, and 2 is assigned to the image identifier of the partial image including managed person P.

[0076] FIG. 23 shows an example of a behavior detection log summarizing the results of behavior detection by the work performance verification device of the present disclosure. The behavior detection log 146 includes behaviors (standing conversations) associated with image identifiers for each partial image extracted from frames captured at the detection time (14:00). The behavior detection log 146 also includes the detection time for each behavior associated with the image identifier. In this verification example, a description using the verification log will be omitted. In this verification example, it is assumed that managed persons B and P were detected to be standing and talking for one hour. In this verification example, if non-work-related behavior continues beyond the break determination time (e.g., 30 minutes), it is assumed that the managed person is determined to be taking a break. Furthermore, if non-work-related behavior exceeds the break determination time but no break time is reported, there is serious doubt about the reported work data of the managed person who reported it.

[0077] FIG. 24 is a conceptual diagram illustrating an example of verification information related to the verification results obtained by the work performance verification device according to the present disclosure displayed on the screen of a terminal device used by an administrator who manages the managed persons. The screen of the terminal device 105S used by the administrator S displays verification information related to the verification results regarding the work performance of the managed person B on the work date to be verified (March 22, 2024). The screen of the terminal device 105S displays verification information related to the verification result, such as, "Mr. B was chatting with Mr. P from 1:30 PM to 2:30 PM on March 22." The screen of the terminal device 105S also displays information indicating a discrepancy between the reported work data and the actual work hours, such as, "Mr. B did not report the time period from 1:30 PM to 2:30 PM as a break." Furthermore, the screen of the terminal device 105S displays action recommendation information urging the administrator S to take action, such as, "Please ask Mr. B to check the actual work hours." By viewing the screen of the terminal device 105S, the administrator S is motivated to check the actual work hours of the managed person B. In this verification example, similarly to verification example 1, the verification result and information including the evidence image of the verification result may be configured to be displayed on the screen of the terminal device 105S.

[0078] FIG. 25 is a conceptual diagram illustrating an example in which verification information related to the verification result by the work performance verification device according to the present disclosure is displayed on the screen of a terminal device used by a managed person. The screen of terminal device 105B used by managed person B displays verification information related to the verification result regarding managed person B's work performance on the work date being verified (March 22, 2024). The screen of terminal device 105B displays verification information related to the verification result, such as, "You were chatting with Mr. P from 1:30 PM to 2:30 PM on March 22." The screen of terminal device 105B also displays information indicating a discrepancy between the reported work data and the actual work situation, such as, "You did not report the time period from 1:30 PM to 2:30 PM as a break time." Furthermore, the screen of terminal device 105B displays behavior recommendation information encouraging managed person B to take action, such as, "Please report the time period from 1:30 PM to 2:30 PM as a break time." By viewing the screen of terminal device 105B, managed person B is motivated to record his or her break time. In this verification example, the verification result and information including the evidence image of the verification result may also be configured to be displayed on the screen of the terminal device 105B.

[0079] [Application example] Next, application examples using the behavior detected by the work performance verification device 10 of this embodiment will be described with reference to the drawings. In the following, an example will be given in which information corresponding to the work identified by the detected behavior is visualized. In the following application example, an example will be shown in which information corresponding to the work identified by the detected behavior is displayed on the screen of a terminal device used by the person to be managed.

[0080] <Application example 1> 26 is an example of a work record table in which work identified by behavior detected by the work performance verification device in Application Example 1 is recorded. For example, the work record table 184 is generated by the verification unit 18. For example, a component (work recording unit) that generates the work record table 184 may be added to the work performance verification device 10.

[0081] The work record table 184 records the names of the managed individuals and the tasks identified by the actions detected by the managed individuals, in association with each other. The tasks include PC operation, filing documents, answering the phone, holding meetings, etc. The tasks recorded in the work record table 184 are identified by actions detected from frames constituting video data captured by the surveillance camera 160. Therefore, the tasks recorded in the work record table 184 do not accurately represent the tasks actually performed by the managed individuals. The work record table 184 may record the percentage of tasks performed on a specific workday or the percentage of tasks performed over a specific period. For example, the work record table 184 may record the percentage of tasks performed over a specific period, such as annually, semiannually, quarterly, monthly, or weekly. For example, the work record table 184 may record the percentage of tasks performed over a specific day of the week, as determined by the actions detected by the managed individuals.

[0082] FIG. 27 is a conceptual diagram illustrating an example in which information regarding the proportion of work identified by behavior detected by a work performance verification device according to the present disclosure is displayed on the screen of a terminal device. The screen of the terminal device 100B used by the managed person B clearly displays information regarding the proportion of work, such as "This is your proportion of work." The screen of the terminal device 100B also displays a pie chart indicating the proportion of work. The screen of the terminal device 100B also displays information according to the managed person B's work tendencies, such as "You spend a lot of time organizing documents and answering phone calls." The screen of the terminal device 100B also displays behavior recommendation information encouraging the managed person B to improve their work-related behavior, such as "Try to improve your work efficiency a little more." By viewing the screen of the terminal device 100B, the managed person B is motivated to improve their work situation.

[0083] <Application example 2> 28 is an example of a work record table in which work situations identified by actions detected by the work performance verification device in application example 2 are recorded. For example, the work record table 185 is generated by the verification unit 18. For example, a component (work recording unit) that generates the work record table 185 may be added to the work performance verification device 10.

[0084] The work record table 185 records the names of the managed individuals and the work statuses identified by the behaviors detected for those individuals in association with each other. The work statuses include periods of concentration during which the managed individuals concentrate on their work and periods of distraction during which they do not. The tasks include PC operation, filing, answering the phone, meetings, etc. The work statuses can be determined by characteristic behaviors that indicate the level of concentration for each task. For example, in the case of PC operation, if the managed individual is staring at the screen of the terminal device 100, the level of concentration is high. For example, if the managed individual is operating the keyboard or mouse while staring at the screen of the terminal device 100, the level of concentration is high. On the other hand, if the managed individual is looking away from the screen of the terminal device 100, the level of concentration is low. For example, the level of concentration of the managed individual on a task is determined using a pre-trained machine learning model. For example, the level of concentration may be determined by an upper limit criterion for determining periods of concentration and a lower limit criterion for determining periods of distraction. A time period in which the concentration level exceeds the upper limit criterion for a predetermined period of time or more is a concentrated time period. A time period in which the concentration level falls below the lower limit criterion for a predetermined period of time or more is a distracted time period. The work status recorded in the work record table 185 is identified based on the concentration level in the work identified by the behavior detected from the frames constituting the video data captured by the surveillance camera 160. Therefore, the work status recorded in the work record table 185 does not accurately represent the work status of the person under management. The work record table 185 may record the work status on a specific workday or for a specific period of time. For example, the work record table 185 records the work status for a specific period of time, such as annually, semi-annually, quarterly, monthly, or weekly. For example, the work record table 185 may record the work status identified by the behavior detected for each specific day of the week.

[0085] FIG. 29 is a conceptual diagram illustrating an example of information about a work situation identified by behavior detected by a work performance verification device according to the present disclosure, displayed on the screen of a terminal device. The screen of the terminal device 100C used by the managed person C clearly displays information about the work situation, such as "This is your work situation." The screen of the terminal device 100C also displays a bar graph indicating the work situation. The bar graph clearly displays periods of concentration and periods of distraction. The screen of the terminal device 100C also displays information corresponding to the work situation trends of the managed person C, such as "You tend to lose concentration in the afternoon." The screen of the terminal device 100C also displays behavior recommendation information, such as "Reconsider your work style," encouraging the managed person C to improve their work-related behavior. By viewing the screen of the terminal device 100C, the managed person C is motivated to improve their work situation. For example, if the managed person is frequently spoken to by others, the administrator of the managed person may be notified of this. For example, the level of concentration may be calculated based on the number of times the managed person is spoken to by others.

[0086] (operation) Next, an example of the operation of the work management system in this embodiment will be described with reference to the drawings. Here, the operation of the work management device 150 and the work performance verification device 10 provided in the work management system will be described separately. In the following, an example will be given in which the work performance of a person to be managed is verified for each work day. The work performance of a person to be managed may also be verified over multiple work days.

[0087] [Work management device] Fig. 30 is a flowchart showing an example of the operation of the work management device according to the present disclosure. In describing the processing according to the flowchart in Fig. 30, the components of the work management device 150 will be the main focus. The main focus of the processing according to the flowchart in Fig. 30 may be the work management device 150.

[0088] 30, the working time acquisition unit 151 acquires the reporting data input by the managed subject (step S111). The timing at which the working time acquisition unit 151 acquires the reporting data is set arbitrarily. For example, the working time acquisition unit 151 acquires the reporting data input for each managed subject one by one. The working time acquisition unit 151 may be configured to acquire the reporting data input for each managed subject at a preset timing.

[0089] Next, the working time acquisition unit 151 records the acquired reporting data in the recording unit 153 (step S112).

[0090] Next, the work data generation unit 155 generates work data for each person to be managed using the declaration data recorded in the recording unit 153 (step S113).

[0091] Once work data has been generated for all managed persons (Yes in step S114), work data output unit 157 transmits the generated work data for each managed person to work performance verification device 10 (step S115). Work data output unit 157 may be configured to transmit work data for each managed person individually to work performance verification device 10. Work data output unit 157 may be configured to transmit work data in response to a request from work performance verification device 10. On the other hand, if work data has not been generated for all managed persons (No in step S114), the process returns to step S113.

[0092] [Work performance verification device] Fig. 31 is a flowchart showing an example of the operation of the work performance verification device according to the present disclosure. In the description of the processing according to the flowchart in Fig. 31, the work performance verification device 10 will be the main subject. The main subject of the processing according to the flowchart in Fig. 31 may be a component of the work performance verification device 10.

[0093] In FIG. 31, first, the work performance verification device 10 acquires video data of the floor to be managed (step S121). The work performance verification device 10 acquires video data transmitted from the work management device 150. The work performance verification device 10 may be configured to request the work management device 150 to transmit video data including frames captured at the time to be verified. The video data includes frames captured at the time to be verified. The work performance verification device 10 may be configured to acquire frames included in the video data.

[0094] Next, the work record verification device 10 executes a behavior identification process (step S122). Details of the behavior identification process in step S122 will be described later with reference to FIG.

[0095] Next, the work performance verification device 10 verifies the work situation of each managed person based on the recorded behavior of each managed person (step S123).

[0096] When the verification of the work status for all managed persons is completed (Yes in step S124), the work performance verification device 10 outputs verification information including the verification results (step S125). The work performance verification device 10 may be configured to output verification information for each managed person. On the other hand, when the verification of the work status for all managed persons is not completed (No in step S124), the process returns to step S123.

[0097] <Behavior recognition processing> Fig. 32 is a flowchart showing an example of behavior identification processing by the work performance verification device according to the present disclosure. The behavior identification processing in Fig. 32 relates to the behavior identification processing in step S122 in Fig. 31. In describing the processing according to the flowchart in Fig. 32, the components of the work performance verification device 10 will be the main focus. The main focus of the processing according to the flowchart in Fig. 32 may be the work performance verification device 10.

[0098] In FIG. 32, first, the object detection unit 12 uses the object detection AI 120 to detect a person in each frame constituting the video data (step S131).

[0099] Next, the object detection unit 12 generates a partial image of the area where the person is detected (step S132).

[0100] Next, the object detection unit 12 assigns the verification target time and an image identifier to the partial image (step S133).

[0101] Next, the face authentication unit 13 uses the face authentication engine 130 to identify a person included in the partial image (step S134). The process of step S134 may be executed after the process of step S135. Furthermore, the process of step S134 may be executed in parallel with the process of step S135.

[0102] Next, the behavior detection unit 14 uses the behavior detection AI 140 to identify the behavior of the person included in the partial image (step S135). The process of step S135 may be executed before the process of step S134. Furthermore, the process of step S135 may be executed in parallel with the process of step S134.

[0103] Next, the verification unit 18 associates the managed person identified by face authentication with the behavior identified by behavior detection, and identifies the behavior of the managed person at the verification target time (step S136).

[0104] Next, the verification unit 18 records the behavior of the person to be managed at the identified verification time (step S137).

[0105] When the behavior of all managed subjects has been recorded (Yes in step S138), the process proceeds to step S123 in Fig. 32. On the other hand, if the behavior of all managed subjects has not been recorded (No in step S138), the process returns to step S134.

[0106] (Variation) Next, modified examples of the work management system in this embodiment will be described with reference to the drawings. Three modified examples will be shown below. The modified examples of the work management system in this embodiment are not limited to the following three modified examples.

[0107] [Variation 1] First, the work management system according to Modification 1 will be described with reference to the drawings. The work management system according to this modification (FIG. 33) differs from the system configuration in FIG. 1 in that some of the components (functions) included in the work performance verification device 10 are located near the surveillance camera 160. Below, differences from the system configuration in FIG. 1 will be mainly described.

[0108] 33 is a conceptual diagram showing an example of an installation environment for a work management system according to the present disclosure. The work management system according to this modification includes a work management device 150, a monitoring camera 160, a detection device 101, and a work performance verification device 10-1. The work management system according to this modification may be configured with only the work management device 150, the monitoring camera 160, the detection device 101, and the work performance verification device 10-1, without including the monitoring camera 160. The work management system according to this modification is also connected to a terminal device 100 used by the person to be managed. In the following, a description of the work management device 150 and the monitoring camera 160 will be omitted.

[0109] The detection device 101 is placed near the surveillance camera 160. The detection device 101 has the object detection function of the work performance verification device 10 (FIG. 1). The detection device 101 is configured as an edge computer. The detection device 101 is connected to the surveillance camera 160 by wiring or the like (not shown). The detection device 101 may be connected to the surveillance camera 160 wirelessly. For example, the detection device 101 is connected to the surveillance camera 160 wirelessly by short-range wireless communication. The detection device 101 may be built into the surveillance camera 160. For example, the detection device 101 is realized by a computer having information processing capabilities. The detection device 101 is also connected to the work performance verification device 10-1 via a network NW.

[0110] 34 is a block diagram showing an example of the configuration of a detection device according to the present disclosure. The detection device 101 includes an image acquisition unit 11, an object detection unit 12, and a partial image transmission unit 123. The detection device 101 also includes an object detection AI 120.

[0111] The image acquisition unit 11, the object detection unit 12, and the object detection AI 120 have the same functions as those shown in Fig. 4. The partial image transmission unit 123 transmits the partial image to which the image identifier has been assigned to the work performance verification device 10-1. The partial image transmission unit 123 may be configured to transmit only the partial image extracted from the frame captured at the time to be verified.

[0112] 35 is a block diagram showing an example of the configuration of a work performance verification device according to the present disclosure. The work performance verification device 10-1 includes a partial image acquisition unit 113, a face authentication unit 13, a behavior detection unit 14, a work data acquisition unit 16, a memory unit 17, a verification unit 18, and an output unit 19. The work performance verification device 10-1 also includes a face authentication engine 130 and a behavior detection AI 140. The face authentication engine 130 and the behavior detection AI 140 may be configured external to the work performance verification device 10-1. In this case, the work performance verification device 10-1 accesses the face authentication engine 130 and the behavior detection AI 140 via an interface such as an API (Application Programming Interface).

[0113] The partial image acquisition unit 113 acquires the partial image transmitted from the detection device 101. The partial image acquisition unit 113 may be configured to receive only the partial image extracted from the frame captured at the time to be verified. The face authentication unit 13, the behavior detection unit 14, the work data acquisition unit 16, the memory unit 17, the verification unit 18, the output unit 19, the face authentication engine 130, and the behavior detection AI 140 have the same functions as the configuration shown in FIG. 4.

[0114] In this modification, the detection device 101 extracts partial images from frames constituting video data captured by the surveillance camera 160. The detection device 101 transmits the extracted partial images to the work performance verification device 10-1. Therefore, according to this modification, the communication load can be reduced compared to when video data captured by the surveillance camera 160 is transmitted to the work performance verification device 10. The processing load of the object detection AI 120 is smaller than that of the processing by the face recognition engine 130 and the behavior detection AI 140. The performance required of the computer on which the object detection AI 120 is implemented is not as high as that required of the computer on which the face recognition engine 130 and the behavior detection AI 140 are implemented. Therefore, the object detection AI 120 can be implemented in small hardware with relatively low performance.

[0115] [Variation 2] Next, a work performance verification device according to Modification 2 will be described with reference to the drawings. The work performance verification device according to Modification 2 (FIG. 36) differs from the work performance verification device of FIG. 4 in that it detects behavior corresponding to the movement of a person included in a partial image. The following mainly describes the differences from the work performance verification device 10 of FIG. 4.

[0116] 36 is a block diagram showing an example of the configuration of a work performance verification device according to the present disclosure. The work performance verification device 10-2 includes an image acquisition unit 11, an object detection unit 12, a face authentication unit 13, a behavior detection unit 14-2, a work data acquisition unit 16, a memory unit 17, a verification unit 18, and an output unit 19. The work performance verification device 10-2 also includes an object detection AI 120, a face authentication engine 130, a motion detection AI 141, and a behavior detection AI 142. The object detection AI 120, the face authentication engine 130, the motion detection AI 141, and the behavior detection AI 142 may be configured external to the work performance verification device 10-2. In this case, the work performance verification device 10-2 accesses the object detection AI 120, the face authentication engine 130, the motion detection AI 141, and the behavior detection AI 142 via an interface such as an API (Application Programming Interface). The image acquisition unit 11, object detection unit 12, face authentication unit 13, work data acquisition unit 16, memory unit 17, verification unit 18, output unit 19, behavior detection AI 140, and face authentication engine 130 have the same functions as the configuration shown in Fig. 4. As with Variation 1, a configuration with an object detection function may be placed near the surveillance camera 160.

[0117] FIG. 37 is a conceptual diagram illustrating an example of motion detection and behavior detection according to the present disclosure. The behavior detection unit 14-2 acquires a partial image to which an image identifier is assigned. The behavior detection unit 14-2 detects the motion of a person included in the partial image to which the image identifier is assigned using a motion detection technique. The behavior detection unit 14-2 detects the motion of a person included in the partial image to which the image identifier is assigned using a motion detection AI 141. The motion detection AI 141 is a model that extracts skeletal data of a person. For example, the motion detection AI 141 is a trained model that extracts skeletal data of a person included in image data in response to input image data. For example, the motion detection AI 141 extracts skeletal data of a person included in the partial image using a three-dimensional skeletal estimation technique. Furthermore, the behavior detection unit 14-2 detects the behavior of a person included in the partial image using the behavior detection technique. The behavior detection unit 14-2 detects a motion according to the skeletal data of the person extracted from the partial image using a behavior detection AI 142. For example, the behavior detection AI 142 is a trained model that extracts behaviors corresponding to input skeletal data. The behavior detection unit 14-2 associates behaviors detected in partial images to which image identifiers have been assigned with the image identifiers. The behavior detection unit 14-2 sends data (behavior data) in which the behaviors identified from the partial images are associated with the image identifiers of the partial images to the verification unit 18.

[0118] For example, the motion detection AI 141 is a trained model constructed using a machine learning technique. For example, the motion detection AI 141 is a model trained using a dataset in which each frame (image data) constituting video data is used as an explanatory variable and the skeleton of a person included in each frame is used as a target variable (label), as training data. For example, the motion detection AI 141 is a trained model trained using a CNN technique. For example, the motion detection AI 141 is a trained model trained using a PCA technique. For example, the motion detection AI 141 is a trained model trained using a VAE technique. For example, the motion detection AI 141 is a trained model trained using a conditional GAN ​​technique. The above techniques are merely examples and do not limit the techniques for training the motion detection AI 141.

[0119] For example, the behavior detection AI 142 is a trained model constructed using a machine learning technique. For example, the behavior detection AI 142 is a model trained using a dataset in which skeletal data is used as an explanatory variable and human behavior is used as a target variable (label) as training data. For example, the behavior detection AI 142 is a trained model trained using a CNN technique. For example, the behavior detection AI 142 is a trained model trained using a PCA technique. For example, the behavior detection AI 142 is a trained model trained using a VAE technique. For example, the behavior detection AI 142 is a trained model trained using a conditional GAN ​​technique. The above techniques are merely examples and do not limit the techniques for training the behavior detection AI 142.

[0120] FIG. 38 shows an example of a behavior detection log summarizing the results of behavior detection by the work performance verification device according to the present disclosure. The behavior detection log 147 includes behaviors associated with image identifiers for each partial image extracted from a frame captured at the detection time (17:00). The behavior detection log 147 also includes the detection time for each behavior associated with the image identifier. By utilizing action recognition using skeletal data, the behavior of the managed person can be detected based on the skeletal movements of the managed person. The behavior detection log 147 includes not only the behavior of simply operating a PC, but also behaviors such as keyboard operation, mouse operation, and screen browsing. In other words, according to this modification, the behavior of the managed person can be verified in more detail based on the behavior detected based on skeletal movements.

[0121] [Variation 3] Next, a work performance verification device according to Modification 3 will be described with reference to the drawings. The work performance verification device according to this modification (FIG. 39) differs from the work performance verification device 10 in FIG. 4 in that it detects the behavior of the person to be managed using operation information (terminal operation information) of the terminal device 100 by the person to be managed. The following mainly describes the differences from the work performance verification device 10 in FIG. 4.

[0122] 39 is a block diagram showing an example of the configuration of a work performance verification device according to the present disclosure. The work performance verification device 10-3 includes an image acquisition unit 11, an operation information acquisition unit 117, an object detection unit 12, a face authentication unit 13, a behavior detection unit 14, a work data acquisition unit 16, a memory unit 17, a verification unit 18-3, and an output unit 19. The work performance verification device 10-3 also includes an object detection AI 120, a face authentication engine 130, and a behavior detection AI 140. The object detection AI 120, the face authentication engine 130, and the behavior detection AI 140 may be configured external to the work performance verification device 10-3. In this case, the work performance verification device 10-3 accesses the object detection AI 120, the face authentication engine 130, and the behavior detection AI 140 via an interface such as an API (Application Programming Interface). The image acquisition unit 11, object detection unit 12, face authentication unit 13, behavior detection unit 14, work data acquisition unit 16, memory unit 17, output unit 19, object detection AI 120, face authentication engine 130, and behavior detection AI 140 have the same functions as the configuration shown in Fig. 4. As with Modification 1, a configuration with object detection functionality may be placed near the surveillance camera 160. Furthermore, the behavior detection technology using skeletal data of Modification 2 may be combined with the method of this modification.

[0123] The operation information acquisition unit 117 is connected to the terminal device 100. The operation information acquisition unit 117 acquires terminal operation information (terminal operation log) including the operation history of the terminal device 100 by the managed person. The terminal operation log includes terminal operation information indicating what applications were used on the terminal device 100. For example, the terminal operation log includes terminal operation information such as document creation, email creation, web browsing, etc. The terminal operation log also includes terminal operation information such as no operation, which means that no operation was performed by the managed person.

[0124] FIG. 40 is a table showing an example of a terminal operation log acquired by a work performance verification device according to the present disclosure. The terminal operation log 171 includes terminal operation information indicating the terminal operations of each managed person at the time to be verified (17:00). The terminal operation log 171 also includes the time to be verified. By utilizing behavior recognition using the terminal operation log 171, the behavior of the managed person can be detected according to the operation of the terminal device of the managed person. The terminal operation log 171 includes not only behavior such as PC operation, but also behaviors such as document creation, email creation, no operation, web browsing, etc. In other words, according to this modification, the behavior of the managed person can be verified in more detail.

[0125] As described above, the work management system of this embodiment includes a work management device and a work performance verification device. The work management device acquires work data including start times and end times declared by the managed person. The work management device transmits the acquired work data to the work performance verification device. The work performance verification device includes an image acquisition unit, an identification unit, a work data acquisition unit, a verification unit, and an output unit. The image acquisition unit acquires frames constituting video data captured on a managed floor where the managed person works. The identification unit identifies the managed person corresponding to a person included in a partial image extracted from the acquired frames and the behavior of the person included in the partial image. The work data acquisition unit acquires work data including start times and end times declared by the identified managed person. The verification unit verifies the work status of the managed person based on the difference between the actual work hours identified based on the behavior of the identified managed person and the reported work hours identified based on the reported work data. The verification unit generates verification information including the verification results regarding the work status of the managed person. The output unit outputs the generated verification information including the verification results regarding the work status of the managed person.

[0126] The work management device of this embodiment transmits work data including start and end times declared by the managed subject to a work performance verification device. The work performance verification device of this embodiment extracts partial images containing people from frames captured on the managed floor. The work performance verification device of this embodiment associates the managed subject corresponding to the person included in the extracted partial image with the person's behavior to identify the behavior of the managed subject. The work performance verification device of this embodiment verifies the work data declared by the managed subject based on the behavior of the managed subject identified using frames captured on the managed floor. Therefore, according to this embodiment, the behavior of the managed subject performing work can be accurately traced, thereby accurately verifying the work performance of the managed subject.

[0127] In one aspect of this embodiment, the identification unit includes an object detection unit, a face authentication unit, and a behavior detection unit. The object detection unit detects a person from a frame using object detection AI (Artificial Intelligence). An image area including the detected person is extracted as a partial image. The face authentication unit authenticates the managed person corresponding to the person included in the partial image using a face authentication engine. The behavior detection unit detects the behavior of the person included in the partial image using the behavior detection AI. According to this aspect, the object detection AI, face authentication engine, and behavior detection AI can be used to accurately trace the behavior of the managed person performing their work, thereby enabling accurate verification of the work performance of the managed person.

[0128] In one aspect of this embodiment, if work-related behavior is identified during a time period not included in the reported working hours, the verification unit generates verification information including a verification result indicating that the work data is questionable. The output unit transmits the verification information including the verification result indicating that the reported work data is questionable to a terminal device used by an administrator who manages the work status of the managed employee. According to this aspect, the administrator can be notified that the managed employee may have performed work based on work-related behavior performed by the managed employee during a time period not included in the reported working hours.

[0129] In one aspect of this embodiment, the verification unit determines that the declared work data is suspicious if work-related behavior by the managed individual is identified at a time that deviates by more than a reference time from the start time included in the declared work data. The verification unit also determines that the declared work data is suspicious if work-related behavior by the managed individual is identified at a time that deviates by more than a reference time from the end time included in the declared work data. In these cases, the verification unit generates verification information including a verification result indicating that the work data declared by the managed individual is suspicious. The output unit transmits the verification information including the verification result indicating that the declared work data is suspicious to a terminal device used by an administrator who manages the work status of the managed individual. According to this aspect, the administrator can be notified that the managed individual may have performed work based on work-related behavior performed by the managed individual during a time period that deviates from the declared working hours.

[0130] In one aspect of this embodiment, if the verification unit identifies non-work-related behavior by the managed individual during the reported working time, it generates verification information including a verification result indicating that the reported work data is questionable. The output unit transmits the verification information including the verification result indicating that the reported work data is questionable to a terminal device used by an administrator who manages the work status of the managed individual. According to this aspect, it is possible to notify the administrator that the managed individual may have engaged in non-work-related behavior based on the non-work-related behavior engaged in by the managed individual during the reported working time.

[0131] In one aspect of this embodiment, the behavior detection unit extracts skeletal data of a person included in a partial image using a motion detection AI that extracts skeletal data of a person included in image data. The behavior detection unit detects behavior corresponding to the extracted skeletal data using a motion detection AI that extracts behavior according to the skeletal data. According to this aspect, the behavior of the managed person can be verified in more detail based on the behavior of the managed person detected according to skeletal movement.

[0132] In one aspect of the present embodiment, the system includes an operation information acquisition unit that acquires terminal operation information including an operation history of a terminal device used by the managed person. The verification unit uses the terminal operation information to recognize behavior indicating the operation content of the terminal device by the managed person. According to this aspect, the behavior of the managed person can be verified in more detail based on the behavior of the managed person detected in accordance with the operation content of the terminal device.

[0133] In this embodiment, an example of managing the work status of a managed person who performs work using a terminal device is shown. The method of this embodiment can also be applied to managing the work status of other work. For example, the method of this embodiment can be applied to managing the work status of reception work at a company. For example, the method of this embodiment can be applied to managing the work status of counter work at a post office, bank, or city hall. For example, the method of this embodiment can be applied to managing the work status of work in a factory, warehouse, construction site, or the like. For example, the method of this embodiment can be applied to managing the work status of medical professionals engaged in healthcare-related work in medical settings such as hospitals. For example, the method of this embodiment can be applied to managing the work status of employees working in restaurants. For example, the method of this embodiment can be applied to managing the work status of staff who manage entrance, provide guidance, and manage safety at event venues. In this way, the method of this embodiment can be applied to managing the work status of a variety of work. Note that the work listed here is merely an example and does not limit the work to which the method of this embodiment can be applied.

[0134] The method of this embodiment can be applied to other purposes besides managing work status. For example, the method of this embodiment can be applied to managing the learning status of students in schools. For example, the method of this embodiment can be applied to managing the hospital life of patients admitted to hospitals. For example, the method of this embodiment can be applied to behavior management of residents admitted to facilities. In this way, the method of this embodiment can be applied to any behavior management related to a person's behavior.

[0135] The method of this embodiment may be used to manage the resting status of a managed person during break time. For example, for a managed person who is often detected to be engaged in work-related activities during break time, the method may be configured to notify not only the manager but also a health management center or the like that manages the health status of the managed person. With this configuration, the health status of the managed person can also be managed.

[0136] (Second embodiment) Next, a work performance verification device according to a second embodiment will be described with reference to the drawings. The work performance verification device according to this embodiment has a simplified configuration of the work performance verification device according to the first embodiment.

[0137] (composition) 41 is a conceptual diagram showing an example of the configuration of a work performance verification device according to the present disclosure. Work performance verification device 20 includes image acquisition unit 21, recognition unit 25, work data acquisition unit 26, verification unit 28, and output unit 29.

[0138] The image acquisition unit 21 acquires frames constituting video data captured of the managed floor where the managed individual works. The identification unit 25 identifies the managed individual corresponding to the person included in the partial image extracted from the acquired frame and the behavior of the person included in the partial image. The work data acquisition unit 26 acquires work data including the start time and end time declared by the identified managed individual. The verification unit 28 verifies the work status of the managed individual based on the difference between the actual work hours identified based on the behavior of the identified managed individual and the reported work hours identified by the reported work data. The verification unit 28 generates verification information including the verification result regarding the work status of the managed individual. The output unit 29 outputs the generated verification information including the verification result regarding the work status of the managed individual.

[0139] (operation) Next, an example of the operation of the work performance verification device 20 in this embodiment will be described with reference to the drawings. Fig. 42 is a flowchart showing an example of the operation of the work performance verification device in the present disclosure. In describing the processing according to the flowchart in Fig. 42, the components of the work performance verification device 20 will be the main focus. The main focus of the processing according to the flowchart in Fig. 42 may be the work performance verification device 20.

[0140] The image acquisition unit 21 acquires frames constituting video data captured on a floor to be managed where a person to be managed works (step S21).

[0141] The identification unit 25 identifies the managed person corresponding to the person included in the partial image extracted from the acquired frame and the behavior of the person included in the partial image (step S22).

[0142] The work data acquisition unit 26 acquires work data including the start time and end time declared by the identified managed person (step S23).

[0143] The verification unit 28 verifies the working status of the managed person based on the difference between the actual working hours determined based on the behavior of the identified managed person and the reported working hours determined based on the reported working data (step S24).

[0144] The verification unit 28 generates verification information including the verification result regarding the working status of the person to be managed (step S25).

[0145] The output unit 29 outputs the generated verification information including the verification result regarding the working status of the person to be managed (step S26).

[0146] The work performance verification device of this embodiment extracts partial images containing people from frames captured on a managed floor. The work performance verification device of this embodiment associates the managed person corresponding to the person included in the extracted partial image with the person's behavior, thereby identifying the behavior of the managed person. The work performance verification device of this embodiment verifies the work data declared by the managed person based on the behavior of the managed person identified using frames captured on the managed floor. Therefore, according to this embodiment, the behavior of the managed person performing work can be accurately traced, thereby accurately verifying the work performance of the managed person.

[0147] (Hardware) Next, a hardware configuration for executing the control and processing in the present disclosure will be described with reference to the drawings. Here, an information processing device 90 (computer) in Fig. 43 is given as an example of such a hardware configuration. The information processing device 90 in Fig. 43 is an example of a configuration for executing the control and processing in the present disclosure and does not limit the scope of the present disclosure.

[0148] As shown in Fig. 43, an information processing device 90 includes a processor 91, a memory 92, an auxiliary storage device 93, an input / output interface 95, and a communication interface 96. In Fig. 43, interface is abbreviated as I / F (Interface). The processor 91, memory 92, auxiliary storage device 93, input / output interface 95, and communication interface 96 are connected to each other via a bus 98 so as to be able to communicate data with each other. The processor 91, memory 92, auxiliary storage device 93, and input / output interface 95 are also connected to a network such as the Internet or an intranet via the communication interface 96.

[0149] The processor 91 loads a program (instructions) stored in an auxiliary storage device 93 or the like into the memory 92. For example, the program is a software program for executing the control and processing in the present disclosure. The processor 91 executes the program loaded into the memory 92. The processor 91 executes the program to execute the control and processing in the present disclosure.

[0150] The memory 92 is a storage device having an area in which a program is loaded. The processor 91 loads a program stored in an auxiliary storage device 93 or the like into the memory 92. The memory 92 is realized by a volatile memory such as a DRAM (Dynamic Random Access Memory). Alternatively, a non-volatile memory such as an MRAM (Magnetoresistive Random Access Memory) may be used as the memory 92.

[0151] The auxiliary storage device 93 stores various data such as programs. For example, the auxiliary storage device 93 is realized by a local disk such as a hard disk or flash memory. Note that it is also possible to configure the system so that various data is stored in the memory 92, and omit the auxiliary storage device 93.

[0152] The input / output interface 95 is an interface for connecting the information processing device 90 to peripheral devices based on standards and specifications. The communication interface 96 is an interface for connecting to external systems and devices via a network such as the Internet or an intranet based on standards and specifications. The input / output interface 95 and the communication interface 96 may be a common interface for connecting to external devices.

[0153] Input devices such as a keyboard, mouse, and touch panel may be connected to the information processing device 90 as needed. These input devices are used to input information and settings. When a touch panel is used as the input device, a screen having the function of the touch panel serves as the interface. The processor 91 and the input devices are connected via an input / output interface 95.

[0154] The information processing device 90 may be equipped with a display device for displaying information. When a display device is equipped, the information processing device 90 is equipped with a display control device (not shown) for controlling the display of the display device. The information processing device 90 and the display device are connected via an input / output interface 95.

[0155] The information processing device 90 may be equipped with a drive device. The drive device acts as an intermediary between the processor 91 and a recording medium (program recording medium) to read data and programs stored on the recording medium and to write processing results of the information processing device 90 to the recording medium. The information processing device 90 and the drive device are connected via an input / output interface 95.

[0156] The above is an example of a hardware configuration for enabling the control and processing in the present disclosure. The hardware configuration in Figure 43 is an example of a hardware configuration for executing the control and processing in the present disclosure, and does not limit the scope of the present disclosure. A program that causes a computer to execute the control and processing in the present disclosure is also included in the scope of the present disclosure.

[0157] A program recording medium on which a program for executing the processing in this embodiment is recorded is also included in the scope of the present invention. For example, the program recording medium is a computer-readable non-transitory recording medium. The recording medium can be realized as an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The recording medium may also be realized as a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card. The recording medium may also be realized as a magnetic recording medium such as a flexible disk or other recording medium.

[0158] The components in the present disclosure may be combined in any manner. The components in the present disclosure may be realized by software. The components in the present disclosure may be realized by circuits.

[0159] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0160] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) an image acquisition unit that acquires frames constituting video data captured on a floor on which a person to be managed works; an identification unit that identifies the managed person corresponding to a person included in a partial image extracted from the acquired frame and the behavior of the person included in the partial image; a work data acquisition unit that acquires work data including a start time and an end time declared by the identified managed person; a verification unit that verifies the working status of the managed person based on the difference between the actual working hours determined in accordance with the identified behavior of the managed person and the declared working hours determined based on the declared work data, and generates verification information including the verification result regarding the working status of the managed person; an output unit that outputs verification information including the generated verification result regarding the working status of the person to be managed. (Appendix 2) The identification unit an object detection unit that detects a person from the frame using object detection AI (Artificial Intelligence) and extracts an image area including the detected person as the partial image; a face authentication unit that authenticates the person to be managed corresponding to the person included in the partial image using a face authentication engine; A work performance verification device as described in Appendix 1, having a behavior detection unit that detects the behavior of a person included in the partial image using behavior detection AI. (Appendix 3) The verification unit If work-related activity is identified during a time period not included in the reported working hours, verification information is generated that includes a verification result indicating that the work data is suspicious; The output unit A work performance verification device as described in Appendix 2 that transmits verification information including a verification result indicating that there is doubt about the declared work data to a terminal device used by an administrator who manages the work status of the managed person. (Appendix 4) The verification unit If a work-related activity by the managed person is identified at a time that deviates from the start time included in the declared work data by more than a reference time, or at a time that deviates from the end time included in the declared work data by more than a reference time, generating verification information including a verification result indicating that there is a doubt about the declared work data, The output unit A work performance verification device as described in Appendix 3 that transmits verification information including a verification result indicating that there is doubt about the declared work data to a terminal device used by an administrator who manages the work status of the managed person. (Appendix 5) The verification unit If an action unrelated to work by the person to be managed is identified during the reported working hours, verification information is generated including a verification result indicating that there is a doubt about the reported work data; The output unit A work performance verification device as described in Appendix 2 that transmits verification information including a verification result indicating that there is doubt about the declared work data to a terminal device used by an administrator who manages the work status of the managed person. (Appendix 6) The behavior detection unit Extracting skeletal data of a person included in the partial image using a motion detection AI that extracts skeletal data of a person included in the image data; A work performance verification device as described in Appendix 2, which uses a behavior detection AI that extracts behavior according to skeletal data to detect behavior corresponding to the extracted skeletal data. (Appendix 7) an operation information acquisition unit that acquires terminal operation information including an operation history of a terminal device used by the person to be managed, The verification unit 3. A work performance verification device according to claim 2, which uses the terminal operation information to recognize behavior indicating the operation content of the terminal device by the person to be managed. (Appendix 8) A work performance verification device according to any one of Supplementary Notes 1 to 7; A work management system comprising: a work management device that acquires work data including start times and end times declared by a person to be managed, and transmits the acquired work data to the work performance verification device. (Appendix 9) The computer Acquire frames constituting video data captured on the floor where the managed person works, Identifying the person to be managed corresponding to a person included in a partial image extracted from the acquired frame and the behavior of the person included in the partial image; Acquire work data including a start time and an end time declared by the identified person to be managed; Verifying the working status of the person to be managed based on the difference between the actual working hours specified in accordance with the behavior of the identified person to be managed and the declared working hours specified by the declared work data; generating verification information including a verification result regarding the working status of the person to be managed; A work performance verification method that outputs verification information including the generated verification result regarding the work status of the person to be managed. (Appendix 10) A process of acquiring frames constituting video data captured on a floor where a person to be managed works; a process of identifying the managed person corresponding to a person included in a partial image extracted from the acquired frame and the behavior of the person included in the partial image; A process of acquiring work data including a start time and an end time declared by the identified person to be managed; A process of verifying the working status of the person to be managed based on the difference between the actual working hours specified in accordance with the behavior of the identified person to be managed and the declared working hours specified in the declared work data; A process of generating verification information including a verification result regarding the working status of the person to be managed; and outputting verification information including the generated verification result regarding the working status of the person to be managed. Furthermore, some or all of the configurations described in Supplementary Notes 2 to 8, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 9 and 10 in the same dependent relationship as Supplementary Notes 2 to 8. Furthermore, not limited to Supplementary Notes 1, 9, and 10, but within the scope of each of the above-mentioned embodiments, some or all of the configurations described as Supplements may also be dependent on various hardware, software, various recording means for recording software, or systems. [Explanation of symbols]

[0161] 10, 20 Work performance verification device 11, 21 Image acquisition unit 12 Object detection unit 13 Face Recognition Unit 14 Behavior detection unit 15, 25 Identification section 16 Work Data Acquisition Department 17 Memory section 18 Verification Department 19 Output section 100 Terminal Device 101 Detection device 113 Partial image acquisition unit 117 Operation information acquisition unit 120 Object Detection AI 123 Partial Image Transmission Unit 130 Face Recognition Engine 140, 142 Behavior detection AI 141 Motion Detection AI 150 Work management device 151 Working Hours Acquisition Department 153 Recording Department 155 Work Data Generation Unit 157 Work Data Output Unit 160 surveillance cameras

Claims

1. an image acquisition unit that acquires frames constituting video data captured on a floor on which a person to be managed works; an identification unit that identifies the managed person corresponding to a person included in a partial image extracted from the acquired frame and the behavior of the person included in the partial image; a work data acquisition unit that acquires work data including a start time and an end time declared by the identified managed person; a verification unit that verifies the working status of the managed person based on the difference between the actual working hours specified in accordance with the identified behavior of the managed person and the declared working hours specified by the declared work data, and generates verification information including the verification result regarding the working status of the managed person; an output unit that outputs verification information including the generated verification result regarding the working status of the person to be managed.

2. The identification unit an object detection unit that detects a person from the frame using object detection AI (Artificial Intelligence) and extracts an image area including the detected person as the partial image; a face authentication unit that authenticates the person to be managed corresponding to the person included in the partial image using a face authentication engine; The work performance verification device according to claim 1 , further comprising: a behavior detection unit that detects the behavior of a person included in the partial image using behavior detection AI.

3. The verification unit If work-related activity is identified during a time period not included in the reported working hours, verification information is generated that includes a verification result indicating that the work data is suspicious; The output unit The work performance verification device according to claim 2 transmits verification information including a verification result indicating that there is doubt about the declared work data to a terminal device used by an administrator who manages the work status of the managed person.

4. The verification unit If a work-related activity by the managed person is identified at a time that deviates from the start time included in the declared work data by more than a reference time, or at a time that deviates from the end time included in the declared work data by more than a reference time, generating verification information including a verification result indicating that there is a doubt about the declared work data, The output unit The work performance verification device according to claim 3 transmits verification information including a verification result indicating that there is doubt about the declared work data to a terminal device used by an administrator who manages the work status of the managed person.

5. The verification unit If an action unrelated to work by the person to be managed is identified during the reported working hours, verification information is generated including a verification result indicating that there is a doubt about the reported work data; The output unit The work performance verification device according to claim 2 transmits verification information including a verification result indicating that there is doubt about the declared work data to a terminal device used by an administrator who manages the work status of the managed person.

6. The behavior detection unit Extracting skeletal data of a person included in the partial image using a motion detection AI that extracts skeletal data of a person included in the image data; The work performance verification device according to claim 2, wherein behavior corresponding to the extracted skeleton data is detected using a behavior detection AI that extracts behavior according to the skeleton data.

7. an operation information acquisition unit that acquires terminal operation information including an operation history of a terminal device used by the person to be managed, The verification unit 3. The work performance verification device according to claim 2, wherein the terminal operation information is used to recognize behavior indicating the operation of the terminal device by the person to be managed.

8. The work performance verification device according to any one of claims 1 to 7, A work management system comprising: a work management device that acquires work data including start times and end times declared by a person to be managed, and transmits the acquired work data to the work performance verification device.

9. The computer Acquire frames constituting video data captured on the floor where the managed person works, Identifying the person to be managed corresponding to a person included in a partial image extracted from the acquired frame and the behavior of the person included in the partial image; Acquire work data including a start time and an end time declared by the identified person to be managed; Verifying the working status of the person to be managed based on the difference between the actual working hours specified in accordance with the behavior of the identified person to be managed and the declared working hours specified by the declared work data; generating verification information including a verification result regarding the working status of the person to be managed; A work performance verification method that outputs verification information including the generated verification result regarding the work status of the person to be managed.

10. A process of acquiring frames constituting video data captured on a floor where a person to be managed works; a process of identifying the managed person corresponding to a person included in a partial image extracted from the acquired frame and the behavior of the person included in the partial image; A process of acquiring work data including a start time and an end time declared by the identified person to be managed; A process of verifying the working status of the person to be managed based on the difference between the actual working hours specified in accordance with the behavior of the identified person to be managed and the declared working hours specified in the declared work data; A process of generating verification information including a verification result regarding the working status of the person to be managed; and outputting verification information including the generated verification result regarding the working status of the person to be managed.

Images