Evaluation systems, evaluation methods, computer programs

Abstract

The objective is to provide an evaluation system that can suitably collect and evaluate the data necessary for evaluation. [Solution] It is an evaluation system, It comprises a management information acquisition unit, a detailed information acquisition unit, and an evaluation unit. The aforementioned management information acquisition unit acquires history management information including the event type, The detailed information acquisition unit acquires detailed information corresponding to the history management information from the data management device that acquired the history management information, based on the history management information relating to the event to be analyzed, which is identified based on the event type. The evaluation unit outputs evaluation information based on the detailed information. Evaluation system.

Description

Technical Field

[0001] The present invention relates to an evaluation system, an evaluation method, and a computer program.

Background Art

[0002] In recent years, workers often perform their work using software tools. In such cases, it has become difficult for users who manage workers to evaluate the working conditions and productivity of workers. Therefore, technologies for evaluating the working conditions of workers have been developed.

[0003] Patent Document 1 discloses a technique for evaluating the capabilities related to the work of members from conversation data in a chat system.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the invention described in Patent Document 1, no disclosure is made regarding a specific method for collecting and evaluating various data necessary for evaluation, such as conversation data in a chat system.

[0006] Therefore, an object of the present invention is to provide an evaluation system capable of suitably collecting and evaluating data necessary for evaluation.

Means for Solving the Problems

[0007] [1] An evaluation system, comprising a management information acquisition unit, a detailed information acquisition unit, and an evaluation unit, The aforementioned management information acquisition unit acquires history management information including the event type, The detailed information acquisition unit acquires detailed information corresponding to the history management information from the tool providing device that acquired the history management information, based on the history management information relating to the event to be analyzed, which is identified based on the event type. The evaluation unit outputs evaluation information based on the detailed information. Evaluation system. [2] The evaluation system according to [1], wherein the detailed information acquisition unit converts the detailed information based on the type of tool from which the detailed information was acquired and the status of the event identified based on the detailed information. [3] The conversion includes a process to remove information not subject to analysis and / or a conversion of the data structure, as described in [2]. [4] The information excluded from analysis includes any of the following: the reply portion, the quoted portion, or special characters, as described in [3]. [5] The history management information and the detailed information are information relating to an event in the communication tool, The aforementioned evaluation information includes an evaluation of the user's communication skills, harassment, or misconduct, as described in any of the evaluation systems in [1] to [4]. [5] The evaluation system according to [1] or [2], wherein the history management information includes one or more identifying pieces of information for uniquely identifying the target event. [7] The evaluation system according to any one of [1] to [6], wherein the evaluation unit identifies history management information relating to an active event based on the history management information and outputs workflow evaluation information relating to the evaluation of the workflow by inputting an instruction statement containing the history management information into the generating AI. [8] The evaluation system further includes an index calculation unit, The aforementioned indicator calculation unit calculates user activity indicators based on historical management information related to active events, The evaluation unit outputs the evaluation information relating to the evaluation of the workflow by inputting an instruction statement including the user's activity indicators into a generating AI, as described in any of [1] to [7]. [9] The instruction further includes the vision and mission of the company to which the user belongs, and the user's position, as described in [7].

[10] The evaluation system described in [7] outputs workflow diagram information by inputting the evaluation information into a generating AI.

[11] The evaluation system further includes a display processing unit, The evaluation unit sends an instruction statement, which further includes output format specification information, to the generation AI to output the evaluation information. The evaluation system according to [1], wherein the display processing unit processes the evaluation information to display it based on the format specified by the output format specification information.

[12] The evaluation system according to

[11] , wherein the display process includes a process of converting the structured evaluation information into data for constructing a web page.

[13] An evaluation method, It comprises a management information acquisition process, a text acquisition process, and an evaluation process. In the aforementioned management information acquisition step, history management information including event type is acquired. In the detailed information acquisition step, based on the history management information relating to the event to be analyzed, which is identified based on the event type, detailed information corresponding to the history management information is acquired from the tool-providing device that acquired the history management information. In the evaluation step, evaluation information is output based on the detailed information. Evaluation method. A computer program that causes a computer to execute the evaluation method described in

[14]

[13] . [Effects of the Invention]

[0008] The present invention provides an evaluation system that can suitably collect and evaluate the data necessary for evaluation. [Brief explanation of the drawing]

[0009] [Figure 1] System configuration diagram in one embodiment [Figure 2]Hardware configuration diagram in one embodiment [Figure 3] Diagram showing an example of data configuration in one embodiment [Figure 4] Diagram showing the processing flow in one embodiment [Figure 5] Diagram showing an example of screen display in one embodiment [Figure 6] Diagram showing an example of screen display in one embodiment [Figure 7] Diagram showing an example of screen display in one embodiment [Figure 8] Diagram showing an example of screen display in one embodiment [Figure 9] Diagram showing an example of screen display in one embodiment

Mode for Carrying Out the Invention

[0010] Hereinafter, it will be described in more detail with reference to the accompanying drawings. The drawings show preferred embodiments. However, it can be implemented in many different forms and is not limited to the embodiments described herein.

[0011] For example, in this embodiment, the configuration, operation, etc. of the evaluation system will be described, but the same effects can also be achieved by the method, apparatus, computer program, etc. executed. The program may be provided as a non-transitory computer-readable recording medium or may be provided so as to be downloadable from an external server.

[0012] An evaluation system according to an embodiment of the present invention collects event history information (history management information, detailed information) of tools used by workers during work, particularly web tools such as web applications, and outputs and visualizes evaluation information, thereby enabling the evaluation of workers' work.

[0013] In this embodiment, event history information (history management information, detailed information) is used for various purposes, such as evaluating work performance, harassment, and misconduct, as will be described later. Event history information is information about the history of events, and includes history management information used for managing event history and detailed information about the body of messages in communication tools. If data such as emails is collected to evaluate workers, and all data from the entire email is acquired for evaluation, then data that is not very relevant to the user being evaluated, such as emails received via CC, will be acquired and used for evaluation. If evaluation is performed based on data that includes such data that does not contribute to the evaluation, the amount of computation may increase, and the accuracy of the evaluation may decrease due to data that is not relevant to the user. Therefore, the evaluation system in this embodiment acquires history management information regarding an overview of event history information, and then performs evaluation by identifying and acquiring the information necessary for evaluation (detailed information in this embodiment) based on the acquired history management information. At this time, the history management information is used as information to identify the events necessary for evaluation, and also as information to acquire information related to the events necessary for evaluation. In this embodiment, the history management information is acquired using the technology described in Japanese Patent No. 7603357.

[0014] The evaluation system in this embodiment acquires detailed information for each event, enabling a unique high-context analysis that takes into account information before and after the interaction.

[0015] Furthermore, the tool used by the evaluation system in this invention to acquire event history information is a web tool such as a web application used by the user for business purposes. In this embodiment, it is a communication tool such as an email application or chat tool, cloud storage, or calendar application, but any tool that is used on the web and for business purposes may be used.

[0016] In this embodiment, the evaluation system uses event history information related to events that occur in the tool, such as receiving and sending emails and chats, creating and editing files in cloud storage, and creating and editing appointments in a calendar application, to perform the processing described later. The event history information collected includes information on active events that occur due to user actions such as sending emails and editing files, and passive events that occur without user action, such as receiving emails and notifications of file editing by other users. When it is desired to understand the user's work status and activities, event history information related to passive events is excluded, and processing is performed using event history information related to active events. In this embodiment, one event history information is acquired for each event. In this embodiment, multiple acquired event history information are used to perform the processing described later.

[0017] <System Configuration> Figure 1 is a block diagram showing the configuration of a system according to one embodiment. As shown in Figure 1, the evaluation system 0 comprises an evaluation device 1, a user terminal 2, a tool provision system 3, and a generation AI management device 4, which are connected to each other via a network NW so as to be able to communicate.

[0018] As evaluation device 1, one or more server devices such as general-purpose servers or personal computers can be used.

[0019] As User Terminal 2, a terminal device such as a personal computer, smartphone, or tablet can be used. User Terminal 2 connects to Evaluation Device 1 by executing an evaluation device utilization program such as a browser application and executes the processing described later. In this embodiment, the evaluation device utilization program is a browser application that is pre-installed or downloaded in advance to User Terminal 2, but it may also be a client application downloaded from a program provisioning device (not shown).

[0020] The tool provision system 3 includes a tool provision device 31. One or more server devices, such as general-purpose servers or personal computers, can be used as the tool provision device 31. In this embodiment, the tool provision device 31 provides event history information related to events generated by the tool to the evaluation device 1. Although not shown in Figure 1, there are multiple tool provision systems 3. In this embodiment, the tool provision system 3 is a system that provides functions for web applications such as email applications, chat tools, cloud storage, and calendar applications. In this embodiment, the evaluation device 1 acquires event history information from the tool provision system 3 via the network NW.

[0021] The generation AI management device 4 is a server device that stores the generation AI. In this embodiment, the generation AI management device 4 is a server device of an external generation AI provision system, but it may also be a device within that system. The generation AI is a trained model that, upon receiving an instruction written in natural language or the like, generates a response that includes natural language and images.

[0022] The network (NW) is an IP (Internet Protocol) network, but there are no restrictions on the type of communication protocol, network type, etc.

[0023] <Hardware Configuration> Figure 2 is a hardware configuration diagram of the evaluation device 1 and user terminal 2. Note that the tool provisioning device 31 and the generation AI management device 4 may have the same hardware configuration as the evaluation device 1 shown in Figure 2(b).

[0024] Figure 2(a) is a hardware configuration diagram of the evaluation device 1. As shown in Figure 2(a), the evaluation device 1 comprises a processing unit 101, a storage unit 102, and a communication unit 103, which are used to ensure the operation of each unit and each process.

[0025] The processing unit 101 has a processor such as a CPU that can execute instruction sets, and executes the OS, evaluation programs, etc. The memory unit 102 includes volatile memory such as RAM capable of storing instruction sets, and non-volatile recording media such as HDDs or SSDs capable of storing the OS, evaluation programs, etc. The communication unit 103 has an interface for connecting to the network NW, and performs communication control with the network NW to input and output information.

[0026] Figure 2(b) is a hardware configuration diagram of user terminal 2. As shown in Figure 2(b), user terminal 2 has a processing unit 201, a storage unit 202, a communication unit 203, an input unit 204, and an output unit 205, which are used to perform the functions of each unit and each process.

[0027] The processing unit 201 has a processor such as a CPU capable of executing instruction sets and executes programs such as the OS and evaluation device usage programs. The memory unit 202 has a volatile memory such as RAM that can store instruction sets, and a non-volatile recording medium such as an HDD or SSD that can record application programs (evaluation device usage programs, etc.) that can utilize multiple services including an OS and evaluation services. The communication unit 203 has an interface for connecting to the network NW, and performs communication control with the network NW to input and output information. The input unit 204 has input devices such as touch panels and keyboards, which are operation input devices capable of input processing. The output unit 205 has an output device such as a display device capable of display processing, such as a display.

[0028] The following describes examples of the data structure of various data used in the processing described later in the evaluation system 0. These data may be stored in the storage unit 102 of the evaluation device 1, or they may be stored in an external database and retrieved when needed.

[0029] User information refers to information about a user in evaluation system 0, and includes identification information (such as a user ID), the user's name, and departmental information related to the department to which the user belongs. In addition, one or more credentials are associated with the user information using the identification information such as the user ID.

[0030] The credentials shown in Figure 3(a) are information for authenticating a user in the tool provision system 3, and include the tool type (tool name), user identification information such as a user ID, and a password. In this embodiment, the credentials include a password, but they do not include a password, or they may be in any format, such as an email address, as long as they are information that can authenticate the user. In this embodiment, if a user is using multiple web tools, multiple credentials of different tool types are linked to the user information. If a user has multiple accounts for a particular tool, multiple credentials for the same tool (of the same tool type) may be linked to a single user information. The evaluation system 0 acquires the event history information of users of other tool provision systems 3 identified by the credentials, as information regarding the tool history of the evaluation system user whose user information is linked to the credentials used for acquisition.

[0031] The history management information shown in Figure 3(b) is information for managing the history of events that occurred in the user's tools, and includes date and time information, tool type, event type, event summary (title), event ID, and target ID. The target ID is information about the target (thread, file, channel, etc.) where the event occurred, and includes the channel name and thread ID. In this embodiment, the event history information is stored in association with the user information of the user associated with the credential information used at the time of acquisition.

[0032] Date and time information refers to information regarding the date and time an event occurred. In this embodiment, date and time information includes information regarding the year, month, day, time, and day of the week.

[0033] The tool type refers to the type of tool used to acquire event information, and in this embodiment, this includes email applications, chat tools, cloud storage, online meeting tools, etc. Note that email application A and email application B may be identified as different tool types.

[0034] The event type is the type of event, and in this embodiment, if the tool type is an email application, the event type is set to be appropriate for each tool type, such as receiving or sending emails, or if it is cloud storage, it is set to be file creation or file editing. In this embodiment, event history information where the event type is receiving emails is classified as event history information for passive events, and event history information where the event type is sending emails is classified as event history information for active events. In this embodiment, event history information is classified into event history information for active events and event history information for passive events based on the event type. In this embodiment, event types are classified into either active events or passive events, but there may be event types that fall into other classifications.

[0035] Event content information is information relating to the content of the event, and in this embodiment, if the tool type is email, it may include the subject of the email, if the tool type is cloud storage, it may include the name of the target of the event, such as the file name.

[0036] An event ID is identification information used to identify an event. In this embodiment, the event ID is used to identify event history information (history management information, detailed information) related to events such as the creation and editing of channels and threads in a chat tool, files in cloud storage, and the sending and receiving of emails displayed on a single screen as a thread in an email application. In this embodiment, the event identification information may be in any format, such as the name of the event target or a unique URL (Uniform Resource Locator), as long as it can identify the target of the event that occurred.

[0037] In this embodiment, the evaluation device 1 acquires event history information from multiple tool-providing devices 31, so the content of the history management information that can be acquired will differ depending on the source tool-providing device 31. Therefore, the evaluation device 1 uses one or more pieces of information (date and time information, tool type, etc.) included in the history management information as identification information to uniquely identify an event, identifies information about the event to be evaluated within the tool-providing device 31, and acquires information about that event as detailed information.

[0038] Detailed information refers to information about events that occurred in the user's tool. In this embodiment, detailed information regarding the communication tool includes text information. In addition, detailed information may include information other than the target information (text information), such as information about a series of interactions (for example, threads related to a series of interactions, or chat spaces and channels where the interactions took place). Therefore, the evaluation system 0 in this embodiment processes the acquired detailed information by removing unnecessary parts so that only the target information (text information, etc., in this embodiment) remains. Text information refers to information about the text of interactions that occurred in the user's tool, and in this embodiment, it refers to information about interactions in the communication tool, including the body of an email and messages in a chat. Note that text information is not limited to the examples described here, as long as it refers to interactions in the communication tool, it may also include, for example, meeting minutes such as online meetings created using AI, or transcripts of online phone calls. In addition, text information may include data of the same items as history management information, such as date and time information and identification information, in order to confirm whether it corresponds to history management information.

[0039] In this specification, information relating to events that occur in a user's tool, including history management information and detailed information, is described as event history information. Event history information (history management information, detailed information) obtained using a specific user's credentials in a specific user's tool is stored in the storage unit 102 in association with user information linked to the credentials used during acquisition. In this embodiment, the evaluation system 0 performs the processing described later using the processed event history information, which has undergone preprocessing to be converted into a format that is easy to handle within the system.

[0040] <Functional Configuration> Figure 1 further shows the functional configuration of the evaluation system 0. The evaluation device 1 comprises a management information acquisition unit 11, a detailed information acquisition unit 12, an index calculation unit 13, an evaluation unit 14, and a display processing unit 15. This is a system in which software processing (evaluation programs stored transiently or permanently in the storage unit 102) is concretely realized by hardware (processing unit 101, etc.). In addition, a user terminal 2 may have some of the functional configuration described later and execute some of the processing. In this case, the user terminal 2 stores some of the evaluation programs in the storage unit 202 and executes some of the processing of the evaluation device 1 in the processing unit 201.

[0041] <Processing Flow> The following explanation will use Figure 4 to describe the processing flow in the management information acquisition unit 11 and the detailed information acquisition unit 12.

[0042] The management information acquisition unit 11 acquires history management information from the tool providing device 31 (S101). The detailed information acquisition unit 12 acquires detailed information based on this history management information (S102). In this embodiment, the detailed information acquisition unit 12 uses credentials and an email address to acquire detailed information. The detailed information acquisition unit 12 performs preprocessing on the acquired detailed information, such as cleansing to remove unnecessary data (S103).

[0043] <Management information acquisition unit 11> The management information acquisition unit 11 acquires history management information from one or more tool provision systems 3 based on user information and associated credentials. In this embodiment, the management information acquisition unit 11 stores the acquired history management information in the storage unit 102. The management information acquisition unit 11 acquires history management information about events that occurred in a specific user's web tool using user information and associated credentials. The management information acquisition unit 11 may also acquire data from the tool provision system 3 periodically, or it may be configured to acquire data when an event occurs. In this embodiment, the management information acquisition unit 11 particularly acquires history management information about active events such as sending emails and creating / modifying files in order to evaluate user activity.

[0044] <Detailed information acquisition part 12> The detailed information acquisition unit 12 acquires detailed information (detailed information about the event to be analyzed) corresponding to the history management information, based on the history management information of the event to be analyzed, which is identified based on the event type. An event to be analyzed is an event that occurs in the tool due to user activity, such as sending an email or chat, and the data is used to evaluate the user's activity. In this embodiment, the detailed information acquisition unit 12 acquires corresponding detailed information based on the history management information of the event to be analyzed (e.g., sending an email or chat) identified based on the event type, for the purpose of evaluating communication. The evaluation unit 14 may use the tool type included in the history management information to identify the event to be analyzed and acquire detailed information for a specific event in a specific tool (e.g., sending a message in a chat tool) as the event to be analyzed. Note that the event to be analyzed may be determined for each item to be evaluated. For example, when evaluating overall work ability, all events related to work in all tools used in work, such as sending emails or creating documents, may be considered events to be analyzed, and information related to these events may be acquired for evaluation.

[0045] In this embodiment, the detailed information acquisition unit 12 identifies an event caused by the activity of a specific user whose history is stored in the tool provider 31, based on credentials, email address, and history management information of the identified event to be analyzed, and acquires detailed information about that event.

[0046] In this embodiment, the detailed information acquisition unit 12 identifies the history management information related to the event to be analyzed (such as an email sending event) from a plurality of history management information stored in the storage unit 102 based on the event type, generates a request to acquire detailed information corresponding to that history management information, and obtains detailed information about the event to be analyzed by sending this request to the tool provision system 3. The request sent to the tool provision system 3 at this time is generated based on the user's credentials and history management information and includes information to uniquely identify the detailed information to be acquired (for example, credentials, tool type, date and time information, identification information, etc.).

[0047] In this embodiment, the acquisition unit (management information acquisition unit 11, detailed information acquisition unit 12) performs processing to convert the data acquired from the tool provision system 3 into data suitable for use in the evaluation system 0. The acquisition unit may also perform the conversion based on conversion rules for each tool type.

[0048] Furthermore, in this embodiment, the detailed information obtained from the tool provider 31 has a different data structure depending on the source tool. Therefore, the detailed information acquisition unit 12 in this embodiment performs a process to convert the detailed information based on the type of tool from which the acquired detailed information was obtained. In addition, in the case of email, the data structure changes depending on the situation, such as whether there are recipients other than the recipient (recipients set as CC), whether it is a forwarded email (whether the subject contains "Forward"), or whether it is a reply email. Therefore, the detailed information acquisition unit 12 determines the converted data structure and the range of information that is unnecessary for analysis (information not subject to analysis) based on the situation of the event to be analyzed identified based on the acquired detailed information (for example, whether there are recipients other than the recipient, whether it is a forwarded email, etc.). At this time, the detailed information acquisition unit 12 performs a process related to the conversion of detailed information using conversion rules that correspond to the tool type and the situation of the event. Information not subject to analysis by the detailed information acquisition unit is information that does not contribute to the evaluation of user activity, and in this embodiment, it includes the reply portion of an email, the quoted portion when quoting other messages, other messages in the thread, and special characters.

[0049] In this embodiment, the detailed information acquisition unit 12 identifies the reply portion based on the acquired detailed information, using the date and time information included in the history management information, the sending date and time stated in the email, and tool-specific rules (for example, a reply follows the string ">" or "|"). The unit then performs a process to remove the reply portion. The detailed information acquisition unit 12 also performs a process to remove portions of the detailed information that satisfy conditions based on pre-set rules, such as the area of ​​special characters.

[0050] Furthermore, even when the goal is to obtain information about a single event (for example, a single message), depending on the tool, the information sent based on the request may not be information about the single event being retrieved, such as a series of threads. In such cases, the detailed information acquisition unit 12 identifies the portion of the detailed information related to the event being analyzed based on the detailed information obtained from the tool provider 31, including information such as the date and time the message was sent and the sender, the date and time information included in the history management information and the sender information identified by credentials, and the processing rules specific to each tool, and then processes the system to remove any other unnecessary information.

[0051] The detailed information acquisition unit 12 may store the acquired detailed information in the storage unit 102 in association with the corresponding history management information, or, in order to protect the information, it may not store the detailed information in the server device of the system, and instead acquire the detailed information each time it is needed using the history management information.

[0052] <Indicator calculation unit 13> The indicator calculation unit 13 calculates activity indicators based on event history information. In this embodiment, the indicator calculation unit 13 calculates activity indicators based particularly on history management information. Activity indicators are indicators related to user activity based on event history information, and in this embodiment, include activity volume (number of active events), number of emails sent, number of unreplied messages, message reply time, and time taken. The indicator calculation unit 13 may also calculate activity indicators for a user based on event history information linked to user information, or it may calculate activity indicators for a group of multiple users based on event history information of multiple users, such as users belonging to a specific department. Furthermore, the indicator calculation unit 13 calculates activity indicators based particularly on event history information related to active events.

[0053] <Evaluation Section 14> The evaluation unit 14 outputs evaluation information based on the acquired event history information. In this embodiment, the evaluation unit 14 also sends an instruction statement containing event history information to the generation AI management device 4 and uses the response output by the generation AI as evaluation information. The evaluation unit 14 may use the response output by the generation AI as is as evaluation information, or it may perform calculations or conversions on the values ​​included in the response and output evaluation information. The evaluation unit 14 outputs evaluation information using event history information stored in association with the user information of the user being evaluated.

[0054] Furthermore, the evaluation unit 14 in this embodiment outputs evaluation information related to the evaluation of communication (communication evaluation information) and evaluation information related to the evaluation of workflow (workflow evaluation information). In addition, when outputting this evaluation information, information regarding the premise, such as the philosophy (vision and mission) of the company to which the user being evaluated belongs, and information about the user, such as the user's department and job title, may be input to the generating AI to output the evaluation information.

[0055] <Regarding the output of communication evaluation information> The following describes an example of outputting evaluation information related to communication. The evaluation unit 14 sends an instruction sentence containing detailed information to the generation AI management device 4 and outputs evaluation information. At this time, the evaluation unit 14 outputs evaluation information related to communication by having the generation AI input an instruction sentence that includes detailed information such as messages in the communication tool and instructions in natural language such as "Please evaluate this message on a scale of 1 to 10 for logic, clarity, and professionalism." At this time, the evaluation unit 14 obtains evaluation information by having the generation AI input one or more combinations of history management information and detailed information related to one event (message transmission). Furthermore, the outputted evaluation information may be stored in the storage unit 102 in association with the history management information and detailed information used for its output, so that it is possible to identify which message was evaluated and what evaluation it received.

[0056] In this embodiment, the evaluation unit 14 estimates communication ability and communication-related risks based on detailed information about the communication tool. In the example described later, the evaluation unit 14 estimates risks, particularly harassment and fraudulent accounting, but it may generate evaluation information for any item as long as it is information about items that can be evaluated using the detailed information and the generated AI.

[0057] The evaluation unit 14 detects and evaluates power harassment, sexual harassment, maternity harassment, paternity harassment, moral harassment, technology harassment, remote harassment, smell harassment, social harassment, and gender harassment in the evaluation of harassment.

[0058] The evaluation unit 14 detects and evaluates fraud, particularly accounting fraud, including barter transactions, through transactions, circular transactions, embezzlement, falsified financial statements, related-party transactions, inflated sales and expenses, fictitious transactions, kickbacks, unfair trading, insider trading, and deferred expenses. Specifically, the evaluation unit 14 identifies and evaluates fraudulent statements such as informal exchanges like, "I'd rather talk to you directly about this than by email," unnatural mentions of deadlines or figures like, "There's not much time until the closing date. Please take immediate action to increase sales by another XX yen," shifting responsibility or avoiding instructions like, "I'll leave the method up to you, but achieving the target is essential," and audit-conscious statements like, "Please handle this carefully so it doesn't cause problems during the audit." The evaluation unit 14 then evaluates the risk on a 10-point scale.

[0059] The communication evaluation information in this embodiment includes communication quality, communication efficiency, logic, clarity, professionalism, conciseness, harassment risk, fraudulent accounting risk, text length, and evaluation comments, but may also include other information not described herein, as long as it is a result of evaluating communication abilities and risks. The values ​​output by the generating AI may be used directly as the communication evaluation values ​​(communication evaluation values) for communication quality, communication efficiency, logic, clarity, professionalism, conciseness, harassment risk, fraudulent accounting risk, etc., or values ​​calculated by adding, subtracting, or other operations on the values ​​output by the generating AI may be used as the communication evaluation values. The evaluation unit 14 may also aggregate evaluation values ​​for a predetermined period, or it may aggregate them by department. Furthermore, the evaluation value for each message may be output as the evaluation value. In addition, the communication evaluation information in this embodiment includes the confidence level of the evaluation output by the generating AI and the reason for the evaluation.

[0060] Furthermore, the evaluation unit 14 in this embodiment outputs evaluation information relating to individual communications based on individual messages, and evaluation information relating to communications over a predetermined period, which is output based on a plurality of messages that occurred during a predetermined period.

[0061] Furthermore, the evaluation unit 14 detects high-risk interactions based on detailed information. At this time, the evaluation unit 14 inputs an instruction statement, which includes detailed information and natural language instructions such as "Please determine whether this message poses a high risk of harassment or fraudulent accounting," into the generating AI, and detects high-risk interactions.

[0062] Furthermore, the evaluation unit 14 may generate an evaluation report related to communication based on the detailed information. The evaluation report related to communication is a report that describes the evaluation of communication written in natural language. The evaluation report related to communication in this embodiment includes the evaluation value for each indicator, the confidence level of the evaluation, and the reason for the evaluation.

[0063] <Regarding the output of workflow evaluation information> The following describes an example of outputting evaluation information related to the workflow. The evaluation unit 14 identifies history management information (activity history) related to active and significant events from the history management information, and sends an instruction statement based on this activity history to the generation AI management device 4 to cause the generation AI to output evaluation information related to the workflow (workflow evaluation information). The evaluation unit 14 may also send an instruction statement for performing a workflow evaluation based on the activity indicators calculated by the indicator calculation unit 13 to the generation AI management device 4 to cause the generation AI to output evaluation information related to the workflow (workflow evaluation information). In this case, the evaluation unit 14 outputs evaluation information related to the workflow by having the generation AI input an instruction statement that includes the activity history and a natural language instruction such as, "Please evaluate the workflow of the target user from these indicators." In addition, the evaluation unit 14 may output workflow evaluation information by using an instruction statement that includes information such as the job title of the user to be evaluated and event history information (history management information, detailed information) during the evaluation period in order to generate workflow evaluation information.

[0064] The workflow evaluation information in this embodiment includes workflow quality, workflow efficiency, commitment and professionalism, risk of overwork, security threats, employee turnover risk, and an evaluation report on the workflow as shown in Figure 7. The workflow evaluation information may also include other factors, such as workflow risks, as long as they are the results of an evaluation of the workflow. Furthermore, the workflow evaluation values ​​(workflow evaluation values) for workflow quality, workflow efficiency, commitment and professionalism, risk of overwork, security threats, and employee turnover risk may be the values ​​output by the generating AI, or values ​​calculated by adding or subtracting from the values ​​output by the generating AI may be used as workflow evaluation values. The evaluation unit 14 may also aggregate evaluation values ​​for a predetermined period, or it may aggregate them by department.

[0065] The workflow evaluation report is a report that describes the evaluation of the workflow in natural language. In this embodiment, the workflow evaluation report includes issues (bottlenecks) related to workflow and productivity, proposed workflow improvements, an evaluation of professionalism (commitment), and action plans by job title, but the items to be evaluated are not limited to those described herein.

[0066] Furthermore, the evaluation unit 14 transmits the acquired workflow evaluation information to the generation AI and outputs workflow diagram information. The output workflow diagram information is information about a workflow diagram related to the work performed by the user being evaluated using the tool, and in this embodiment, it includes information about the current workflow and the improved workflow. In addition, the generation AI in this embodiment outputs structured workflow diagram information based on the input output format specification information, but it may also output the diagram showing the workflow itself as workflow diagram information. Structured workflow information is information about a graph expressed in a format that allows the diagram to be represented in text, such as information about a graph written using Mermaid notation.

[0067] The instruction sent to the generation AI includes output format specification information regarding the output format. In this embodiment, the evaluation unit 14 acquires evaluation information in the specified format, such as structured evaluation information, based on this output format specification information. By adopting this configuration in this embodiment, the amount of processing and data exchanged can be reduced compared to outputting in formats such as HTML.

[0068] In outputting the evaluation information described above, the evaluation unit 14 sends an instruction message containing data about the user being evaluated (event history information, activity indicators, and evaluation information) to output the evaluation information and workflow. Furthermore, when conducting an evaluation over a predetermined period, the evaluation unit 14 sends an instruction message containing data corresponding to the evaluation period, such as multiple event history information acquired within the predetermined evaluation period and activity indicators based on multiple event history information within the predetermined period.

[0069] The evaluation information and workflow output by the evaluation unit 14 are stored in the storage unit 102 in association with the user information of the user being evaluated. In addition, evaluation values ​​aggregated for a predetermined period or department, and evaluation information generated based on multiple event history information during a predetermined period, may be stored in association with the unit of evaluation (e.g., department or period).

[0070] In this embodiment, the evaluation unit 14 uses a generation AI stored in an external generation AI management device to perform processing related to the output of evaluation information and workflow output. However, evaluation information and workflow may also be output using methods other than those described herein, such as a trained model that has been trained to output evaluation information.

[0071] <Display Processing Unit 15> The display processing unit 15 displays various screens based on various necessary data, including event history information and evaluation information. When the display processing unit 15 obtains data in a predetermined format, such as structured evaluation information, it displays it in the corresponding format.

[0072] Furthermore, the display processing unit 15 in this embodiment performs display processing when it receives information in a predetermined format, such as a structured evaluation report. This display processing may include processing to convert the information into web page construction data such as HTML or CSS.

[0073] <Explanation of screen display example> The following describes examples of screens displayed by the display processing unit 15 using Figures 5-8. Note that the examples described here are specific to this embodiment, and display processing may be performed based on other information.

[0074] Figure 5 shows an example of the display of the communication evaluation information display screen W1. The communication evaluation information display screen W1 includes a search condition input area W11, a bar graph display area W12, a status display area W13, a weekly evaluation value display area W14, an individual evaluation value display area W15, and a line graph display area W16. The display processing unit 15 processes the display of the communication evaluation information display screen W1 as shown in Figure 5 based on user information, event history information (history management information, detailed information), and evaluation information.

[0075] The search condition input area W11 is an area for inputting the range of information to be displayed on the communication evaluation information display screen W1. In this embodiment, it includes an input field where conditions including department, user, and period (start date, end date, start time, end time) can be entered, and a search button. After the search conditions are entered in the search condition input area W11, when the search button is pressed, the display processing unit 15 processes the communication evaluation information display screen W1 based on the evaluation information output based on the event history information of the set range. When the search button is pressed, processing such as the acquisition of data that satisfies the conditions by the acquisition unit (management information acquisition unit 11, detailed information acquisition unit 12) or the output of evaluation information based on data that corresponds to the specified conditions by the evaluation unit 14 may be performed.

[0076] The bar graph display area W12 is an area for displaying communication evaluation values ​​as a bar graph, and in this embodiment, the communication evaluation values ​​aggregated for each department are displayed. In the example shown in Figure 5, the values ​​for communication quality and communication risk (harassment risk and fraudulent accounting risk in this embodiment) are displayed in the graph, but other communication evaluation values ​​may also be displayed in the graph.

[0077] Status display area W13 is an area for displaying each user's communication evaluation value. In the example shown in Figure 5, the graph has communication quality on the horizontal axis and communication efficiency on the vertical axis. In addition, the icon of each user is displayed at the position where points are plotted based on each user's communication evaluation value, but the user's name may also be displayed.

[0078] The weekly evaluation value display area W14 is an area for displaying a list of communication evaluation values ​​for a predetermined period (1 week). In this embodiment, the period, department, user, communication quality, communication efficiency, logic, clarity, professionalism, conciseness, harassment risk, fraudulent accounting risk, and text length are displayed. When the string "View" is pressed, the evaluation information of the evaluated user for the evaluation period may be displayed in a list. When "View" is pressed and the screen is switched, information with the same target user and period, such as User A's event history information for 2025 / 11 / 18, will be displayed. In the weekly evaluation value display area W14 shown in Figure 5, weekly evaluation values ​​are displayed, but communication evaluation values ​​for periods other than those exemplified here, such as monthly values, may also be displayed.

[0079] The individual evaluation value display area W15 is an area for displaying communication evaluation values ​​for individual events (message sending). In this embodiment, it displays the date and time, department, user, tool, target, communication quality, communication efficiency, logic, clarity, professionalism, conciseness, harassment risk, fraudulent accounting risk, and text length. However, other information not described here, such as the time the event occurred, may also be displayed. The tool displayed here is the tool used to send the message being evaluated. The target is the location (thread, channel) where the message was sent, or the target where the event being evaluated (e.g., sending a message) occurred. When the word "View" is pressed, the communication evaluation report display screen W5, as shown in Figure 9, is displayed as an evaluation report for the target event.

[0080] The line graph display area W16 is the area that displays each user's communication evaluation value as a line graph over time. In the example shown in Figure 5, communication evaluation values ​​related to communication quality and communication efficiency are displayed over time, but for example, evaluation values ​​related to communication risk (fraudulent accounting risk, harassment risk) may also be displayed over time. Also, in the example shown in Figure 5, evaluation values ​​every 7 days are displayed over time, but periods other than those shown in Figure 5 may be used as the unit period for displaying over time, such as daily.

[0081] Figure 6 shows an example of the workflow evaluation information display screen W2. The workflow evaluation information display screen W2 includes a search condition input area W21, a bar graph display area W22, a status display area W23, a monthly evaluation value display area W24, a weekly evaluation value display area W25, and a line graph display area W26. The display processing unit 15 processes the workflow evaluation information display screen W2 as shown in Figure 6 based on user information, event history information (history management information, detailed information), and evaluation information.

[0082] The search condition input area W21, like the search condition input area W11 shown in Figure 5, is an area for inputting the range of information that will be displayed on the workflow evaluation information display screen W2.

[0083] The bar graph display area W22 is an area for displaying workflow evaluation values ​​as a bar graph, and in this embodiment, workflow evaluation values ​​aggregated for each department are displayed. In the example shown in Figure 6, evaluation values ​​related to workflow quality and work style risks (overwork risk, employee turnover risk, security threat) are displayed, but workflow evaluation values ​​other than those shown here may also be displayed in the graph.

[0084] Status display area W23 is an area for displaying workflow evaluation values ​​for each user. In the example shown in Figure 6, the graph has workflow quality on the horizontal axis and workflow efficiency on the vertical axis. In addition, the icon of each user is displayed at the position where points are plotted based on each user's communication evaluation value, but the user name may also be displayed.

[0085] The monthly evaluation display area W24 and the weekly evaluation display area W25 are areas for displaying a list of workflow evaluation values ​​for a predetermined period (one week and one month in the example in Figure 6), and in this embodiment, these include period, department, user, workflow quality, workflow efficiency, commitment and professionalism, risk of overwork, security threats, and turnover risk.

[0086] The line graph display area W26 is the area that displays each user's workflow evaluation values ​​as a line graph over time. In the example shown in Figure 6, evaluation values ​​related to workflow quality and the risk of overwork are displayed, but evaluation values ​​other than those shown in Figure 6, such as the risk of employee turnover or security threats, may also be displayed. In addition, in the example shown in Figure 6, evaluation values ​​are displayed in 7-day intervals over time, but periods other than those shown in the example in Figure 5 may be used as the unit period for displaying over time, such as daily.

[0087] In this embodiment, when the word "View" is pressed in the monthly evaluation value display area W24, the display switches to the workflow evaluation report display screen W3, as shown in Figure 7. Similarly, when the word "View" is pressed in the weekly evaluation value display area W25, the display switches to the workflow display screen W4, as shown in Figure 8. When "View" is pressed and the screen switches, the same information for the target user and period, such as evaluation information and workflow for user D from 2025 / 11 / 03 to 2025 / 11 / 10, is displayed.

[0088] Furthermore, in this embodiment, when a check is made on the checklist related to department, job type, or user, the items of information displayed in the weekly evaluation value display area W14, the individual evaluation value display area W15, the monthly evaluation value display area W24, and the weekly evaluation value display area W25 are changed. At this time, if a check is made on the department and user, the department and user are displayed, and so on, the checked items are displayed.

[0089] Figure 7 shows an example of the workflow evaluation report display screen W3. The workflow evaluation report display screen W3 is a screen for displaying workflow evaluation information, and in particular, a screen for displaying evaluation reports. In this embodiment, the workflow evaluation report display screen W3 is a screen that can switch and display workflow evaluation information for multiple periods. When a screen element indicating a period, such as "2025 / 12 / 20-2025 / 12 / 27", is pressed, the workflow evaluation information corresponding to that period is displayed. The display processing unit 15 processes the display of a screen as shown in Figure 7 based on the workflow evaluation information.

[0090] Figure 8 shows an example of the workflow display screen W4. The workflow display screen W4 is a screen for displaying the workflow related to the activities of the user being evaluated, and in this embodiment, it is a screen that displays the current workflow and the improved workflow.

[0091] Figure 9 shows an example of the display of the communication evaluation report display screen W5. The communication evaluation report display screen W5 is a screen for displaying communication evaluation information, and in particular a screen for displaying the communication evaluation report. In this embodiment, for each specific event (email sending), the evaluation value, confidence level, and evaluation reason for each communication evaluation item are displayed. The display processing unit 15 processes the display of the screen shown in Figure 9 based on the communication evaluation information.

[0092] <Examples> The following describes examples of verification conducted by the inventors. While the present invention is based on the following experimental results, the evaluation items are not limited to harassment or accounting fraud, and the results and other values ​​are not limited to the following forms.

[0093] In the verification described below, messages (email, chat messages, etc.) were input into the evaluation system, and the results of determining whether or not there was a risk were compared with the correct answer (the risk category and the risk level within that category). The results shown in paragraphs 0094 to 0097 below are the results obtained when the generation AI, whose evaluation accuracy was confirmed by unit testing, was deployed in the evaluation system, and test data was sent via communication tools such as email for evaluation.

[0094] In a test using 288 data points for verifying harassment and accounting fraud risks (data mixing high-risk messages that could constitute harassment or accounting fraud with low-risk, normal messages), the AI ​​successfully identified approximately 97% of the cases correctly. In particular, the accuracy rate for cases identified as "risky" by the AI ​​was 99%. The following provides a detailed explanation of the verification results regarding the detection of harassment and accounting fraud.

[0095] [1] Verification of harassment Based on 288 data points, an examination of harassment detection revealed 70 true negatives (24.3%), 208 true positives (72.2%), 8 false negatives (2.8%), and 2 false positives (0.7%).

[0096] [2] Verification of accounting fraud Based on 288 data points, an examination of fraud detection revealed 68 true negatives (23.6%), 214 true positives (74.3%), 2 false negatives (0.7%), and 4 false positives (1.4%).

[0097] By comparing the results of the verification described above with the general passing criteria, it was demonstrated that AI-based detection has a certain degree of reliability in detecting risks such as harassment and fraudulent accounting in messages. The following shows a comparison between the general passing score and the results of verification using the system in question. While the typical passing score for accuracy is 80-90%, the accuracy rate in our verification test was an exceptionally high 96-98%. While the general acceptable threshold for the failure rate is 10-15% or less, the failure rate in our verification was 1-4%, indicating a low risk in practical operation. While a typical acceptable false positive rate is around 20-30%, the false positive rate in our verification process was a low 1-2%, suggesting improved work efficiency.

[0098] Furthermore, in paragraph 0099 below, we will show the results of actual operational tests evaluated by the evaluation system on real-world messages generated by business activities, rather than test data, at a time separate from the integration tests described above. Based on 858 data points, the results showed 854 true negatives (99.5%), 2 true positives (0.2%), 0 false negatives (0.0%), and 2 false positives (0.2%).

[0099] Based on the results of unit and integration tests, it was confirmed that the system exhibits a high accuracy rate, no missed detections, and an extremely low false positive rate (1-2%) in actual production environments. Therefore, it was confirmed that this system is an evaluation system that is easy to deploy in real-world situations, as only the notified content needs to be verified in actual operation. The verification described above confirmed that the evaluation system is a system that contributes to operational efficiency by allowing personnel to concentrate only on cases that the AI ​​has determined to be "risky." [Explanation of Symbols]

[0100] 0 Evaluation System 1. Evaluation device 11 Management information acquisition department 12 Detailed information acquisition section 13 Indicator calculation section 14. Evaluation Department 15 Display Processing Unit 2 User terminals 3. Tool Provision System 31 Tool-providing device 4 Generation AI management device

Claims

1. It is an evaluation system, It comprises a management information acquisition unit, a detailed information acquisition unit, and an evaluation unit. The aforementioned management information acquisition unit acquires history management information, which includes the event type and is information relating to the summary of the event history information, from a tool providing device that holds event history information relating to events that occurred in the web tool used by workers to perform their duties. The detailed information acquisition unit identifies the history management information related to the event to be analyzed based on the event type, and acquires the detailed information necessary for evaluation from the tool providing device regarding the event history information corresponding to the history management information. The evaluation unit outputs evaluation information relating to the evaluation of work, which includes at least one of the following: the evaluation of the worker's work ability, harassment, or misconduct, by transmitting an instruction statement containing the detailed information to the generating AI. Evaluation system.

2. The detailed information acquisition unit transforms the detailed information based on the type of tool from which the detailed information was acquired, the status of the event identified based on the detailed information, and a transformation rule corresponding to the type of tool and / or the status of the event. The evaluation system according to claim 1.

3. The aforementioned transformation includes a process to remove information that is not subject to analysis and / or a transformation of the data structure. The evaluation system according to claim 2.

4. The aforementioned information excluded from analysis includes any of the following: replies, quoted portions, or special characters. The evaluation system according to claim 3.

5. The aforementioned history management information and the aforementioned detailed information are information relating to an event in the communication tool, The aforementioned evaluation information includes an evaluation of the user's communication skills, harassment, or misconduct. The evaluation system according to claim 1.

6. The aforementioned history management information includes one or more identifying pieces of information for uniquely identifying the target event. The evaluation system according to claim 1.

7. The evaluation unit identifies history management information related to active events based on the history management information, and outputs workflow evaluation information related to workflow evaluation by inputting an instruction statement containing the history management information to the generation AI. The evaluation system according to claim 1.

8. The evaluation system further includes an index calculation unit, The aforementioned indicator calculation unit calculates user activity indicators based on historical management information related to active events, The evaluation unit outputs workflow evaluation information related to workflow evaluation by inputting an instruction statement including the user's activity indicators to the generating AI. The evaluation system according to claim 1.

9. The aforementioned instructions further include the vision and mission of the company to which the user belongs, and the user's position. The evaluation system according to claim 7.

10. By inputting the aforementioned evaluation information into the generating AI, workflow diagram information is output. The evaluation system according to claim 7.

11. The evaluation system further includes a display processing unit, The evaluation unit sends an instruction statement, which further includes output format specification information, to the generating AI to output the evaluation information. The display processing unit processes the evaluation information to display it based on the format specified by the output format specification information. The evaluation system according to claim 1.

12. The display process includes a process of converting the structured evaluation information into data for constructing a web page. The evaluation system according to claim 11.

13. An evaluation method performed by a computer, It comprises a management information acquisition process, a detailed information acquisition process, and an evaluation process. In the aforementioned management information acquisition step, from a tool providing device that holds event history information related to events that occurred on a web tool used by workers to perform their duties, history management information is acquired, which includes the event type and is information relating to an overview of the event history information. In the detailed information acquisition step, historical management information relating to the event to be analyzed is identified based on the event type, and detailed information necessary for evaluation is acquired from the tool provider device for the event history information corresponding to the historical management information. In the evaluation process, an instruction document containing the detailed information is sent to the generating AI to output evaluation information related to the evaluation of the worker's work performance, including at least one of the evaluations of harassment and misconduct. Evaluation method.

14. A computer program that causes the computer to execute the evaluation method described in claim 13.

Images