Methods, systems, and computer program products for evaluating the labor efficiency of knowledge work such as software development.

Abstract

The present invention provides a method, system, and computer program product for evaluating the labor efficiency of knowledge work, such as software development work. The method includes the step of extracting and receiving event data from one or more data sources. The event data relates to activities performed by knowledge workers, and event records are obtained. Each event record includes identification information of each worker associated with the event and a time interval or point in time when the event occurred. The method further includes the step of calculating activity records associated with the time intervals within a predetermined period from the event records. The activity records include information about events that occurred based on the event records. The method further includes the step of allocating work effort based on the time intervals of the activity records, associating costs with the activities, and obtaining effort values ​​indicating the amount of effort in each time interval.

Description

Technical Field

[0001] (Reference to Related Applications) This application claims the priority of U.S. Application Serial No. 18 / 192,953, filed on March 30, 2023. The disclosure of the same application is hereby incorporated herein by reference in its entirety.

[0002] At least one embodiment of the present invention relates to a method, a system, and a computer program product for evaluating the labor efficiency of knowledge work, particularly software development work.

Background Art

[0003] In many organizations, labor costs are an important expense. Therefore, it is important to understand whether labor is being performed efficiently in order to minimize the costs required to produce valuable products and services for customers.

[0004] Evaluating labor efficiency in modern knowledge work such as software development is much more difficult than evaluating the efficiency of repetitive and substitutable work such as conventional efficiency analysis in the manufacturing industry. Software development and other knowledge work involve a lot of thinking, communication, testing, and experimentation, so it can be difficult to even grasp what people are working on at a given time.

[0005] One of the solutions that has been adopted for work tracking is to have workers record their working hours and the task content during that time. Although this provides the necessary information, manual input itself is very costly. Furthermore, time recording data tends to be coarse-grained and cannot capture activities in units of minutes such as interruptions due to email correspondence. Finally, this type of collection method is easily affected by human errors in accurately remembering and describing the work content.

[0006] To address the limitations of manual time-tracking solutions, systems have been developed that automatically attribute effort to specific tasks within project management systems. In these systems, workers are assigned tasks with unique identifiers and metadata, indicating the higher-level initiative to which they belong. Aggregating this information allows for the calculation of total effort towards different goals, which can be used to inform accounting decisions, such as determining how much of software development costs or other research and development expenses should be treated as capital expenditures.

[0007] By leveraging data from other sources such as calendars, version control systems, document management systems, and communication systems, it becomes possible to supplement the information within the version control system itself and provide further evidence of who was responsible for what during specific time periods.

[0008] Despite existing methods for identifying assigned tasks, a significant gap remains in understanding whether employees are actually using their time efficiently. Simply being assigned to a specific project does not provide information about the nature of the activities performed during that period. Without this detailed information about the nature of the activities, analytical systems cannot provide information about how much time was used productively or unproductively.

[0009] The following U.S. patents relate to at least one aspect of the present invention: 10,318,248, 10,860,314, 11,188,323, and 11,488,081. [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] An object of at least one embodiment of the present invention is to provide a method, system, and computer program product for evaluating labor or time efficiency by accurately calculating information about work activities in each time interval for each worker. [Means for solving the problem]

[0011] To achieve the above and other objectives of at least one embodiment of the present invention, a method for evaluating the labor efficiency of knowledge work is provided. The method includes the step of extracting and receiving event data from one or more data sources. The data relates to activities performed by knowledge workers, and event records are obtained. Each event record includes identification information of each worker associated with the event, and a time interval or point in time when the event occurred. Furthermore, the method includes the step of calculating activity records associated with the time intervals over a predetermined period from the event records. The activity records include information about events that occurred based on the event records. Furthermore, the method includes the step of allocating work effort based on the time intervals from the activity records, associating costs with the activities, and calculating labor values ​​indicating the amount of effort for each time interval.

[0012] This method may further include a step of classifying the activity records.

[0013] This method may further include steps to supplement the activity record and calculate additional information about the activity based on the amount of effort invested in the activity and the activity metadata.

[0014] This method may further include a step of classifying the augmented activity records.

[0015] This method may further include the step of presenting the classified activity records to the user.

[0016] The categorized activity records may be presented to the user in the form of a report.

[0017] The categorized and enhanced activity records may be presented to the user in the form of a report.

[0018] This method may further include the step of processing the event data to add or modify metadata related to each event and obtaining pre-processed event data.

[0019] The process of generating activity records may be performed using the pre-processed event data.

[0020] The aforementioned knowledge work may also include software development work.

[0021] Furthermore, in order to achieve the above and other objectives of at least one embodiment of the present invention, a system for evaluating the labor efficiency of knowledge work is provided. The system includes at least one hardware processor and at least one storage device connected to the at least one hardware processor for storing instructions that, when executed by the one or more hardware processors, are operable to cause the one or more hardware processors to perform processing. This processing includes receiving event data extracted from one or more data sources. The data relates to activities performed by knowledge workers, and event records are obtained. Each event record includes identification information of each worker associated with the event, and a time interval or point in time when the event occurred. Furthermore, this processing also includes calculating activity records associated with the time intervals over a predetermined period from the event records. The activity records include information related to the events that occurred during or near the time intervals. Furthermore, this processing includes allocating work effort based on the time intervals from the activity records, associating costs with the activities, and obtaining effort values ​​indicating the amount of effort in each time interval.

[0022] This process may further include a process of classifying the activity records.

[0023] This process may further include a process of enhancing the activity records and calculating additional information regarding the activities based on the amount of labor input into the activities and the activity metadata.

[0024] This process may further include a process of classifying the enhanced activity records.

[0025] This process may further include a process of presenting the classified activity records to the user.

[0026] The classified activity records may be presented to the user in the form of a report.

[0027] The classified and enhanced activity records may be presented to the user.

[0028] This process may further include a process of processing the event data to add or modify metadata related to each event and obtaining preprocessed event data.

[0029] The process of generating the activity records may be performed using the preprocessed event data.

[0030] The knowledge work may be software development work.

[0031] Furthermore, to achieve the above and other objectives of at least one embodiment of the present invention, a computer-readable storage medium is provided. The storage medium stores machine-executable instruction programs, thereby performing processing. These processing includes receiving event data extracted from one or more data sources. The data relates to activities performed by knowledge workers, and event records are obtained. Each event record includes identification information of each worker associated with the event, and a time interval or point in time when the event occurred. Furthermore, the processing also includes calculating activity records associated with the time intervals over a predetermined period from the event records. The activity records include information about events that occurred based on the event records. Furthermore, the processing includes allocating work effort based on the time intervals from the activity records, associating costs with the activities, and obtaining effort values ​​indicating the amount of effort in each time interval.

[0032] The aforementioned knowledge work may also include software development work. [Brief explanation of the drawing]

[0033] [Figure 1] This is a block diagram and flowchart illustrating the steps of a method according to at least one embodiment of the present invention. [Figure 2] This is a schematic block diagram showing conventional hardware for performing a step of a method according to at least one embodiment of the present invention. [Figure 3A] This is a partial cross-sectional schematic screenshot showing an exemplary time allocation report provided by at least one embodiment of the present invention. [Figure 3B] This is a partial cross-sectional schematic screenshot showing an exemplary time allocation report provided by at least one embodiment of the present invention. [Figure 3C]This is a partial cross-sectional schematic screenshot showing an exemplary time allocation report provided by at least one embodiment of the present invention. [Figure 3D] This is a partial cross-sectional schematic screenshot showing an exemplary time allocation report provided by at least one embodiment of the present invention. [Modes for carrying out the invention]

[0034] Where necessary, detailed embodiments of the present invention are disclosed herein. However, it should be understood that the disclosed embodiments are merely examples of the present invention, and that the present invention can be carried out in various different forms. The drawings are not necessarily to scale. Some features may be exaggerated or reduced in order to illustrate the details of certain components. Accordingly, certain structural and functional details disclosed herein should not be interpreted restrictively, but merely provide a representative basis for those skilled in the art to carry out the present invention in various forms.

[0035] At least one embodiment of the present invention provides a method, system, and computer program for evaluating the effectiveness of work effort. This is achieved by accurately calculating information about both tasks and activities occurring for each worker in each time interval, and then displaying that information through a user interface that helps visualize the amount of work effort that is adding value to the customer and the amount of work effort that is contributing to the team's speed.

[0036] The remainder of this specification describes methods, systems, and computer programs for evaluating the effectiveness of work labor. Each section outlines the inputs, outputs, and processing steps associated with a particular method process. These, combined, obtain lower-level information about worker activity from various data sources and generate higher-level reports on the efficiency of work time. The sequence of processes is shown in Figure 1.

[0037] Figure 2 shows a block diagram of system hardware components that can be used to implement the various services and processing devices described herein. Bus 110 serves as the main information highway interconnecting the other hardware components shown. CPU 112 is the central processing unit of the system that performs the calculations and logical operations necessary to run one or more programs. CPU 112, either alone or in combination with one or more of the other components disclosed in Figure 2, is the same as the terms used in this disclosure, such as processing device, computing device, or processor. Read-only memory (ROM) 114 and random-access memory (RAM) 216 constitute an exemplary memory device.

[0038] The controller 118 provides an interface between the system bus 110 and one or more tangible, non-temporary, and optional computer-readable memory devices 120. These memory devices 120 may include, for example, external or internal DVD or CD-ROM drives, hard drives, flash memory, USB drives, etc. As mentioned above, these various drives and controllers are optional devices. Furthermore, the memory devices 120 may be configured to include individual files for storing any software modules or instructions, auxiliary data, common files for storing results or groups of auxiliary data, or one or more databases for storing the result information, auxiliary data, and related information described above.

[0039] Program instructions, software, or interactive modules for performing any of the methods and systems described herein are stored in ROM 114 and / or RAM 116. Optionally, program instructions are stored in tangible computer-readable media such as compact disks, digital disks, flash memory, memory cards, USB drives, Blu-ray discs, and / or other recording media.

[0040] An optional display interface 122 allows information from bus 110 to be displayed on display 124 in audio, visual, graphic, or alphanumeric format. Communication with external devices is performed using various communication ports 128. An example of a communication port 128 is connected to a communication network such as the Internet or a local area network.

[0041] The hardware may also include an interface 130 that enables the reception of data from input devices 134 such as a keyboard 132, mouse, joystick, touchscreen, remote control, pointing device, video input device, and audio input device.

[0042] A "computer" refers to a device equipped with a processor and tangible computer-readable memory. Memory stores programming instructions, which, when executed by the processor, cause the computer to perform one or more operations according to those instructions. Examples of computer devices include personal computers, servers, mainframes, game systems, televisions, and portable electronic devices such as smartphones, PDAs, cameras, tablet computers, laptops, and media players.

[0043] A "knowledge worker" refers to a person whose occupation involves the development or application of knowledge. This includes, for example, those engaged in the planning, acquisition, retrieval, analysis, organization, storage, programming, distribution, marketing, and other contributions to information transformation and commercialization, as well as those engaged in utilizing the knowledge thus generated (often the same). Knowledge workers also include those working in the information technology field, such as programmers, system analysts, technical writers, academic experts, and researchers. The term is also frequently used to refer to people in fields outside of information technology.

[0044] 1. Extract event data from the data source. Referring to Figure 1, a method of at least one embodiment of the present invention is schematically shown in 100. The data extraction step 10 of method 100 is performed to obtain event information from a data source. This step 10 may obtain data from various different source types and systems. The following list of data sources is for illustrative purposes only and is not intended to be exhaustive or restrictive. At least one embodiment of the present invention is extendable to benefit from other data sources related to activities performed by an individual.

[0045] The following data source types are common sources used to provide input events in at least one embodiment of the present invention. The data extraction process collects raw data from these systems, such as through APIs or file exports, and generates event records having a start time and end time interval or a specific point in time, and metadata about the events, including personal identification information associated with the event and other information about the type of event. • Project Management / Ticket Tracking Systems - These systems maintain various information about individual activities related to tasks, bugs, etc., in the form of tickets. They have fields indicating when a ticket was assigned to an individual, when a ticket transitioned to an in-progress state, and when it was completed or reassigned. Event records are generated for ongoing time intervals. In addition, individuals can comment on or edit tickets, which constitutes another type of activity at the time the comment or edit was made. • Version control and code review systems - One common event in version control systems is a changeset, or "commit," which represents an update made by an individual at a specific point in time. This is a common type of event. In code review systems, users can perform structured operations such as adding comments and approving or rejecting change requests, which represent other types of events. • Document management systems and office applications – Similarly, systems for editing documents, spreadsheets, design files, and other deliverables often provide data similar to the "commits" in version control systems. This includes information about the changes made and who made them at a specific point in time, with varying levels of granularity depending on the software. More specific information, such as the author's name and the time of the last modification, may also be available. These changes and comments can be handled in the same way as in version control systems and code review systems. • File Management and Backup System - The cloud file storage management system independently records author information, modification time, and version history for each file, regardless of the application used for editing. This can be used as a source of change events for applications that do not provide data or for applications where it is inconvenient to collect data from the file management and backup system. • Calendar and scheduling management systems – These systems hold information such as scheduled meetings, appointments, and outings, which function as events based on time intervals. Note that within the methods described herein, or as a pre-processing step for handling events that are not actual activities, such as "focus time" or "no-meeting blocks," there may be processing steps to select or ignore specific events. • Communication tools (including direct messages, group chats, and email) - In addition to email, various other communication tools have emerged. These typically contain some form of user identification, content, and timestamp associated with the message, which can be used as input events. • Video and audio communication systems – Activity on these systems indicates active human-to-human communication. They can be used to confirm that meetings originated from calendar data sources actually occurred, and they can also independently indicate direct communication activity (including ad-hoc events not present in calendar events). • Device-based activity tracking and keylogging systems – Systems exist that record detailed activity related to keystrokes, mouse movements, and other user inputs at the level of a specific application or the entire computing device. Device agent software can also record activity at the file system level and network level. This data can be used directly as individual point-in-time events or aggregated into time intervals in which multiple user input events occurred. • Time Management and HR Systems – Individuals can manually or automatically record their actual work hours in various ways, with the option to add metadata about specific activities or projects. HR systems also include data on employee vacations and travel. This time data provides input events and helps corroborate the start and end times of other activities. • Paging and Notification Systems – Software engineers and others responsible for handling time-sensitive requests during system failures may receive pages and notifications from paging systems. Furthermore, these systems often have event statuses indicating the time an alert occurred and the time the page was closed by the person who received it. This data can be used to generate event information as point-in-time or interval events, depending on the record regarding the page's closing time. It can also be combined with subsequent point-in-time events, such as code commits or emails, that are thought to be associated with the response to the page or alert. • Physical Location Information - Employee badges and mobile devices may record information about an individual's physical location at a given time, such as through badge swiping or Wi-Fi access point responses. Depending on the job, this may provide various pieces of information about what an individual is doing at a given time and may be used to corroborate or replace other data sources, such as start and end times of work or actual attendance at meetings on the calendar.

[0046] 2. Event preprocessing In one embodiment of the present invention, after events have been extracted from various data sources, it may be beneficial to apply a preprocessing step (block 12 in Figure 1) to the events to add or modify metadata associated with each event.

[0047] In a typical preprocessing step, identifiers representing the same person in different data sources are associated by mapping the identification information in each input data source to a standardized identifier. There are many ways to associate identifiers, and at least one embodiment of the present invention does not depend on a specific identifier association method, but common methods include exact matching of email addresses, or exact or approximate matching of first, last, and middle names. Another possible method is to analyze common elements in identifiers associated with data records in different sources, for example, using a code commit that references a ticket ID, it can be inferred that the code creator identifier and the ticket assignee identifier are likely to be the same person and should therefore be linked.

[0048] Another common preprocessing step is extracting data references and links between different sources. Data records may contain complete URIs to related data in other sources, or they may simply contain tokens in the text that identify records in other sources, such as ticket identifiers placed in the titles or bodies of code change requests or code commit messages.

[0049] By employing more advanced methods, it is possible to infer data relationships between different sources and include this relationship information as a preprocessing step. For example, this can be done by analyzing text and keywords within meeting names, code comments, and ticket descriptions, and determining a high degree of relevance even if explicit URIs or identifier token links are not present.

[0050] Individual events can also be preprocessed independently of events from other data sources. In some embodiments, code and text changes are preprocessed to determine the type and nature of the changes. For example, different event metadata is generated during the preprocessing stage to indicate whether the change set involved deleting existing code, modifying code, or writing new code. Furthermore, the creation date and author of the code being modified can also be factors in determining the type of activity.

[0051] The pre-processing steps for code / text changes may involve further analysis of other attributes of the deleted or added code / text to generate additional metadata about the changes (such as bug fixes, refactoring, new code, whether it is copied code, programming language, framework, changes in complexity metrics, and other attributes). This list is illustrative and not exhaustive, and at least one embodiment of the invention described herein works with any kind of pre-processing method that helps to infer information from an event to determine the nature of its underlying activity.

[0052] 3. Activity calculation from events (Figure 1, Block 14) A. Overview In at least one embodiment of the present invention, time periods are allocated to tasks and activities. This is intended to identify the most appropriate tasks to which an individual's activities belong, even when records obtained from various input data sources are inconsistent, incomplete, or redundant, and to determine metadata associated with the activities most likely to have been performed during a particular time interval.

[0053] B. Input The primary input to this process or method is multiple event records from one or more data sources. Each event record contains identifying information for individuals associated with the event. Multi-person events, such as meetings, can be represented by multiple event records, one for each individual involved.

[0054] This method has an optional auxiliary input for the time period used to calculate the output activity records. If this is omitted, the method may generate output covering a range of activities related to the time period covered by the input event records.

[0055] Each event record must include either a time interval indicating a start and end time, such as the start and end times of a meeting, or a single point in time indicating when the event occurred, such as the time the email was sent.

[0056] Event records may contain other metadata that describes the event's attributes, which are used to determine the time to which the event belongs. For example, indicating the number of words in an email or the number of lines of code in a code changeset is useful in accurately estimating the start time of such a point-in-time event.

[0057] Metadata can also indicate the type of event, which can influence how activity time intervals are calculated. For example, instant messages, on average, take less time to compose than emails, so they may be allocated a shorter time interval. Different types of activities may be assigned different priorities. For example, in a project management system, a ticket marked "in progress" for a week has a lower priority than a meeting taking place that week. Therefore, the meeting's time is allocated to the event record, while the meeting's time is excluded from the time interval of the lower-priority ticket.

[0058] C. Output This method outputs activity records that describe the tasks and activities an individual performed over different periods of time.

[0059] Each record identifies the individuals involved in an activity. If multiple people participate in similar activities (such as attending the same meeting), separate activity records are created.

[0060] Each activity record must specify the activity's time interval, along with its start and end times. The duration of the time intervals in the output activity records may be limited to cover only the periods during which the individual was most likely to have actually worked. This is intended to ensure that the total time across all activity records within a given time window reflects the individual's actual working time.

[0061] However, this method does not need to limit its output to only the time intervals in which an individual is most likely to have actually been working. In another embodiment, it is also possible to output activity records over the entire time range, including nights and weekends, to avoid any gaps in time. Subsequent methods for calculating weighted effort associated with activity time intervals to compensate for fluctuations in working time and non-working time, or work intensity, are described herein.

[0062] This method can generate output activity records indicating "unknown," "missing," or "idle" (hereinafter referred to as "empty time") to show that there was little or no activity during a particular interval, or it can simply create gaps between activity records to indicate periods of no activity or low activity.

[0063] If the entire duration of the option is entered, this method may either truncate time intervals that extend outside that period or create a record that fills in blank time at the beginning or end of the period.

[0064] Each activity record may specify a maximum of one task identifier, and the task identifier may be omitted to indicate that it does not belong to a specific known task. The task identifier links to an identifier in the project management system or ticket tracking system. The task identifier is derived from the input event record associated with the task.

[0065] If an individual is working on multiple tasks during the same period, or if an individual's activities need to be allocated to multiple tasks, this method may output multiple duplicate activity records (one record per task), and optionally, an allocation ratio indicating the proportion of effort to be allocated to each task during the period may be specified. This ratio may total 100%, or less than 100%, leaving a portion of the interval unallocated. If different types of activities occur concurrently within the same task, an optional weighted multiple activity record may be output.

[0066] Finally, this method may include activity metadata related to the estimated activity in the output activity record, or it may omit all activity metadata to indicate "unknown," "missing," or "idle" activities. Activity metadata may include information related to event records that serve as evidence of the activity. It may also indicate the type of activity, such as a meeting, coding, or writing an email.

[0067] This method may output multiple activity records in relation to a single input event record. This is useful for situations where certain types of activities are interrupted by other shorter activities during a given period, such as meetings or emails occurring within a continuous block of work for a task.

[0068] Activity record output may include information related to multiple input event records. For example, if a meeting occurs during a time period when an individual has an ongoing task assigned to them, the record can include activity metadata related to the meeting and distribute that activity to the assigned task.

[0069] Output activity records do not need to include metadata related to input event records that occurred during the time interval specified in the output activity record. This method may refer to event records that occurred before and after the specified time interval for the generation of activity metadata and task allocation. For example, even if an individual does not have any tasks assigned and marked as in progress at the time of a code commit, if a task assigned later that day transitions directly from "to-do" to "done" (i.e., indicating that it was not marked as in progress at the start of the work), the event record related to that latter task state transition may be referenced in the activity metadata and used as the source for the task identifier in the activity record.

[0070] D. Implementation of the method Because the nature of input data from various types of systems and various providers of each type of system is diverse, there are many possible embodiments of this method of generating activity records from input event records. The key attributes of the possible embodiments of this method are described below. First, the nature of common types of input events is described in detail, and then the method of generating output activity records from these events is described.

[0071] <1. Time point change processing> A type of input event record is a time record indicating that an individual created some kind of content. This could be entirely new content (e.g., an email or instant message) or a set of changes based on existing content (e.g., a code commit or a document edit). These types of events share similar characteristics and can be handled in a similar way. These are called time changes.

[0072] First, these time-stamp changes usually have a single author, but there may be multiple authors, for example, when two people are collaboratively editing the code. In this case, multiple event records are generated, each with metadata indicating the relationship.

[0073] These timestamp changes only show activities that occurred before that timestamp, and do not show activities that occurred afterward.

[0074] For a specific point in time change, there are several possibilities regarding when an individual actually spent effort on the change. First, the individual may have started working immediately after the previous event ended and continued working until the point in time change. In that case, the appropriate output activity record interval would be from the end of the previous event to the point in time change.

[0075] Furthermore, there may have been intermediate activities that were not present in any of the input data sources, such as other people stopping by and talking about unrelated topics.

[0076] Furthermore, it's possible that the time point change operation started before the previous event record. This is especially likely with large, infrequent time point changes. For example, it's natural to assume that a code commit with 1000 lines of changes started before the 10-word instant message sent 5 minutes before the commit.

[0077] While the end date of the final segment is clear due to the time shift, determining the appropriate start time for the change work is difficult, including whether or not it was interrupted by other activities. Returning to the instant message example, it's highly likely that the message was sent in the middle of a code commit, but the amount of time that should be allocated to the message is unclear.

[0078] <2. Meeting / Conversation Handling> Another common input event record indicates actual communication between two or more individuals, such as scheduled meetings or impromptu phone calls.

[0079] While this type of event record is usually easy to process because its start and end times are clearly defined, there are a few exceptional cases that require special handling using this method.

[0080] Firstly, it's not always clear whether someone attended a particular meeting, as they may either be absent or attend without formally accepting the invitation. Furthermore, some individuals may receive duplicate meeting invitations.

[0081] Finally, in work meetings where participants are engaged in collaborative activities such as sending emails or editing documents, other activities may be underway during the meeting, potentially leading to changes in the meeting schedule.

[0082] In this embodiment of the method, in such complex situations, it is necessary to determine, if any, the output activity record to be output to the meeting.

[0083] <3. Task Management and Processing of Progress Status> Project management systems and ticket tracking systems can provide data on when a task is assigned to an individual and when a task transitions to a "in progress" or "completed" status.

[0084] If an individual has only one ongoing task assigned to them, this is a strong indicator that it is the most suitable task for allocating an activity record in the output. However, the situation becomes more complex if an individual has multiple ongoing tasks assigned to them, or if they have no ongoing tasks at all.

[0085] In such cases, this method requires determining which task is most likely to be associated with an activity, or whether there are any tasks that should be associated with an activity record.

[0086] <4. Activity tracking or handling of sudden time changes> Activity tracking systems that record user input events to computing devices are extremely reliable sources of information due to their fine granularity. Instantaneous change events, such as instant messages sent at intervals of a few seconds, serve a similar function. Sudden events that occurred within a certain period are highly likely to indicate that the user was actually active and fully focused at that time.

[0087] If this type of information is available, it is likely to be more accurate and comprehensive than data from other sources.

[0088] <5. Time log processing> Time log data is information about work content and time spent, manually reported by individuals. Compared to other data sources, it has both high and low reliability aspects. First, human-recorded time logs may contain information not found in other sources. For example, information about the topic of a one-off phone call and whether it was related to the task at hand. Also, time logs can capture time that does not appear in other data sources, such as the time spent sitting and thinking about how to solve a problem.

[0089] On the other hand, time logs have several inaccuracies. First, because they rely on human memory, human error is likely to occur when recalling the start time of an activity or the content and type of tasks and activities during the time log period.

[0090] Finally, time logs have limited granularity because very short activities, such as quick replies to instant messages or brief conversations with people passing by your desk, are not easily recorded.

[0091] <6. Rule-Based Approach> One way to implement this method is to adopt a rule-based approach that translates input event records into output activity records based on heuristics derived from domain knowledge and policies related to the activity types mentioned above.

[0092] One example of a rules-based approach is to treat all types of code commit events as activity records that begin at the end of the previous code commit, but with the condition that they do not exceed one day and do not go back further than the start time of the current day. This is based on the assumption that individuals typically do not spend more than one day on a single code commit, and that they usually submit a code commit before starting the next task, rather than working on multiple code commits in parallel.

[0093] Rules can be applied to different types of events, prioritizing them or using them to complement each other. For example, meeting intervals may take precedence over time intervals between code commits, because it is assumed that individuals were unlikely to have been working on code commits during meeting times. Time logs that coincide with changes in the status of assigned tickets may take precedence over time intervals derived from code commits at a specific point in time, which would otherwise take precedence.

[0094] Regarding time shifts, if there is no event immediately preceding the establishment of the activity interval, a model of the time required for the time shift may also be adopted in the rules. These models may have fixed values ​​based on the activity type (for example, if there is no preceding event, an instant message is considered a 30-second activity) or they may be variable models that depend on event attributes. For example, the processing time for an email with no preceding event can be modeled as a function based on the number of words.

[0095] The fixed or simply variable models used in this method may be created without data, based on domain expertise, or they may be created as a result of surveys on durations based on general or organization-specific activity types and other variables, or as a result of analyzing other data.

[0096] In one embodiment of this method, users of the method may also set rule-related parameters based on specific circumstances or policies. For example, this could be the case if a user is aware of a unique work practice within their organization that limits the time between code commits, or if an accounting firm sets acceptable time limits before and after a task is marked as in progress when attributing time to a task for research and development tax credit purposes.

[0097] <7. Machine Learning Models> In addition to heuristics and rules that define how to generate activity records from event records, this method may rely in part or in whole on one or more machine learning models.

[0098] In this case, the machine learning model can be trained using either additional data or only existing event record data. In the latter case, the algorithm can model how various metadata associated with code commits affects the time spent on those commits, using only assumption-based constraints (for example, the maximum time spent on a commit is the time elapsed since the previous commit, but the actual commit time is unknown).

[0099] Another approach to training a model is to leverage more detailed data. For example, by asking users to record their work time or by implementing an activity tracking system on their devices, it may be possible to establish reliable training data about the times when various activities related to a specific task were performed. This data can then be used to label lower-granular data (such as code commits, calendar events, and ticket tracking data) and generate a machine learning model. This model can be applied to historical data or other lower-granular data where detailed tracking is not possible.

[0100] User feedback is also a means of training learning algorithms. While it takes various forms, it could involve having users verify the accuracy of specific data points, or manually scrutinizing large datasets and providing correct labels for machine learning algorithms.

[0101] <8. Input data pre-fetching window> In different embodiments of this method, different results may be generated if a certain amount of event record input data is required beyond the end of the output period, or if the data is changed within a certain time frame after the end of the provision period. The size of this time frame is called the look-ahead window.

[0102] For example, if a project management system ticket skips the normal "In Progress" status and goes directly from "Pending" to "Completed," it may be providing activity information from an earlier point in time than the timestamp of the status change event. To improve the accuracy of the methods described herein, it is beneficial to check data for a certain period of time in advance.

[0103] On the other hand, there are also advantages to limiting the time frame during which future events will affect activity records for the current period. For example, this is useful in creating reports used for accounting, where a key requirement is to ensure that those reports are not altered after the books for the period in question have been "closed."

[0104] The size of the look-ahead window effectively sets an upper limit on the amount of time attributable to a time change. This is because if no other events are detected within the look-ahead window, this method may output a different activity record. Events occurring after the end of the current look-ahead window can only affect the activity record within the scope of the look-ahead window set before their occurrence.

[0105] In various embodiments of this method, a fixed or variable lookahead window constrains the output data, but a larger and more varied lookahead window can be used during training to improve the accuracy of the machine learning model.

[0106] <9. Hybrid Approach> While some embodiments of the methods described herein may employ a fully rule-based or fully machine learning-based approach, a hybrid solution may be preferred. In a hybrid approach, certain rules may override model outputs, or the model may be applied to compute time intervals for specific subsets of input data. For example, time intervals to which changes in time are attributed may be calculated based on metadata or individual creators.

[0107] In any case, at least one embodiment of the present invention is not bound by a particular implementation of the method for generating activity records from event records, and is compatible with any implementation of the method.

[0108] 4. Allocating effort to time intervals A. Overview The method described in the previous section outputs activity records associated with time intervals within a period. While this may be useful for knowing when people were performing various activities, associating costs with those activities requires allocating individual efforts to those time intervals. The next step in the method of at least one embodiment of the present invention (i.e., block 16 in Figure 1) is a method for allocating the efforts people have spent to periods.

[0109] <1. Input and Output> The effort allocation method requires only one or more activity time intervals as input. Other metadata associated with the activity time intervals (such as the individual who performed the activity, the type of activity, and related tasks) is already provided by the activity record generation method described above, but is an optional input to this method and may be used to improve accuracy.

[0110] The labor allocation method outputs a value indicating the amount of labor for each of the one or more activity time intervals provided. The labor output value can be expressed in units of work time such as man-hours, as a percentage of labor over a period of time, or in other units useful for representing the allocation of labor resources. A monetary reward value can be derived by multiplying the labor value by the worker's compensation.

[0111] In one embodiment of this method, for each of the one or more input time intervals, there may be one output value representing the proportion of the amount of labor for the entire given interval. This allows the time within a particular time window to be allocated by providing all activities within that time window as input to the labor allocation method. By multiplying the output allocation value by the labor cost in that time window, the cost associated with each input activity interval can be calculated.

[0112] In another embodiment of this method, the output is in units of labor. This method is useful because it does not require looking ahead to future time intervals to determine the labor allocated to the current interval. (In embodiments that output the labor ratio within a time window, the entire time window must be complete in order to know the correct values ​​for preceding intervals within that time window.)

[0113] Even in embodiments that output labor units without lookahead, as long as it is guaranteed that a certain number of units will be output within a predetermined period (e.g., 5 business days every 7 calendar days), it can be used to ensure 100% allocation of labor during a specific period. The output unit can then be multiplied by a reward amount corresponding to the time granularity (e.g., per business day). For 100% allocation to be achieved, the input data must reliably cover 100% of the time on the calendar, without any gaps or overlaps (i.e., time intervals where data does not exist are given as "empty" activity interval inputs to fill in the gaps).

[0114] In another embodiment, the output may consist of a number of work units that vary over time periods, which is suitable for hourly workers, shift workers, or workers with variable compensation.

[0115] In some embodiments, 100% of the labor is reliably allocated to a specific time interval, but this is not a requirement. In at least one embodiment, the system works even if the labor units do not directly correspond to the reward or 100% allocation of labor. Over-allocation or under-allocation of labor can be corrected later by summing up all allocations and dividing by the total, thereby achieving 100% allocation. Alternatively, inaccuracies in allocation can be accumulated and adjusted later (for example, to amortize holiday days evenly over the year). Finally, it is not always necessary to adjust for over-allocation or under-allocation, and in at least one embodiment, a useful output as a whole can still be produced.

[0116] <2. Allocation Modeling> This section describes how this effort allocation method uses a model to calculate effort values ​​for input time intervals.

[0117] (a. Fixed model) A simple implementation would allow for the equal distribution of work effort across all provided time intervals. However, this can lead to undesirable results, as people do not work 24 hours a day, and especially in input intervals that span off-duty hours, they may not actually be working. This can degrade the accuracy of results, especially in input time intervals of less than a few days that span nights or weekends, as these intervals often represent the majority of a working week on the calendar, but in reality, no work is done during this time.

[0118] Another implementation involves hardcoding a rule-based work schedule, such as Monday through Friday from 9 am to 5 pm. This might work if workers adhere strictly to the schedule, but since most modern knowledge workers do not, inaccuracies arise. Furthermore, even if a fixed work schedule accurately reflects actual working hours, this allocation may not accurately reflect the amount of effort being put in, as workers' energy levels fluctuate throughout the day.

[0119] (b. Schedule model) A more accurate effort allocation model can be trained based on historical data to determine the amount of effort to be allocated to each segment.

[0120] One approach to this training is to construct a schedule that reflects the amount of effort previously observed at specific times of day, days of the week, months, and / or years. The granularity of the period can be arbitrary, but for example, the schedule model may include a list of values ​​for each hour of the week. These values ​​indicate the amount of work to allocate to the intervals covering that period, and the sum of the values ​​for the entire 168 hours (24 x 7) of the week becomes the total work time (e.g., 40 hours).

[0121] By using such a scheduling model, this method processes the input time intervals and generates the total amount of work effort for each given interval by multiplying each period within those intervals by the ratio of work time to calendar time.

[0122] There are several ways to train such scheduling models based on past activity from various data sources. One possible approach is to observe events at specific points in time and the density of those events in a given time period in the past to infer the schedule of an individual, team, or organization. The advantage of this approach is that it can naturally account for fluctuations in productivity within working hours.

[0123] Another approach involves using reliable information, such as time logs and activity monitor data, to build schedules based on past activities. The advantage of this method is that it allows you to visualize start times and breaks in work, which cannot be obtained by simply observing point-in-time events.

[0124] It should be noted that the scheduling model does not depend on any information other than the start and end times of the activities. Since the allocation is determined based on the expected work time, knowledge of the nature of the activities is not required to calculate the effort allocation.

[0125] (c. Input Variable Model) Another approach to allocating work effort to activity segments is to consider information from the activity segment metadata rather than simply looking at the schedule.

[0126] In this type of input variable model, the allocation method calculates the amount of effort to allocate to the time interval of an activity, using the type and attributes of that activity.

[0127] For example, activities supported by reliable data regarding whether a user was actually working, such as time logs or activity monitoring data, may be taken into consideration by the model. Work effort may be allocated to segments with reliable data and less or no work effort may be allocated to segments without such data.

[0128] Even when reliable data is unavailable, events such as time changes indicate that an individual was working at least at that point in time, and can therefore be used to allocate workload to time periods surrounding those events, even if the scheduling model typically indicates that work would not be performed during those times.

[0129] All types of input variable models can be implemented using fixed rules, machine learning algorithms, or a hybrid of both. Machine learning (ML) models can be trained using highly reliable data such as work logs and activity monitoring data, or using direct labels / feedback to the system. ML models can be used in a hybrid approach, calculating effort for a subset of activity types or intervals based on fixed rules, or they can be used to fully implement a method for calculating effort for activity interval inputs.

[0130] 5. Reinforcement of activity effort When a method is applied to calculate activity records from event records and allocate effort based on intervals from those activity records, the amount of effort associated with each activity record is obtained.

[0131] At this point, in some embodiments of the present invention, it may be desirable to apply a post-processing augmentation step (block 18 in Figure 1) to calculate more relevant information about the activity based on the amount of labor invested in the activity and the activity metadata. There are many types of augmentation that can be implemented, and in at least one embodiment, it is not dependent on a particular type of augmentation, but some useful types of augmentation are listed below.

[0132] A. Strengthening the time budget One way to reinforce this is to compare the amount of effort associated with one or more activity segments with the time budget associated with a specific task or other activity metadata.

[0133] This time budget is set based on goals such as limiting the number of meetings per week. Alternatively, the time budget may be derived from other activity metadata, such as the number of changed words or changed lines in a change set, based on the estimated time required depending on the size of the change set, and may be set by rules or derived from statistical analysis of historical data.

[0134] Time budgets can also be set based on estimates for specific work tickets and may be expressed in abstract units such as work hours or story points. For estimates, which are in units different from work effort values, the budget can be calculated by setting a target budget threshold based on a statistical analysis of work effort compared to abstract estimates obtained from historical data. For task-related time budgets, all types of activities associated with the task may be considered for budgeting, or only a specific subset of activities obtained through metadata-based filtering may be included, for example, by including only actual work time related to code commits and excluding meetings.

[0135] A method for reinforcing the time budget may involve outputting information about how much of the input activity effort is within budget or over budget, and splitting the input data into multiple outputs representing within-budget activities and over-budget activities.

[0136] B. Strengthening the difficulty level After calculating the effort required for an activity, it may be even more beneficial to apply augmentation methods that calculate the efficiency and difficulty level of the activity based on the effort and other metadata.

[0137] For example, activity metadata may include information about the number of words or lines changed as part of the change set. A high effort per change unit may indicate a more difficult activity compared to an activity with a lower effort cost per change unit.

[0138] In another embodiment, by comparing the amount of work required to the scale of the change set, it is possible to identify changes that are likely to have been automatically generated or copied because they exceed what a human can manually perform with a certain amount of work (e.g., creating a 5,000-word document in 5 minutes).

[0139] The outputs of these difficulty augmentation methods may be used in downstream reports to classify different types of activities. They may also be used to divide activities into multiple segments based on effort allocation methods inferred from effort and activity metadata. For example, a writing activity segment can be divided into writing time and editing time based on the number of words written, effort, and a comparison model of writing time and editing time against word count and elapsed time.

[0140] C. Reinforce the timing Another reinforcement method involves calculating efficiency or efficiency penalties related to the timing and duration of activities.

[0141] The aforementioned method of allocating effort to time intervals assumes that time is largely substitutable and already calculates how much effort could have been allocated to other activities but was consumed by one activity. However, this effort allocation method does not aim to evaluate the degree of productivity actually achieved in relation to the amount of effort consumed. For this purpose, a timing augmentation method can be applied.

[0142] Certain types of activities, such as writing and software development, require concentration, and it is well known that there is a startup period during which the person performing the activity becomes accustomed to the task and productivity decreases. Even short interruptions can negatively impact the working memory associated with the task, potentially impairing productivity for a longer period than the interruption itself would suggest.

[0143] Timing augmentation methods may attempt to account for overhead associated with task initiation time, context switching, or other factors that affect productivity due to the duration or time of day of the work interval.

[0144] The timing augmentation method may generate an output that shows the amount of effort taken into account overhead associated with start / fall times and other timing-based inefficiencies in an activity. Furthermore, an activity may be divided into multiple activities, each representing the time spent on the productive and overhead portions of the task.

[0145] 6. Activity Classification Following the steps of calculating activities from an event, allocating effort to activity time intervals, and post-processing and augmenting activity records as necessary, the next step in at least one embodiment of the present invention (block 20 in Figure 1) is to apply an activity classification method to activity records called augmented activity records, to which data and augmentation attributes have been added.

[0146] While raw data on activity intervals, the amount of effort involved in each, and attributes added through augmentation may be useful in some situations, they are most effectively utilized and aggregated for reporting when applying classification methods to augmented activity records and outputting group keys for each record that are used to aggregate records sharing the same values ​​or to filter specific records.

[0147] Furthermore, in at least one embodiment of the present invention, multiple classification methods may be applied to activity records for the purpose of displaying data along multiple dimensions in a report, or for the purpose of filtering records that have a specific value in one classification method.

[0148] There are many ways to classify activities for various purposes, and at least one embodiment of the present invention functions independently of a specific classification method, but here we will describe a general classification method and its application examples.

[0149] 1.Period Activities are categorized into specific groups based on their duration, which is a common way to classify data. Any size of duration can be used, but common durations include days, weeks, months, quarters, and years.

[0150] 2. Sprint Teams that adopt "sprints" as part of an agile process should ideally categorize and aggregate activities on a sprint basis to facilitate comparisons between sprints.

[0151] 3. Release / Version Some teams use release numbers or versions to define periods in their software development process, either instead of or in addition to sprints, and such teams may categorize their activities in this field.

[0152] 4. Activity Type One basic way to classify activities is to group them directly by the type of activity indicated in the activity metadata within the activity record. Examples of activity types include being out of the office, attending a meeting, coding, and sending emails. Activity types may also be further subdivided based on other attributes, such as attributes added during post-processing. For example, coding time can be divided into normal coding time and coding time exceeding the time budget estimate.

[0153] 5. Individuals, teams, and groups Identifying information of individuals who performed an activity, or groups of multiple individuals grouped by attributes such as team, department, job, length of service, or salary, represent another dimension that can be used for classification.

[0154] 6. Task Identifier It may be desirable to group activities based on assigned task identifiers to understand the amount of activity associated with each task.

[0155] 7. Task Type Another approach to classifying activities is to categorize them based on task attributes such as task type or labels. This helps to understand the amount of activity associated with specific types of work indicated in the task definition, such as bug fixing, refactoring, or planning.

[0156] 8. Board or project A task may be associated with one or more "board" or "project" entities within the project management system, which are independent of other fields within the task. Boards or projects linked to the task identifier in the activity record can be used for classification.

[0157] 9. Task Groups, Epics, and Initiatives Another common concept in project management / task tracking software is "epic" or "initiative." This is essentially a parent container that houses a set of tasks, and may itself be part of a higher-level parent container. Activities may be categorized based on epics or other task groups associated with task identifiers.

[0158] 10. Delete Code vs. Classification Code vs. New Code Attributes associated with code commits, indicating the type of change, can also be used to categorize activities. The amount of effort spent writing new code, deleting or modifying existing code is useful as an indicator of the difficulty of modifying existing code and whether developers are adequately investing in refactoring in maintenance work aligned with organizational goals.

[0159] 11. Code Lifetime Another useful classification method is based on the lifespan of code added or removed by coding activities. Data on the amount and timing of code removed before being merged into another branch indicates the degree of "code replacement" or rework that occurred in a particular coding activity. Furthermore, the percentage of code removed over time after being integrated into the main codebase can be used to classify coding activities by quality; code rewritten in a short period of time is likely to indicate a high number of bugs or other quality issues.

[0160] 12. Merge / Deployment Status The version control system allows you to track which branch coding activity has been integrated into. Furthermore, information from the deployment system allows you to verify whether specific code has been released to the customer.

[0161] It can be helpful to categorize coding activities based on factors such as whether they were merged into a common branch, the time taken to merge, whether they were deployed, and the time taken until the initial deployment.

[0162] 13. Other code attributes Other attributes of the code, such as the programming language and framework used, are also useful for classification. One application of this classification is to track progress toward reducing the amount of work required to address low-quality code by refactoring code in frequently used areas.

[0163] 14. Capital Expenditures vs. Operating Expenses A higher-level activity classification determines whether an activity represents capitalizable effort for accounting purposes or must be classified as an operating expense. There are various rules defining the types of capitalizable activities, and these rules may differ between compliance with generally accepted accounting principles (GAAP) and calculation of R&D tax credits.

[0164] In the activity classification of capital expenditures (CapEx), whether the effort invested in a task should be capitalized is inferred from one or more attributes, or from subcategories such as task classification by epic or task type.

[0165] 15. Value-added activities and non-value-added activities Another higher-level classification method is to determine whether an activity is a value-added activity or a non-value-added activity. The thinking behind this classification is that it may be possible to reduce some or all of the non-value-added activities through improvement without compromising the value delivered to the customer.

[0166] For example, in a software engineering team, time spent on certain types of meetings, filling out expense reports, or fixing bugs may be considered non-value-added.

[0167] The output of this type of method does not have to be binary; it may be a numerical value indicating the amount of value added by the activity, or a limited set of values ​​(e.g., "low", "medium", "high").

[0168] The specific method for classifying activities as either value-added or non-value-added can be defined based on fixed rules derived from domain expertise, or it can be provided as a configuration option by the user of at least one embodiment.

[0169] 16. Velocity Contribution vs. Overhead Kick Another higher-level classification approach is to classify activities based on whether they contribute to the team's "velocity," that is, the team's defined rate of work completion over time.

[0170] This classification considers factors such as whether a task associated with a task identifier was added after the sprint began or completed by the end of the sprint. Additionally, time exceeding the time budget derived from time estimates may be considered overhead rather than contributing to velocity. Depending on the desired outcome, scope creep metrics, such as tasks added to an epic or project beyond its estimated total size, can also be used to indicate activities that do not contribute to velocity.

[0171] To reiterate, the method for classifying activities may be defined based on domain expertise, or it may be set by the system users.

[0172] 7. Activity Report After the augmented activity records are categorized in some way, they are presented to the user in the form of reports via documents, websites, mobile applications, or other media, as shown in step 22 of Figure 1. Alternatively, the results can be provided through an application programming interface (API), making them available to other computer programs. This list of delivery methods is illustrative and not exhaustive, and the final activity information may be reported in any format.

[0173] A. Data aggregation The final step, the activity report, begins with data aggregation. This is similar to creating a pivot table from raw data in spreadsheet software. In this data aggregation step, activities are aggregated along the graph dimensions and, if necessary, grouped into zero or more specified groups based on the output of the classification step. Duration is a commonly used graph dimension, along with other classifications that correlate with time, such as sprints and releases, but any graph dimension can be used.

[0174] The data may be filtered by graph dimensions or classification groups using standard filtering operators to include or exclude specific values, or by using value ranges to specify values ​​to include or exclude in the result set.

[0175] While any field in the augmented activity record may be displayed as a value, users may typically prefer to see aggregate function values ​​such as mean, minimum, maximum, standard deviation, and percentiles for each graph dimension (i.e., for each period), related to one or more of the following: 1. Absolute labor volume, such as full-time equivalent personnel labor or man-days. 2. Labor as a percentage of available labor 3. Total activity time as an absolute value (length of the time interval for aggregation before calculating effort) 4. Total activity time as a percentage

[0176] Furthermore, users may want to report the total number of activity records, or specific counts based on different types of activity metadata properties, such as code commits, pull requests, tickets, and individuals.

[0177] The output of the aggregation process is a series of rows, each consisting of one column for each desired aggregate value, with columns representing graph dimensional values ​​and additional classification group values. This output can be represented in a spreadsheet, exposed through standard methods of providing tabular data via an API, or provided as input for a software user interface or document graphics (such as a graph in a spreadsheet).

[0178] B. Data Presentation In at least one embodiment of the present invention, a data presentation step may be included, if necessary, via documentation or a user interface. This data presentation step is optional if the data is provided via an API or as a plain data file such as a CSV file.

[0179] Data presentation is carried out using any method available in standard spreadsheet applications to display tabular data. Typically, data is presented horizontally along graph dimensions (e.g., time period, sprint, release), with different bars displayed for each graph series value derived from the classification.

[0180] In addition to these standard data presentation methods, in at least one embodiment of the present invention, other presentation methods may be used to more effectively convey certain types of information.

[0181] First, in general, in addition to non-binary classifications such as task type (e.g., bug, feature) or activity type (e.g., meeting, coding), certain higher-level binary classifications such as capital expenditure / operating expenses, value / non-value, and velocity / overhead can be useful for users to review. Figure 3A shows how to display two data dimensions on a graph, displaying graph series values ​​with one binary classification value (velocity contribution) above the vertically centered horizontal axis and the other classification value (overhead) below the horizontal graph axis, making it easy for users to visualize which binary classification each graph series value belongs to.

[0182] Furthermore, users may find it useful to view data associated with classification groups different from those used in the primary visualization in the graph legend. One option to achieve this is to provide a tree user interface that visualizes a subset of data related to a specific group. Figures 3B, 3C, and 3D show individual subgraphs that expand all activities grouped by activity type over time, displaying slices of activity per task for different types of tasks and different epics. Expanding an epic allows viewing activities for a specific task or subtask. The specific classification group displayed in the tree structure is arbitrary, and at least one embodiment of the present invention works regardless of the method or depth of tree expansion. These figures are simply to illustrate how to make data from additional classification dimensions readily available in the user interface.

[0183] While exemplary embodiments are described above, they are not intended to describe all forms of the invention. Rather, the terminology used herein is for illustrative purposes only, not limitation, and it should be understood that various modifications are possible without departing from the spirit and scope of the invention. Furthermore, it is possible to form further embodiments of the invention by combining features of various embodiments.

Claims

1. A method for evaluating the labor efficiency of knowledge work, A step of extracting and receiving event data from one or more data sources and obtaining event records, wherein the data relates to activities performed by knowledge workers, and each event record includes the identification information of each worker associated with the event and the time interval or point in time when the event occurred. A step of calculating an activity record associated with the time interval over a predetermined period from the event record, wherein the activity record includes information about an event that occurred based on the event record. A method comprising the steps of allocating work effort based on the time intervals of the activity record, associating costs with the activities, and obtaining effort values ​​that indicate the amount of effort in each time interval.

2. The method according to claim 1, further comprising the step of classifying the activity records.

3. The method according to claim 1, further comprising the steps of supplementing the activity record and calculating additional information relating to the activity based on the amount of labor invested in the activity and the activity metadata.

4. The method according to claim 3, further comprising the step of classifying the augmented activity records.

5. The method according to claim 2, further comprising the step of presenting the classified activity records to the user.

6. The method according to claim 5, wherein the classified activity records are presented to the user in the form of a report.

7. The method according to claim 4, wherein the classified and enhanced activity records are presented to the user.

8. A method further comprising the steps of processing the event data to add or modify metadata related to each event and obtaining the pre-processed event data.

9. The method according to claim 8, wherein the step of calculating the activity record is performed using the pre-processed event data.

10. The method according to claim 1, wherein the knowledge work is software development work.

11. A system for evaluating the labor efficiency of knowledge work, At least one hardware processor, When connected to at least one hardware processor and executed by one or more hardware processors, A process for extracting and receiving event data from one or more data sources and obtaining event records, wherein the data relates to activities performed by knowledge workers, and each event record includes the identification information of each worker associated with the event and the time interval or point in time when the event occurred. A process for calculating an activity record associated with the time interval during a predetermined period from the event record, wherein the activity record includes information about the event that occurred during or near the time interval. A system comprising at least one storage device for storing instructions that cause one or more hardware processors to perform the following processes: allocating work effort based on the time intervals of the activity record, associating costs with the activities, and obtaining effort values ​​indicating the amount of effort in each time interval.

12. The system according to claim 11, further comprising a process for classifying the activity records.

13. The system according to claim 11, further comprising a process for supplementing the activity record and calculating additional information about the activity based on the amount of labor invested in the activity and the activity metadata.

14. The system according to claim 13, further comprising a process for classifying the augmented activity records.

15. The system according to claim 12, further comprising a process for presenting the classified activity records to the user.

16. The system according to claim 15, wherein the classified activity records are presented to the user in the form of a report.

17. The system according to claim 14, wherein the classified and enhanced activity records are presented to the user in the form of a report.

18. The system according to claim 11, further comprising processing the event data to add or modify metadata related to each event and obtaining the pre-processed event data.

19. The system according to claim 18, wherein the process of generating an activity record is performed using the pre-processed event data.

20. The system according to claim 11, wherein the knowledge work is software development work.

21. It stores a program of machine-executable instructions, A process for extracting and receiving event data from one or more data sources and obtaining event records, wherein the data relates to activities performed by knowledge workers, and each event record includes the identification information of each worker associated with the event and the time interval or point in time when the event occurred. A step of calculating an activity record associated with the time interval over a predetermined period from the event record, wherein the activity record includes a process that includes information about an event that occurred based on the event record. A computer-readable storage medium that performs the following processes: allocating work effort based on the time intervals of the activity record, associating costs with the activities, and obtaining effort values ​​indicating the amount of effort in each time interval.

22. The storage medium according to claim 21, wherein the knowledge work is software development work.

Images