system

Abstract

To provide a system.SOLUTION: A system includes: first means for collecting voice data when interviewing a user; second means for generating a prompt sentence for outputting a summary of a literal record of a user at the time of an interview on the basis of text data converted from the voice data; third means for generating the summary of the literal record by inputting the generated prompt sentence to a regenerative AI model; and fourth means for notifying the user of the generated summary of the literal record.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional creation of interview records has problems such as a large burden on the line length and difficulty in concentrating on observing non-verbal information during interviews. In addition, the environment for efficiently utilizing interview records and conducting continuous employee training has not been established.

Means for Solving the Problems

[0005] By developing an interview implementation support tool that utilizes generative AI and implementing it in a personnel management system, the creation and summarization of verbatim records of interviews are automated, and the interview records are stored in the personnel management system. As a result, the burden of creating interview records with a long line length is reduced, and it becomes possible to concentrate on observing non-verbal information during interviews. Furthermore, by automatically reflecting it in the ToDo list and calendar, an environment for conducting continuous employee training by utilizing the implementation records is established. [Brief explanation of the drawing]

[0006] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 1 of Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2 of Embodiment 2. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2. [Figure 15] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 3 of Example 3. [Figure 16] This is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3. [Figure 17] This is a sequence diagram showing the processing flow of the data processing system in Example 1 of the Form 1 when an emotion engine is combined. [Figure 18] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Modes for carrying out the invention]

[0007] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0008] First, let's explain the terminology used in the following explanation.

[0009] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (TENSOR PROCESSING UNIT®).

[0010] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0011] In the following embodiments, the tagged storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0012] In the following embodiments, the tagged communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0014] [First Embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0019] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0027] "Example of form 1"

[0028] One embodiment of this invention involves developing an interview support tool utilizing generative AI and implementing it in a human resources management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript. When the interview ends, the generative AI summarizes the transcript and stores the summary in the human resources management system. It also has a function to automatically reflect the content of the interview in a to-do list and calendar. As a result, the burden of creating interview records is reduced for line managers, allowing them to concentrate on observing nonverbal information during the interview.

[0029] "Example of form 2"

[0030] Another embodiment of the present invention involves creating an environment for continuous employee development by utilizing implementation records. Specifically, interview records are stored in a human resources management system, and these records are used to design employee skill development and career paths. For example, employee strengths and weaknesses are extracted from interview records, and individual development plans are created based on these. In addition, employee interests and aspirations are understood from interview records, and career paths are proposed based on these. This makes it possible to provide development tailored to each individual employee.

[0031] The following describes the processing flow for each example of the form.

[0032] "Example of form 1"

[0033] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[0034] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the summary in the HR management system.

[0035] Step 3: It also includes a function that automatically updates the to-do list and calendar based on the content of the interview.

[0036] "Example of form 2"

[0037] Step 1: Store interview records in the HR management system and use them to design employee skill development and career paths.

[0038] Step 2: Extract the employee's strengths and weaknesses from the interview records and create an individualized training plan based on that.

[0039] Step 3: Understand the employee's interests and aspirations from the interview records, and propose a career path based on that.

[0040] (Example 1)

[0041] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0042] Traditional interview record-keeping is often done manually, placing a significant burden on line managers and supervisors. Furthermore, the lack of time to observe nonverbal information during interviews can lead to a decline in interview quality. Additionally, the difficulty in efficiently managing interview content and reflecting it in to-do lists and calendars can result in insufficient follow-up after interviews.

[0043] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0044] This invention includes a server that develops and implements an interview support tool utilizing generative AI into a human resources management system; a server that uses generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system; a server that collects audio data at the start of an interview and transmits it to the server; a server that uses speech recognition software to convert the audio data into text data; a server that uses a generative AI model to summarize the verbatim transcript; a server that stores the summarized verbatim transcript in a database; a server that automatically updates the to-do list and calendar based on the interview content; and a server that notifies the user of the summarized verbatim transcript and update information. This reduces the burden of creating interview records and allows for focus on observing nonverbal information during interviews. Furthermore, because interview content can be efficiently managed and automatically reflected in the to-do list and calendar, follow-up after interviews becomes easier.

[0045] "Generative AI" refers to a system that uses artificial intelligence technology to generate and process data.

[0046] An "interview support tool" is a combination of software or hardware designed to assist with the progress and record-keeping of interviews.

[0047] A "human resources management system" is an integrated system for managing employee information, evaluation, and training.

[0048] A "verbatim transcript" is a record that transcribes the audio of an interview or meeting exactly as it was spoken.

[0049] A "summary" refers to a concise compilation of detailed information, such as verbatim transcripts.

[0050] "Speech recognition software" is software used to convert speech data into text data.

[0051] "Generative AI model" refers to the core algorithms and pre-trained models of generative AI.

[0052] A "database" is a system for efficiently storing, managing, and retrieving data.

[0053] A "to-do list" is a list used to manage tasks and appointments in a list format.

[0054] A "calendar" is a tool for managing appointments and events based on dates and times.

[0055] A "terminal" refers to a device such as a computer or smartphone that is directly operated by the user.

[0056] A "server" is a computer system used to process and store data over a network.

[0057] "Nonverbal information" refers to information conveyed through means other than words, such as facial expressions and gestures.

[0058] A "notification" is a message or alert sent from a system to a user to inform them of information.

[0059] This invention develops an interview support tool utilizing generative AI and implements it into a human resources management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript, and when the interview ends, it summarizes the transcript and stores the summary in the human resources management system. It also includes a function to automatically reflect the content of the interview in a to-do list and calendar.

[0060] Hardware and software to be used

[0061] server

[0062] The server provides computing resources to run generative AI models. Specifically, it uses speech recognition software to convert audio data into text data and a generative AI model (e.g., OpenAI®'s GPT-4®) to summarize the verbatim transcript. The server also stores the summarized verbatim transcript in the human resources management system database and automatically updates to-do lists and calendars based on the interview content.

[0063] terminal

[0064] The terminal is a device used by the user to conduct interviews. When an interview begins, the terminal uses its built-in microphone to collect audio data and transmits it to the server in real time. When the interview ends, the terminal receives a summarized transcript and updated to-do list / calendar information from the server and notifies the user.

[0065] User

[0066] The user is the person conducting the interview. The user initiates the interview via their device and does not need to perform any operations during the interview. Once the interview is complete, the user reviews the summarized transcript via their device and manages updates to their to-do list and calendar.

[0067] Specific example

[0068] Example of a prompt

[0069] "Please summarize the following interview content: 'Hello, today I'd like to discuss your work progress. First, could you tell me about the progress of your project last week?'"

[0070] Processing flow

[0071] 1. Start of interview: The user launches the dedicated application on their device and clicks the "Start Interview" button.

[0072] 2. Voice data collection: The device uses its built-in microphone to collect voice data and transmits it to the server in real time.

[0073] 3. Transcript creation: The server uses speech recognition software to convert the audio data into text data.

[0074] 4. Summary generation: The server uses a generation AI model to summarize the verbatim transcript.

[0075] 5. Data storage: The server stores the summarized verbatim transcripts in the human resources management system database.

[0076] 6. To-Do List & Calendar Update: The server automatically updates the to-do list and calendar based on the interview content.

[0077] 7. User Notifications: The device notifies the user of summarized verbatim transcripts and update information.

[0078] This system reduces the burden of creating interview records, allowing you to focus on observing nonverbal information during interviews. Furthermore, it efficiently manages interview content and automatically reflects it in to-do lists and calendars, making post-interview follow-up easier.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] Interview begins

[0082] Subject: User

[0083] The user launches a dedicated application on their terminal and clicks the "Start Interview" button. This causes the terminal to send an interview start request to the server. The input is the user's action, and the output is the interview start request. Specifically, the terminal displays "Interview started."

[0084] Step 2:

[0085] Audio data collection

[0086] Subject: terminal

[0087] The terminal begins collecting audio data using its built-in microphone as soon as the interview starts. The collected audio data is sent to the server in real time. The input is the user's voice, and the output is the audio data sent to the server. Specifically, the terminal displays "Collecting audio data" and shows the collection status in real time.

[0088] Step 3:

[0089] Transcript creation

[0090] Subject: Server

[0091] The server receives audio data sent from the terminal and converts it into text data using speech recognition software. The input is audio data, and the output is text data. Specifically, the server displays "Creating verbatim transcript" and updates the progress in real time.

[0092] Step 4:

[0093] Summary generation

[0094] Subject: Server

[0095] The server uses a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The input is the text data of the verbatim transcript, and the output is the summarized text. Specifically, the server displays "Generating summary" and updates the progress in real time.

[0096] Step 5:

[0097] Data storage

[0098] Subject: Server

[0099] The server stores the summarized verbatim transcript in the human resources management system's database. The input is the summarized text, and the output is the record stored in the database. Specifically, the server displays "Storing data" and, once storage is complete, displays "Data storage complete."

[0100] Step 6:

[0101] To-do list / calendar update

[0102] Subject: Server

[0103] The server automatically updates the to-do list and calendar based on the interview content. The input is a summarized verbatim transcript, and the output is the updated to-do list and calendar. Specifically, the server displays "Updating to-do list and calendar" and "Update complete" when the update is finished.

[0104] Step 7:

[0105] User notifications

[0106] Subject: terminal

[0107] The terminal receives summarized verbatim transcripts and updated to-do list / calendar information from the server and notifies the user. The input is the notification information from the server, and the output is the notification to the user. Specifically, the terminal displays "Summary and update information received" and notifies the user.

[0108] (Application Example 1)

[0109] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0110] Traditional interview support tools often require manual tasks such as creating and summarizing interview records and updating to-do lists and calendars, which is time-consuming and labor-intensive. Similar problems exist in interviews between factory workers and robots, making efficient work instructions and schedule management difficult.

[0111] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for developing an interview support tool utilizing generation AI and implementing it in the personnel management system, means for using generation AI to create verbatim transcripts and summaries of interviews and storing the interview records in the personnel management system, means for automatically reflecting them in to-do lists and calendars, means for supporting interviews between workers and robots in the factory, and means for automatically updating work instructions and schedules based on the interview content. As a result, the burden of creating interview records is reduced, and efficient work instructions and schedule management become possible.

[0112] "Generative AI" refers to a system that uses artificial intelligence technology to generate data and automate specific tasks.

[0113] An "interview support tool" is a software or hardware system designed to assist with the progress and record-keeping of interviews.

[0114] A "human resources management system" is a system used by companies and organizations to manage employee information and to perform labor management and talent development.

[0115] A "verbatim transcript" is a document that meticulously records the content of conversations during interviews, meetings, or other similar events.

[0116] A "summary" is a document that extracts the key points from detailed records, such as verbatim transcripts, and presents them concisely.

[0117] A "to-do list" is a list of tasks or work items that need to be done.

[0118] A "calendar" is a tool for managing appointments and events based on dates and times.

[0119] A "worker" is an employee who actually performs work in a factory or on a construction site.

[0120] A "robot" is a mechanical device that automatically performs tasks according to programmed instructions.

[0121] "Work instructions" are specific instructions or orders for performing a particular task.

[0122] A "schedule" is a plan of tasks or events that should be completed within a specific period of time.

[0123] As an embodiment of this invention, a system is constructed to support interviews between factory workers and robots. The specific method of implementation is shown below.

[0124] System Configuration

[0125] 1. Hardware Configuration

[0126] Microphone: Used to collect interview audio.

[0127] Computer: Used to process audio data and run generative AI models.

[0128] Robot: An automated machine or device that performs tasks within a factory.

[0129] 2. Software Configuration

[0130] OpenAI API: Used for creating verbatim transcripts and generating summaries of audio.

[0131] Python (registered trademark): A programming language used for data processing and file manipulation.

[0132] Human Resources Management System: A system for storing interview records and summaries and reflecting them in to-do lists and calendars.

[0133] Data processing and data calculation

[0134] 1. Collection of audio data and creation of verbatim transcripts.

[0135] The server collects the interview audio through the microphone.

[0136] The collected audio data is converted to text using the OpenAI API, and a verbatim transcript is created.

[0137] 2. Summary of the verbatim transcript

[0138] The server summarizes the generated verbatim transcript using the OpenAI API.

[0139] The summarized text is stored in the human resources management system.

[0140] 3. Reflection in To-Do list and calendar

[0141] The server extracts key tasks from the summarized text and adds them to a to-do list.

[0142] Additionally, the schedule will be updated based on the interview content and reflected in the calendar.

[0143] Specific example

[0144] For example, if during an interview someone says, "We need to check the next maintenance schedule," that statement will be recorded in the verbatim transcript. After the interview, the server will summarize the transcript and generate a summary titled "Check the next maintenance schedule." This summary will be stored in the HR management system, and "Check the next maintenance schedule" will be automatically added to the ToDo list.

[0145] Example of a prompt

[0146] Examples of prompt sentences for summarizing verbatim transcripts are as follows:

[0147] Please summarize the following verbatim transcript:

[0148] [Verbatim text]

[0149] summary:

[0150] The following is an example of a prompt message for extracting items from a to-do list.

[0151] Extract the to-do list items from the following summary:

[0152] [Summary text]

[0153] To-do list:

[0154] In this way, it becomes possible to efficiently support meetings between factory workers and robots, and to automate work instructions and schedule management.

[0155] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0156] Step 1:

[0157] The server collects interview audio via a microphone. The input is the interview audio, and the output is audio data. Specifically, the microphone captures the audio during the interview in real time and sends that audio data to the server.

[0158] Step 2:

[0159] The server converts the collected audio data into text using the OpenAI API and creates a verbatim transcript. The input is audio data, and the output is the verbatim transcript text. Specifically, the audio data is input into the OpenAI speech recognition model, and the generated text is saved as a verbatim transcript.

[0160] Step 3:

[0161] The server summarizes the generated verbatim transcript using the OpenAI API. The input is the verbatim transcript text, and the output is the summarized text. Specifically, prompt sentences for summarizing the verbatim transcript text are input to the OpenAI text generation model, and the generated summary is obtained.

[0162] Step 4:

[0163] The server stores the summarized text in the human resources management system. The input is the summarized text, and the output is the summarized data stored in the human resources management system. Specifically, it performs the operation of saving the summarized text to the human resources management system's database.

[0164] Step 5:

[0165] The server extracts important tasks from the summarized text and adds them to a to-do list. The input is the summarized text, and the output is the updated to-do list. Specifically, the summarized text is input to an OpenAI text generation model as prompts for task extraction, and the generated tasks are added to the to-do list.

[0166] Step 6:

[0167] The server updates the schedule based on the interview content and reflects it in the calendar. The input is a summary text, and the output is the updated calendar. Specifically, it extracts schedule-related information from the summary text and adds that information to the calendar system.

[0168] Step 7:

[0169] The user checks their updated to-do list and calendar and performs the necessary tasks. The input is the updated to-do list and calendar, and the output is the user's completed tasks. Specifically, the user logs into the system, checks their to-do list and calendar, and performs the instructed tasks.

[0170] (Example 2)

[0171] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0172] Traditional human resource management systems often required manual processes for creating and storing interview records and developing employee training plans, resulting in time-consuming and labor-intensive tasks. Furthermore, they lacked sufficient opportunities to identify employees' strengths and weaknesses based on interview records, create individualized training plans, and propose career paths, making effective, personalized employee development challenging.

[0173] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0174] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that utilizes generative AI to create verbatim transcripts and summaries of interviews and stores the interview records in the human resources management system; a server that analyzes the interview records to extract the strengths and weaknesses of employees; a server that creates individual training plans based on the extracted data; a server that proposes career paths based on the interests and aspirations of employees; and a server that automatically reflects the results in to-do lists and calendars. As a result, the creation and storage of interview records are automated, and the extraction of employee strengths and weaknesses, the creation of individual training plans, and the proposal of career paths can be carried out efficiently, enabling effective training tailored to each employee.

[0175] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[0176] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[0177] A "human resources management system" is software or a system used to manage employee information and support human resources tasks such as training and performance evaluation.

[0178] "Transcript creation" is the process of recording the content of an interview exactly as it was said.

[0179] "Summarizing" is the process of concisely summarizing the content of an interview.

[0180] An "interview record" is data that records the content of an interview.

[0181] "Storage" is the process of saving data and making it accessible as needed.

[0182] "Analysis" is the process of analyzing data and extracting useful information.

[0183] "Strengths" refer to the skills and abilities that employees excel at.

[0184] A "weakness" refers to the skills or abilities that an employee struggles with.

[0185] A "training plan" is a plan designed to support employees in improving their skills and advancing their careers.

[0186] "Interest" refers to the areas or activities that employees are interested in.

[0187] "Orientation" refers to the direction and goals that employees strive for.

[0188] A "career path" is the outline of the jobs and roles that an employee should pursue in the future.

[0189] A "to-do list" is a list of tasks that need to be done.

[0190] A "calendar" is a tool for managing appointments and schedules.

[0191] This invention develops an interview support tool that utilizes generative AI and implements it into a human resources management system to efficiently create and store interview records, develop employee training plans, and propose career paths. A specific embodiment of this system is described below.

[0192] Hardware and software to be used

[0193] Hardware: Servers, terminals (PCs, tablets, etc.)

[0194] Software: Human resource management systems, generative AI models (e.g., OpenAI GPT-4)

[0195] Program processing

[0196] 1. Entering interview records

[0197] After the interview, the user uses a device (PC or tablet) to enter the interview record into the human resources management system.

[0198] Specifically, the user accesses an input form in the human resources management system and enters the interview details in text format.

[0199] Example: "The user uses a terminal to input the interview record for employee A. Strengths: project management, weaknesses: presentation skills."

[0200] 2. Storage of interview records

[0201] The server receives interview records submitted by users and stores them in the human resources management system's database.

[0202] Specifically, the server converts the input data into the appropriate format and saves it to the database.

[0203] Example: "The server receives the interview record and saves it to the database."

[0204] 3. Data Extraction and Analysis

[0205] The server analyzes the stored interview records to extract the employee's strengths and weaknesses.

[0206] Specifically, the server inputs the interview records into a generated AI model (e.g., OpenAI GPT-4) and retrieves the analysis results.

[0207] Example: "The server inputs interview records into a generated AI model and extracts employee A's strengths and weaknesses."

[0208] 4. Creating a training plan

[0209] The server creates individual training plans based on the extracted data.

[0210] In terms of specific operations, the server uses the analysis results of the generated AI model to suggest training and workshops necessary for improving employees' skills.

[0211] Example: "The server proposes project management training, leveraging employee A's strengths. To address their weaknesses, it plans training to improve their presentation skills."

[0212] 5. Proposed Career Paths

[0213] The server suggests appropriate career paths based on employees' interests and aspirations.

[0214] Specifically, the server uses the analysis results of the generated AI model to design future career paths for employees and reflects the proposed content in the human resources management system.

[0215] Example: "Since employee A is interested in project management, the server suggests a career path to becoming a project manager in the future."

[0216] Specific example

[0217] Example of interview record entry:

[0218] "The user uses a terminal to input interview records for employee B. Strengths include data analysis, weaknesses include teamwork."

[0219] Examples of prompts for a generative AI model:

[0220] "Based on the following interview record, identify employee B's strengths and weaknesses and propose an appropriate training plan. Interview record: Strengths: data analysis, Weaknesses: teamwork."

[0221] This system automates the creation and storage of interview records, enabling efficient identification of employees' strengths and weaknesses, creation of individual training plans, and proposal of career paths, thereby facilitating effective training tailored to each employee.

[0222] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0223] Step 1:

[0224] Entering interview records

[0225] After the interview, the user uses a device (PC or tablet) to enter the interview record into the human resources management system.

[0226] Input: Interview content (text format), employee's strengths and weaknesses, and other interview information.

[0227] Specific operation: The user accesses an input form in the HR management system and enters the interview details in text format. For example, they might enter, "Entering interview record for employee A. Strengths: project management, Weaknesses: presentation skills."

[0228] Output: The entered interview record is sent to the HR management system.

[0229] Step 2:

[0230] Storage of interview records

[0231] The server receives interview records submitted by users and stores them in the human resources management system's database.

[0232] Input: Interview records submitted by the user.

[0233] Specific operation: The server converts the input data into the appropriate format and saves it to the database. For example, the server receives an interview record and saves "Employee A's Interview Record" to the database.

[0234] Output: Interview records stored in the database.

[0235] Step 3:

[0236] Data extraction and analysis

[0237] The server analyzes the stored interview records to extract the employee's strengths and weaknesses.

[0238] Input: Interview records stored in the database.

[0239] Specific operation: The server inputs interview records into a generating AI model (e.g., OpenAI GPT-4) and obtains the analysis results. For example, the server inputs "Employee A's interview records" into the generating AI model and extracts strengths and weaknesses.

[0240] Output: Employee strengths and weaknesses as analyzed.

[0241] Step 4:

[0242] Creating a training plan

[0243] The server creates individual training plans based on the extracted data.

[0244] Input: Employee strengths and weaknesses as analyzed.

[0245] Specific operation: Based on the analysis results of the generated AI model, the server proposes training and workshops necessary for improving employees' skills. For example, the server might create a training plan such as, "Leverage employee A's strengths and propose project management training. To address their weaknesses, plan training to improve their presentation skills."

[0246] Output: Individual training plan.

[0247] Step 5:

[0248] Career path proposals

[0249] The server suggests appropriate career paths based on employees' interests and aspirations.

[0250] Input: Employees' interests and preferences as analyzed.

[0251] Specific operation: Based on the analysis results of the generated AI model, the server designs future career paths for employees and reflects the proposed content in the human resources management system. For example, the server might propose a career path such as, "Since employee A is interested in project management, we propose a career path to become a project manager in the future."

[0252] Output: Proposed career paths.

[0253] (Application Example 2)

[0254] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0255] Traditional HR management systems made creating and managing interview records cumbersome, leaving line managers with little time to observe nonverbal information during interviews. Furthermore, there was a lack of systems capable of efficiently collecting and analyzing robot work records to propose optimal work assignments and improvement plans. This made it difficult to improve employee and robot skills and to efficiently allocate tasks.

[0256] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0257] This invention includes means for the server to develop an interview support tool utilizing generative AI and implement it in a human resources management system; means for using generative AI to create verbatim transcripts and summaries of interviews and store the interview records in the human resources management system; means for automatically reflecting the results in to-do lists and calendars; means for collecting robot work records and analyzing efficiency and error rates; means for identifying the robot's strengths and weaknesses and proposing optimal work assignments; and means for creating individual improvement plans. As a result, the burden of creating interview records is reduced, line managers can concentrate on nonverbal information, and it becomes possible to analyze the robot's work efficiency and error rates and propose optimal work assignments and improvement plans.

[0258] "Generative AI" refers to a system that uses artificial intelligence technology to generate data and automate or assist with specific tasks.

[0259] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[0260] A "human resources management system" is a system that manages employee information and streamlines human resources tasks such as evaluation, training, and placement.

[0261] "Vertical transcript creation" is the process of recording conversations, such as interviews and meetings, exactly as they happened.

[0262] A "summary" refers to a condensed version of a long text or conversation.

[0263] A "to-do list" is a tool for managing tasks and appointments in a list format.

[0264] "Automatically add to calendar" is a feature that automatically adds specific events or tasks to your calendar.

[0265] "Robot work log" refers to a detailed record of the tasks performed by the robot.

[0266] "Efficiency" is an indicator that shows how effectively a task or process is performed.

[0267] The "error rate" is an indicator that shows how often errors occur in a task or process.

[0268] A "specialized task" refers to a task that a particular robot can perform with higher efficiency than other tasks.

[0269] A "task that a robot is not good at" refers to a task that a particular robot can perform with lower efficiency than other tasks.

[0270] "Optimal work arrangement" refers to positioning robots and employees in a way that allows them to perform their tasks most efficiently.

[0271] An "improvement plan" refers to a specific plan aimed at improving work efficiency and reducing error rates.

[0272] The system for implementing this invention develops an interview implementation support tool that utilizes generative AI and implements it in the human resource management system. Specifically, it uses generative AI to create a verbatim record and summary of the interview, and stores the interview record in the human resource management system. It also includes a function to automatically reflect it in the ToDo list and calendar.

[0273] Furthermore, it also includes a function to collect the work records of robots and analyze the efficiency and error rate. By doing so, it can identify the tasks that robots are good at and those they are not good at, and propose an optimal work arrangement. It also includes a function to create individual improvement plans.

[0274] Hardware and Software to be Used

[0275] Hardware: Server, factory robot, user terminal

[0276] Software: Python, JSON, generative AI model

[0277] Data Processing and Data Calculation

[0278] The server creates a verbatim record of the interview using the generative AI model and summarizes it. These data are stored in the human resource management system in JSON format. Furthermore, based on the information extracted from the interview record, it is automatically reflected in the ToDo list and calendar.

[0279] The work records of robots are collected to calculate the efficiency and error rate. These data are analyzed by the server to identify the tasks that robots are good at and those they are not good at. Based on the analysis results, an optimal work arrangement and individual improvement plans are generated.

[0280] Specific Example

[0281] For example, consider a scenario where the record of a factory robot performing "assembly work" is saved, and an improvement plan is created based on that record. The work records of the robot are collected as follows.

[0282] Example of prompt text:

[0283] Robot ID: robot_1

[0284] Task: Assembly

[0285] Efficiency: 0.9

[0286] Error rate: 0.05

[0287] By inputting this prompt text into the generative AI model, the process of saving, analyzing the work records of the robot, and creating improvement plans is automated. As a result, the workload of creating interview records is reduced, the line length can focus on non-verbal information, and it becomes possible to analyze the work efficiency and error rate of the robot and propose optimal work arrangements and improvement plans.

[0288] The flow of the specific process in Application Example 2 will be described using FIG. 14.

[0289] Step 1:

[0290] The server creates a verbatim record of the interview using the generative AI model. As input, it receives the audio data of the interview and generates a verbatim record using the generative AI model. As output, the verbatim record is obtained in text format.

[0291] Step 2:

[0292] The server summarizes the generated verbatim record. As input, it receives the text data of the verbatim record and applies a summarization algorithm. As output, the summarized text is obtained.

[0293] Step 3:

[0294] The server stores verbatim transcripts and summaries in JSON format in the human resources management system. It receives the text data of the verbatim transcripts and summaries as input and converts it to JSON format. As output, the JSON data is stored in the human resources management system.

[0295] Step 4:

[0296] The server automatically updates the to-do list and calendar based on information extracted from the interview records. It receives JSON data of the interview records as input and converts it into the to-do list and calendar format. As output, new tasks and events are added to the to-do list and calendar.

[0297] Step 5:

[0298] The server collects robot work records. It receives work data (work ID, efficiency, error rate, etc.) transmitted from the robot as input. The collected work records are stored in a database as output.

[0299] Step 6:

[0300] The server analyzes the collected work records. It receives work records stored in a database as input and calculates efficiency and error rates. The analysis results are output.

[0301] Step 7:

[0302] The server identifies the tasks that the robot is good at and bad at. It receives analysis results as input and applies an algorithm to identify the robot's strengths and weaknesses. The output is a list of tasks the robot is good at and bad at.

[0303] Step 8:

[0304] The server proposes an optimal work arrangement. As input, it receives a list of proficient and unproficient tasks, and applies an algorithm to calculate the optimal work arrangement. As output, a proposal for the optimal work arrangement is obtained.

[0305] Step 9:

[0306] The server creates an individual improvement plan. As input, it receives the analysis results and the proposal for the optimal work arrangement, and applies an algorithm to generate an improvement plan. As output, an individual improvement plan is obtained.

[0307] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion identification model 59 and perform specific processing using the user's emotions.

[0308] "Form Example 1"

[0309] [[ID=二十]]As a form example 1 of this invention, there is a system combined with an emotion engine. In this system, a face-to-face interview implementation support tool utilizing generative AI is developed and implemented in the personnel management system. The generative AI creates a verbatim record and summary of the interview, and stores the results in the personnel management system. Furthermore, it also has a function to automatically reflect the content of the interview in the ToDo list and calendar. By combining an emotion engine with this system, it becomes possible to recognize the emotions of the user during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the tone of the user's voice, expression, choice of words, etc., and the results are added to the interview record. As a result, not only the content of the interview but also the user's emotional state at that time can be grasped, enabling deeper understanding and response.

[0310] "Form Example 2"

[0311] As an example of the second form of this invention, there is a system that reduces the burden on line managers in creating interview records, allowing them to concentrate on observing nonverbal information during interviews. In this system, a generating AI creates and summarizes the verbatim transcript of the interview, and stores the results in the personnel management system. Furthermore, it also has a function to automatically reflect the content of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, it analyzes emotions from the user's tone of voice, facial expressions, and word choice, and adds the results to the interview record. As a result, the burden on line managers in creating interview records is reduced, allowing them to concentrate on observing nonverbal information during interviews.

[0312] "Example of form 4"

[0313] As an example of the fourth form of this invention, there is a system that creates an environment for continuous employee training by utilizing implementation records. In this system, a generating AI creates and summarizes verbatim transcripts of interviews and stores the results in a human resources management system. Furthermore, it also has a function to automatically reflect the content of the interviews in a to-do list and calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, it analyzes emotions from the user's tone of voice, facial expressions, and word choice, and adds the results to the interview record. This makes it possible to create an environment for continuous employee training by utilizing the interview records.

[0314] The following describes the processing flow for each example of the form.

[0315] "Example of form 1"

[0316] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[0317] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[0318] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[0319] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[0320] Step 5: The results of the emotional engine analysis are reflected in the interview record.

[0321] "Example of form 2"

[0322] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[0323] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[0324] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[0325] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[0326] Step 5: The results of the emotion engine analysis are reflected in the interview record, reducing the burden on the line manager in creating the interview record and allowing them to focus on observing nonverbal information during the interview.

[0327] "Example of form 4"

[0328] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[0329] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[0330] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[0331] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[0332] Step 5: The results of the emotion engine analysis are reflected in the interview records, creating an environment where continuous employee development can be carried out by utilizing these records.

[0333] (Example 1)

[0334] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0335] Traditional interview record-keeping was largely manual, placing a significant burden on line managers and supervisors. Furthermore, accurately recording nonverbal information and emotional states during interviews was difficult, resulting in a lack of data necessary to improve interview quality. Additionally, reflecting interview content in to-do lists and calendars was a time-consuming process, highlighting the need for more efficient management.

[0336] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0337] This invention includes means for developing and implementing an interview support tool utilizing generative AI into a human resources management system, means for using generative AI to create verbatim transcripts and summaries of interviews and storing the interview records in the human resources management system, means for automatically reflecting the results in to-do lists and calendars, means for using an emotion engine to recognize the emotional state during the interview and reflecting that information in the interview records, and means for automating the entire process from the start to the end of the interview. As a result, the burden of creating interview records is reduced, detailed interview records including nonverbal information and emotional states become possible, and the content of interviews can be managed more efficiently.

[0338] "Generative AI" refers to a system that uses artificial intelligence technology to generate text and data.

[0339] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[0340] A "human resources management system" is an integrated system for managing employee information, evaluation, and training.

[0341] A "verbatim transcript" is a document that meticulously records the content of conversations during interviews, meetings, or other similar events.

[0342] A "summary" is a document that extracts the key points from detailed records, such as verbatim transcripts, and summarizes them concisely.

[0343] A "to-do list" is a list that displays tasks or action items that need to be completed in a list format.

[0344] A "calendar" is a tool that displays dates and times for schedule management.

[0345] An "emotional engine" is a technology or system for analyzing emotional states based on factors such as voice, facial expressions, and word choice.

[0346] "Nonverbal information" refers to information conveyed through means other than words, and includes facial expressions, gestures, and tone of voice.

[0347] "Process automation" refers to the automatic execution of a series of tasks that were previously performed manually by a system.

[0348] This invention develops an interview support tool utilizing generative AI and implements it into a human resources management system. A specific embodiment of this system is described below.

[0349] System Configuration

[0350] The system consists of three main elements: a server, a terminal, and a user. The server is equipped with a generative AI model and an emotion engine, and is responsible for creating verbatim transcripts of interviews, summarizing them, recognizing emotional states, and automatically updating to to-do lists and calendars. The terminal is responsible for collecting audio data from interviews and sending it to the server. The user conducts interviews and operates the system.

[0351] Hardware and software to be used

[0352] Server: A server with high-performance computing capabilities (e.g., a cloud server)

[0353] Generative AI models: Natural language processing models such as OpenAI's GPT-4.

[0354] Emotion engine: Emotion recognition API, etc.

[0355] Human Resources Management System: Integrated System

[0356] To-do list and calendar: Schedule management tools

[0357] Device: A computer or tablet equipped with a microphone for collecting audio data.

[0358] Data processing and data calculation

[0359] When a user begins a conversation, the device collects audio data and sends it to the server in real time. The server inputs the received audio data into a generating AI model to produce a verbatim transcript. The generated transcript is temporarily stored on the server.

[0360] Once the interview is complete, the server uses the generation AI model again to summarize the verbatim transcript. The summarized content is stored in the HR management system. Furthermore, the server analyzes the interview content and automatically reflects it in to-do lists and calendars.

[0361] Using an emotion engine, the server analyzes the user's tone of voice, facial expressions, and word choice to recognize their emotional state. This emotional information is also added to the verbatim transcript and stored in the personnel management system.

[0362] Examples of specific cases and prompt statements

[0363] As a concrete example, consider the following scenario.

[0364] scenario:

[0365] The user conducts a meeting with a subordinate. During the meeting, the subordinate reports on the project's progress and sets several tasks as the next steps. The subordinate also expresses concerns about the project.

[0366] Example of a prompt:

[0367] "We will now begin the interview with your subordinate. Please create a verbatim transcript and summarize it. Also, analyze their emotional state during the interview and reflect this in your to-do list and calendar."

[0368] This system reduces the burden of creating interview records for users, allowing them to focus on observing nonverbal information during interviews. It also enables efficient management of interview content and the retention of detailed records, including emotional states.

[0369] The flow of the specific processing in Example 1 will be explained using Figure 15.

[0370] Step 1:

[0371] The user initiates the interview. The user clicks the "Start Interview" button on the terminal, notifying the system that they wish to begin the interview. The input is the user's action, and the output is the signal that the interview has started. The terminal displays the message "Starting interview."

[0372] Step 2:

[0373] The terminal collects audio data in real time during the interview. The input is the user's voice, and the output is the collected audio data. The terminal uses its built-in microphone to collect audio data and sends it to the server in real time. The server stores the received audio data in a buffer.

[0374] Step 3:

[0375] The server inputs the received audio data into a generating AI model to generate a verbatim transcript. The input is audio data, and the output is the generated verbatim transcript. The server inputs the audio data stored in the buffer into the generating AI model to generate a verbatim transcript. The generated verbatim transcript is stored in the server's temporary storage.

[0376] Step 4:

[0377] Once the interview is complete, the server uses the generative AI model again to summarize the verbatim transcript. The input is the verbatim transcript, and the output is the summarized verbatim transcript. The server re-inputs the verbatim transcript into the generative AI model to generate the summary. The summarized content is stored in temporary storage.

[0378] Step 5:

[0379] The server stores the summarized verbatim transcript in the human resources management system. The input is the summarized verbatim transcript, and the output is the data stored in the human resources management system. The server saves the summarized verbatim transcript to the human resources management system's database. Once saving is complete, the user receives a notification stating, "The summary has been saved."

[0380] Step 6:

[0381] The server analyzes the summarized verbatim transcript and adds the necessary information to the to-do list and calendar. The input is the summarized verbatim transcript, and the output is an updated to-do list and calendar. For example, action items such as "Schedule the next meeting" or "Complete a specific task" are automatically added.

[0382] Step 7:

[0383] The server uses an emotion engine to analyze the user's emotional state from audio data and verbatim transcripts. The input is audio data and verbatim transcripts, and the output is recognized emotional information. For example, if a user expresses "anxiety," this information is recognized by the emotion engine.

[0384] Step 8:

[0385] The server adds the recognized sentiment information to the verbatim transcript and stores it in the personnel management system. The input is the recognized sentiment information, and the output is the updated verbatim transcript. The server adds the recognized sentiment information to the verbatim transcript and saves it to the personnel management system's database. Once saving is complete, the user is notified that "Sentiment information has been saved."

[0386] (Application Example 1)

[0387] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0388] Conventional interview support systems often require manual tasks such as creating verbatim transcripts, summarizing interviews, and recognizing emotions, leading to a heavy workload in interview record-keeping. Furthermore, it can be difficult to focus on observing nonverbal information during interviews, potentially lowering the quality of the interview. Additionally, the lack of features to automatically reflect interview content in to-do lists or calendars resulted in insufficient follow-up after interviews. To address these challenges, an efficient interview support system utilizing generative AI is necessary.

[0389] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0390] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that utilizes generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system; a server that automatically reflects the results in to-do lists and calendars; a server that combines an emotion engine to recognize the user's emotions during the interview and reflects that information in the interview records; a server that uses speech recognition technology to transcribe the interview audio into verbatim transcripts; a server that uses a generative AI model to summarize the verbatim transcripts and analyze emotions; and a server that saves the results of the summarization and emotion analysis as calendars and interview records. This reduces the burden of creating interview records and allows for greater focus on observing nonverbal information during interviews. Furthermore, it enables more efficient follow-up after interviews and improves the quality of interviews.

[0391] "Generative AI" refers to a system that uses artificial intelligence technology to generate, analyze, and summarize data.

[0392] An "interview support tool" is a combination of software or hardware designed to assist in the progress of an interview and efficiently create records.

[0393] A "human resources management system" is a system used by companies and organizations to manage employee information and to conduct evaluations and training.

[0394] A "verbatim transcript" is a record that transcribes the audio of an interview or meeting exactly as it was spoken.

[0395] A "summary" is a shortened version of long-form data, such as verbatim transcripts, that extracts the most important points.

[0396] "Interview records" refer to documents or data that record the content of an interview, including verbatim transcripts, summaries, and sentiment analysis results.

[0397] A "to-do list" is a list of tasks that need to be done.

[0398] A "calendar" is a tool for managing appointments and schedules.

[0399] An "emotion engine" is a system that analyzes emotions from voice and text data and outputs the results.

[0400] "Speech recognition technology" is a technology that converts speech into text.

[0401] A "generative AI model" is a specific implementation of generative AI, a machine learning model trained to perform a particular task.

[0402] "Nonverbal information" refers to information conveyed through means other than words, including facial expressions, tone of voice, and gestures.

[0403] As an example of how to implement this invention, a factory robot interview support system will be used. This system utilizes generative AI to create verbatim transcripts of interviews, summarize them, recognize emotions, and automatically update to to-do lists and calendars.

[0404] The server will develop an interview support tool utilizing generative AI and implement it into the human resources management system. Specifically, it will use speech recognition technology to transcribe interview audio into verbatim text, summarize the transcript using a generative AI model, and analyze emotions. This will reduce the burden of creating interview records and allow staff to focus on observing nonverbal information during interviews.

[0405] The server summarizes the verbatim transcript using a generative AI model and analyzes the sentiment. For example, Library A is used as the generative AI model. Library B is used for speech recognition technology. This automates the creation and summarization of the interview transcript.

[0406] The server combines an emotion engine to recognize the user's emotions during the interview and reflects that information in the interview record. A predetermined pipeline is used as the emotion engine. This allows for the understanding of not only the content of the interview but also the user's emotional state at that time.

[0407] The server saves summaries and sentiment analysis results as calendars and interview records. The saved data is automatically reflected in to-do lists and calendars. This allows for more efficient follow-up after interviews and improves the quality of the interviews.

[0408] As a concrete example, a summary can be obtained by inputting the following prompt sentence into the AI ​​model:

[0409] Example of a prompt:

[0410] "Today, I'd like to discuss the efficiency of the new production line. While the current production speed isn't meeting our target, there are several areas for improvement. First, we need to review the machine maintenance schedule. We also need to update the worker training program."

[0411] By inputting this prompt into the AI ​​model, a summary can be obtained. This allows for an efficient understanding of the interview content and enables quick action to be taken as needed.

[0412] The flow of a specific process in Application Example 1 will be explained using Figure 16.

[0413] Step 1:

[0414] The server uses speech recognition technology to transcribe the interview audio into text as a verbatim transcript.

[0415] Input: Audio data of the interview

[0416] Output: Verbatim transcript (text data)

[0417] Specific operation: The server converts the audio data acquired from the microphone into text. This records the content of the interview as a verbatim transcript.

[0418] Step 2:

[0419] The server uses a generative AI model to summarize the verbatim transcript.

[0420] Input: Verbatim transcript (text data)

[0421] Output: Summary (text data)

[0422] Specific operation: The server summarizes the verbatim transcript. Specifically, it extracts the key points from the transcript and summarizes them concisely.

[0423] Step 3:

[0424] The server uses an emotion engine to analyze the emotions in the verbatim transcript.

[0425] Input: Verbatim transcript (text data)

[0426] Output: Sentiment analysis results (text data)

[0427] Specific operation: The server analyzes the emotions in the verbatim transcript. Specifically, it recognizes emotions from the user's tone of voice and word choice, and outputs the results.

[0428] Step 4:

[0429] The server saves the summary and sentiment analysis results as a calendar and interview records.

[0430] Input: Summary (text data), Sentiment analysis results (text data)

[0431] Output: Calendar entries, interview records (data in JSON format)

[0432] Specific operation: The server saves the summary and sentiment analysis results in JSON format and automatically reflects them in the calendar and to-do list. This allows for efficient follow-up after interviews.

[0433] Step 5:

[0434] The user reviews the interview record and takes the necessary actions.

[0435] Input: Calendar entries, interview records (data in JSON format)

[0436] Output: Action plan (text data)

[0437] Specific operation: The user reviews the interview records and calendar entries provided by the server and plans the necessary actions. This enables specific responses based on the content of the interview.

[0438] (Example 2)

[0439] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0440] Traditional methods for creating interview records required line managers to manually create the records, which was a significant burden. Furthermore, it was difficult to appropriately observe and reflect nonverbal information during interviews (such as tone of voice and facial expressions). Additionally, there was no established environment for utilizing interview records for continuous employee development.

[0441] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for developing an interview implementation support tool utilizing generation AI and implementing it in the personnel management system; means for using generation AI to create verbatim transcripts and summaries of interviews and storing the interview records in the personnel management system; means for collecting the tone of voice and facial expressions of the user during the interview, analyzing emotions using an emotion engine, and reflecting the results in the interview records; means for automatically reflecting the interview content in a to-do list or calendar; and means for storing the verbatim transcripts, summaries, and emotion analysis results in the personnel management system. As a result, the burden of creating interview records on line managers is reduced, allowing them to concentrate on observing nonverbal information during interviews, and it becomes possible to create an environment for continuous employee training by utilizing the interview records.

[0442] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[0443] "Interview support tools" is a general term for software or hardware used to support the progress and record-keeping of interviews.

[0444] A "human resources management system" is an integrated software system used for managing employee information, evaluation, and training.

[0445] A "verbatim transcript" is text data that meticulously records the content of an interview or conversation.

[0446] A "summary" is text data that extracts important points and keywords from a verbatim transcript and presents them concisely.

[0447] An "emotion engine" is software or hardware that analyzes data such as voice and facial expressions to recognize a user's emotions.

[0448] "Nonverbal information" refers to information conveyed through means other than words, and specifically includes tone of voice, facial expressions, and gestures.

[0449] A "to-do list" is a tool for managing tasks and appointments in a list format.

[0450] A "calendar" is a tool for managing appointments and events based on dates and times.

[0451] "Vertical transcription" is the process of converting audio data into text and meticulously recording the content of interviews and conversations.

[0452] "Interview records" refer to data that records the content of an interview, including verbatim transcripts, summaries, and sentiment analysis results.

[0453] "Sentiment analysis" is the process of recognizing a user's emotions by analyzing data such as voice and facial expressions.

[0454] "Automatic reflection" refers to the process of automatically reflecting specific data or information into other systems or tools.

[0455] This invention is a system that uses a generation AI-powered interview support tool to create verbatim transcripts, summaries, and sentiment analyses of interviews, stores this data in a human resources management system, and automatically reflects it in to-do lists and calendars. A specific embodiment of this system is described below.

[0456] Hardware and software to be used

[0457] Hardware: Servers, terminals (PCs, tablets, smartphones)

[0458] Software: Generative AI models (e.g., GPT-4), emotion engines, human resource management systems

[0459] Data processing and data calculation workflow

[0460] 1. Collection of audio data from interviews

[0461] The user starts the interview.

[0462] The device collects audio data of the interview in real time. Specifically, it records audio using the microphone on a PC, tablet, or smartphone.

[0463] The device sends the collected audio data to the server.

[0464] 2. Creating a verbatim transcript of the audio data

[0465] The server sends the received audio data to the AI ​​model for generation.

[0466] The generative AI model converts audio data into text and creates a verbatim transcript. Specifically, it uses speech recognition technology to convert speech into text.

[0467] 3. Summary of the verbatim transcript

[0468] This process summarizes verbatim transcripts generated by generative AI models. Specifically, it extracts key points and keywords to generate a concise summary.

[0469] The server receives the summarized text and proceeds to the next processing step.

[0470] 4. Sentiment analysis

[0471] The device collects the user's voice tone and facial expressions during the interview. Specifically, it uses the camera and microphone to record changes in facial expressions and voice.

[0472] The server sends the collected data to the emotion engine.

[0473] The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from factors such as tone of voice, facial expressions, and word choice.

[0474] The server adds the results of the sentiment analysis to the verbatim transcript.

[0475] 5. Data Storage

[0476] The server stores verbatim transcripts, summaries, and sentiment analysis results in the human resources management system. Specifically, it stores them in a database.

[0477] The human resources management system manages the stored data and makes it accessible as needed.

[0478] 6. Reflection in To-Do list and calendar

[0479] The server analyzes the interview content and generates tasks that are reflected in the to-do list and calendar.

[0480] The HR management system automatically adds generated tasks to to-do lists and calendars. Specifically, it reflects them in employees' schedules and task management tools.

[0481] Specific example

[0482] For example, when employee A has a meeting with line manager B, the following specific actions are taken:

[0483] 1. The users (employee A and line manager B) begin the interview, and the terminal (PC) starts recording the audio.

[0484] 2. The device transmits the recorded audio data to the server in real time.

[0485] 3. The server sends the audio data to the AI ​​model that generates the verbatim transcript.

[0486] 4. A generative AI model summarizes the verbatim transcript and extracts the key points.

[0487] 5. The terminal collects the facial expressions and tone of voice of employee A during the interview, and the server sends this information to the emotion engine.

[0488] 6. The emotion engine analyzes the emotions and adds the results to the verbatim transcript.

[0489] 7. The server stores verbatim transcripts, summaries, and sentiment analysis results in the human resources management system.

[0490] 8. The HR management system analyzes the interview content and generates tasks that are reflected in the to-do list and calendar.

[0491] Example of a prompt

[0492] "Create a verbatim transcript and summary of the meeting between employee A and line manager B. Also, analyze employee A's emotions during the meeting and add your findings to the transcript."

[0493] This system reduces the burden on line managers in creating interview records, allowing them to focus on observing nonverbal information during interviews. Furthermore, it enables the creation of an environment for continuous employee development by utilizing interview records.

[0494] The flow of the specific processing in Example 2 will be explained using Figure 17.

[0495] Step 1:

[0496] The user starts the interview. The device collects the interview audio data in real time. Specifically, it records the audio using the microphone on a PC, tablet, or smartphone. The device sends the collected audio data to the server. The input is the interview audio data, and the output is the audio data sent to the server.

[0497] Step 2:

[0498] The server sends the received audio data to a generative AI model. The generative AI model converts the audio data into text and creates a verbatim transcript. Specifically, it uses speech recognition technology to convert speech into text. The input is the audio data sent to the server, and the output is the generated verbatim transcript.

[0499] Step 3:

[0500] This process summarizes the verbatim transcript generated by the generative AI model. Specifically, it extracts key points and keywords and generates a concise summary. The server receives the summarized text and proceeds to the next processing step. The input is the generated verbatim transcript, and the output is the summarized text.

[0501] Step 4:

[0502] The device collects the user's voice tone and facial expressions during the interview. Specifically, it uses a camera and microphone to record changes in facial expressions and voice. The server sends the collected data to the emotion engine. The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from voice tone, facial expressions, and word choice. The input is the collected voice tone and facial expression data, and the output is the result of the emotion analysis.

[0503] Step 5:

[0504] The server adds the sentiment analysis results to the verbatim transcript. Specifically, it integrates the sentiment analysis results into the verbatim transcript in text format. The input is the sentiment analysis results and the verbatim transcript, and the output is the verbatim transcript with the sentiment analysis added.

[0505] Step 6:

[0506] The server stores the verbatim transcripts, summaries, and sentiment analysis results in the human resources management system. Specifically, it stores them in a database. The human resources management system manages the stored data and makes it accessible as needed. The inputs are the verbatim transcripts, summaries, and sentiment analysis results with added sentiment analysis, and the output is the data stored in the human resources management system.

[0507] Step 7:

[0508] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar. The HR management system automatically adds the generated tasks to the to-do list and calendar. Specifically, it reflects this in the employee's schedule and task management tool. The input is the analyzed interview content, and the output is the tasks reflected in the to-do list and calendar.

[0509] (Application Example 2)

[0510] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0511] Traditional interview record systems required manual transcription, summarization, and emotional assessment, placing a significant burden on line managers and supervisors. Furthermore, the lack of effective use of interview records for employee skill development and career path design made creating individual training plans difficult. Additionally, there was a problem with ineffective training and skill development support for factory robot operators.

[0512] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0513] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that uses generative AI to create verbatim transcripts and summaries of interviews and stores the interview records in the human resources management system; a server that automatically reflects the results in to-do lists and calendars; a server that uses an emotion engine to recognize emotions during interviews and reflects that information in the interview records; and a server that records interviews and training sessions conducted by factory robot operators to support skill development and career path design for the operators. This reduces the burden of creating interview records, allows for focus on observing nonverbal information during interviews, and enables the creation of an environment for continuous employee training by utilizing the records.

[0514] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[0515] A "meeting support tool" is software or a system designed to assist in recording and analyzing interviews.

[0516] A "human resources management system" is a system that manages employee information and supports skill development and career path design.

[0517] "Transcript creation" is the process of meticulously recording the content of an interview or conversation.

[0518] "Summarizing" is the process of condensing the content of a long text or conversation into a shorter form.

[0519] A "to-do list" is a tool for managing tasks that need to be done in a list format.

[0520] A "calendar" is a tool for managing schedules and appointments.

[0521] An "emotion engine" is a system that analyzes and recognizes emotions from speech and text.

[0522] "Interview records" refer to documents or data that record the content of an interview.

[0523] A "factory robot" is an automated machine used to perform tasks within a factory.

[0524] An "operator" is a person who operates a machine or system.

[0525] A "training session" is a place for training to improve specific skills or knowledge.

[0526] "Skill improvement" is the process of improving specific skills or abilities.

[0527] A "career path" refers to the line of work and positions that an employee should pursue in the future.

[0528] In order to implement this invention, the following system configuration and program are required.

[0529] System Configuration

[0530] The server will develop and implement an interview support tool utilizing generative AI. Using generative AI, it will create and summarize verbatim transcripts of interviews and store the interview records in the HR management system. Furthermore, it will have a function to automatically reflect these records in to-do lists and calendars. It will also use an emotion engine to recognize emotions during interviews and reflect that information in the interview records. It will record interviews and training sessions conducted by factory robot operators to support skill development and career path design for operators.

[0531] Hardware and software to be used

[0532] Hardware: Microphone, computer

[0533] Software: Python, SpeechRecognition library, Transformers library, EmotionRecognition library

[0534] Program Processing Description

[0535] The server reads the audio file and retrieves the audio data. Next, it converts the audio data into text. It uses a generative AI model to summarize the text and an emotion engine to analyze the emotions in the text. Based on this information, it creates an interview record and saves it in JSON format. The interview record is stored in the HR management system and automatically reflected in the to-do list and calendar.

[0536] Specific example

[0537] Operators conduct interviews regarding factory robot operation, and these interviews are recorded. When the recorded audio files are input into a server, a verbatim transcript and summary of the interview are generated, and the operator's emotions are also analyzed. This facilitates skill development and career path design for operators.

[0538] Example of a prompt

[0539] Please summarize the following text:

[0540] "Today we discussed the operation of the factory robots. Operator A seems to lack confidence in certain operations. However, he is very skilled in other operations, and is particularly strong in troubleshooting. In future training, we plan to create a program to address A's weaknesses."

[0541] The flow of a specific process in Application Example 2 will be explained using Figure 18.

[0542] Step 1:

[0543] The server reads audio files and retrieves audio data. Specifically, it uploads audio files recorded using a microphone to the server and retrieves audio data using the SpeechRecognition library. The input is an audio file, and the output is audio data.

[0544] Step 2:

[0545] The server converts the acquired audio data into text. Specifically, it uses the SpeechRecognition library to convert the audio data into text. The input is audio data, and the output is text data.

[0546] Step 3:

[0547] The server summarizes text data using a generative AI model. Specifically, it uses a summarization model from the Transformers library to shorten the text data. The input is text data, and the output is the summarized text.

[0548] Step 4:

[0549] The server analyzes emotions from text data using an emotion engine. Specifically, it uses the EmotionRecognition library to extract emotions from text data. The input is text data, and the output is emotion data.

[0550] Step 5:

[0551] The server creates an interview record based on summarized text and sentiment data. Specifically, it combines the summarized text and sentiment data to generate the interview record in JSON format. The input is summarized text and sentiment data, and the output is the interview record.

[0552] Step 6:

[0553] The server stores the created interview records in the human resources management system. Specifically, it saves the generated JSON-formatted interview records to the database. The input is the interview records, and the output is the records stored in the database.

[0554] Step 7:

[0555] The server automatically updates the to-do list and calendar based on the interview records. Specifically, it adds tasks and schedules extracted from the interview records to the to-do list and calendar. The input is the interview records, and the output is the updated to-do list and calendar.

[0556] Step 8:

[0557] The user records interviews and training sessions conducted by factory robot operators, supporting their skill development and career path design. Specifically, it creates individual training plans based on interview records to improve the operators' skills. The input is the interview records, and the output is the training plan.

[0558] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0559] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0560] Other examples of generative AI include Gemini® (registered trademark) (Internet search). <url: https: gemini.google.com ?hl="ja">) are some examples.

[0561] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0562] [Second Embodiment]

[0563] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0564] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0565] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0566] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0567] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0568] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0569] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0570] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0571] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0572] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0573] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0574] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0575] "Example of form 1"

[0576] One embodiment of this invention involves developing an interview support tool utilizing generative AI and implementing it in a human resources management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript. When the interview ends, the generative AI summarizes the transcript and stores the summary in the human resources management system. It also has a function to automatically reflect the content of the interview in a to-do list and calendar. As a result, the burden of creating interview records is reduced for line managers, allowing them to concentrate on observing nonverbal information during the interview.

[0577] "Example of form 2"

[0578] Another embodiment of the present invention involves creating an environment for continuous employee development by utilizing implementation records. Specifically, interview records are stored in a human resources management system, and these records are used to design employee skill development and career paths. For example, employee strengths and weaknesses are extracted from interview records, and individual development plans are created based on these. In addition, employee interests and aspirations are understood from interview records, and career paths are proposed based on these. This makes it possible to provide development tailored to each individual employee.

[0579] The following describes the processing flow for each example of the form.

[0580] "Example of form 1"

[0581] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[0582] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the summary in the HR management system.

[0583] Step 3: It also includes a function that automatically updates the to-do list and calendar based on the content of the interview.

[0584] "Example of form 2"

[0585] Step 1: Store interview records in the HR management system and use them to design employee skill development and career paths.

[0586] Step 2: Extract the employee's strengths and weaknesses from the interview records and create an individualized training plan based on that.

[0587] Step 3: Understand the employee's interests and aspirations from the interview records, and propose a career path based on that.

[0588] (Example 1)

[0589] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0590] Traditional interview record-keeping is often done manually, placing a significant burden on line managers and supervisors. Furthermore, the lack of time to observe nonverbal information during interviews can lead to a decline in interview quality. Additionally, the difficulty in efficiently managing interview content and reflecting it in to-do lists and calendars can result in insufficient follow-up after interviews.

[0591] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0592] This invention includes a server that develops and implements an interview support tool utilizing generative AI into a human resources management system; a server that uses generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system; a server that collects audio data at the start of an interview and transmits it to the server; a server that uses speech recognition software to convert the audio data into text data; a server that uses a generative AI model to summarize the verbatim transcript; a server that stores the summarized verbatim transcript in a database; a server that automatically updates the to-do list and calendar based on the interview content; and a server that notifies the user of the summarized verbatim transcript and update information. This reduces the burden of creating interview records and allows for focus on observing nonverbal information during interviews. Furthermore, because interview content can be efficiently managed and automatically reflected in the to-do list and calendar, follow-up after interviews becomes easier.

[0593] "Generative AI" refers to a system that uses artificial intelligence technology to generate and process data.

[0594] An "interview support tool" is a combination of software or hardware designed to assist with the progress and record-keeping of interviews.

[0595] A "human resources management system" is an integrated system for managing employee information, evaluation, and training.

[0596] A "verbatim transcript" is a record that transcribes the audio of an interview or meeting exactly as it was spoken.

[0597] A "summary" refers to a concise compilation of detailed information, such as verbatim transcripts.

[0598] "Speech recognition software" is software used to convert speech data into text data.

[0599] "Generative AI model" refers to the core algorithms and pre-trained models of generative AI.

[0600] A "database" is a system for efficiently storing, managing, and retrieving data.

[0601] A "to-do list" is a list used to manage tasks and appointments in a list format.

[0602] A "calendar" is a tool for managing appointments and events based on dates and times.

[0603] A "terminal" refers to a device such as a computer or smartphone that is directly operated by the user.

[0604] A "server" is a computer system used to process and store data over a network.

[0605] "Nonverbal information" refers to information conveyed through means other than words, such as facial expressions and gestures.

[0606] A "notification" is a message or alert sent from a system to a user to inform them of information.

[0607] This invention develops an interview support tool utilizing generative AI and implements it into a human resources management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript, and when the interview ends, it summarizes the transcript and stores the summary in the human resources management system. It also includes a function to automatically reflect the content of the interview in a to-do list and calendar.

[0608] Hardware and software to be used

[0609] server

[0610] The server provides computing resources to run generative AI models. Specifically, it uses speech recognition software to convert audio data into text data and a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The server also stores the summarized transcript in the human resources management system's database and automatically updates to-do lists and calendars based on the interview content.

[0611] terminal

[0612] The terminal is a device used by the user to conduct interviews. When an interview begins, the terminal uses its built-in microphone to collect audio data and transmits it to the server in real time. When the interview ends, the terminal receives a summarized transcript and updated to-do list / calendar information from the server and notifies the user.

[0613] User

[0614] The user is the person conducting the interview. The user initiates the interview via their device and does not need to perform any operations during the interview. Once the interview is complete, the user reviews the summarized transcript via their device and manages updates to their to-do list and calendar.

[0615] Specific example

[0616] Example of a prompt

[0617] "Please summarize the following interview content: 'Hello, today I'd like to discuss your work progress. First, could you tell me about the progress of your project last week?'"

[0618] Processing flow

[0619] 1. Start of interview: The user launches the dedicated application on their device and clicks the "Start Interview" button.

[0620] 2. Voice data collection: The device uses its built-in microphone to collect voice data and transmits it to the server in real time.

[0621] 3. Transcript creation: The server uses speech recognition software to convert the audio data into text data.

[0622] 4. Summary generation: The server uses a generation AI model to summarize the verbatim transcript.

[0623] 5. Data storage: The server stores the summarized verbatim transcripts in the human resources management system database.

[0624] 6. To-Do List & Calendar Update: The server automatically updates the to-do list and calendar based on the interview content.

[0625] 7. User Notifications: The device notifies the user of summarized verbatim transcripts and update information.

[0626] This system reduces the burden of creating interview records, allowing you to focus on observing nonverbal information during interviews. Furthermore, it efficiently manages interview content and automatically reflects it in to-do lists and calendars, making post-interview follow-up easier.

[0627] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0628] Step 1:

[0629] Interview begins

[0630] Subject: User

[0631] The user launches a dedicated application on their terminal and clicks the "Start Interview" button. This causes the terminal to send an interview start request to the server. The input is the user's action, and the output is the interview start request. Specifically, the terminal displays "Interview started."

[0632] Step 2:

[0633] Audio data collection

[0634] Subject: terminal

[0635] The terminal begins collecting audio data using its built-in microphone as soon as the interview starts. The collected audio data is sent to the server in real time. The input is the user's voice, and the output is the audio data sent to the server. Specifically, the terminal displays "Collecting audio data" and shows the collection status in real time.

[0636] Step 3:

[0637] Transcript creation

[0638] Subject: Server

[0639] The server receives audio data sent from the terminal and converts it into text data using speech recognition software. The input is audio data, and the output is text data. Specifically, the server displays "Creating verbatim transcript" and updates the progress in real time.

[0640] Step 4:

[0641] Summary generation

[0642] Subject: Server

[0643] The server uses a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The input is the text data of the verbatim transcript, and the output is the summarized text. Specifically, the server displays "Generating summary" and updates the progress in real time.

[0644] Step 5:

[0645] Data storage

[0646] Subject: Server

[0647] The server stores the summarized verbatim transcript in the human resources management system's database. The input is the summarized text, and the output is the record stored in the database. Specifically, the server displays "Storing data" and, once storage is complete, displays "Data storage complete."

[0648] Step 6:

[0649] To-do list / calendar update

[0650] Subject: Server

[0651] The server automatically updates the to-do list and calendar based on the interview content. The input is a summarized verbatim transcript, and the output is the updated to-do list and calendar. Specifically, the server displays "Updating to-do list and calendar" and "Update complete" when the update is finished.

[0652] Step 7:

[0653] User notifications

[0654] Subject: terminal

[0655] The terminal receives summarized verbatim transcripts and updated to-do list / calendar information from the server and notifies the user. The input is the notification information from the server, and the output is the notification to the user. Specifically, the terminal displays "Summary and update information received" and notifies the user.

[0656] (Application Example 1)

[0657] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0658] Traditional interview support tools often require manual tasks such as creating and summarizing interview records and updating to-do lists and calendars, which is time-consuming and labor-intensive. Similar problems exist in interviews between factory workers and robots, making efficient work instructions and schedule management difficult.

[0659] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for developing an interview support tool utilizing generation AI and implementing it in the personnel management system, means for using generation AI to create verbatim transcripts and summaries of interviews and storing the interview records in the personnel management system, means for automatically reflecting them in to-do lists and calendars, means for supporting interviews between workers and robots in the factory, and means for automatically updating work instructions and schedules based on the interview content. As a result, the burden of creating interview records is reduced, and efficient work instructions and schedule management become possible.

[0660] "Generative AI" refers to a system that uses artificial intelligence technology to generate data and automate specific tasks.

[0661] An "interview support tool" is a software or hardware system designed to assist with the progress and record-keeping of interviews.

[0662] A "human resources management system" is a system used by companies and organizations to manage employee information and to perform labor management and talent development.

[0663] A "verbatim transcript" is a document that meticulously records the content of conversations during interviews, meetings, or other similar events.

[0664] A "summary" is a document that extracts the key points from detailed records, such as verbatim transcripts, and presents them concisely.

[0665] A "to-do list" is a list of tasks or work items that need to be done.

[0666] A "calendar" is a tool for managing appointments and events based on dates and times.

[0667] A "worker" is an employee who actually performs work in a factory or on a construction site.

[0668] A "robot" is a mechanical device that automatically performs tasks according to programmed instructions.

[0669] "Work instructions" are specific instructions or orders for performing a particular task.

[0670] A "schedule" is a plan of tasks or events that should be completed within a specific period of time.

[0671] As an embodiment of this invention, a system is constructed to support interviews between factory workers and robots. The specific method of implementation is shown below.

[0672] System Configuration

[0673] 1. Hardware Configuration

[0674] Microphone: Used to collect interview audio.

[0675] Computer: Used to process audio data and run generative AI models.

[0676] Robot: An automated machine or device that performs tasks within a factory.

[0677] 2. Software Configuration

[0678] OpenAI API: Used for creating verbatim transcripts and generating summaries of audio.

[0679] Python: A programming language used for data processing and file manipulation.

[0680] Human Resources Management System: A system for storing interview records and summaries and reflecting them in to-do lists and calendars.

[0681] Data processing and data calculation

[0682] 1. Collection of audio data and creation of verbatim transcripts.

[0683] The server collects the interview audio through the microphone.

[0684] The collected audio data is converted to text using the OpenAI API, and a verbatim transcript is created.

[0685] 2. Summary of the verbatim transcript

[0686] The server summarizes the generated verbatim transcript using the OpenAI API.

[0687] The summarized text is stored in the human resources management system.

[0688] 3. Reflection in To-Do list and calendar

[0689] The server extracts key tasks from the summarized text and adds them to a to-do list.

[0690] Additionally, the schedule will be updated based on the interview content and reflected in the calendar.

[0691] Specific example

[0692] For example, if during an interview someone says, "We need to check the next maintenance schedule," that statement will be recorded in the verbatim transcript. After the interview, the server will summarize the transcript and generate a summary titled "Check the next maintenance schedule." This summary will be stored in the HR management system, and "Check the next maintenance schedule" will be automatically added to the ToDo list.

[0693] Example of a prompt

[0694] Examples of prompt sentences for summarizing verbatim transcripts are as follows:

[0695] Please summarize the following verbatim transcript:

[0696] [Verbatim text]

[0697] summary:

[0698] The following is an example of a prompt message for extracting items from a to-do list.

[0699] Extract the to-do list items from the following summary:

[0700] [Summary text]

[0701] To-do list:

[0702] In this way, it becomes possible to efficiently support meetings between factory workers and robots, and to automate work instructions and schedule management.

[0703] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0704] Step 1:

[0705] The server collects interview audio via a microphone. The input is the interview audio, and the output is audio data. Specifically, the microphone captures the audio during the interview in real time and sends that audio data to the server.

[0706] Step 2:

[0707] The server converts the collected audio data into text using the OpenAI API and creates a verbatim transcript. The input is audio data, and the output is the verbatim transcript text. Specifically, the audio data is input into the OpenAI speech recognition model, and the generated text is saved as a verbatim transcript.

[0708] Step 3:

[0709] The server summarizes the generated verbatim transcript using the OpenAI API. The input is the verbatim transcript text, and the output is the summarized text. Specifically, prompt sentences for summarizing the verbatim transcript text are input to the OpenAI text generation model, and the generated summary is obtained.

[0710] Step 4:

[0711] The server stores the summarized text in the human resources management system. The input is the summarized text, and the output is the summarized data stored in the human resources management system. Specifically, it performs the operation of saving the summarized text to the human resources management system's database.

[0712] Step 5:

[0713] The server extracts important tasks from the summarized text and adds them to a to-do list. The input is the summarized text, and the output is the updated to-do list. Specifically, the summarized text is input to an OpenAI text generation model as prompts for task extraction, and the generated tasks are added to the to-do list.

[0714] Step 6:

[0715] The server updates the schedule based on the interview content and reflects it in the calendar. The input is a summary text, and the output is the updated calendar. Specifically, it extracts schedule-related information from the summary text and adds that information to the calendar system.

[0716] Step 7:

[0717] The user checks their updated to-do list and calendar and performs the necessary tasks. The input is the updated to-do list and calendar, and the output is the user's completed tasks. Specifically, the user logs into the system, checks their to-do list and calendar, and performs the instructed tasks.

[0718] (Example 2)

[0719] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0720] Traditional human resource management systems often required manual processes for creating and storing interview records and developing employee training plans, resulting in time-consuming and labor-intensive tasks. Furthermore, they lacked sufficient opportunities to identify employees' strengths and weaknesses based on interview records, create individualized training plans, and propose career paths, making effective, personalized employee development challenging.

[0721] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0722] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that utilizes generative AI to create verbatim transcripts and summaries of interviews and stores the interview records in the human resources management system; a server that analyzes the interview records to extract the strengths and weaknesses of employees; a server that creates individual training plans based on the extracted data; a server that proposes career paths based on the interests and aspirations of employees; and a server that automatically reflects the results in to-do lists and calendars. As a result, the creation and storage of interview records are automated, and the extraction of employee strengths and weaknesses, the creation of individual training plans, and the proposal of career paths can be carried out efficiently, enabling effective training tailored to each employee.

[0723] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[0724] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[0725] A "human resources management system" is software or a system used to manage employee information and support human resources tasks such as training and performance evaluation.

[0726] "Transcript creation" is the process of recording the content of an interview exactly as it was said.

[0727] "Summarizing" is the process of concisely summarizing the content of an interview.

[0728] An "interview record" is data that records the content of an interview.

[0729] "Storage" is the process of saving data and making it accessible as needed.

[0730] "Analysis" is the process of analyzing data and extracting useful information.

[0731] "Strengths" refer to the skills and abilities that employees excel at.

[0732] A "weakness" refers to the skills or abilities that an employee struggles with.

[0733] A "training plan" is a plan designed to support employees in improving their skills and advancing their careers.

[0734] "Interest" refers to the areas or activities that employees are interested in.

[0735] "Orientation" refers to the direction and goals that employees strive for.

[0736] A "career path" is the outline of the jobs and roles that an employee should pursue in the future.

[0737] A "to-do list" is a list of tasks that need to be done.

[0738] A "calendar" is a tool for managing appointments and schedules.

[0739] This invention develops an interview support tool that utilizes generative AI and implements it into a human resources management system to efficiently create and store interview records, develop employee training plans, and propose career paths. A specific embodiment of this system is described below.

[0740] Hardware and software to be used

[0741] Hardware: Servers, terminals (PCs, tablets, etc.)

[0742] Software: Human resource management systems, generative AI models (e.g., OpenAI GPT-4)

[0743] Program processing

[0744] 1. Entering interview records

[0745] After the interview, the user uses a device (PC or tablet) to enter the interview record into the human resources management system.

[0746] Specifically, the user accesses an input form in the human resources management system and enters the interview details in text format.

[0747] Example: "The user uses a terminal to input the interview record for employee A. Strengths: project management, weaknesses: presentation skills."

[0748] 2. Storage of interview records

[0749] The server receives interview records submitted by users and stores them in the human resources management system's database.

[0750] Specifically, the server converts the input data into the appropriate format and saves it to the database.

[0751] Example: "The server receives the interview record and saves it to the database."

[0752] 3. Data Extraction and Analysis

[0753] The server analyzes the stored interview records to extract the employee's strengths and weaknesses.

[0754] Specifically, the server inputs the interview records into a generated AI model (e.g., OpenAI GPT-4) and retrieves the analysis results.

[0755] Example: "The server inputs interview records into a generated AI model and extracts employee A's strengths and weaknesses."

[0756] 4. Creating a training plan

[0757] The server creates individual training plans based on the extracted data.

[0758] In terms of specific operations, the server uses the analysis results of the generated AI model to suggest training and workshops necessary for improving employees' skills.

[0759] Example: "The server proposes project management training, leveraging employee A's strengths. To address their weaknesses, it plans training to improve their presentation skills."

[0760] 5. Proposed Career Paths

[0761] The server suggests appropriate career paths based on employees' interests and aspirations.

[0762] Specifically, the server uses the analysis results of the generated AI model to design future career paths for employees and reflects the proposed content in the human resources management system.

[0763] Example: "Since employee A is interested in project management, the server suggests a career path to becoming a project manager in the future."

[0764] Specific example

[0765] Example of interview record entry:

[0766] "The user uses a terminal to input interview records for employee B. Strengths include data analysis, weaknesses include teamwork."

[0767] Examples of prompts for a generative AI model:

[0768] "Based on the following interview record, identify employee B's strengths and weaknesses and propose an appropriate training plan. Interview record: Strengths: data analysis, Weaknesses: teamwork."

[0769] This system automates the creation and storage of interview records, enabling efficient identification of employees' strengths and weaknesses, creation of individual training plans, and proposal of career paths, thereby facilitating effective training tailored to each employee.

[0770] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0771] Step 1:

[0772] Entering interview records

[0773] After the interview, the user uses a device (PC or tablet) to enter the interview record into the human resources management system.

[0774] Input: Interview content (text format), employee's strengths and weaknesses, and other interview information.

[0775] Specific operation: The user accesses an input form in the HR management system and enters the interview details in text format. For example, they might enter, "Entering interview record for employee A. Strengths: project management, Weaknesses: presentation skills."

[0776] Output: The entered interview record is sent to the HR management system.

[0777] Step 2:

[0778] Storage of interview records

[0779] The server receives interview records submitted by users and stores them in the human resources management system's database.

[0780] Input: Interview records submitted by the user.

[0781] Specific operation: The server converts the input data into the appropriate format and saves it to the database. For example, the server receives an interview record and saves "Employee A's Interview Record" to the database.

[0782] Output: Interview records stored in the database.

[0783] Step 3:

[0784] Data extraction and analysis

[0785] The server analyzes the stored interview records to extract the employee's strengths and weaknesses.

[0786] Input: Interview records stored in the database.

[0787] Specific operation: The server inputs interview records into a generating AI model (e.g., OpenAI GPT-4) and obtains the analysis results. For example, the server inputs "Employee A's interview records" into the generating AI model and extracts strengths and weaknesses.

[0788] Output: Employee strengths and weaknesses as analyzed.

[0789] Step 4:

[0790] Creating a training plan

[0791] The server creates individual training plans based on the extracted data.

[0792] Input: Employee strengths and weaknesses as analyzed.

[0793] Specific operation: Based on the analysis results of the generated AI model, the server proposes training and workshops necessary for improving employees' skills. For example, the server might create a training plan such as, "Leverage employee A's strengths and propose project management training. To address their weaknesses, plan training to improve their presentation skills."

[0794] Output: Individual training plan.

[0795] Step 5:

[0796] Career path proposals

[0797] The server suggests appropriate career paths based on employees' interests and aspirations.

[0798] Input: Employees' interests and preferences as analyzed.

[0799] Specific operation: Based on the analysis results of the generated AI model, the server designs future career paths for employees and reflects the proposed content in the human resources management system. For example, the server might propose a career path such as, "Since employee A is interested in project management, we propose a career path to become a project manager in the future."

[0800] Output: Proposed career paths.

[0801] (Application Example 2)

[0802] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0803] Traditional HR management systems made creating and managing interview records cumbersome, leaving line managers with little time to observe nonverbal information during interviews. Furthermore, there was a lack of systems capable of efficiently collecting and analyzing robot work records to propose optimal work assignments and improvement plans. This made it difficult to improve employee and robot skills and to efficiently allocate tasks.

[0804] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0805] This invention includes means for the server to develop an interview support tool utilizing generative AI and implement it in a human resources management system; means for using generative AI to create verbatim transcripts and summaries of interviews and store the interview records in the human resources management system; means for automatically reflecting the results in to-do lists and calendars; means for collecting robot work records and analyzing efficiency and error rates; means for identifying the robot's strengths and weaknesses and proposing optimal work assignments; and means for creating individual improvement plans. As a result, the burden of creating interview records is reduced, line managers can concentrate on nonverbal information, and it becomes possible to analyze the robot's work efficiency and error rates and propose optimal work assignments and improvement plans.

[0806] "Generative AI" refers to a system that uses artificial intelligence technology to generate data and automate or assist with specific tasks.

[0807] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[0808] A "human resources management system" is a system that manages employee information and streamlines human resources tasks such as evaluation, training, and placement.

[0809] "Vertical transcript creation" is the process of recording conversations, such as interviews and meetings, exactly as they happened.

[0810] A "summary" refers to a condensed version of a long text or conversation.

[0811] A "to-do list" is a tool for managing tasks and appointments in a list format.

[0812] "Automatically add to calendar" is a feature that automatically adds specific events or tasks to your calendar.

[0813] "Robot work log" refers to a detailed record of the tasks performed by the robot.

[0814] "Efficiency" is an indicator that shows how effectively a task or process is performed.

[0815] The "error rate" is an indicator that shows how often errors occur in a task or process.

[0816] A "specialized task" refers to a task that a particular robot can perform with higher efficiency than other tasks.

[0817] A "task that a robot is not good at" refers to a task that a particular robot can perform with lower efficiency than other tasks.

[0818] "Optimal work arrangement" refers to positioning robots and employees in a way that allows them to perform their tasks most efficiently.

[0819] An "improvement plan" refers to a specific plan aimed at improving work efficiency and reducing error rates.

[0820] The system for implementing this invention involves developing an interview support tool utilizing generative AI and implementing it in a human resources management system. Specifically, it uses generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system. It also includes a function to automatically reflect the information in to-do lists and calendars.

[0821] Furthermore, it includes a function to collect robot work records and analyze efficiency and error rates. This allows for understanding the robot's strengths and weaknesses and proposing optimal work assignments. It also includes a function to create individual improvement plans.

[0822] Hardware and software to be used

[0823] Hardware: Servers, factory robots, user terminals

[0824] Software: Python, JSON, Generative AI Model

[0825] Data processing and data calculation

[0826] The server generates a verbatim transcript of the interview using an AI model and then summarizes it. This data is stored in JSON format in the HR management system. Furthermore, information extracted from the interview records is automatically reflected in to-do lists and calendars.

[0827] Robot work logs are collected to calculate efficiency and error rates. This data is analyzed on a server to identify the robot's strengths and weaknesses in different tasks. Based on the analysis, optimal work assignments and individual improvement plans are generated.

[0828] Specific example

[0829] For example, consider a scenario where records of a factory robot's "assembly work" are saved, and improvement plans are created based on those records. The robot's work records are collected as follows:

[0830] Example of a prompt:

[0831] Robot ID: robot_1

[0832] Task: Assembly

[0833] Efficiency: 0.9

[0834] Error rate: 0.05

[0835] By inputting this prompt into the generating AI model, the process of saving, analyzing, and creating improvement plans for robot work records is automated. This reduces the burden of creating interview records, allowing line managers to focus on nonverbal information, and enables them to analyze robot work efficiency and error rates, and propose optimal work assignments and improvement plans.

[0836] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0837] Step 1:

[0838] The server generates a verbatim transcript of the interview using a generative AI model. It receives audio data of the interview as input and generates the verbatim transcript using the generative AI model. The output is the verbatim transcript in text format.

[0839] Step 2:

[0840] The server summarizes the generated verbatim transcript. It receives the text data of the verbatim transcript as input and applies a summarization algorithm. The output is the summarized text.

[0841] Step 3:

[0842] The server stores verbatim transcripts and summaries in JSON format in the human resources management system. It receives the text data of the verbatim transcripts and summaries as input and converts it to JSON format. As output, the JSON data is stored in the human resources management system.

[0843] Step 4:

[0844] The server automatically updates the to-do list and calendar based on information extracted from the interview records. It receives JSON data of the interview records as input and converts it into the to-do list and calendar format. As output, new tasks and events are added to the to-do list and calendar.

[0845] Step 5:

[0846] The server collects robot work records. It receives work data (work ID, efficiency, error rate, etc.) transmitted from the robot as input. The collected work records are stored in a database as output.

[0847] Step 6:

[0848] The server analyzes the collected work records. It receives work records stored in a database as input and calculates efficiency and error rates. The analysis results are output.

[0849] Step 7:

[0850] The server identifies the tasks that the robot is good at and bad at. It receives analysis results as input and applies an algorithm to identify the robot's strengths and weaknesses. The output is a list of tasks the robot is good at and bad at.

[0851] Step 8:

[0852] The server proposes the optimal work assignment. It receives a list of tasks the server is good at and bad at as input, and applies an algorithm to calculate the optimal work assignment. The output is a proposed optimal work assignment.

[0853] Step 9:

[0854] The server creates individual improvement plans. It receives analysis results and optimal work allocation suggestions as input and applies an algorithm to generate improvement plans. The output is an individual improvement plan.

[0855] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0856] "Example of form 1"

[0857] One example of this invention is a system that incorporates an emotion engine. In this system, a tool for supporting interviews using generative AI is developed and implemented in a human resources management system. The generative AI creates and summarizes verbatim transcripts of interviews and stores the results in the human resources management system. Furthermore, it also has a function to automatically reflect the content of the interview in a to-do list and calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, and word choice, and the results are added to the interview record. This makes it possible to grasp not only the content of the interview but also the user's emotional state at that time, enabling a deeper understanding and response.

[0858] "Example of form 2"

[0859] As an example of the second form of this invention, there is a system that reduces the burden on line managers in creating interview records, allowing them to concentrate on observing nonverbal information during interviews. In this system, a generating AI creates and summarizes the verbatim transcript of the interview, and stores the results in the personnel management system. Furthermore, it also has a function to automatically reflect the content of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, it analyzes emotions from the user's tone of voice, facial expressions, and word choice, and adds the results to the interview record. As a result, the burden on line managers in creating interview records is reduced, allowing them to concentrate on observing nonverbal information during interviews.

[0860] "Example of form 4"

[0861] As an example of the fourth form of this invention, there is a system that creates an environment for continuous employee training by utilizing implementation records. In this system, a generating AI creates and summarizes verbatim transcripts of interviews and stores the results in a human resources management system. Furthermore, it also has a function to automatically reflect the content of the interviews in a to-do list and calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, it analyzes emotions from the user's tone of voice, facial expressions, and word choice, and adds the results to the interview record. This makes it possible to create an environment for continuous employee training by utilizing the interview records.

[0862] The following describes the processing flow for each example of the form.

[0863] "Example of form 1"

[0864] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[0865] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[0866] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[0867] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[0868] Step 5: The results of the emotional engine analysis are reflected in the interview record.

[0869] "Example of form 2"

[0870] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[0871] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[0872] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[0873] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[0874] Step 5: The results of the emotion engine analysis are reflected in the interview record, reducing the burden on the line manager in creating the interview record and allowing them to focus on observing nonverbal information during the interview.

[0875] "Example of form 4"

[0876] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[0877] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[0878] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[0879] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[0880] Step 5: The results of the emotion engine analysis are reflected in the interview records, creating an environment where continuous employee development can be carried out by utilizing these records.

[0881] (Example 1)

[0882] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0883] Traditional interview record-keeping was largely manual, placing a significant burden on line managers and supervisors. Furthermore, accurately recording nonverbal information and emotional states during interviews was difficult, resulting in a lack of data necessary to improve interview quality. Additionally, reflecting interview content in to-do lists and calendars was a time-consuming process, highlighting the need for more efficient management.

[0884] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0885] This invention includes means for developing and implementing an interview support tool utilizing generative AI into a human resources management system, means for using generative AI to create verbatim transcripts and summaries of interviews and storing the interview records in the human resources management system, means for automatically reflecting the results in to-do lists and calendars, means for using an emotion engine to recognize the emotional state during the interview and reflecting that information in the interview records, and means for automating the entire process from the start to the end of the interview. As a result, the burden of creating interview records is reduced, detailed interview records including nonverbal information and emotional states become possible, and the content of interviews can be managed more efficiently.

[0886] "Generative AI" refers to a system that uses artificial intelligence technology to generate text and data.

[0887] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[0888] A "human resources management system" is an integrated system for managing employee information, evaluation, and training.

[0889] A "verbatim transcript" is a document that meticulously records the content of conversations during interviews, meetings, or other similar events.

[0890] A "summary" is a document that extracts the key points from detailed records, such as verbatim transcripts, and summarizes them concisely.

[0891] A "to-do list" is a list that displays tasks or action items that need to be completed in a list format.

[0892] A "calendar" is a tool that displays dates and times for schedule management.

[0893] An "emotional engine" is a technology or system for analyzing emotional states based on factors such as voice, facial expressions, and word choice.

[0894] "Nonverbal information" refers to information conveyed through means other than words, and includes facial expressions, gestures, and tone of voice.

[0895] "Process automation" refers to the automatic execution of a series of tasks that were previously performed manually by a system.

[0896] This invention develops an interview support tool utilizing generative AI and implements it into a human resources management system. A specific embodiment of this system is described below.

[0897] System Configuration

[0898] The system consists of three main elements: a server, a terminal, and a user. The server is equipped with a generative AI model and an emotion engine, and is responsible for creating verbatim transcripts of interviews, summarizing them, recognizing emotional states, and automatically updating to to-do lists and calendars. The terminal is responsible for collecting audio data from interviews and sending it to the server. The user conducts interviews and operates the system.

[0899] Hardware and software to be used

[0900] Server: A server with high-performance computing capabilities (e.g., a cloud server)

[0901] Generative AI models: Natural language processing models such as OpenAI's GPT-4.

[0902] Emotion engine: Emotion recognition API, etc.

[0903] Human Resources Management System: Integrated System

[0904] To-do list and calendar: Schedule management tools

[0905] Device: A computer or tablet equipped with a microphone for collecting audio data.

[0906] Data processing and data calculation

[0907] When a user begins a conversation, the device collects audio data and sends it to the server in real time. The server inputs the received audio data into a generating AI model to produce a verbatim transcript. The generated transcript is temporarily stored on the server.

[0908] Once the interview is complete, the server uses the generation AI model again to summarize the verbatim transcript. The summarized content is stored in the HR management system. Furthermore, the server analyzes the interview content and automatically reflects it in to-do lists and calendars.

[0909] Using an emotion engine, the server analyzes the user's tone of voice, facial expressions, and word choice to recognize their emotional state. This emotional information is also added to the verbatim transcript and stored in the personnel management system.

[0910] Examples of specific cases and prompt statements

[0911] As a concrete example, consider the following scenario.

[0912] scenario:

[0913] The user conducts a meeting with a subordinate. During the meeting, the subordinate reports on the project's progress and sets several tasks as the next steps. The subordinate also expresses concerns about the project.

[0914] Example of a prompt:

[0915] "We will now begin the interview with your subordinate. Please create a verbatim transcript and summarize it. Also, analyze their emotional state during the interview and reflect this in your to-do list and calendar."

[0916] This system reduces the burden of creating interview records for users, allowing them to focus on observing nonverbal information during interviews. It also enables efficient management of interview content and the retention of detailed records, including emotional states.

[0917] The flow of the specific processing in Example 1 will be explained using Figure 15.

[0918] Step 1:

[0919] The user initiates the interview. The user clicks the "Start Interview" button on the terminal, notifying the system that they wish to begin the interview. The input is the user's action, and the output is the signal that the interview has started. The terminal displays the message "Starting interview."

[0920] Step 2:

[0921] The terminal collects audio data in real time during the interview. The input is the user's voice, and the output is the collected audio data. The terminal uses its built-in microphone to collect audio data and sends it to the server in real time. The server stores the received audio data in a buffer.

[0922] Step 3:

[0923] The server inputs the received audio data into a generating AI model to generate a verbatim transcript. The input is audio data, and the output is the generated verbatim transcript. The server inputs the audio data stored in the buffer into the generating AI model to generate a verbatim transcript. The generated verbatim transcript is stored in the server's temporary storage.

[0924] Step 4:

[0925] Once the interview is complete, the server uses the generative AI model again to summarize the verbatim transcript. The input is the verbatim transcript, and the output is the summarized verbatim transcript. The server re-inputs the verbatim transcript into the generative AI model to generate the summary. The summarized content is stored in temporary storage.

[0926] Step 5:

[0927] The server stores the summarized verbatim transcript in the human resources management system. The input is the summarized verbatim transcript, and the output is the data stored in the human resources management system. The server saves the summarized verbatim transcript to the human resources management system's database. Once saving is complete, the user receives a notification stating, "The summary has been saved."

[0928] Step 6:

[0929] The server analyzes the summarized verbatim transcript and adds the necessary information to the to-do list and calendar. The input is the summarized verbatim transcript, and the output is an updated to-do list and calendar. For example, action items such as "Schedule the next meeting" or "Complete a specific task" are automatically added.

[0930] Step 7:

[0931] The server uses an emotion engine to analyze the user's emotional state from audio data and verbatim transcripts. The input is audio data and verbatim transcripts, and the output is recognized emotional information. For example, if a user expresses "anxiety," this information is recognized by the emotion engine.

[0932] Step 8:

[0933] The server adds the recognized sentiment information to the verbatim transcript and stores it in the personnel management system. The input is the recognized sentiment information, and the output is the updated verbatim transcript. The server adds the recognized sentiment information to the verbatim transcript and saves it to the personnel management system's database. Once saving is complete, the user is notified that "Sentiment information has been saved."

[0934] (Application Example 1)

[0935] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0936] Conventional interview support systems often require manual tasks such as creating verbatim transcripts, summarizing interviews, and recognizing emotions, leading to a heavy workload in interview record-keeping. Furthermore, it can be difficult to focus on observing nonverbal information during interviews, potentially lowering the quality of the interview. Additionally, the lack of features to automatically reflect interview content in to-do lists or calendars resulted in insufficient follow-up after interviews. To address these challenges, an efficient interview support system utilizing generative AI is necessary.

[0937] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0938] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that utilizes generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system; a server that automatically reflects the results in to-do lists and calendars; a server that combines an emotion engine to recognize the user's emotions during the interview and reflects that information in the interview records; a server that uses speech recognition technology to transcribe the interview audio into verbatim transcripts; a server that uses a generative AI model to summarize the verbatim transcripts and analyze emotions; and a server that saves the results of the summarization and emotion analysis as calendars and interview records. This reduces the burden of creating interview records and allows for greater focus on observing nonverbal information during interviews. Furthermore, it enables more efficient follow-up after interviews and improves the quality of interviews.

[0939] "Generative AI" refers to a system that uses artificial intelligence technology to generate, analyze, and summarize data.

[0940] An "interview support tool" is a combination of software or hardware designed to assist in the progress of an interview and efficiently create records.

[0941] A "human resources management system" is a system used by companies and organizations to manage employee information and to conduct evaluations and training.

[0942] A "verbatim transcript" is a record that transcribes the audio of an interview or meeting exactly as it was spoken.

[0943] A "summary" is a shortened version of long-form data, such as verbatim transcripts, that extracts the most important points.

[0944] "Interview records" refer to documents or data that record the content of an interview, including verbatim transcripts, summaries, and sentiment analysis results.

[0945] A "to-do list" is a list of tasks that need to be done.

[0946] A "calendar" is a tool for managing appointments and schedules.

[0947] An "emotion engine" is a system that analyzes emotions from voice and text data and outputs the results.

[0948] "Speech recognition technology" is a technology that converts speech into text.

[0949] A "generative AI model" is a specific implementation of generative AI, a machine learning model trained to perform a particular task.

[0950] "Nonverbal information" refers to information conveyed through means other than words, including facial expressions, tone of voice, and gestures.

[0951] As an example of how to implement this invention, a factory robot interview support system will be used. This system utilizes generative AI to create verbatim transcripts of interviews, summarize them, recognize emotions, and automatically update to to-do lists and calendars.

[0952] The server will develop an interview support tool utilizing generative AI and implement it into the human resources management system. Specifically, it will use speech recognition technology to transcribe interview audio into verbatim text, summarize the transcript using a generative AI model, and analyze emotions. This will reduce the burden of creating interview records and allow staff to focus on observing nonverbal information during interviews.

[0953] The server summarizes the verbatim transcript using a generative AI model and analyzes the sentiment. For example, Library A is used as the generative AI model. Library B is used for speech recognition technology. This automates the creation and summarization of the interview transcript.

[0954] The server combines an emotion engine to recognize the user's emotions during the interview and reflects that information in the interview record. A predetermined pipeline is used as the emotion engine. This allows for the understanding of not only the content of the interview but also the user's emotional state at that time.

[0955] The server saves summaries and sentiment analysis results as calendars and interview records. The saved data is automatically reflected in to-do lists and calendars. This allows for more efficient follow-up after interviews and improves the quality of the interviews.

[0956] As a concrete example, a summary can be obtained by inputting the following prompt sentence into the AI ​​model:

[0957] Example of a prompt:

[0958] "Today, I'd like to discuss the efficiency of the new production line. While the current production speed isn't meeting our target, there are several areas for improvement. First, we need to review the machine maintenance schedule. We also need to update the worker training program."

[0959] By inputting this prompt into the AI ​​model, a summary can be obtained. This allows for an efficient understanding of the interview content and enables quick action to be taken as needed.

[0960] The flow of a specific process in Application Example 1 will be explained using Figure 16.

[0961] Step 1:

[0962] The server uses speech recognition technology to transcribe the interview audio into text as a verbatim transcript.

[0963] Input: Audio data of the interview

[0964] Output: Verbatim transcript (text data)

[0965] Specific operation: The server converts the audio data acquired from the microphone into text. This records the content of the interview as a verbatim transcript.

[0966] Step 2:

[0967] The server uses a generative AI model to summarize the verbatim transcript.

[0968] Input: Verbatim transcript (text data)

[0969] Output: Summary (text data)

[0970] Specific operation: The server summarizes the verbatim transcript. Specifically, it extracts the key points from the transcript and summarizes them concisely.

[0971] Step 3:

[0972] The server uses an emotion engine to analyze the emotions in the verbatim transcript.

[0973] Input: Verbatim transcript (text data)

[0974] Output: Sentiment analysis results (text data)

[0975] Specific operation: The server analyzes the emotions in the verbatim transcript. Specifically, it recognizes emotions from the user's tone of voice and word choice, and outputs the results.

[0976] Step 4:

[0977] The server saves the summary and sentiment analysis results as a calendar and interview records.

[0978] Input: Summary (text data), Sentiment analysis results (text data)

[0979] Output: Calendar entries, interview records (data in JSON format)

[0980] Specific operation: The server saves the summary and sentiment analysis results in JSON format and automatically reflects them in the calendar and to-do list. This allows for efficient follow-up after interviews.

[0981] Step 5:

[0982] The user reviews the interview record and takes the necessary actions.

[0983] Input: Calendar entries, interview records (data in JSON format)

[0984] Output: Action plan (text data)

[0985] Specific operation: The user reviews the interview records and calendar entries provided by the server and plans the necessary actions. This enables specific responses based on the content of the interview.

[0986] (Example 2)

[0987] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0988] Traditional methods for creating interview records required line managers to manually create the records, which was a significant burden. Furthermore, it was difficult to appropriately observe and reflect nonverbal information during interviews (such as tone of voice and facial expressions). Additionally, there was no established environment for utilizing interview records for continuous employee development.

[0989] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for developing an interview implementation support tool utilizing generation AI and implementing it in the personnel management system; means for using generation AI to create verbatim transcripts and summaries of interviews and storing the interview records in the personnel management system; means for collecting the tone of voice and facial expressions of the user during the interview, analyzing emotions using an emotion engine, and reflecting the results in the interview records; means for automatically reflecting the interview content in a to-do list or calendar; and means for storing the verbatim transcripts, summaries, and emotion analysis results in the personnel management system. As a result, the burden of creating interview records on line managers is reduced, allowing them to concentrate on observing nonverbal information during interviews, and it becomes possible to create an environment for continuous employee training by utilizing the interview records.

[0990] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[0991] "Interview support tools" is a general term for software or hardware used to support the progress and record-keeping of interviews.

[0992] A "human resources management system" is an integrated software system used for managing employee information, evaluation, and training.

[0993] A "verbatim transcript" is text data that meticulously records the content of an interview or conversation.

[0994] A "summary" is text data that extracts important points and keywords from a verbatim transcript and presents them concisely.

[0995] An "emotion engine" is software or hardware that analyzes data such as voice and facial expressions to recognize a user's emotions.

[0996] "Nonverbal information" refers to information conveyed through means other than words, and specifically includes tone of voice, facial expressions, and gestures.

[0997] A "to-do list" is a tool for managing tasks and appointments in a list format.

[0998] A "calendar" is a tool for managing appointments and events based on dates and times.

[0999] "Vertical transcription" is the process of converting audio data into text and meticulously recording the content of interviews and conversations.

[1000] "Interview records" refer to data that records the content of an interview, including verbatim transcripts, summaries, and sentiment analysis results.

[1001] "Sentiment analysis" is the process of recognizing a user's emotions by analyzing data such as voice and facial expressions.

[1002] "Automatic reflection" refers to the process of automatically reflecting specific data or information into other systems or tools.

[1003] This invention is a system that uses a generation AI-powered interview support tool to create verbatim transcripts, summaries, and sentiment analyses of interviews, stores this data in a human resources management system, and automatically reflects it in to-do lists and calendars. A specific embodiment of this system is described below.

[1004] Hardware and software to be used

[1005] Hardware: Servers, terminals (PCs, tablets, smartphones)

[1006] Software: Generative AI models (e.g., GPT-4), emotion engines, human resource management systems

[1007] Data processing and data calculation workflow

[1008] 1. Collection of audio data from interviews

[1009] The user starts the interview.

[1010] The device collects audio data of the interview in real time. Specifically, it records audio using the microphone on a PC, tablet, or smartphone.

[1011] The device sends the collected audio data to the server.

[1012] 2. Creating a verbatim transcript of the audio data

[1013] The server sends the received audio data to the AI ​​model for generation.

[1014] The generative AI model converts audio data into text and creates a verbatim transcript. Specifically, it uses speech recognition technology to convert speech into text.

[1015] 3. Summary of the verbatim transcript

[1016] This process summarizes verbatim transcripts generated by generative AI models. Specifically, it extracts key points and keywords to generate a concise summary.

[1017] The server receives the summarized text and proceeds to the next processing step.

[1018] 4. Sentiment analysis

[1019] The device collects the user's voice tone and facial expressions during the interview. Specifically, it uses the camera and microphone to record changes in facial expressions and voice.

[1020] The server sends the collected data to the emotion engine.

[1021] The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from factors such as tone of voice, facial expressions, and word choice.

[1022] The server adds the results of the sentiment analysis to the verbatim transcript.

[1023] 5. Data Storage

[1024] The server stores verbatim transcripts, summaries, and sentiment analysis results in the human resources management system. Specifically, it stores them in a database.

[1025] The human resources management system manages the stored data and makes it accessible as needed.

[1026] 6. Reflection in To-Do list and calendar

[1027] The server analyzes the interview content and generates tasks that are reflected in the to-do list and calendar.

[1028] The HR management system automatically adds generated tasks to to-do lists and calendars. Specifically, it reflects them in employees' schedules and task management tools.

[1029] Specific example

[1030] For example, when employee A has a meeting with line manager B, the following specific actions are taken:

[1031] 1. The users (employee A and line manager B) begin the interview, and the terminal (PC) starts recording the audio.

[1032] 2. The device transmits the recorded audio data to the server in real time.

[1033] 3. The server sends the audio data to the AI ​​model that generates the verbatim transcript.

[1034] 4. A generative AI model summarizes the verbatim transcript and extracts the key points.

[1035] 5. The terminal collects the facial expressions and tone of voice of employee A during the interview, and the server sends this information to the emotion engine.

[1036] 6. The emotion engine analyzes the emotions and adds the results to the verbatim transcript.

[1037] 7. The server stores verbatim transcripts, summaries, and sentiment analysis results in the human resources management system.

[1038] 8. The HR management system analyzes the interview content and generates tasks that are reflected in the to-do list and calendar.

[1039] Example of a prompt

[1040] "Create a verbatim transcript and summary of the meeting between employee A and line manager B. Also, analyze employee A's emotions during the meeting and add your findings to the transcript."

[1041] This system reduces the burden on line managers in creating interview records, allowing them to focus on observing nonverbal information during interviews. Furthermore, it enables the creation of an environment for continuous employee development by utilizing interview records.

[1042] The flow of the specific processing in Example 2 will be explained using Figure 17.

[1043] Step 1:

[1044] The user starts the interview. The device collects the interview audio data in real time. Specifically, it records the audio using the microphone on a PC, tablet, or smartphone. The device sends the collected audio data to the server. The input is the interview audio data, and the output is the audio data sent to the server.

[1045] Step 2:

[1046] The server sends the received audio data to a generative AI model. The generative AI model converts the audio data into text and creates a verbatim transcript. Specifically, it uses speech recognition technology to convert speech into text. The input is the audio data sent to the server, and the output is the generated verbatim transcript.

[1047] Step 3:

[1048] This process summarizes the verbatim transcript generated by the generative AI model. Specifically, it extracts key points and keywords and generates a concise summary. The server receives the summarized text and proceeds to the next processing step. The input is the generated verbatim transcript, and the output is the summarized text.

[1049] Step 4:

[1050] The device collects the user's voice tone and facial expressions during the interview. Specifically, it uses a camera and microphone to record changes in facial expressions and voice. The server sends the collected data to the emotion engine. The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from voice tone, facial expressions, and word choice. The input is the collected voice tone and facial expression data, and the output is the result of the emotion analysis.

[1051] Step 5:

[1052] The server adds the sentiment analysis results to the verbatim transcript. Specifically, it integrates the sentiment analysis results into the verbatim transcript in text format. The input is the sentiment analysis results and the verbatim transcript, and the output is the verbatim transcript with the sentiment analysis added.

[1053] Step 6:

[1054] The server stores the verbatim transcripts, summaries, and sentiment analysis results in the human resources management system. Specifically, it stores them in a database. The human resources management system manages the stored data and makes it accessible as needed. The inputs are the verbatim transcripts, summaries, and sentiment analysis results with added sentiment analysis, and the output is the data stored in the human resources management system.

[1055] Step 7:

[1056] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar. The HR management system automatically adds the generated tasks to the to-do list and calendar. Specifically, it reflects this in the employee's schedule and task management tool. The input is the analyzed interview content, and the output is the tasks reflected in the to-do list and calendar.

[1057] (Application Example 2)

[1058] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1059] Traditional interview record systems required manual transcription, summarization, and emotional assessment, placing a significant burden on line managers and supervisors. Furthermore, the lack of effective use of interview records for employee skill development and career path design made creating individual training plans difficult. Additionally, there was a problem with ineffective training and skill development support for factory robot operators.

[1060] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1061] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that uses generative AI to create verbatim transcripts and summaries of interviews and stores the interview records in the human resources management system; a server that automatically reflects the results in to-do lists and calendars; a server that uses an emotion engine to recognize emotions during interviews and reflects that information in the interview records; and a server that records interviews and training sessions conducted by factory robot operators to support skill development and career path design for the operators. This reduces the burden of creating interview records, allows for focus on observing nonverbal information during interviews, and enables the creation of an environment for continuous employee training by utilizing the records.

[1062] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[1063] A "meeting support tool" is software or a system designed to assist in recording and analyzing interviews.

[1064] A "human resources management system" is a system that manages employee information and supports skill development and career path design.

[1065] "Transcript creation" is the process of meticulously recording the content of an interview or conversation.

[1066] "Summarizing" is the process of condensing the content of a long text or conversation into a shorter form.

[1067] A "to-do list" is a tool for managing tasks that need to be done in a list format.

[1068] A "calendar" is a tool for managing schedules and appointments.

[1069] An "emotion engine" is a system that analyzes and recognizes emotions from speech and text.

[1070] "Interview records" refer to documents or data that record the content of an interview.

[1071] A "factory robot" is an automated machine used to perform tasks within a factory.

[1072] An "operator" is a person who operates a machine or system.

[1073] A "training session" is a place for training to improve specific skills or knowledge.

[1074] "Skill improvement" is the process of improving specific skills or abilities.

[1075] A "career path" refers to the line of work and positions that an employee should pursue in the future.

[1076] In order to implement this invention, the following system configuration and program are required.

[1077] System Configuration

[1078] The server will develop and implement an interview support tool utilizing generative AI. Using generative AI, it will create and summarize verbatim transcripts of interviews and store the interview records in the HR management system. Furthermore, it will have a function to automatically reflect these records in to-do lists and calendars. It will also use an emotion engine to recognize emotions during interviews and reflect that information in the interview records. It will record interviews and training sessions conducted by factory robot operators to support skill development and career path design for operators.

[1079] Hardware and software to be used

[1080] Hardware: Microphone, computer

[1081] Software: Python, SpeechRecognition library, Transformers library, EmotionRecognition library

[1082] Program Processing Description

[1083] The server reads the audio file and retrieves the audio data. Next, it converts the audio data into text. It uses a generative AI model to summarize the text and an emotion engine to analyze the emotions in the text. Based on this information, it creates an interview record and saves it in JSON format. The interview record is stored in the HR management system and automatically reflected in the to-do list and calendar.

[1084] Specific example

[1085] Operators conduct interviews regarding factory robot operation, and these interviews are recorded. When the recorded audio files are input into a server, a verbatim transcript and summary of the interview are generated, and the operator's emotions are also analyzed. This facilitates skill development and career path design for operators.

[1086] Example of a prompt

[1087] Please summarize the following text:

[1088] "Today we discussed the operation of the factory robots. Operator A seems to lack confidence in certain operations. However, he is very skilled in other operations, and is particularly strong in troubleshooting. In future training, we plan to create a program to address A's weaknesses."

[1089] The flow of a specific process in Application Example 2 will be explained using Figure 18.

[1090] Step 1:

[1091] The server reads audio files and retrieves audio data. Specifically, it uploads audio files recorded using a microphone to the server and retrieves audio data using the SpeechRecognition library. The input is an audio file, and the output is audio data.

[1092] Step 2:

[1093] The server converts the acquired audio data into text. Specifically, it uses the SpeechRecognition library to convert the audio data into text. The input is audio data, and the output is text data.

[1094] Step 3:

[1095] The server summarizes text data using a generative AI model. Specifically, it uses a summarization model from the Transformers library to shorten the text data. The input is text data, and the output is the summarized text.

[1096] Step 4:

[1097] The server analyzes emotions from text data using an emotion engine. Specifically, it uses the EmotionRecognition library to extract emotions from text data. The input is text data, and the output is emotion data.

[1098] Step 5:

[1099] The server creates an interview record based on summarized text and sentiment data. Specifically, it combines the summarized text and sentiment data to generate the interview record in JSON format. The input is summarized text and sentiment data, and the output is the interview record.

[1100] Step 6:

[1101] The server stores the created interview records in the human resources management system. Specifically, it saves the generated JSON-formatted interview records to the database. The input is the interview records, and the output is the records stored in the database.

[1102] Step 7:

[1103] The server automatically updates the to-do list and calendar based on the interview records. Specifically, it adds tasks and schedules extracted from the interview records to the to-do list and calendar. The input is the interview records, and the output is the updated to-do list and calendar.

[1104] Step 8:

[1105] The user records interviews and training sessions conducted by factory robot operators, supporting their skill development and career path design. Specifically, it creates individual training plans based on interview records to improve the operators' skills. The input is the interview records, and the output is the training plan.

[1106] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1107] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1108] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[1109] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1110] [Third Embodiment]

[1111] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1112] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1113] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1114] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1115] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1116] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1117] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1118] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1119] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1120] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1121] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1122] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[1123] "Example of form 1"

[1124] One embodiment of this invention involves developing an interview support tool utilizing generative AI and implementing it in a human resources management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript. When the interview ends, the generative AI summarizes the transcript and stores the summary in the human resources management system. It also has a function to automatically reflect the content of the interview in a to-do list and calendar. As a result, the burden of creating interview records is reduced for line managers, allowing them to concentrate on observing nonverbal information during the interview.

[1125] "Example of form 2"

[1126] Another embodiment of the present invention involves creating an environment for continuous employee development by utilizing implementation records. Specifically, interview records are stored in a human resources management system, and these records are used to design employee skill development and career paths. For example, employee strengths and weaknesses are extracted from interview records, and individual development plans are created based on these. In addition, employee interests and aspirations are understood from interview records, and career paths are proposed based on these. This makes it possible to provide development tailored to each individual employee.

[1127] The following describes the processing flow for each example of the form.

[1128] "Example of form 1"

[1129] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[1130] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the summary in the HR management system.

[1131] Step 3: It also includes a function that automatically updates the to-do list and calendar based on the content of the interview.

[1132] "Example of form 2"

[1133] Step 1: Store interview records in the HR management system and use them to design employee skill development and career paths.

[1134] Step 2: Extract the employee's strengths and weaknesses from the interview records and create an individualized training plan based on that.

[1135] Step 3: Understand the employee's interests and aspirations from the interview records, and propose a career path based on that.

[1136] (Example 1)

[1137] Next, we will describe Embodiment 1 of Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1138] Traditional interview record-keeping is often done manually, placing a significant burden on line managers and supervisors. Furthermore, the lack of time to observe nonverbal information during interviews can lead to a decline in interview quality. Additionally, the difficulty in efficiently managing interview content and reflecting it in to-do lists and calendars can result in insufficient follow-up after interviews.

[1139] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1140] This invention includes a server that develops and implements an interview support tool utilizing generative AI into a human resources management system; a server that uses generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system; a server that collects audio data at the start of an interview and transmits it to the server; a server that uses speech recognition software to convert the audio data into text data; a server that uses a generative AI model to summarize the verbatim transcript; a server that stores the summarized verbatim transcript in a database; a server that automatically updates the to-do list and calendar based on the interview content; and a server that notifies the user of the summarized verbatim transcript and update information. This reduces the burden of creating interview records and allows for focus on observing nonverbal information during interviews. Furthermore, because interview content can be efficiently managed and automatically reflected in the to-do list and calendar, follow-up after interviews becomes easier.

[1141] "Generative AI" refers to a system that uses artificial intelligence technology to generate and process data.

[1142] An "interview support tool" is a combination of software or hardware designed to assist with the progress and record-keeping of interviews.

[1143] A "human resources management system" is an integrated system for managing employee information, evaluation, and training.

[1144] A "verbatim transcript" is a record that transcribes the audio of an interview or meeting exactly as it was spoken.

[1145] A "summary" refers to a concise compilation of detailed information, such as verbatim transcripts.

[1146] "Speech recognition software" is software used to convert speech data into text data.

[1147] "Generative AI model" refers to the core algorithms and pre-trained models of generative AI.

[1148] A "database" is a system for efficiently storing, managing, and retrieving data.

[1149] A "to-do list" is a list used to manage tasks and appointments in a list format.

[1150] A "calendar" is a tool for managing appointments and events based on dates and times.

[1151] A "terminal" refers to a device such as a computer or smartphone that is directly operated by the user.

[1152] A "server" is a computer system used to process and store data over a network.

[1153] "Nonverbal information" refers to information conveyed through means other than words, such as facial expressions and gestures.

[1154] A "notification" is a message or alert sent from a system to a user to inform them of information.

[1155] This invention develops an interview support tool utilizing generative AI and implements it into a human resources management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript, and when the interview ends, it summarizes the transcript and stores the summary in the human resources management system. It also includes a function to automatically reflect the content of the interview in a to-do list and calendar.

[1156] Hardware and software to be used

[1157] server

[1158] The server provides computing resources to run generative AI models. Specifically, it uses speech recognition software to convert audio data into text data and a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The server also stores the summarized transcript in the human resources management system's database and automatically updates to-do lists and calendars based on the interview content.

[1159] terminal

[1160] The terminal is a device used by the user to conduct interviews. When an interview begins, the terminal uses its built-in microphone to collect audio data and transmits it to the server in real time. When the interview ends, the terminal receives a summarized transcript and updated to-do list / calendar information from the server and notifies the user.

[1161] User

[1162] The user is the person conducting the interview. The user initiates the interview via their device and does not need to perform any operations during the interview. Once the interview is complete, the user reviews the summarized transcript via their device and manages updates to their to-do list and calendar.

[1163] Specific example

[1164] Example of a prompt

[1165] "Please summarize the following interview content: 'Hello, today I'd like to discuss your work progress. First, could you tell me about the progress of your project last week?'"

[1166] Processing flow

[1167] 1. Start of interview: The user launches the dedicated application on their device and clicks the "Start Interview" button.

[1168] 2. Voice data collection: The device uses its built-in microphone to collect voice data and transmits it to the server in real time.

[1169] 3. Transcript creation: The server uses speech recognition software to convert the audio data into text data.

[1170] 4. Summary generation: The server uses a generation AI model to summarize the verbatim transcript.

[1171] 5. Data storage: The server stores the summarized verbatim transcripts in the human resources management system database.

[1172] 6. To-Do List & Calendar Update: The server automatically updates the to-do list and calendar based on the interview content.

[1173] 7. User Notifications: The device notifies the user of summarized verbatim transcripts and update information.

[1174] This system reduces the burden of creating interview records, allowing you to focus on observing nonverbal information during interviews. Furthermore, it efficiently manages interview content and automatically reflects it in to-do lists and calendars, making post-interview follow-up easier.

[1175] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1176] Step 1:

[1177] Interview begins

[1178] Subject: User

[1179] The user launches a dedicated application on their terminal and clicks the "Start Interview" button. This causes the terminal to send an interview start request to the server. The input is the user's action, and the output is the interview start request. Specifically, the terminal displays "Interview started."

[1180] Step 2:

[1181] Audio data collection

[1182] Subject: terminal

[1183] The terminal begins collecting audio data using its built-in microphone as soon as the interview starts. The collected audio data is sent to the server in real time. The input is the user's voice, and the output is the audio data sent to the server. Specifically, the terminal displays "Collecting audio data" and shows the collection status in real time.

[1184] Step 3:

[1185] Transcript creation

[1186] Subject: Server

[1187] The server receives audio data sent from the terminal and converts it into text data using speech recognition software. The input is audio data, and the output is text data. Specifically, the server displays "Creating verbatim transcript" and updates the progress in real time.

[1188] Step 4:

[1189] Summary generation

[1190] Subject: Server

[1191] The server uses a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The input is the text data of the verbatim transcript, and the output is the summarized text. Specifically, the server displays "Generating summary" and updates the progress in real time.

[1192] Step 5:

[1193] Data storage

[1194] Subject: Server

[1195] The server stores the summarized verbatim transcript in the human resources management system's database. The input is the summarized text, and the output is the record stored in the database. Specifically, the server displays "Storing data" and, once storage is complete, displays "Data storage complete."

[1196] Step 6:

[1197] To-do list / calendar update

[1198] Subject: Server

[1199] The server automatically updates the to-do list and calendar based on the interview content. The input is a summarized verbatim transcript, and the output is the updated to-do list and calendar. Specifically, the server displays "Updating to-do list and calendar" and "Update complete" when the update is finished.

[1200] Step 7:

[1201] User notifications

[1202] Subject: terminal

[1203] The terminal receives summarized verbatim transcripts and updated to-do list / calendar information from the server and notifies the user. The input is the notification information from the server, and the output is the notification to the user. Specifically, the terminal displays "Summary and update information received" and notifies the user.

[1204] (Application Example 1)

[1205] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1206] Traditional interview support tools often require manual tasks such as creating and summarizing interview records and updating to-do lists and calendars, which is time-consuming and labor-intensive. Similar problems exist in interviews between factory workers and robots, making efficient work instructions and schedule management difficult.

[1207] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for developing an interview support tool utilizing generation AI and implementing it in the personnel management system, means for using generation AI to create verbatim transcripts and summaries of interviews and storing the interview records in the personnel management system, means for automatically reflecting them in to-do lists and calendars, means for supporting interviews between workers and robots in the factory, and means for automatically updating work instructions and schedules based on the interview content. As a result, the burden of creating interview records is reduced, and efficient work instructions and schedule management become possible.

[1208] "Generative AI" refers to a system that uses artificial intelligence technology to generate data and automate specific tasks.

[1209] An "interview support tool" is a software or hardware system designed to assist with the progress and record-keeping of interviews.

[1210] A "human resources management system" is a system used by companies and organizations to manage employee information and to perform labor management and talent development.

[1211] A "verbatim transcript" is a document that meticulously records the content of conversations during interviews, meetings, or other similar events.

[1212] A "summary" is a document that extracts the key points from detailed records, such as verbatim transcripts, and presents them concisely.

[1213] A "to-do list" is a list of tasks or work items that need to be done.

[1214] A "calendar" is a tool for managing appointments and events based on dates and times.

[1215] A "worker" is an employee who actually performs work in a factory or on a construction site.

[1216] A "robot" is a mechanical device that automatically performs tasks according to programmed instructions.

[1217] "Work instructions" are specific instructions or orders for performing a particular task.

[1218] A "schedule" is a plan of tasks or events that should be completed within a specific period of time.

[1219] As an embodiment of this invention, a system is constructed to support interviews between factory workers and robots. The specific method of implementation is shown below.

[1220] System Configuration

[1221] 1. Hardware Configuration

[1222] Microphone: Used to collect interview audio.

[1223] Computer: Used to process audio data and run generative AI models.

[1224] Robot: An automated machine or device that performs tasks within a factory.

[1225] 2. Software Configuration

[1226] OpenAI API: Used for creating verbatim transcripts and generating summaries of audio.

[1227] Python: A programming language used for data processing and file manipulation.

[1228] Human Resources Management System: A system for storing interview records and summaries and reflecting them in to-do lists and calendars.

[1229] Data processing and data calculation

[1230] 1. Collection of audio data and creation of verbatim transcripts.

[1231] The server collects the interview audio through the microphone.

[1232] The collected audio data is converted to text using the OpenAI API, and a verbatim transcript is created.

[1233] 2. Summary of the verbatim transcript

[1234] The server summarizes the generated verbatim transcript using the OpenAI API.

[1235] The summarized text is stored in the human resources management system.

[1236] 3. Reflection in To-Do list and calendar

[1237] The server extracts key tasks from the summarized text and adds them to a to-do list.

[1238] Additionally, the schedule will be updated based on the interview content and reflected in the calendar.

[1239] Specific example

[1240] For example, if during an interview someone says, "We need to check the next maintenance schedule," that statement will be recorded in the verbatim transcript. After the interview, the server will summarize the transcript and generate a summary titled "Check the next maintenance schedule." This summary will be stored in the HR management system, and "Check the next maintenance schedule" will be automatically added to the ToDo list.

[1241] Example of a prompt

[1242] Examples of prompt sentences for summarizing verbatim transcripts are as follows:

[1243] Please summarize the following verbatim transcript:

[1244] [Verbatim text]

[1245] summary:

[1246] The following is an example of a prompt message for extracting items from a to-do list.

[1247] Extract the to-do list items from the following summary:

[1248] [Summary text]

[1249] To-do list:

[1250] In this way, it becomes possible to efficiently support meetings between factory workers and robots, and to automate work instructions and schedule management.

[1251] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1252] Step 1:

[1253] The server collects interview audio via a microphone. The input is the interview audio, and the output is audio data. Specifically, the microphone captures the audio during the interview in real time and sends that audio data to the server.

[1254] Step 2:

[1255] The server converts the collected audio data into text using the OpenAI API and creates a verbatim transcript. The input is audio data, and the output is the verbatim transcript text. Specifically, the audio data is input into the OpenAI speech recognition model, and the generated text is saved as a verbatim transcript.

[1256] Step 3:

[1257] The server summarizes the generated verbatim transcript using the OpenAI API. The input is the verbatim transcript text, and the output is the summarized text. Specifically, prompt sentences for summarizing the verbatim transcript text are input to the OpenAI text generation model, and the generated summary is obtained.

[1258] Step 4:

[1259] The server stores the summarized text in the human resources management system. The input is the summarized text, and the output is the summarized data stored in the human resources management system. Specifically, it performs the operation of saving the summarized text to the human resources management system's database.

[1260] Step 5:

[1261] The server extracts important tasks from the summarized text and adds them to a to-do list. The input is the summarized text, and the output is the updated to-do list. Specifically, the summarized text is input to an OpenAI text generation model as prompts for task extraction, and the generated tasks are added to the to-do list.

[1262] Step 6:

[1263] The server updates the schedule based on the interview content and reflects it in the calendar. The input is a summary text, and the output is the updated calendar. Specifically, it extracts schedule-related information from the summary text and adds that information to the calendar system.

[1264] Step 7:

[1265] The user checks their updated to-do list and calendar and performs the necessary tasks. The input is the updated to-do list and calendar, and the output is the user's completed tasks. Specifically, the user logs into the system, checks their to-do list and calendar, and performs the instructed tasks.

[1266] (Example 2)

[1267] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1268] Traditional human resource management systems often required manual processes for creating and storing interview records and developing employee training plans, resulting in time-consuming and labor-intensive tasks. Furthermore, they lacked sufficient opportunities to identify employees' strengths and weaknesses based on interview records, create individualized training plans, and propose career paths, making effective, personalized employee development challenging.

[1269] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1270] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that utilizes generative AI to create verbatim transcripts and summaries of interviews and stores the interview records in the human resources management system; a server that analyzes the interview records to extract the strengths and weaknesses of employees; a server that creates individual training plans based on the extracted data; a server that proposes career paths based on the interests and aspirations of employees; and a server that automatically reflects the results in to-do lists and calendars. As a result, the creation and storage of interview records are automated, and the extraction of employee strengths and weaknesses, the creation of individual training plans, and the proposal of career paths can be carried out efficiently, enabling effective training tailored to each employee.

[1271] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[1272] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[1273] A "human resources management system" is software or a system used to manage employee information and support human resources tasks such as training and performance evaluation.

[1274] "Transcript creation" is the process of recording the content of an interview exactly as it was said.

[1275] "Summarizing" is the process of concisely summarizing the content of an interview.

[1276] An "interview record" is data that records the content of an interview.

[1277] "Storage" is the process of saving data and making it accessible as needed.

[1278] "Analysis" is the process of analyzing data and extracting useful information.

[1279] "Strengths" refer to the skills and abilities that employees excel at.

[1280] A "weakness" refers to the skills or abilities that an employee struggles with.

[1281] A "training plan" is a plan designed to support employees in improving their skills and advancing their careers.

[1282] "Interest" refers to the areas or activities that employees are interested in.

[1283] "Orientation" refers to the direction and goals that employees strive for.

[1284] A "career path" is the outline of the jobs and roles that an employee should pursue in the future.

[1285] A "to-do list" is a list of tasks that need to be done.

[1286] A "calendar" is a tool for managing appointments and schedules.

[1287] This invention develops an interview support tool that utilizes generative AI and implements it into a human resources management system to efficiently create and store interview records, develop employee training plans, and propose career paths. A specific embodiment of this system is described below.

[1288] Hardware and software to be used

[1289] Hardware: Servers, terminals (PCs, tablets, etc.)

[1290] Software: Human resource management systems, generative AI models (e.g., OpenAI GPT-4)

[1291] Program processing

[1292] 1. Entering interview records

[1293] After the interview, the user uses a device (PC or tablet) to enter the interview record into the human resources management system.

[1294] Specifically, the user accesses an input form in the human resources management system and enters the interview details in text format.

[1295] Example: "The user uses a terminal to input the interview record for employee A. Strengths: project management, weaknesses: presentation skills."

[1296] 2. Storage of interview records

[1297] The server receives interview records submitted by users and stores them in the human resources management system's database.

[1298] Specifically, the server converts the input data into the appropriate format and saves it to the database.

[1299] Example: "The server receives the interview record and saves it to the database."

[1300] 3. Data Extraction and Analysis

[1301] The server analyzes the stored interview records to extract the employee's strengths and weaknesses.

[1302] Specifically, the server inputs the interview records into a generated AI model (e.g., OpenAI GPT-4) and retrieves the analysis results.

[1303] Example: "The server inputs interview records into a generated AI model and extracts employee A's strengths and weaknesses."

[1304] 4. Creating a training plan

[1305] The server creates individual training plans based on the extracted data.

[1306] In terms of specific operations, the server uses the analysis results of the generated AI model to suggest training and workshops necessary for improving employees' skills.

[1307] Example: "The server proposes project management training, leveraging employee A's strengths. To address their weaknesses, it plans training to improve their presentation skills."

[1308] 5. Proposed Career Paths

[1309] The server suggests appropriate career paths based on employees' interests and aspirations.

[1310] Specifically, the server uses the analysis results of the generated AI model to design future career paths for employees and reflects the proposed content in the human resources management system.

[1311] Example: "Since employee A is interested in project management, the server suggests a career path to becoming a project manager in the future."

[1312] Specific example

[1313] Example of interview record entry:

[1314] "The user uses a terminal to input interview records for employee B. Strengths include data analysis, weaknesses include teamwork."

[1315] Examples of prompts for a generative AI model:

[1316] "Based on the following interview record, identify employee B's strengths and weaknesses and propose an appropriate training plan. Interview record: Strengths: data analysis, Weaknesses: teamwork."

[1317] This system automates the creation and storage of interview records, enabling efficient identification of employees' strengths and weaknesses, creation of individual training plans, and proposal of career paths, thereby facilitating effective training tailored to each employee.

[1318] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1319] Step 1:

[1320] Entering interview records

[1321] After the interview, the user uses a device (PC or tablet) to enter the interview record into the human resources management system.

[1322] Input: Interview content (text format), employee's strengths and weaknesses, and other interview information.

[1323] Specific operation: The user accesses an input form in the HR management system and enters the interview details in text format. For example, they might enter, "Entering interview record for employee A. Strengths: project management, Weaknesses: presentation skills."

[1324] Output: The entered interview record is sent to the HR management system.

[1325] Step 2:

[1326] Storage of interview records

[1327] The server receives interview records submitted by users and stores them in the human resources management system's database.

[1328] Input: Interview records submitted by the user.

[1329] Specific operation: The server converts the input data into the appropriate format and saves it to the database. For example, the server receives an interview record and saves "Employee A's Interview Record" to the database.

[1330] Output: Interview records stored in the database.

[1331] Step 3:

[1332] Data extraction and analysis

[1333] The server analyzes the stored interview records to extract the employee's strengths and weaknesses.

[1334] Input: Interview records stored in the database.

[1335] Specific operation: The server inputs interview records into a generating AI model (e.g., OpenAI GPT-4) and obtains the analysis results. For example, the server inputs "Employee A's interview records" into the generating AI model and extracts strengths and weaknesses.

[1336] Output: Employee strengths and weaknesses as analyzed.

[1337] Step 4:

[1338] Creating a training plan

[1339] The server creates individual training plans based on the extracted data.

[1340] Input: Employee strengths and weaknesses as analyzed.

[1341] Specific operation: Based on the analysis results of the generated AI model, the server proposes training and workshops necessary for improving employees' skills. For example, the server might create a training plan such as, "Leverage employee A's strengths and propose project management training. To address their weaknesses, plan training to improve their presentation skills."

[1342] Output: Individual training plan.

[1343] Step 5:

[1344] Career path proposals

[1345] The server suggests appropriate career paths based on employees' interests and aspirations.

[1346] Input: Employees' interests and preferences as analyzed.

[1347] Specific operation: Based on the analysis results of the generated AI model, the server designs future career paths for employees and reflects the proposed content in the human resources management system. For example, the server might propose a career path such as, "Since employee A is interested in project management, we propose a career path to become a project manager in the future."

[1348] Output: Proposed career paths.

[1349] (Application Example 2)

[1350] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1351] Traditional HR management systems made creating and managing interview records cumbersome, leaving line managers with little time to observe nonverbal information during interviews. Furthermore, there was a lack of systems capable of efficiently collecting and analyzing robot work records to propose optimal work assignments and improvement plans. This made it difficult to improve employee and robot skills and to efficiently allocate tasks.

[1352] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1353] This invention includes means for the server to develop an interview support tool utilizing generative AI and implement it in a human resources management system; means for using generative AI to create verbatim transcripts and summaries of interviews and store the interview records in the human resources management system; means for automatically reflecting the results in to-do lists and calendars; means for collecting robot work records and analyzing efficiency and error rates; means for identifying the robot's strengths and weaknesses and proposing optimal work assignments; and means for creating individual improvement plans. As a result, the burden of creating interview records is reduced, line managers can concentrate on nonverbal information, and it becomes possible to analyze the robot's work efficiency and error rates and propose optimal work assignments and improvement plans.

[1354] "Generative AI" refers to a system that uses artificial intelligence technology to generate data and automate or assist with specific tasks.

[1355] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[1356] A "human resources management system" is a system that manages employee information and streamlines human resources tasks such as evaluation, training, and placement.

[1357] "Vertical transcript creation" is the process of recording conversations, such as interviews and meetings, exactly as they happened.

[1358] A "summary" refers to a condensed version of a long text or conversation.

[1359] A "to-do list" is a tool for managing tasks and appointments in a list format.

[1360] "Automatically add to calendar" is a feature that automatically adds specific events or tasks to your calendar.

[1361] "Robot work log" refers to a detailed record of the tasks performed by the robot.

[1362] "Efficiency" is an indicator that shows how effectively a task or process is performed.

[1363] The "error rate" is an indicator that shows how often errors occur in a task or process.

[1364] A "specialized task" refers to a task that a particular robot can perform with higher efficiency than other tasks.

[1365] A "task that a robot is not good at" refers to a task that a particular robot can perform with lower efficiency than other tasks.

[1366] "Optimal work arrangement" refers to positioning robots and employees in a way that allows them to perform their tasks most efficiently.

[1367] An "improvement plan" refers to a specific plan aimed at improving work efficiency and reducing error rates.

[1368] The system for implementing this invention involves developing an interview support tool utilizing generative AI and implementing it in a human resources management system. Specifically, it uses generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system. It also includes a function to automatically reflect the information in to-do lists and calendars.

[1369] Furthermore, it includes a function to collect robot work records and analyze efficiency and error rates. This allows for understanding the robot's strengths and weaknesses and proposing optimal work assignments. It also includes a function to create individual improvement plans.

[1370] Hardware and software to be used

[1371] Hardware: Servers, factory robots, user terminals

[1372] Software: Python, JSON, Generative AI Model

[1373] Data processing and data calculation

[1374] The server generates a verbatim transcript of the interview using an AI model and then summarizes it. This data is stored in JSON format in the HR management system. Furthermore, information extracted from the interview records is automatically reflected in to-do lists and calendars.

[1375] Robot work logs are collected to calculate efficiency and error rates. This data is analyzed on a server to identify the robot's strengths and weaknesses in different tasks. Based on the analysis, optimal work assignments and individual improvement plans are generated.

[1376] Specific example

[1377] For example, consider a scenario where records of a factory robot's "assembly work" are saved, and improvement plans are created based on those records. The robot's work records are collected as follows:

[1378] Example of a prompt:

[1379] Robot ID: robot_1

[1380] Task: Assembly

[1381] Efficiency: 0.9

[1382] Error rate: 0.05

[1383] By inputting this prompt into the generating AI model, the process of saving, analyzing, and creating improvement plans for robot work records is automated. This reduces the burden of creating interview records, allowing line managers to focus on nonverbal information, and enables them to analyze robot work efficiency and error rates, and propose optimal work assignments and improvement plans.

[1384] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1385] Step 1:

[1386] The server generates a verbatim transcript of the interview using a generative AI model. It receives audio data of the interview as input and generates the verbatim transcript using the generative AI model. The output is the verbatim transcript in text format.

[1387] Step 2:

[1388] The server summarizes the generated verbatim transcript. It receives the text data of the verbatim transcript as input and applies a summarization algorithm. The output is the summarized text.

[1389] Step 3:

[1390] The server stores verbatim transcripts and summaries in JSON format in the human resources management system. It receives the text data of the verbatim transcripts and summaries as input and converts it to JSON format. As output, the JSON data is stored in the human resources management system.

[1391] Step 4:

[1392] The server automatically updates the to-do list and calendar based on information extracted from the interview records. It receives JSON data of the interview records as input and converts it into the to-do list and calendar format. As output, new tasks and events are added to the to-do list and calendar.

[1393] Step 5:

[1394] The server collects robot work records. It receives work data (work ID, efficiency, error rate, etc.) transmitted from the robot as input. The collected work records are stored in a database as output.

[1395] Step 6:

[1396] The server analyzes the collected work records. It receives work records stored in a database as input and calculates efficiency and error rates. The analysis results are output.

[1397] Step 7:

[1398] The server identifies the tasks that the robot is good at and bad at. It receives analysis results as input and applies an algorithm to identify the robot's strengths and weaknesses. The output is a list of tasks the robot is good at and bad at.

[1399] Step 8:

[1400] The server proposes the optimal work assignment. It receives a list of tasks the server is good at and bad at as input, and applies an algorithm to calculate the optimal work assignment. The output is a proposed optimal work assignment.

[1401] Step 9:

[1402] The server creates individual improvement plans. It receives analysis results and optimal work allocation suggestions as input and applies an algorithm to generate improvement plans. The output is an individual improvement plan.

[1403] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1404] "Example of form 1"

[1405] One example of this invention is a system that incorporates an emotion engine. In this system, a tool for supporting interviews using generative AI is developed and implemented in a human resources management system. The generative AI creates and summarizes verbatim transcripts of interviews and stores the results in the human resources management system. Furthermore, it also has a function to automatically reflect the content of the interview in a to-do list and calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, and word choice, and the results are added to the interview record. This makes it possible to grasp not only the content of the interview but also the user's emotional state at that time, enabling a deeper understanding and response.

[1406] "Example of form 2"

[1407] As an example of the second form of this invention, there is a system that reduces the burden on line managers in creating interview records, allowing them to concentrate on observing nonverbal information during interviews. In this system, a generating AI creates and summarizes the verbatim transcript of the interview, and stores the results in the personnel management system. Furthermore, it also has a function to automatically reflect the content of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, it analyzes emotions from the user's tone of voice, facial expressions, and word choice, and adds the results to the interview record. As a result, the burden on line managers in creating interview records is reduced, allowing them to concentrate on observing nonverbal information during interviews.

[1408] "Example of form 4"

[1409] As an example of the fourth form of this invention, there is a system that creates an environment for continuous employee training by utilizing implementation records. In this system, a generating AI creates and summarizes verbatim transcripts of interviews and stores the results in a human resources management system. Furthermore, it also has a function to automatically reflect the content of the interviews in a to-do list and calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, it analyzes emotions from the user's tone of voice, facial expressions, and word choice, and adds the results to the interview record. This makes it possible to create an environment for continuous employee training by utilizing the interview records.

[1410] The following describes the processing flow for each example of the form.

[1411] "Example of form 1"

[1412] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[1413] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[1414] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[1415] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[1416] Step 5: The results of the emotional engine analysis are reflected in the interview record.

[1417] "Example of form 2"

[1418] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[1419] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[1420] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[1421] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[1422] Step 5: The results of the emotion engine analysis are reflected in the interview record, reducing the burden on the line manager in creating the interview record and allowing them to focus on observing nonverbal information during the interview.

[1423] "Example of form 4"

[1424] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[1425] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the result in the HR management system.

[1426] Step 3: The contents of the meeting will be automatically reflected in your to-do list and calendar.

[1427] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, and word choice.

[1428] Step 5: The results of the emotion engine analysis are reflected in the interview records, creating an environment where continuous employee development can be carried out by utilizing these records.

[1429] (Example 1)

[1430] Next, we will describe Embodiment 1 of Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1431] Traditional interview record-keeping was largely manual, placing a significant burden on line managers and supervisors. Furthermore, accurately recording nonverbal information and emotional states during interviews was difficult, resulting in a lack of data necessary to improve interview quality. Additionally, reflecting interview content in to-do lists and calendars was a time-consuming process, highlighting the need for more efficient management.

[1432] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1433] This invention includes means for developing and implementing an interview support tool utilizing generative AI into a human resources management system, means for using generative AI to create verbatim transcripts and summaries of interviews and storing the interview records in the human resources management system, means for automatically reflecting the results in to-do lists and calendars, means for using an emotion engine to recognize the emotional state during the interview and reflecting that information in the interview records, and means for automating the entire process from the start to the end of the interview. As a result, the burden of creating interview records is reduced, detailed interview records including nonverbal information and emotional states become possible, and the content of interviews can be managed more efficiently.

[1434] "Generative AI" refers to a system that uses artificial intelligence technology to generate text and data.

[1435] A "meeting support tool" is software or a system designed to assist in conducting and recording interviews.

[1436] A "human resources management system" is an integrated system for managing employee information, evaluation, and training.

[1437] A "verbatim transcript" is a document that meticulously records the content of conversations during interviews, meetings, or other similar events.

[1438] A "summary" is a document that extracts the key points from detailed records, such as verbatim transcripts, and summarizes them concisely.

[1439] A "to-do list" is a list that displays tasks or action items that need to be completed in a list format.

[1440] A "calendar" is a tool that displays dates and times for schedule management.

[1441] An "emotional engine" is a technology or system for analyzing emotional states based on factors such as voice, facial expressions, and word choice.

[1442] "Nonverbal information" refers to information conveyed through means other than words, and includes facial expressions, gestures, and tone of voice.

[1443] "Process automation" refers to the automatic execution of a series of tasks that were previously performed manually by a system.

[1444] This invention develops an interview support tool utilizing generative AI and implements it into a human resources management system. A specific embodiment of this system is described below.

[1445] System Configuration

[1446] The system consists of three main elements: a server, a terminal, and a user. The server is equipped with a generative AI model and an emotion engine, and is responsible for creating verbatim transcripts of interviews, summarizing them, recognizing emotional states, and automatically updating to to-do lists and calendars. The terminal is responsible for collecting audio data from interviews and sending it to the server. The user conducts interviews and operates the system.

[1447] Hardware and software to be used

[1448] Server: A server with high-performance computing capabilities (e.g., a cloud server)

[1449] Generative AI models: Natural language processing models such as OpenAI's GPT-4.

[1450] Emotion engine: Emotion recognition API, etc.

[1451] Human Resources Management System: Integrated System

[1452] To-do list and calendar: Schedule management tools

[1453] Device: A computer or tablet equipped with a microphone for collecting audio data.

[1454] Data processing and data calculation

[1455] When a user begins a conversation, the device collects audio data and sends it to the server in real time. The server inputs the received audio data into a generating AI model to produce a verbatim transcript. The generated transcript is temporarily stored on the server.

[1456] Once the interview is complete, the server uses the generation AI model again to summarize the verbatim transcript. The summarized content is stored in the HR management system. Furthermore, the server analyzes the interview content and automatically reflects it in to-do lists and calendars.

[1457] Using an emotion engine, the server analyzes the user's tone of voice, facial expressions, and word choice to recognize their emotional state. This emotional information is also added to the verbatim transcript and stored in the personnel management system.

[1458] Examples of specific cases and prompt statements

[1459] As a concrete example, consider the following scenario.

[1460] scenario:

[1461] The user conducts a meeting with a subordinate. During the meeting, the subordinate reports on the project's progress and sets several tasks as the next steps. The subordinate also expresses concerns about the project.

[1462] Example of a prompt:

[1463] "We will now begin the interview with your subordinate. Please create a verbatim transcript and summarize it. Also, analyze their emotional state during the interview and reflect this in your to-do list and calendar."

[1464] This system reduces the burden of creating interview records for users, allowing them to focus on observing nonverbal information during interviews. It also enables efficient management of interview content and the retention of detailed records, including emotional states.

[1465] The flow of the specific processing in Example 1 will be explained using Figure 15.

[1466] Step 1:

[1467] The user initiates the interview. The user clicks the "Start Interview" button on the terminal, notifying the system that they wish to begin the interview. The input is the user's action, and the output is the signal that the interview has started. The terminal displays the message "Starting interview."

[1468] Step 2:

[1469] The terminal collects audio data in real time during the interview. The input is the user's voice, and the output is the collected audio data. The terminal uses its built-in microphone to collect audio data and sends it to the server in real time. The server stores the received audio data in a buffer.

[1470] Step 3:

[1471] The server inputs the received audio data into a generating AI model to generate a verbatim transcript. The input is audio data, and the output is the generated verbatim transcript. The server inputs the audio data stored in the buffer into the generating AI model to generate a verbatim transcript. The generated verbatim transcript is stored in the server's temporary storage.

[1472] Step 4:

[1473] Once the interview is complete, the server uses the generative AI model again to summarize the verbatim transcript. The input is the verbatim transcript, and the output is the summarized verbatim transcript. The server re-inputs the verbatim transcript into the generative AI model to generate the summary. The summarized content is stored in temporary storage.

[1474] Step 5:

[1475] The server stores the summarized verbatim transcript in the human resources management system. The input is the summarized verbatim transcript, and the output is the data stored in the human resources management system. The server saves the summarized verbatim transcript to the human resources management system's database. Once saving is complete, the user receives a notification stating, "The summary has been saved."

[1476] Step 6:

[1477] The server analyzes the summarized verbatim transcript and adds the necessary information to the to-do list and calendar. The input is the summarized verbatim transcript, and the output is an updated to-do list and calendar. For example, action items such as "Schedule the next meeting" or "Complete a specific task" are automatically added.

[1478] Step 7:

[1479] The server uses an emotion engine to analyze the user's emotional state from audio data and verbatim transcripts. The input is audio data and verbatim transcripts, and the output is recognized emotional information. For example, if a user expresses "anxiety," this information is recognized by the emotion engine.

[1480] Step 8:

[1481] The server adds the recognized sentiment information to the verbatim transcript and stores it in the personnel management system. The input is the recognized sentiment information, and the output is the updated verbatim transcript. The server adds the recognized sentiment information to the verbatim transcript and saves it to the personnel management system's database. Once saving is complete, the user is notified that "Sentiment information has been saved."

[1482] (Application Example 1)

[1483] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1484] Conventional interview support systems often require manual tasks such as creating verbatim transcripts, summarizing interviews, and recognizing emotions, leading to a heavy workload in interview record-keeping. Furthermore, it can be difficult to focus on observing nonverbal information during interviews, potentially lowering the quality of the interview. Additionally, the lack of features to automatically reflect interview content in to-do lists or calendars resulted in insufficient follow-up after interviews. To address these challenges, an efficient interview support system utilizing generative AI is necessary.

[1485] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1486] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that utilizes generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system; a server that automatically reflects the results in to-do lists and calendars; a server that combines an emotion engine to recognize the user's emotions during the interview and reflects that information in the interview records; a server that uses speech recognition technology to transcribe the interview audio into verbatim transcripts; a server that uses a generative AI model to summarize the verbatim transcripts and analyze emotions; and a server that saves the results of the summarization and emotion analysis as calendars and interview records. This reduces the burden of creating interview records and allows for greater focus on observing nonverbal information during interviews. Furthermore, it enables more efficient follow-up after interviews and improves the quality of interviews.

[1487] "Generative AI" refers to a system that uses artificial intelligence technology to generate, analyze, and summarize data.

[1488] An "interview support tool" is a combination of software or hardware designed to assist in the progress of an interview and efficiently create records.

[1489] A "human resources management system" is a system used by companies and organizations to manage employee information and to conduct evaluations and training.

[1490] A "verbatim transcript" is a record that transcribes the audio of an interview or meeting exactly as it was spoken.

[1491] A "summary" is a shortened version of long-form data, such as verbatim transcripts, that extracts the most important points.

[1492] "Interview records" refer to documents or data that record the content of an interview, including verbatim transcripts, summaries, and sentiment analysis results.

[1493] A "to-do list" is a list of tasks that need to be done.

[1494] A "calendar" is a tool for managing appointments and schedules.

[1495] An "emotion engine" is a system that analyzes emotions from voice and text data and outputs the results.

[1496] "Speech recognition technology" is a technology that converts speech into text.

[1497] A "generative AI model" is a specific implementation of generative AI, a machine learning model trained to perform a particular task.

[1498] "Nonverbal information" refers to information conveyed through means other than words, including facial expressions, tone of voice, and gestures.

[1499] As an example of how to implement this invention, a factory robot interview support system will be used. This system utilizes generative AI to create verbatim transcripts of interviews, summarize them, recognize emotions, and automatically update to to-do lists and calendars.

[1500] The server will develop an interview support tool utilizing generative AI and implement it into the human resources management system. Specifically, it will use speech recognition technology to transcribe interview audio into verbatim text, summarize the transcript using a generative AI model, and analyze emotions. This will reduce the burden of creating interview records and allow staff to focus on observing nonverbal information during interviews.

[1501] The server summarizes the verbatim transcript using a generative AI model and analyzes the sentiment. For example, Library A is used as the generative AI model. Library B is used for speech recognition technology. This automates the creation and summarization of the interview transcript.

[1502] The server combines an emotion engine to recognize the user's emotions during the interview and reflects that information in the interview record. A predetermined pipeline is used as the emotion engine. This allows for the understanding of not only the content of the interview but also the user's emotional state at that time.

[1503] The server saves summaries and sentiment analysis results as calendars and interview records. The saved data is automatically reflected in to-do lists and calendars. This allows for more efficient follow-up after interviews and improves the quality of the interviews.

[1504] As a concrete example, a summary can be obtained by inputting the following prompt sentence into the AI ​​model:

[1505] Example of a prompt:

[1506] "Today, I'd like to discuss the efficiency of the new production line. While the current production speed isn't meeting our target, there are several areas for improvement. First, we need to review the machine maintenance schedule. We also need to update the worker training program."

[1507] By inputting this prompt into the AI ​​model, a summary can be obtained. This allows for an efficient understanding of the interview content and enables quick action to be taken as needed.

[1508] The flow of a specific process in Application Example 1 will be explained using Figure 16.

[1509] Step 1:

[1510] The server uses speech recognition technology to transcribe the interview audio into text as a verbatim transcript.

[1511] Input: Audio data of the interview

[1512] Output: Verbatim transcript (text data)

[1513] Specific operation: The server converts the audio data acquired from the microphone into text. This records the content of the interview as a verbatim transcript.

[1514] Step 2:

[1515] The server uses a generative AI model to summarize the verbatim transcript.

[1516] Input: Verbatim transcript (text data)

[1517] Output: Summary (text data)

[1518] Specific operation: The server summarizes the verbatim transcript. Specifically, it extracts the key points from the transcript and summarizes them concisely.

[1519] Step 3:

[1520] The server uses an emotion engine to analyze the emotions in the verbatim transcript.

[1521] Input: Verbatim transcript (text data)

[1522] Output: Sentiment analysis results (text data)

[1523] Specific operation: The server analyzes the emotions in the verbatim transcript. Specifically, it recognizes emotions from the user's tone of voice and word choice, and outputs the results.

[1524] Step 4:

[1525] The server saves the summary and sentiment analysis results as a calendar and interview records.

[1526] Input: Summary (text data), Sentiment analysis results (text data)

[1527] Output: Calendar entries, interview records (data in JSON format)

[1528] Specific operation: The server saves the summary and sentiment analysis results in JSON format and automatically reflects them in the calendar and to-do list. This allows for efficient follow-up after interviews.

[1529] Step 5:

[1530] The user reviews the interview record and takes the necessary actions.

[1531] Input: Calendar entries, interview records (data in JSON format)

[1532] Output: Action plan (text data)

[1533] Specific operation: The user reviews the interview records and calendar entries provided by the server and plans the necessary actions. This enables specific responses based on the content of the interview.

[1534] (Example 2)

[1535] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1536] Traditional methods for creating interview records required line managers to manually create the records, which was a significant burden. Furthermore, it was difficult to appropriately observe and reflect nonverbal information during interviews (such as tone of voice and facial expressions). Additionally, there was no established environment for utilizing interview records for continuous employee development.

[1537] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for developing an interview implementation support tool utilizing generation AI and implementing it in the personnel management system; means for using generation AI to create verbatim transcripts and summaries of interviews and storing the interview records in the personnel management system; means for collecting the tone of voice and facial expressions of the user during the interview, analyzing emotions using an emotion engine, and reflecting the results in the interview records; means for automatically reflecting the interview content in a to-do list or calendar; and means for storing the verbatim transcripts, summaries, and emotion analysis results in the personnel management system. As a result, the burden of creating interview records on line managers is reduced, allowing them to concentrate on observing nonverbal information during interviews, and it becomes possible to create an environment for continuous employee training by utilizing the interview records.

[1538] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[1539] "Interview support tools" is a general term for software or hardware used to support the progress and record-keeping of interviews.

[1540] A "human resources management system" is an integrated software system used for managing employee information, evaluation, and training.

[1541] A "verbatim transcript" is text data that meticulously records the content of an interview or conversation.

[1542] A "summary" is text data that extracts important points and keywords from a verbatim transcript and presents them concisely.

[1543] An "emotion engine" is software or hardware that analyzes data such as voice and facial expressions to recognize a user's emotions.

[1544] "Nonverbal information" refers to information conveyed through means other than words, and specifically includes tone of voice, facial expressions, and gestures.

[1545] A "to-do list" is a tool for managing tasks and appointments in a list format.

[1546] A "calendar" is a tool for managing appointments and events based on dates and times.

[1547] "Vertical transcription" is the process of converting audio data into text and meticulously recording the content of interviews and conversations.

[1548] "Interview records" refer to data that records the content of an interview, including verbatim transcripts, summaries, and sentiment analysis results.

[1549] "Sentiment analysis" is the process of recognizing a user's emotions by analyzing data such as voice and facial expressions.

[1550] "Automatic reflection" refers to the process of automatically reflecting specific data or information into other systems or tools.

[1551] This invention is a system that uses a generation AI-powered interview support tool to create verbatim transcripts, summaries, and sentiment analyses of interviews, stores this data in a human resources management system, and automatically reflects it in to-do lists and calendars. A specific embodiment of this system is described below.

[1552] Hardware and software to be used

[1553] Hardware: Servers, terminals (PCs, tablets, smartphones)

[1554] Software: Generative AI models (e.g., GPT-4), emotion engines, human resource management systems

[1555] Data processing and data calculation workflow

[1556] 1. Collection of audio data from interviews

[1557] The user starts the interview.

[1558] The device collects audio data of the interview in real time. Specifically, it records audio using the microphone on a PC, tablet, or smartphone.

[1559] The device sends the collected audio data to the server.

[1560] 2. Creating a verbatim transcript of the audio data

[1561] The server sends the received audio data to the AI ​​model for generation.

[1562] The generative AI model converts audio data into text and creates a verbatim transcript. Specifically, it uses speech recognition technology to convert speech into text.

[1563] 3. Summary of the verbatim transcript

[1564] This process summarizes verbatim transcripts generated by generative AI models. Specifically, it extracts key points and keywords to generate a concise summary.

[1565] The server receives the summarized text and proceeds to the next processing step.

[1566] 4. Sentiment analysis

[1567] The device collects the user's voice tone and facial expressions during the interview. Specifically, it uses the camera and microphone to record changes in facial expressions and voice.

[1568] The server sends the collected data to the emotion engine.

[1569] The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from factors such as tone of voice, facial expressions, and word choice.

[1570] The server adds the results of the sentiment analysis to the verbatim transcript.

[1571] 5. Data Storage

[1572] The server stores verbatim transcripts, summaries, and sentiment analysis results in the human resources management system. Specifically, it stores them in a database.

[1573] The human resources management system manages the stored data and makes it accessible as needed.

[1574] 6. Reflection in To-Do list and calendar

[1575] The server analyzes the interview content and generates tasks that are reflected in the to-do list and calendar.

[1576] The HR management system automatically adds generated tasks to to-do lists and calendars. Specifically, it reflects them in employees' schedules and task management tools.

[1577] Specific example

[1578] For example, when employee A has a meeting with line manager B, the following specific actions are taken:

[1579] 1. The users (employee A and line manager B) begin the interview, and the terminal (PC) starts recording the audio.

[1580] 2. The device transmits the recorded audio data to the server in real time.

[1581] 3. The server sends the audio data to the AI ​​model that generates the verbatim transcript.

[1582] 4. A generative AI model summarizes the verbatim transcript and extracts the key points.

[1583] 5. The terminal collects the facial expressions and tone of voice of employee A during the interview, and the server sends this information to the emotion engine.

[1584] 6. The emotion engine analyzes the emotions and adds the results to the verbatim transcript.

[1585] 7. The server stores verbatim transcripts, summaries, and sentiment analysis results in the human resources management system.

[1586] 8. The HR management system analyzes the interview content and generates tasks that are reflected in the to-do list and calendar.

[1587] Example of a prompt

[1588] "Create a verbatim transcript and summary of the meeting between employee A and line manager B. Also, analyze employee A's emotions during the meeting and add your findings to the transcript."

[1589] This system reduces the burden on line managers in creating interview records, allowing them to focus on observing nonverbal information during interviews. Furthermore, it enables the creation of an environment for continuous employee development by utilizing interview records.

[1590] The flow of the specific processing in Example 2 will be explained using Figure 17.

[1591] Step 1:

[1592] The user starts the interview. The device collects the interview audio data in real time. Specifically, it records the audio using the microphone on a PC, tablet, or smartphone. The device sends the collected audio data to the server. The input is the interview audio data, and the output is the audio data sent to the server.

[1593] Step 2:

[1594] The server sends the received audio data to a generative AI model. The generative AI model converts the audio data into text and creates a verbatim transcript. Specifically, it uses speech recognition technology to convert speech into text. The input is the audio data sent to the server, and the output is the generated verbatim transcript.

[1595] Step 3:

[1596] This process summarizes the verbatim transcript generated by the generative AI model. Specifically, it extracts key points and keywords and generates a concise summary. The server receives the summarized text and proceeds to the next processing step. The input is the generated verbatim transcript, and the output is the summarized text.

[1597] Step 4:

[1598] The device collects the user's voice tone and facial expressions during the interview. Specifically, it uses a camera and microphone to record changes in facial expressions and voice. The server sends the collected data to the emotion engine. The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from voice tone, facial expressions, and word choice. The input is the collected voice tone and facial expression data, and the output is the result of the emotion analysis.

[1599] Step 5:

[1600] The server adds the sentiment analysis results to the verbatim transcript. Specifically, it integrates the sentiment analysis results into the verbatim transcript in text format. The input is the sentiment analysis results and the verbatim transcript, and the output is the verbatim transcript with the sentiment analysis added.

[1601] Step 6:

[1602] The server stores the verbatim transcripts, summaries, and sentiment analysis results in the human resources management system. Specifically, it stores them in a database. The human resources management system manages the stored data and makes it accessible as needed. The inputs are the verbatim transcripts, summaries, and sentiment analysis results with added sentiment analysis, and the output is the data stored in the human resources management system.

[1603] Step 7:

[1604] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar. The HR management system automatically adds the generated tasks to the to-do list and calendar. Specifically, it reflects this in the employee's schedule and task management tool. The input is the analyzed interview content, and the output is the tasks reflected in the to-do list and calendar.

[1605] (Application Example 2)

[1606] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1607] Traditional interview record systems required manual transcription, summarization, and emotional assessment, placing a significant burden on line managers and supervisors. Furthermore, the lack of effective use of interview records for employee skill development and career path design made creating individual training plans difficult. Additionally, there was a problem with ineffective training and skill development support for factory robot operators.

[1608] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1609] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that uses generative AI to create verbatim transcripts and summaries of interviews and stores the interview records in the human resources management system; a server that automatically reflects the results in to-do lists and calendars; a server that uses an emotion engine to recognize emotions during interviews and reflects that information in the interview records; and a server that records interviews and training sessions conducted by factory robot operators to support skill development and career path design for the operators. This reduces the burden of creating interview records, allows for focus on observing nonverbal information during interviews, and enables the creation of an environment for continuous employee training by utilizing the records.

[1610] "Generative AI" refers to a system that uses artificial intelligence technology to generate and analyze data.

[1611] A "meeting support tool" is software or a system designed to assist in recording and analyzing interviews.

[1612] A "human resources management system" is a system that manages employee information and supports skill development and career path design.

[1613] "Transcript creation" is the process of meticulously recording the content of an interview or conversation.

[1614] "Summarizing" is the process of condensing the content of a long text or conversation into a shorter form.

[1615] A "to-do list" is a tool for managing tasks that need to be done in a list format.

[1616] A "calendar" is a tool for managing schedules and appointments.

[1617] An "emotion engine" is a system that analyzes and recognizes emotions from speech and text.

[1618] "Interview records" refer to documents or data that record the content of an interview.

[1619] A "factory robot" is an automated machine used to perform tasks within a factory.

[1620] An "operator" is a person who operates a machine or system.

[1621] A "training session" is a place for training to improve specific skills or knowledge.

[1622] "Skill improvement" is the process of improving specific skills or abilities.

[1623] A "career path" refers to the line of work and positions that an employee should pursue in the future.

[1624] In order to implement this invention, the following system configuration and program are required.

[1625] System Configuration

[1626] The server will develop and implement an interview support tool utilizing generative AI. Using generative AI, it will create and summarize verbatim transcripts of interviews and store the interview records in the HR management system. Furthermore, it will have a function to automatically reflect these records in to-do lists and calendars. It will also use an emotion engine to recognize emotions during interviews and reflect that information in the interview records. It will record interviews and training sessions conducted by factory robot operators to support skill development and career path design for operators.

[1627] Hardware and software to be used

[1628] Hardware: Microphone, computer

[1629] Software: Python, SpeechRecognition library, Transformers library, EmotionRecognition library

[1630] Program Processing Description

[1631] The server reads the audio file and retrieves the audio data. Next, it converts the audio data into text. It uses a generative AI model to summarize the text and an emotion engine to analyze the emotions in the text. Based on this information, it creates an interview record and saves it in JSON format. The interview record is stored in the HR management system and automatically reflected in the to-do list and calendar.

[1632] Specific example

[1633] Operators conduct interviews regarding factory robot operation, and these interviews are recorded. When the recorded audio files are input into a server, a verbatim transcript and summary of the interview are generated, and the operator's emotions are also analyzed. This facilitates skill development and career path design for operators.

[1634] Example of a prompt

[1635] Please summarize the following text:

[1636] "Today we discussed the operation of the factory robots. Operator A seems to lack confidence in certain operations. However, he is very skilled in other operations, and is particularly strong in troubleshooting. In future training, we plan to create a program to address A's weaknesses."

[1637] The flow of a specific process in Application Example 2 will be explained using Figure 18.

[1638] Step 1:

[1639] The server reads audio files and retrieves audio data. Specifically, it uploads audio files recorded using a microphone to the server and retrieves audio data using the SpeechRecognition library. The input is an audio file, and the output is audio data.

[1640] Step 2:

[1641] The server converts the acquired audio data into text. Specifically, it uses the SpeechRecognition library to convert the audio data into text. The input is audio data, and the output is text data.

[1642] Step 3:

[1643] The server summarizes text data using a generative AI model. Specifically, it uses a summarization model from the Transformers library to shorten the text data. The input is text data, and the output is the summarized text.

[1644] Step 4:

[1645] The server analyzes emotions from text data using an emotion engine. Specifically, it uses the EmotionRecognition library to extract emotions from text data. The input is text data, and the output is emotion data.

[1646] Step 5:

[1647] The server creates an interview record based on summarized text and sentiment data. Specifically, it combines the summarized text and sentiment data to generate the interview record in JSON format. The input is summarized text and sentiment data, and the output is the interview record.

[1648] Step 6:

[1649] The server stores the created interview records in the human resources management system. Specifically, it saves the generated JSON-formatted interview records to the database. The input is the interview records, and the output is the records stored in the database.

[1650] Step 7:

[1651] The server automatically updates the to-do list and calendar based on the interview records. Specifically, it adds tasks and schedules extracted from the interview records to the to-do list and calendar. The input is the interview records, and the output is the updated to-do list and calendar.

[1652] Step 8:

[1653] The user records interviews and training sessions conducted by factory robot operators, supporting their skill development and career path design. Specifically, it creates individual training plans based on interview records to improve the operators' skills. The input is the interview records, and the output is the training plan.

[1654] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1655] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1656] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[1657] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1658] [Fourth Embodiment]

[1659] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1660] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1661] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1662] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1663] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1664] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1665] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1666] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1667] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1668] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1669] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1670] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1671] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[1672] "Example of form 1"

[1673] One embodiment of this invention involves developing an interview support tool utilizing generative AI and implementing it in a human resources management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript. When the interview ends, the generative AI summarizes the transcript and stores the summary in the human resources management system. It also has a function to automatically reflect the content of the interview in a to-do list and calendar. As a result, the burden of creating interview records is reduced for line managers, allowing them to concentrate on observing nonverbal information during the interview.

[1674] "Example of form 2"

[1675] Another embodiment of the present invention involves creating an environment for continuous employee development by utilizing implementation records. Specifically, interview records are stored in a human resources management system, and these records are used to design employee skill development and career paths. For example, employee strengths and weaknesses are extracted from interview records, and individual development plans are created based on these. In addition, employee interests and aspirations are understood from interview records, and career paths are proposed based on these. This makes it possible to provide development tailored to each individual employee.

[1676] The following describes the processing flow for each example of the form.

[1677] "Example of form 1"

[1678] Step 1: Once the interview begins, the generating AI starts creating a verbatim transcript.

[1679] Step 2: Once the interview is complete, the generating AI summarizes the verbatim transcript and stores the summary in the HR management system.

[1680] Step 3: It also includes a function that automatically updates the to-do list and calendar based on the content of the interview.

[1681] "Example of form 2"

[1682] Step 1: Store interview records in the HR management system and use them to design employee skill development and career paths.

[1683] Step 2: Extract the employee's strengths and weaknesses from the interview records and create an individualized training plan based on that.

[1684] Step 3: Understand the employee's interests and aspirations from the interview records, and propose a career path based on that.

[1685] (Example 1)

[1686] Next, we will describe Embodiment 1 of Example Form 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1687] Traditional interview record-keeping is often done manually, placing a significant burden on line managers and supervisors. Furthermore, the lack of time to observe nonverbal information during interviews can lead to a decline in interview quality. Additionally, the difficulty in efficiently managing interview content and reflecting it in to-do lists and calendars can result in insufficient follow-up after interviews.

[1688] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1689] This invention includes a server that develops and implements an interview support tool utilizing generative AI into a human resources management system; a server that uses generative AI to create and summarize verbatim transcripts of interviews and stores the interview records in the human resources management system; a server that collects audio data at the start of an interview and transmits it to the server; a server that uses speech recognition software to convert the audio data into text data; a server that uses a generative AI model to summarize the verbatim transcript; a server that stores the summarized verbatim transcript in a database; a server that automatically updates the to-do list and calendar based on the interview content; and a server that notifies the user of the summarized verbatim transcript and update information. This reduces the burden of creating interview records and allows for focus on observing nonverbal information during interviews. Furthermore, because interview content can be efficiently managed and automatically reflected in the to-do list and calendar, follow-up after interviews becomes easier.

[1690] "Generative AI" refers to a system that uses artificial intelligence technology to generate and process data.

[1691] An "interview support tool" is a combination of software or hardware designed to assist with the progress and record-keeping of interviews.

[1692] A "human resources management system" is an integrated system for managing employee information, evaluation, and training.

[1693] A "verbatim transcript" is a record that transcribes the audio of an interview or meeting exactly as it was spoken.

[1694] A "summary" refers to a concise compilation of detailed information, such as verbatim transcripts.

[1695] "Speech recognition software" is software used to convert speech data into text data.

[1696] "Generative AI model" refers to the core algorithms and pre-trained models of generative AI.

[1697] A "database" is a system for efficiently storing, managing, and retrieving data.

[1698] A "to-do list" is a list used to manage tasks and appointments in a list format.

[1699] A "calendar" is a tool for managing appointments and events based on dates and times.

[1700] A "terminal" refers to a device such as a computer or smartphone that is directly operated by the user.

[1701] A "server" is a computer system used to process and store data over a network.

[1702] "Nonverbal information" refers to information conveyed through means other than words, such as facial expressions and gestures.

[1703] A "notification" is a message or alert sent from a system to a user to inform them of information.

[1704] This invention develops an interview support tool utilizing generative AI and implements it into a human resources management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript, and when the interview ends, it summarizes the transcript and stores the summary in the human resources management system. It also includes a function to automatically reflect the content of the interview in a to-do list and calendar.

[1705] Hardware and software to be used

[1706] server

[1707] The server provides computing resources to run generative AI models. Specifically, it uses speech recognition software to convert audio data into text data and a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The server also stores the summarized transcript in the human resources management system's database and automatically updates to-do lists and calendars based on the interview content.

[1708] terminal

[1709] The terminal is a device used by the user to conduct interviews. When an interview begins, the terminal uses its built-in microphone to collect audio data and transmits it to the server in real time. When the interview ends, the terminal receives a summarized transcript and updated to-do list / calendar information from the server and notifies the user.

[1710] User

[1711] The user is the person conducting the interview. The user initiates the interview via their device and does not need to perform any operations during the interview. Once the interview is complete, the user reviews the summarized transcript via their device and manages updates to their to-do list and calendar.

[1712] Specific example

[1713] Example of a prompt

[1714] "Please summarize the following interview content: 'Hello, today I'd like to discuss your work progress. First, could you tell me about the progress of your project last week?'"

[1715] Processing flow

[1716] 1. Start of interview: The user launches the dedicated application on their device and clicks the "Start Interview" button.

[1717] 2. Voice data collection: The device uses its built-in microphone to collect voice data and transmits it to the server in real time.

[1718] 3. Transcript creation: The server uses speech recognition software to convert the audio data into text data.

[1719] 4. Summary generation: The server uses a generation AI model to summarize the verbatim transcript.

[1720] 5. Data storage: The server stores the summarized verbatim transcripts in the human resources management system database.

[1721] 6. To-Do List & Calendar Update: The server automatically updates the to-do list and calendar based on the interview content.

[1722] 7. User Notifications: The device notifies the user of summarized verbatim transcripts and update information.

[1723] This system reduces the burden of creating interview records, allowing you to focus on observing nonverbal information during interviews. Furthermore, it efficiently manages interview content and automatically reflects it in to-do lists and calendars, making post-interview follow-up easier.

[1724] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1725] Step 1:

[1726] Interview begins

[1727] Subject: User

[1728] The user launches a dedicated application on their terminal and clicks the "Start Interview" button. This causes the terminal to send an interview start request to the server. The input is the user's action, and the output is the interview start request. Specifically, the terminal displays "Interview started."

[1729] Step 2:

[1730] Audio data collection

[1731] Subject: terminal

[1732] The terminal begins collecting audio data using its built-in microphone as soon as the interview starts. The collected audio data is sent to the server in real time. The input is the user's voice, and the output is the audio data sent to the server. Specifically, the terminal displays "Collecting audio data" and shows the collection status in real time.

[1733] Step 3:

[1734] Transcript creation

[1735] Subject: Server

[1736] The server receives audio data sent from the terminal and converts it into text data using speech recognition software. The input is audio data, and the output is text data. Specifically, the server displays "Creating verbatim transcript" and updates the progress in real time.

[1737] Step 4:

[1738] Summary generation

[1739] Subject: Server

[1740] The server uses a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The input is the text data of the verbatim transcript, and the output is the summarized text. Specifically, the server displays "Generating summary" and updates the progress in real time.

[1741] Step 5:

[1742] Data storage

[1743] Subject: Server

[1744] The server stores the summarized verbatim transcript in the human resources management system's database. The input is the summarized text, and the output is the record stored in the database. Specifically, the server displays "Storing data" and, once storage is complete, displays "Data storage complete."

[1745] Step 6:

[1746] To-do list / calendar update

[1747] Subject: Server

[1748] The server automatically updates the to-do list and calendar based on the interview content. The input is a summarized verbatim transcript, and the output is the updated to-do list and calendar. Specifically, the server displays "Updating to-do list and calendar" and "Update complete" when the update is finished.

[1749] Step 7:

[1750] User notifications

[1751] Subject: terminal

[1752] The terminal receives summarized verbatim transcripts and updated to-do list / calendar information from the server and notifies the user. The input is the notification information from the server, and the output is the notification to the user. Specifically, the terminal displays "Summary and update information received" and notifies the user.

[1753] (Application Example 1)

[1754] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1755] Traditional interview support tools often require manual tasks such as creating and summarizing interview records and updating to-do lists and calendars, which is time-consuming and labor-intensive. Similar problems exist in interviews between factory workers and robots, making efficient work instructions and schedule management difficult.

[1756] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for developing an interview support tool utilizing generation AI and implementing it in the personnel management system, means for using generation AI to create verbatim transcripts and summaries of interviews and storing the interview records in the personnel management system, means for automatically reflecting them in to-do lists and calendars, means for supporting interviews between workers and robots in the factory, and means for automatically updating work instructions and schedules based on the interview content. As a result, the burden of creating interview records is reduced, and efficient work instructions and schedule management become possible.

[1757] "Generative AI" refers to a system that uses artificial intelligence technology to generate data and automate specific tasks.

[1758] An "interview support tool" is a software or hardware system designed to assist with the progress and record-keeping of interviews.

[1759] A "human resources management system" is a system used by companies and organizations to manage employee information and to perform labor management and talent development.

[1760] A "verbatim transcript" is a document that meticulously records the content of conversations during interviews, meetings, or other similar events.

[1761] A "summary" is a document that extracts the key points from detailed records, such as verbatim transcripts, and presents them concisely.

[1762] A "to-do list" is a list of tasks or work items that need to be done.

[1763] A "calendar" is a tool for managing appointments and events based on dates and times.

[1764] A "worker" is an employee who actually performs work in a factory or on a construction site.

[1765] A "robot" is a mechanical device that automatically performs tasks according to programmed instructions.

[1766] "Work instructions" are specific instructions or orders for performing a particular task.

[1767] A "schedule" is a plan of tasks or events that should be completed within a specific period of time.

[1768] As an embodiment of this invention, a system is constructed to support interviews between factory workers and robots. The specific method of implementation is shown below.

[1769] System Configuration

[1770] 1. Hardware Configuration

[1771] Microphone: Used to collect interview audio.

[1772] Computer: Used to process audio data and run generative AI models.

[1773] Robot: An automated machine or device that performs tasks within a factory.

[1774] 2. Software Configuration

[1775] OpenAI API: Used for creating verbatim transcripts and generating summaries of audio.

[1776] Python: A programming language used for data processing and file manipulation.

[1777] Human Resources Management System: A system for storing interview records and summaries and reflecting them in to-do lists and calendars.

[1778] Data processing and data calculation

[1779] 1. Collection of audio data and creation of verbatim transcripts.

[1780] The server collects the interview audio through the microphone.

[1781] The collected audio data is converted to text using the OpenAI API, and a verbatim transcript is created.

[1782] 2. Summary of the verbatim transcript

[1783] The server summarizes the generated verbatim transcript using the OpenAI API.

[1784] The summarized text is stored in the human resources management system.

[1785] 3. Reflection in To-Do list and calendar

[1786] The server extracts key tasks from the summarized text and adds them to a to-do list.

[1787] Additionally, the schedule will be updated based on the interview content and reflected in the calendar.

[1788] Specific example

[1789] For example, if during an interview someone says, "We need to check the next maintenance schedule," that statement will be recorded in the verbatim transcript. After the interview, the server will summarize the transcript and generate a summary titled "Check the next maintenance schedule." This summary will be stored in the HR management system, and "Check the next maintenance schedule" will be automatically added to the ToDo list.

[1790] Example of a prompt

[1791] Examples of prompt sentences for summarizing verbatim transcripts are as follows:

[1792] Please summarize the following verbatim transcript:

[1793] [Verbatim text]

[1794] summary:

[1795] The following is an example of a prompt message for extracting items from a to-do list.

[1796] Extract the to-do list items from the following summary:

[1797] [Summary text]

[1798] To-do list:

[1799] In this way, it becomes possible to efficiently support meetings between factory workers and robots, and to automate work instructions and schedule management.

[1800] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1801] Step 1:

[1802] The server collects interview audio via a microphone. The input is the interview audio, and the output is audio data. Specifically, the microphone captures the audio during the interview in real time and sends that audio data to the server.

[1803] Step 2:

[1804] The server converts the collected audio data into text using the OpenAI API and creates a verbatim transcript. The input is audio data, and the output is the verbatim transcript text. Specifically, the audio data is input into the OpenAI speech recognition model, and the generated text is saved as a verbatim transcript.

[1805] Step 3:

[1806] The server summarizes the generated verbatim transcript using the OpenAI API. The input is the verbatim transcript text, and the output is the summarized text. Specifically, prompt sentences for summarizing the verbatim transcript text are input to the OpenAI text generation model, and the generated summary is obtained.

[1807] Step 4:

[1808] The server stores the summarized text in the human resources management system. The input is the summarized text, and the output is the summarized data stored in the human resources management system. Specifically, it performs the operation of saving the summarized text to the human resources management system's database.

[1809] Step 5:

[1810] The server extracts important tasks from the summarized text and adds them to a to-do list. The input is the summarized text, and the output is the updated to-do list. Specifically, the summarized text is input to an OpenAI text generation model as prompts for task extraction, and the generated tasks are added to the to-do list.

[1811] Step 6:

[1812] The server updates the schedule based on the interview content and reflects it in the calendar. The input is a summary text, and the output is the updated calendar. Specifically, it extracts schedule-related information from the summary text and adds that information to the calendar system.

[1813] Step 7:

[1814] The user checks their updated to-do list and calendar and performs the necessary tasks. The input is the updated to-do list and calendar, and the output is the user's completed tasks. Specifically, the user logs into the system, checks their to-do list and calendar, and performs the instructed tasks.

[1815] (Example 2)

[1816] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1817] Traditional human resource management systems often required manual processes for creating and storing interview records and developing employee training plans, resulting in time-consuming and labor-intensive tasks. Furthermore, they lacked sufficient opportunities to identify employees' strengths and weaknesses based on interview records, create individualized training plans, and propose career paths, making effective, personalized employee development challenging.

[1818] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1819] This invention includes a server that develops an interview support tool utilizing generative AI and implements it in a human resources management system; a server that utilizes generative AI to create verbatim transcripts and summaries of interviews and stores the interview records in the human resources management system; a server that analyzes the interview records to extract the strengths and weaknesses of employees; a server that creates individual training plans based on the extracted data; a server that proposes career paths based on the interests and aspirations of employees; and a server that automatically reflects the results in to-do lists and calendars. As a result, the creation and storage of interview records are automated, and the extraction of employee strengths and weaknesses, the creation of individual ...

Claims

[Claim 1] A means of collecting audio data during user interviews, A means for inputting the aforementioned audio data into a generating AI model, and for the generating AI model to convert the aforementioned audio data into verbatim text data, A means for inputting a prompt sentence instructing the generation AI model to summarize the verbatim transcript, and for the generation AI model to generate a summary of the verbatim transcript, The means for inputting the aforementioned audio data and the aforementioned verbatim transcript into an emotion engine, and for the emotion engine to recognize the user's emotional state, Means for storing the interview record, which includes a summary of the verbatim transcript and the emotional state of the user, The means for inputting the interview record into the generating AI model, and for the generating AI model to extract the user's strengths, which are the skills or abilities the user excels at, and the user's weaknesses, which are the skills or abilities the user is not good at, A means for inputting the user's strengths and weaknesses into the generating AI model, and for the generating AI model to propose training or workshops necessary for the user's skill improvement, A system that includes this.

Images