System

Abstract

To provide a system.SOLUTION: A system includes: first means for collecting voice data when interviewing a user; second means for generating a prompt sentence for outputting a summary of a literal record of a user at the time of an interview on the basis of text data converted from the voice data; third means for generating the summary of the literal record by inputting the generated prompt sentence to a regenerative AI model; and fourth means for notifying the user of the generated summary of the literal record.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional method of creating interview records placed a heavy burden on line managers, making it difficult for them to concentrate on observing non-verbal information during interviews. In addition, there was no environment in place for efficiently utilizing interview records and for continuous employee development. [Means for solving the problem]

[0005] By developing an interview implementation support tool that utilizes generative AI and implementing it in the personnel management system, the creation of verbatim interview records and summaries can be automated, and interview records can be stored in the personnel management system. This reduces the burden on line managers of creating interview records, allowing them to focus on observing non-verbal information during interviews. Furthermore, by automatically reflecting these records in to-do lists and calendars, an environment can be created for continuous employee development using the implementation records. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] As an embodiment of the present invention, an interview implementation support tool that utilizes generative AI is developed and implemented in a personnel management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript. When the interview ends, the generative AI summarizes the verbatim transcript and stores the summary in the personnel management system. It also has a function that automatically reflects the contents of the interview in a to-do list or calendar. This reduces the burden on line managers of creating interview records, allowing them to concentrate on observing nonverbal information during the interview.

[0029] "Example 2"

[0030] In another embodiment of the present invention, the implementation records are utilized to create an environment for continuous employee development. Specifically, interview records are stored in a personnel management system, and based on that, employee skills are improved and career paths are designed. For example, the strengths and weaknesses of employees are extracted from the interview records, and based on that, individual development plans are created. In addition, the interests and inclinations of employees are understood from the interview records, and career paths are proposed based on that. This makes it possible to provide development tailored to each individual employee.

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: When the interview begins, the generative AI starts creating a transcript.

[0034] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the summary in the HR management system.

[0035] Step 3: It also has a function that automatically updates your to-do list and calendar based on the contents of the interview.

[0036] "Example 2"

[0037] Step 1: Store the interview records in a human resources management system and use them to improve employee skills and design career paths.

[0038] Step 2: Extract the employee's strengths and weaknesses from the interview records and create an individual development plan based on that.

[0039] Step 3: Understand the employee's interests and aspirations from the interview records and use that to suggest a career path.

[0040] Example 1

[0041] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0042] Traditional interview records are often created manually, placing a heavy burden on line managers and supervisors. In addition, there is no time to observe non-verbal information during the interview, which can lead to a decline in the quality of the interview. Furthermore, it is difficult to efficiently manage the interview content and reflect it in a to-do list or calendar, which can lead to insufficient follow-up after the interview.

[0043] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0044] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for utilizing the generative AI to create and summarize a verbatim record of an interview and store the interview record in the personnel management system, a means for a terminal to collect voice data at the start of an interview and send it to the server, a means for the server to convert the voice data into text data using voice recognition software, a means for summarizing the verbatim record using a generative AI model, a means for storing the summarized verbatim record in a database, a means for automatically updating a to-do list or a calendar based on the interview content, and a means for the terminal to notify the user of the summarized verbatim record and update information. This reduces the burden of creating interview records, making it possible to concentrate on observing non-verbal information during interviews. In addition, the contents of the interview can be efficiently managed and automatically reflected in a to-do list or a calendar, making it easier to follow up after the interview.

[0045] "Generative AI" is a system that uses artificial intelligence technology to generate and process data.

[0046] An "interview implementation support tool" is a combination of software and / or hardware that assists in the conduct of interviews and the creation of records.

[0047] A "human resources management system" is an integrated system for managing employee information, evaluating, and training.

[0048] A verbatim transcript is a recording of an interview, meeting, etc., transcribed verbatim.

[0049] A "summary" is a concise summary of detailed information, such as a verbatim transcript.

[0050] "Speech recognition software" refers to software for converting voice data into text data.

[0051] "Generative AI model" refers to the core algorithm or trained model of generative AI.

[0052] A "database" is a system for efficiently storing, managing, and searching data.

[0053] A "ToDo list" is a list for managing tasks and schedules that need to be done in a list format.

[0054] A "calendar" is a tool for managing schedules and events based on dates and times.

[0055] A "terminal" is a device such as a computer or smartphone that is directly operated by a user.

[0056] A "server" is a computer system for processing and storing data over a network.

[0057] "Nonverbal information" refers to information that is conveyed by means other than words, such as facial expressions and gestures.

[0058] A "notification" is a message or alert sent from the system to inform the user of information.

[0059] This invention involves developing an interview support tool that utilizes generative AI and implementing it in a personnel management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript, and when the interview ends, it summarizes the verbatim transcript and stores the summary in the personnel management system. It also has a function that automatically reflects the contents of the interview in a to-do list or calendar.

[0060] Hardware and Software Used

[0061] server

[0062] The server provides the computational resources to run the generative AI model, converts the voice data into text data using speech recognition software, and summarizes the transcripts using a generative AI model (e.g., OpenAI's GPT-4). The server also stores the summarized transcripts in the HR management system's database and automatically updates to-do lists and calendars based on the interview content.

[0063] Terminal

[0064] The terminal is a device on which the user conducts the interview. When the interview begins, the terminal collects voice data using a built-in microphone and transmits it to the server in real time. When the interview ends, the terminal receives a summarized transcript and updated to-do list and calendar information from the server and notifies the user.

[0065] User

[0066] The user is the person conducting the interview. The user initiates the interview through the device and does not need to do anything during the interview. Once the interview is over, the user can view the summarized transcript and manage to-do lists and calendar updates through the device.

[0067] Examples

[0068] Examples of prompt statements

[0069] "Please summarize the interview below. Interview content: 'Hello, today I would like to discuss your work progress. First, please tell me how the project went last week.'"

[0070] Processing flow

[0071] 1. Starting the interview: The user launches the dedicated application on the device and clicks the "Start interview" button.

[0072] 2. Voice data collection: The device uses the built-in microphone to collect voice data and transmits it to the server in real time.

[0073] 3. Transcription: The server uses voice recognition software to convert the audio data into text.

[0074] 4. Summary generation: The server summarizes the transcript using a generative AI model.

[0075] 5. Data storage: The server stores the summarized transcripts in the HR management system database.

[0076] 6. ToDo list / calendar update: The server automatically updates the toDo list and calendar based on the meeting contents.

[0077] 7. User Notification: The terminal notifies the user with the summarized transcript and updates.

[0078] This system reduces the burden of creating interview records and allows you to concentrate on observing nonverbal information during interviews. It also makes it easy to follow up after interviews, as the contents of interviews can be efficiently managed and automatically reflected in to-do lists and calendars.

[0079] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0080] Step 1:

[0081] Interview begins

[0082] Subject: User

[0083] The user starts the dedicated application on the terminal and clicks the "Start interview" button. This causes the terminal to send a request to start the interview to the server. The input is the user's operation, and the output is the request to start the interview. As a specific operation, the terminal displays "Interview has started."

[0084] Step 2:

[0085] Audio data collection

[0086] Subject: Terminal

[0087] As soon as the interview begins, the terminal starts collecting voice data using the built-in microphone. The collected voice data is sent to the server in real time. The input is the user's voice, and the output is the voice data sent to the server. In concrete terms, the terminal displays "collecting voice data" and shows the collection status in real time.

[0088] Step 3:

[0089] Transcription

[0090] Subject: Server

[0091] The server receives the voice data sent from the terminal and converts it into text data using voice recognition software. The input is voice data and the output is text data. In concrete terms, the server displays "Creating a transcript" and updates the progress in real time.

[0092] Step 4:

[0093] Summary Generation

[0094] Subject: Server

[0095] The server uses a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The input is the text data of the verbatim transcript, and the output is the summarized text. The server displays "Generating summary" and provides real-time updates on the progress.

[0096] Step 5:

[0097] Data storage

[0098] Subject: Server

[0099] The server stores the summarized transcript in the database of the human resources management system. The input is the summarized text and the output is the record stored in the database. In concrete terms, the server displays "Saving data" and, when storage is complete, displays "Data storage completed."

[0100] Step 6:

[0101] ToDo list / Calendar update

[0102] Subject: Server

[0103] The server automatically updates the to-do list and calendar based on the contents of the meeting. The input is a summarized verbatim transcript, and the output is an updated to-do list and calendar. Specifically, the server displays "Updating to-do list and calendar" and then displays "Update completed" when the update is complete.

[0104] Step 7:

[0105] User Notifications

[0106] Subject: Terminal

[0107] The terminal receives the summarized verbatim transcript and updated ToDo list / calendar information from the server and notifies the user. The input is the notification information from the server, and the output is the notification to the user. As a specific operation, the terminal displays "Summary and updated information received" and notifies the user.

[0108] (Application example 1)

[0109] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] Conventional interview support tools have the problem that tasks such as creating interview records, summarizing, and updating to-do lists and calendars are often done manually, which takes time and effort. Similar problems exist in interviews between workers and robots in factories, making it difficult to provide efficient work instructions and manage schedules.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and storing the interview record in the personnel management system, a means for automatically reflecting the results in a ToDo list or a calendar, a means for supporting interviews between workers and robots in a factory, and a means for automatically updating work instructions and schedules based on the interview contents. This reduces the burden of creating interview records, and enables efficient work instructions and schedule management.

[0112] "Generative AI" is a system that uses artificial intelligence technology to generate data and automate certain tasks.

[0113] An "interview implementation support tool" is a software or hardware system that supports the progress of interviews and the creation of records.

[0114] A "human resources management system" is a system used to manage employee information in a company or organization, and to manage labor and develop human resources.

[0115] A verbatim transcript is a document that records exactly what was said during an interview or meeting.

[0116] A "summary" is a document that extracts important points from a detailed record such as a verbatim transcript and summarizes them concisely.

[0117] A "ToDo list" is a list of tasks or work items that need to be done.

[0118] A "calendar" is a tool for managing schedules and events based on dates and times.

[0119] "Workers" are employees who actually perform work in a factory or on-site.

[0120] A "robot" is a mechanical device that performs tasks automatically according to programmed instructions.

[0121] A "work instruction" is a specific instruction or command for carrying out a specific task.

[0122] A "schedule" is a plan of tasks or events to be performed within a specific period of time.

[0123] As an embodiment of the present invention, a system for supporting interviews between workers and robots in a factory is constructed. A specific implementation method thereof is shown below.

[0124] System Configuration

[0125] 1. Hardware Configuration

[0126] Microphone: Used to collect interview audio.

[0127] Computer: Used to process the audio data and run generative AI models.

[0128] Robot: An automated machine that performs tasks in a factory.

[0129] 2. Software Configuration

[0130] OpenAI API: Used for audio transcription and summary generation.

[0131] Python(registered trademark): A programming language used for data processing and file manipulation.

[0132] Human Resources Management System: A system for storing interview records and summaries, and reflecting them in to-do lists and calendars.

[0133] Data processing and calculation

[0134] 1. Collecting audio data and transcribing

[0135] The server collects the interview audio through a microphone.

[0136] The collected audio data is converted into text using the OpenAI API, and a verbatim transcript is created.

[0137] 2. Summary of the Verbatim

[0138] The server summarizes the generated transcripts using the OpenAI API.

[0139] The summarized text is stored in the personnel management system.

[0140] 3. Adding to your ToDo list and calendar

[0141] The server extracts important tasks from the summarized text and adds them to a to-do list.

[0142] In addition, the schedule will be updated based on the contents of the interview and reflected on the calendar.

[0143] Examples

[0144] For example, if during an interview someone says, "We need to check the next maintenance schedule," that content is recorded in the transcript. After the interview ends, the server summarizes the transcript and generates a summary called "Check the next maintenance schedule." This summary is stored in the personnel management system, and "Check the next maintenance schedule" is automatically added to the to-do list.

[0145] Examples of prompt statements

[0146] An example prompt for summarizing a transcript is as follows:

[0147] Please summarize the following transcript:

[0148] [Transcript text]

[0149] summary:

[0150] An example prompt for retrieving to-do list items is:

[0151] Extract to-do list items from the summary below:

[0152] [Summary text]

[0153] To-do list:

[0154] In this way, it is possible to efficiently support meetings between workers and robots within the factory and automate work instructions and schedule management.

[0155] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0156] Step 1:

[0157] The server collects the interview voice through a microphone. The input is the interview voice, and the output is the voice data. Specifically, the microphone captures the voice during the interview in real time and transmits the voice data to the server.

[0158] Step 2:

[0159] The server converts the collected voice data into text using the OpenAI API and creates a transcript. The input is the voice data, and the output is the transcript text. Specifically, the voice data is input into OpenAI's speech recognition model, and the generated text is saved as a transcript.

[0160] Step 3:

[0161] The server summarizes the generated transcripts using the OpenAI API. The input is the transcript text, and the output is the summary text. Specifically, a prompt for summarizing the transcript text is input to the OpenAI text generation model, and the generated summary is obtained.

[0162] Step 4:

[0163] The server stores the summarized text in the human resources management system. The input is the summarized text, and the output is the summary data stored in the human resources management system. Specifically, the server performs an operation to store the summarized text in the database of the human resources management system.

[0164] Step 5:

[0165] The server extracts important tasks from the summarized text and adds them to the to-do list. The input is the summarized text, and the output is an updated to-do list. Specifically, the summary text is input to OpenAI's text generation model as a prompt for task extraction, and the generated tasks are added to the to-do list.

[0166] Step 6:

[0167] The server updates the schedule based on the meeting contents and reflects it in the calendar. The input is the summary text and the output is the updated calendar. Specifically, the server extracts schedule-related information from the summary text and adds that information to the calendar system.

[0168] Step 7:

[0169] The user checks the updated ToDo list and calendar and performs the necessary tasks. The input is the updated ToDo list and calendar, and the output is the user's execution of the tasks. Specifically, the user logs into the system, checks the ToDo list and calendar, and performs the tasks instructed.

[0170] Example 2

[0171] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] In conventional personnel management systems, the creation and storage of interview records and the creation of employee training plans were often done manually, which was time-consuming and labor-intensive. In addition, there was a lack of sufficient extraction of employees' strengths and weaknesses based on interview records, creation of individual training plans, and career path suggestions, making it difficult to provide effective training tailored to each employee.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0174] In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and store the interview record in the personnel management system, a means for analyzing the interview record and extracting the strengths and weaknesses of employees, a means for creating an individual training plan based on the extracted data, a means for proposing a career path based on the interests and inclinations of employees, and a means for automatically reflecting the results in a to-do list or a calendar. This automates the creation and storage of interview records, and makes it possible to efficiently extract the strengths and weaknesses of employees, create individual training plans, and propose career paths, enabling effective training tailored to each employee.

[0175] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[0176] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[0177] A "human resources management system" is software or a system for managing employee information and supporting human resources tasks such as training and evaluation.

[0178] "Verbatim transcription" is the process of recording the exact verbatim content of an interview.

[0179] "Summarizing" is the process of concisely summarizing the contents of an interview.

[0180] "Interview records" are data that record the contents of an interview.

[0181] "Storage" is the process of storing data and making it accessible when needed.

[0182] "Analysis" is the process of analyzing data and extracting useful information.

[0183] "Strengths" are the skills and abilities that employees excel at.

[0184] "Weaknesses" are skills or abilities that an employee is not good at.

[0185] A "development plan" is a plan to support employees in improving their skills and growing their careers.

[0186] "Interests" are areas or activities that interest employees.

[0187] "Orientation" refers to the direction or goal that an employee aims for.

[0188] A "career path" is the path of the job or role that an employee will pursue in the future.

[0189] A "ToDo list" is a list of tasks that need to be done.

[0190] A "calendar" is a tool for managing appointments and schedules.

[0191] This invention is a system that develops an interview implementation support tool that utilizes generative AI and implements it in a personnel management system to efficiently create and store interview records, create employee training plans, and propose career paths. A specific embodiment of this system is described below.

[0192] Hardware and Software Used

[0193] Hardware: Servers, terminals (PCs, tablets, etc.)

[0194] Software: Human resource management systems, generative AI models (e.g. OpenAI GPT-4)

[0195] Program Processing

[0196] 1. Entering interview records

[0197] After the interview, the user uses a terminal (PC or tablet) to enter the interview record into the human resources management system.

[0198] Specifically, the user accesses the input form of the personnel management system and inputs the details of the interview in text format.

[0199] Example: "A user uses a terminal to enter interview records for employee A. His strengths are project management and his weaknesses are presentation skills."

[0200] 2. Keeping records of interviews

[0201] The server receives the interview records sent by the users and stores them in the database of the personnel management system.

[0202] Specifically, the server converts the input data into an appropriate format and stores it in a database.

[0203] Example: "The server receives the interview notes and stores them in a database."

[0204] 3. Data extraction and analysis

[0205] The server analyzes the stored interview records and extracts employees' strengths and weaknesses.

[0206] Specifically, the server inputs the interview records into a generative AI model (e.g., OpenAI GPT-4) and obtains the analysis results.

[0207] Example: "The server inputs the interview records into the generative AI model and extracts the strengths and weaknesses of employee A."

[0208] 4. Creating a training plan

[0209] The server creates an individual training plan based on the extracted data.

[0210] Specifically, the server will suggest the training and education necessary to improve employee skills based on the analysis results of the generative AI model.

[0211] Example: "The server will propose training in project management to utilize Employee A's strengths, and to address Employee A's weaknesses, he will plan training to improve presentation skills."

[0212] 5. Career path suggestions

[0213] The server suggests appropriate career paths based on employees' interests and aspirations.

[0214] Specifically, the server designs future career paths for employees based on the analysis results of the generative AI model and reflects the suggestions in the human resources management system.

[0215] Example: "Since Employee A is interested in project management, the server will suggest that Employee A should become a project manager in the future."

[0216] Examples

[0217] Example of interview record entry:

[0218] "The user uses the terminal to input the interview record of employee B. The strength is data analysis, and the weakness is teamwork."

[0219] Example prompt for a generative AI model:

[0220] "Based on the following interview record, please identify Employee B's strengths and weaknesses and propose an appropriate training plan. Interview record: Strengths are data analysis, and weaknesses are teamwork."

[0221] This system automates the creation and storage of interview records, making it possible to efficiently identify employees' strengths and weaknesses, create individual training plans, and suggest career paths, enabling effective training tailored to each individual employee.

[0222] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0223] Step 1:

[0224] Enter interview records

[0225] After the interview, the user uses a terminal (PC or tablet) to enter the interview record into the human resources management system.

[0226] Input: Interview details (in text format), employee strengths and weaknesses, and other interview information.

[0227] Specific operation: The user accesses the input form of the personnel management system and inputs the interview details in text format. For example, the user inputs, "Enter the interview record of employee A. His strengths are project management and his weaknesses are presentation skills."

[0228] Output: The entered interview record is sent to the HR management system.

[0229] Step 2:

[0230] Keeping records of interviews

[0231] The server receives the interview records sent by the users and stores them in the database of the personnel management system.

[0232] Input: Interview notes submitted by user.

[0233] Specific operation: The server converts the input data into an appropriate format and stores it in the database. For example, the server receives the interview record and stores "Employee A's interview record" in the database.

[0234] Output: Interview notes stored in a database.

[0235] Step 3:

[0236] Data extraction and analysis

[0237] The server analyzes the stored interview records and extracts employees' strengths and weaknesses.

[0238] Input: Interview notes stored in database.

[0239] Specific operation: The server inputs the interview record into the generative AI model (e.g., OpenAI GPT-4) and obtains the analysis results. For example, the server inputs "Employee A's interview record" into the generative AI model and extracts strengths and weaknesses.

[0240] Output: Employee strengths and weaknesses as analyzed results.

[0241] Step 4:

[0242] Creating a training plan

[0243] The server creates an individual training plan based on the extracted data.

[0244] Input: Employee strengths and weaknesses as analyzed results.

[0245] Specific operation: Based on the analysis results of the generative AI model, the server proposes the training and education necessary to improve the skills of employees. For example, the server creates a training plan such as, "Utilizing employee A's strengths, we propose training in project management. To compensate for his weaknesses, we plan training to improve presentation skills."

[0246] Output: An individual development plan.

[0247] Step 5:

[0248] Career path suggestions

[0249] The server suggests appropriate career paths based on employees' interests and aspirations.

[0250] Input: Employee interests and inclinations as analyzed results.

[0251] Specific operation: Based on the analysis results of the generative AI model, the server designs future career paths for employees and reflects the proposed content in the personnel management system. For example, the server may suggest a career path such as "Employee A is interested in project management, so we propose a career path to become a project manager in the future."

[0252] Output: A proposed career path.

[0253] (Application example 2)

[0254] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0255] In conventional personnel management systems, creating and managing interview records was a hassle, and line managers had no time to observe non-verbal information during interviews. In addition, there was a lack of a system that efficiently collected and analyzed robot work records and proposed optimal work placements and improvement plans. This made it difficult to improve the skills of employees and robots and to place them in efficient work positions.

[0256] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0257] In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and store the interview record in the personnel management system, a means for automatically reflecting it in a to-do list or a calendar, a means for collecting the robot's work record and analyzing its efficiency and error rate, a means for grasping the robot's good and bad tasks and proposing optimal work allocation, and a means for creating individual improvement plans. This reduces the burden of creating interview records, allowing the line manager to focus on non-verbal information, and makes it possible to analyze the robot's work efficiency and error rate and propose optimal work allocation and improvement plans.

[0258] "Generative AI" is a system that uses artificial intelligence techniques to generate data to automate or assist with specific tasks.

[0259] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[0260] A "human resources management system" is a system for managing employee information and streamlining human resources tasks such as evaluation, training, and placement.

[0261] "Verbatim transcription" is the process of recording the exact words of a conversation, such as an interview or meeting.

[0262] A "summary" is a short summary of a longer piece of text or conversation.

[0263] A "ToDo list" is a tool that allows you to manage tasks and schedules in list format.

[0264] "Auto-add to calendar" is a feature that automatically adds specific events and tasks to your calendar.

[0265] "Robot work record" refers to a detailed record of the work performed by a robot.

[0266] "Efficiency" is a measure of how effectively a task or process is carried out.

[0267] An "error rate" is an indicator that indicates the frequency with which errors occur in a task or process.

[0268] A "good task" refers to a task that a particular robot can perform more efficiently than other tasks.

[0269] A "weak task" refers to a task that a particular robot performs less efficiently than other tasks.

[0270] "Optimal work placement" refers to placing robots and employees so that they can perform their work most efficiently.

[0271] "Improvement plan" refers to a specific plan for improving work efficiency and error rate.

[0272] The system for implementing this invention is to develop an interview support tool that utilizes generative AI and implement it in a personnel management system. Specifically, generative AI is used to create and summarize interview transcripts, and the interview records are stored in the personnel management system. It also includes a function to automatically reflect the results in a to-do list or calendar.

[0273] In addition, the system also includes a function to collect the robot's work records and analyze its efficiency and error rate. This allows the system to understand the tasks that the robot is good at and bad at, and to propose optimal work arrangements. It also includes a function to create individual improvement plans.

[0274] Hardware and Software Used

[0275] Hardware: Servers, factory robots, user terminals

[0276] Software: Python, JSON, generative AI models

[0277] Data processing and calculation

[0278] The server creates and summarizes interview transcripts using generative AI models. These data are stored in the HR management system in JSON format. In addition, information extracted from the interview records is automatically updated in to-do lists and calendars.

[0279] The robot's work records are collected to calculate efficiency and error rates. These data are analyzed on the server to identify tasks that the robot is good at and bad at. Based on the analysis results, optimal work placement and individual improvement plans are generated.

[0280] Examples

[0281] For example, consider a scenario in which a factory robot saves records of its assembly work and creates an improvement plan based on those records. The robot's work records are collected as follows:

[0282] Example prompt:

[0283] Robot ID: robot_1

[0284] Operation: Assembly

[0285] Efficiency: 0.9

[0286] Error rate: 0.05

[0287] By inputting this prompt sentence into the generative AI model, the process of saving and analyzing the robot's work records and creating improvement plans is automated. This reduces the burden of creating interview records and allows line managers to focus on non-verbal information, while also enabling them to analyze the robot's work efficiency and error rate, and propose optimal work placement and improvement plans.

[0288] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0289] Step 1:

[0290] The server creates a transcript of the interview using a generative AI model. As input, it receives the audio data of the interview and generates a transcript using the generative AI model. As output, the transcript is obtained in text format.

[0291] Step 2:

[0292] The server summarizes the generated transcripts: as input it takes the text data of the transcript and applies a summarization algorithm; as output it obtains the summarized text.

[0293] Step 3:

[0294] The server stores the transcripts and summaries in JSON format in the HR management system. As input, it receives the text data of the transcripts and summaries and converts them into JSON format. As output, the JSON format data is stored in the HR management system.

[0295] Step 4:

[0296] The server automatically updates the to-do list and calendar based on the information extracted from the interview record. As input, it receives the JSON data of the interview record and converts it into the format of the to-do list and calendar. As output, new tasks and events are added to the to-do list and calendar.

[0297] Step 5:

[0298] The server collects the robot's work records. As input, it receives the work data (such as work ID, efficiency, error rate, etc.) sent from the robot. As output, the collected work records are stored in a database.

[0299] Step 6:

[0300] The server analyzes the collected work records. As input, it receives the work records stored in the database, calculates the efficiency and error rate, and as output, it obtains the analysis results.

[0301] Step 7:

[0302] The server knows what tasks the robot is good at and bad at. As input, it takes the analysis results and applies an algorithm to identify what tasks the robot is good at and bad at. As output, it gets a list of what tasks the robot is good at and bad at.

[0303] Step 8:

[0304] The server proposes an optimal task placement. As input, it receives a list of good and bad tasks and applies an algorithm to calculate the optimal task placement. As output, it obtains a proposal for the optimal task placement.

[0305] Step 9:

[0306] The server creates an individualized improvement plan. As input, it receives the analysis results and the optimal work placement proposals, and applies an algorithm to generate the improvement plan. As output, it obtains the individualized improvement plan.

[0307] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0308] "Example 1"

[0309] As an example 1 of the present invention, there is a system that combines an emotion engine. In this system, an interview implementation support tool that utilizes a generation AI is developed and implemented in a personnel management system. The generation AI creates and summarizes a verbatim record of the interview, and stores the results in the personnel management system. In addition, it also has a function that automatically reflects the contents of the interview in a to-do list and a calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This makes it possible to grasp not only the contents of the interview but also the user's emotional state at that time, enabling a deeper understanding and response.

[0310] "Example 2"

[0311] As a second embodiment of the present invention, there is a system that reduces the burden on line managers in creating interview records and allows them to concentrate on observing nonverbal information during interviews. In this system, a generation AI creates and summarizes interviews verbatim, and stores the results in a personnel management system. In addition, it also has a function that automatically reflects the contents of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the emotions of the user during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This reduces the burden on line managers in creating interview records and allows them to concentrate on observing nonverbal information during the interview.

[0312] "Example 4"

[0313] As a fourth embodiment of the present invention, there is a system that utilizes implementation records to create an environment for continuous employee development. In this system, a generation AI creates and summarizes interviews verbatim, and stores the results in a personnel management system. In addition, it also has a function to automatically reflect the contents of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the emotions of the user during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This makes it possible to create an environment for continuous employee development by utilizing interview records.

[0314] The process flow of each embodiment will be described below.

[0315] "Example 1"

[0316] Step 1: When the interview begins, the generative AI starts creating a transcript.

[0317] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[0318] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[0319] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[0320] Step 5: The results of the emotion engine analysis are reflected in the interview transcript.

[0321] "Example 2"

[0322] Step 1: When the interview begins, the generative AI starts creating a transcript.

[0323] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[0324] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[0325] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[0326] Step 5: The analysis results of the emotion engine are reflected in the interview record, reducing the line manager's burden of creating interview records and allowing him or her to focus on observing non-verbal information during the interview.

[0327] "Example 4"

[0328] Step 1: When the interview begins, the generative AI starts creating a transcript.

[0329] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[0330] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[0331] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[0332] Step 5: The analysis results of the emotion engine are reflected in the interview records, and the interview records can be used to create an environment for continuous employee development.

[0333] Example 1

[0334] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0335] Previously, interview records were created manually, placing a heavy burden on line managers and administrators. It was also difficult to properly record non-verbal information and emotional states during interviews, resulting in a lack of information to improve the quality of interviews. Furthermore, the task of reflecting the contents of interviews in to-do lists and calendars was time-consuming, and efficient management was required.

[0336] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0337] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create a verbatim transcript and summarize the interview and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or a calendar, a means for recognizing an emotional state during the interview using an emotion engine and reflecting the information in the interview record, and a means for automating a series of processes from the start to the end of the interview. This reduces the burden of creating interview records, enables detailed interview records including non-verbal information and emotional states, and makes it possible to efficiently manage the contents of the interview.

[0338] "Generative AI" is a system that uses artificial intelligence technology to generate text and data.

[0339] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[0340] A "human resources management system" is an integrated system for managing employee information, evaluating, and training.

[0341] A verbatim transcript is a document that records exactly what was said during an interview or meeting.

[0342] A "summary" is a short document that extracts the important points from a detailed record such as a verbatim transcript.

[0343] A "ToDo List" is a list that displays tasks or action items that need to be completed.

[0344] A "calendar" is a tool that displays dates and times for schedule management.

[0345] An "emotion engine" is a technology or system for analyzing emotional states from voice, facial expressions, choice of words, etc.

[0346] "Nonverbal information" is information that is conveyed by means other than words, and includes facial expressions, gestures, and tone of voice.

[0347] "Process automation" refers to the automatic execution of a series of tasks that were previously performed manually by a system.

[0348] This invention relates to developing an interview implementation support tool that utilizes generative AI and implementing it in a personnel management system. A specific embodiment of this system is described below.

[0349] System Configuration

[0350] The system consists of three main components: a server, a terminal, and a user. The server is equipped with a generative AI model and an emotion engine, and is responsible for creating transcripts of interviews, summarizing them, recognizing emotional states, and automatically updating to-do lists and calendars. The terminal is responsible for collecting voice data from interviews and sending it to the server. The user conducts the interviews and operates the system.

[0351] Hardware and Software Used

[0352] Server: A server with high-performance computing power (e.g. cloud server)

[0353] Generative AI models: Natural language processing models such as OpenAI's GPT-4

[0354] Emotion engine: Emotion recognition API, etc.

[0355] Human Resources Management System: Integrated System

[0356] To-do list and calendar: schedule management tools

[0357] Device: A computer or tablet equipped with a microphone to collect audio data.

[0358] Data processing and calculation

[0359] When a user starts an interview, the device collects audio data and transmits it in real time to the server, which then inputs the received audio data into a generative AI model to generate a transcript, which is then temporarily stored on the server.

[0360] Once the interview is over, the server uses the generative AI model again to summarize the transcript, which is then stored in the personnel management system. The server also analyzes the interview content and automatically updates it on to-do lists and calendars.

[0361] Using an emotion engine, the server analyzes the user's tone of voice, facial expressions and word choice to recognize their emotional state. This emotion information is also added to the transcript and stored in the personnel management system.

[0362] Examples of concrete examples and prompts

[0363] As a concrete example, consider the following scenario.

[0364] scenario:

[0365] A user has a meeting with a subordinate. During the meeting, the subordinate reports the progress of the project and sets some tasks as next steps. The subordinate also expresses concerns about the project.

[0366] Example prompt:

[0367] "Start a meeting with your subordinate. Transcribe and summarize the conversation. Analyze the emotional state of the meeting and reflect it on your to-do list and calendar."

[0368] This system reduces the burden of creating interview records and allows users to focus on observing non-verbal information during interviews. It also enables users to efficiently manage interview content and keep detailed records, including emotional states.

[0369] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0370] Step 1:

[0371] The user starts an interview. The user clicks the interview start button on the terminal to notify the system that he or she wants to start the interview. The input is the user's operation, and the output is the signal to start the interview. The terminal displays the message "Starting the interview."

[0372] Step 2:

[0373] The terminal collects voice data during the interview in real time. The input is the user's voice, and the output is the collected voice data. The terminal collects voice data using a built-in microphone and transmits it to the server in real time. The server stores the received voice data in a buffer.

[0374] Step 3:

[0375] The server inputs the received voice data into a generative AI model to generate a verbatim transcript. The input is the voice data and the output is the generated verbatim transcript. The server inputs the buffered voice data into a generative AI model to generate a verbatim transcript. The generated verbatim transcript is stored in the server's temporary storage.

[0376] Step 4:

[0377] Once the interview is over, the server uses the generative AI model again to summarize the transcript. The input is the verbatim transcript and the output is the summarized verbatim transcript. The server re-inputs the verbatim transcript into the generative AI model and generates a summary. The summarized content is saved in temporary storage.

[0378] Step 5:

[0379] The server stores the summarized verbatim transcript in the HRMS. The input is the summarized verbatim transcript and the output is the data stored in the HRMS. The server saves the summarized verbatim transcript in the HRMS database. Once saved, the user is notified "Summary saved."

[0380] Step 6:

[0381] The server parses the condensed verbatim transcript and adds the necessary information to the to-do list or calendar. The input is the condensed verbatim transcript and the output is an updated to-do list or calendar. The server parses the condensed verbatim transcript and adds the necessary information to the to-do list or calendar. For example, action items such as "schedule next meeting" or "complete specific task" are automatically added.

[0382] Step 7:

[0383] The server uses an emotion engine to analyze the user's emotional state from the voice data or the verbatim transcript. The input is the voice data or the verbatim transcript and the output is the recognized emotion information. The server uses an emotion engine to analyze the user's emotional state from the voice data or the verbatim transcript. For example, if the user expresses "anxiety", that information is recognized by the emotion engine.

[0384] Step 8:

[0385] The server adds the recognized emotion information to the transcript and stores it in the HR management system. The input is the recognized emotion information and the output is the updated transcript. The server adds the recognized emotion information to the transcript and stores it in the HR management system database. Once the saving is complete, the user is notified that "emotion information has been saved."

[0386] (Application example 1)

[0387] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] In conventional interview support systems, tasks such as creating verbatim transcripts of interviews, summarizing, and recognizing emotions were often done manually, which led to the problem of a high workload for creating interview records. In addition, it was difficult to concentrate on observing non-verbal information during the interview, which could lead to a decline in the quality of the interview. Furthermore, there was a lack of functionality to automatically reflect the contents of the interview in a to-do list or calendar, which meant that follow-up after the interview was insufficient. In order to solve these issues, an efficient interview support system that utilizes generative AI was needed.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0390] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create and summarize a verbatim record of an interview and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or calendar, a means for combining an emotion engine to recognize the user's emotions during the interview and reflecting the information in the interview record, a means for converting the voice of the interview into text as a verbatim record using speech recognition technology, a means for summarizing the verbatim record and analyzing emotions using a generative AI model, and a means for saving the results of the summary and emotion analysis as a calendar or interview record. This reduces the burden of creating interview records and allows users to concentrate on observing non-verbal information during the interview. Furthermore, follow-up after the interview is performed efficiently, improving the quality of the interview.

[0391] "Generative AI" is a system that uses artificial intelligence techniques to generate, analyze, and summarize data.

[0392] An "interview support tool" is a combination of software and hardware that supports the progress of interviews and efficiently creates records.

[0393] A "human resources management system" is a system used to manage employee information and evaluate and develop employees in a company or organization.

[0394] A verbatim transcript is a recording of an interview, meeting, etc., transcribed verbatim.

[0395] A "summary" is a short summary of long data such as a verbatim transcript, extracting the important points.

[0396] An "interview record" is a document or data that records the contents of an interview, including a verbatim transcript, a summary, or the results of sentiment analysis.

[0397] A "ToDo list" is a list of tasks that need to be done.

[0398] A "calendar" is a tool for managing appointments and schedules.

[0399] An "emotion engine" is a system that analyzes emotions from voice and text data and outputs the results.

[0400] "Speech recognition technology" is a technology that converts speech into text.

[0401] A “generative AI model” is a specific implementation of generative AI: a machine learning model trained to perform a specific task.

[0402] "Nonverbal information" is information that is conveyed by means other than words, and includes facial expressions, tone of voice, gestures, etc.

[0403] As an embodiment of the present invention, a factory robot interview support system will be described as an example. This system utilizes generative AI to create verbatim records of interviews, summarize them, recognize emotions, and automatically reflect them in to-do lists and calendars.

[0404] Sabato will develop an interview support tool that utilizes generative AI and implement it in the personnel management system. Specifically, it will use voice recognition technology to convert the audio of the interview into a verbatim transcript, and then use a generative AI model to summarize the transcript and analyze emotions. This will reduce the burden of creating interview records, and allow employees to focus on observing non-verbal information during the interview.

[0405] The server uses a generative AI model to summarize the transcript and analyze emotions. For example, Library A is used as the generative AI model. For speech recognition technology, Library B is used. This allows the interview transcript to be automatically created and summarized.

[0406] The server recognizes the user's emotions during the interview by combining the emotion engine and reflects the information in the interview record. A specified pipeline is used as the emotion engine. This allows the server to grasp not only the contents of the interview but also the user's emotional state at that time.

[0407] The server saves the summary and the results of the sentiment analysis as a calendar or interview record. The saved data is automatically reflected in the to-do list and calendar. This makes follow-up after the interview more efficient and improves the quality of the interview.

[0408] As a concrete example, a summary can be obtained by inputting the following prompt sentence into a generative AI model:

[0409] Example prompt:

[0410] "Today I would like to discuss the efficiency of our new production line. Although the current production rate is not up to the target, there are some areas for improvement. First, we need to review the maintenance schedule of our machines. Also, we need to update the training program for our workers."

[0411] By inputting this prompt into a generative AI model, a summary can be obtained, allowing the content of the interview to be efficiently understood and necessary actions to be taken quickly.

[0412] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0413] Step 1:

[0414] The server uses speech recognition technology to convert the interview audio into text, verbatim.

[0415] Input: Audio data from the interview

[0416] Output: Verbatim transcription (text data)

[0417] Specific operation: The server converts the voice data acquired from the microphone into text, which records the contents of the interview verbatim.

[0418] Step 2:

[0419] The server summarizes the transcript using a generative AI model.

[0420] Input: Verbatim transcription (text data)

[0421] Output: Summary (text data)

[0422] Specific Actions: The server summarizes the transcript, specifically extracting the key points of the transcript and summarizing them in a concise format.

[0423] Step 3:

[0424] The server analyzes the sentiment of the verbatim transcript using a sentiment engine.

[0425] Input: Verbatim transcription (text data)

[0426] Output: Sentiment analysis results (text data)

[0427] What it does: The server analyzes the emotions in the transcript, specifically recognizing emotions from the user's tone of voice and choice of words, and outputs the results.

[0428] Step 4:

[0429] The server stores the summaries and the results of sentiment analysis as calendars and interview records.

[0430] Input: Summary (text data), sentiment analysis results (text data)

[0431] Output: Calendar entries, meeting notes (data in JSON format)

[0432] How it works: The server saves the summary and the results of sentiment analysis in JSON format and automatically updates the calendar and to-do list, allowing for efficient follow-up after the interview.

[0433] Step 5:

[0434] The user reviews the interview record and takes necessary actions.

[0435] Input: Calendar entries, interview notes (data in JSON format)

[0436] Output: Action plan (text data)

[0437] Specific Actions: The user checks the meeting records and calendar entries provided by the server and plans the necessary actions, allowing specific responses based on the contents of the meeting.

[0438] Example 2

[0439] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0440] In the conventional interview record preparation method, line managers had to prepare interview records manually, which was a heavy burden. It was also difficult to properly observe non-verbal information during interviews (tone of voice, facial expressions, etc.) and reflect them in the records. Furthermore, the environment for utilizing interview records for continuous employee development was not in place.

[0441] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in the personnel management system, a means for using the generation AI to create a verbatim record of the interview and summarize it, and storing the interview record in the personnel management system, a means for collecting the tone of voice and facial expression of the user during the interview, analyzing emotions using an emotion engine, and reflecting the results in the interview record, a means for automatically reflecting the interview content in a ToDo list or a calendar, and a means for storing the results of the verbatim record, summary, and emotion analysis in the personnel management system. This reduces the burden on the line manager in creating interview records, allowing him to concentrate on observing non-verbal information during the interview, and further makes it possible to prepare an environment for continuous employee development by utilizing the interview records.

[0442] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[0443] "Interview implementation support tool" is a general term for software or hardware used to assist in the progress of interviews and the creation of records.

[0444] A "human resources management system" is an integrated software system for managing employee information, evaluating, and training.

[0445] A "verbatim transcript" is text data that records the content of an interview or conversation verbatim.

[0446] A "summary" is text data that is created by extracting important points and keywords from a verbatim transcript and summarizing them concisely.

[0447] An "emotion engine" is software or hardware that analyzes data such as voice and facial expressions and recognizes the user's emotions.

[0448] "Nonverbal information" is information that is conveyed by means other than words, and specifically includes tone of voice, facial expressions, gestures, etc.

[0449] A "ToDo list" is a tool for managing tasks and schedules in list format.

[0450] A "calendar" is a tool for managing schedules and events based on dates and times.

[0451] "Verbatim transcription" is the process of converting audio data into text and recording the verbatim content of an interview or conversation.

[0452] "Interview records" are data that record the contents of an interview, including verbatim transcripts, summaries, and the results of sentiment analysis.

[0453] "Sentiment analysis" is the process of analyzing data such as voice and facial expressions to recognize a user's emotions.

[0454] "Auto-reflection" is the process of automatically reflecting certain data or information in other systems or tools.

[0455] This invention is a system that uses an interview implementation support tool that utilizes generative AI to create a verbatim transcript of an interview, summarize it, and analyze emotions, and then stores the data in a personnel management system and automatically reflects it in a to-do list or calendar. A specific embodiment of this system is described below.

[0456] Hardware and Software Used

[0457] Hardware: Servers, terminals (PCs, tablets, smartphones)

[0458] Software: Generative AI models (e.g. GPT-4), emotion engines, human resource management systems

[0459] Data processing and calculation flow

[0460] 1. Collecting audio data from interviews

[0461] The user starts the interview.

[0462] The device collects audio data from interviews in real time, specifically by recording audio using the microphone on a PC, tablet, or smartphone.

[0463] The terminal transmits the collected voice data to a server.

[0464] 2. Transcription of audio data

[0465] The server sends the received voice data to the generative AI model.

[0466] A generative AI model converts the audio data into text and creates a transcript, using speech recognition technology to convert speech to text.

[0467] 3. Summary of the Transcript

[0468] The generative AI model summarizes the transcript by extracting key points and keywords and generating a concise summary.

[0469] The server receives the summarized text and proceeds to the next processing step.

[0470] 4. Sentiment analysis

[0471] The device collects the user's tone of voice and facial expressions during the interview, specifically by using a camera and microphone to record changes in facial expressions and voice.

[0472] The server sends the collected data to the emotion engine.

[0473] The emotion engine analyzes the user's emotions by recognizing their tone of voice, facial expressions, and choice of words.

[0474] The server adds the results of the sentiment analysis to the transcript.

[0475] 5. Data Retention

[0476] The server stores the transcripts, summaries, and sentiment analysis results in the personnel management system, specifically in a database.

[0477] The HR management system manages the stored data and ensures that it can be accessed when needed.

[0478] 6. Adding to your ToDo list and calendar

[0479] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar.

[0480] The HR management system automatically adds the generated tasks to to-do lists and calendars, and in particular, reflects them in employees' schedules and task management tools.

[0481] Examples

[0482] For example, when employee A has an interview with line manager B, the following specific actions are taken:

[0483] 1. The users (employee A and line manager B) start an interview, and the terminal (PC) records the audio.

[0484] 2. The device transmits the recorded audio data to the server in real time.

[0485] 3. The server sends the audio data to a generative AI model to create a transcript.

[0486] 4. A generative AI model summarizes the transcript and extracts key takeaways.

[0487] 5. The terminal collects Employee A’s facial expressions and tone of voice during the interview, and the server sends them to the emotion engine.

[0488] 6. The sentiment engine analyzes the sentiment and adds the results to the transcript.

[0489] 7. The server stores the transcripts, summaries, and sentiment analysis results in the human resources management system.

[0490] 8. The HR management system analyzes the interview content and generates tasks to be reflected in to-do lists and calendars.

[0491] Examples of prompt statements

[0492] "Please prepare and summarize a transcript of the meeting between Employee A and Line Manager B. Also, please analyze Employee A's emotions during the meeting and add the results to the transcript."

[0493] This system reduces the burden on line managers of creating interview records, allowing them to focus on observing non-verbal information during interviews. It also creates an environment for continuous employee development by utilizing interview records.

[0494] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0495] Step 1:

[0496] The user starts the interview. The device collects the voice data of the interview in real time. Specifically, the device records the voice using the microphone of a PC, tablet, or smartphone. The device sends the collected voice data to the server. The input is the voice data of the interview, and the output is the voice data sent to the server.

[0497] Step 2:

[0498] The server sends the received voice data to the generative AI model. The generative AI model converts the voice data into text and creates a transcript. Specifically, it uses speech recognition technology to convert speech to text. The input is the voice data sent to the server, and the output is the generated transcript.

[0499] Step 3:

[0500] The generative AI model summarizes the transcript by extracting key points and keywords and generating a concise summary. The server receives the summarized text and passes it on to the next processing step. The input is the generated transcript and the output is the summarized text.

[0501] Step 4:

[0502] The device collects the user's tone of voice and facial expressions during the interview. Specifically, it uses a camera and microphone to record facial expressions and changes in voice. The server sends the collected data to the emotion engine. The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from tone of voice, facial expressions, choice of words, etc. The input is the collected tone of voice and facial expression data, and the output is the result of the emotion analysis.

[0503] Step 5:

[0504] The server adds the results of the sentiment analysis to the transcript, i.e. integrates the results of the sentiment analysis into the transcript in text form. The input is the results of the sentiment analysis and the verbatim transcript, and the output is the verbatim transcript with the sentiment analysis added.

[0505] Step 6:

[0506] The server stores the verbatim transcripts, summaries, and the results of sentiment analysis in the HRMS, specifically in a database. The HRMS manages the stored data and makes it accessible when needed. The input is the verbatim transcripts with added sentiment analysis, summaries, and the results of sentiment analysis, and the output is the data stored in the HRMS.

[0507] Step 7:

[0508] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar. The personnel management system automatically adds the generated tasks to the to-do list and calendar. Specifically, it reflects them in the employee's schedule and task management tool. The input is the analyzed interview content, and the output is the tasks reflected in the to-do list and calendar.

[0509] (Application example 2)

[0510] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0511] In conventional interview record creation systems, the creation of verbatim records of interviews, summarization, and emotion recognition were done manually, placing a heavy burden on line managers and administrators. In addition, interview records were not fully utilized to improve employees' skills or design career paths, making it difficult to create individual training plans. Furthermore, there was a problem in that training and skill improvement support for factory robot operators was not effectively provided.

[0512] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0513] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create a verbatim transcript of an interview and summarize it, and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or a calendar, a means for recognizing emotions during an interview using an emotion engine and reflecting the information in the interview record, and a means for recording interviews and training sessions conducted by operators of factory robots and supporting the improvement of the operators' skills and the design of their career paths. This reduces the burden of creating interview records, making it possible to concentrate on observing non-verbal information during interviews, and further makes it possible to create an environment for continuous employee development by utilizing the implementation records.

[0514] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[0515] An "interview implementation support tool" is software or a system that supports the recording and analysis of interviews.

[0516] A "human resources management system" is a system that manages employee information and supports skill development and career path planning.

[0517] "Verbatim transcription" is the process of recording the verbatim contents of an interview or conversation.

[0518] "Summarizing" is the process of making a shorter summary of a longer piece of text or conversation.

[0519] A "ToDo List" is a tool that allows you to manage tasks that need to be done in list format.

[0520] A "calendar" is a tool for managing schedules and appointments.

[0521] An "emotion engine" is a system for analyzing and recognizing emotions from voice and text.

[0522] An "interview record" is a document or data that records the contents of an interview.

[0523] A "factory robot" is an automated machine used to perform tasks in a factory.

[0524] An "operator" is a person who operates a machine or system.

[0525] A "Training Session" is a training session designed to improve a particular skill or knowledge.

[0526] "Upskilling" is the process of improving a particular skill or ability.

[0527] A "career path" is the path to the occupation or position that an employee will pursue in the future.

[0528] In order to implement the present invention, the following system configuration and program are required.

[0529] System Configuration

[0530] The server will develop an interview support tool that utilizes generative AI and implement it in the personnel management system. Using generative AI, it will transcribe and summarize interviews verbatim, and store the interview records in the personnel management system. It will also have a function to automatically reflect these in to-do lists and calendars. It will use an emotion engine to recognize emotions during interviews, and reflect this information in the interview records. It will record interviews and training sessions conducted by factory robot operators, and support the operators in improving their skills and designing their career paths.

[0531] Hardware and Software Used

[0532] Hardware: Microphone, Computer

[0533] Software: Python, SpeechRecognition library, Transformers library, EmotionRecognition library

[0534] Program processing explanation

[0535] The server reads the audio file and obtains the audio data. It then converts the audio data into text. It uses a generative AI model to summarize the text and uses an emotion engine to analyze emotions from the text. Based on this information, it creates an interview record and saves it in JSON format. The interview record is stored in the HR management system and automatically reflected in the to-do list and calendar.

[0536] Examples

[0537] An operator is interviewed about the operation of a factory robot, and the audio is recorded. The recorded audio file is input to a server, which generates a verbatim transcript and summary of the interview, and also analyzes the operator's emotions. This makes it easier to improve the operator's skills and design career paths.

[0538] Examples of prompt statements

[0539] Summarize the following text:

[0540] "Today we discussed the operation of a factory robot. It seems that operator A feels unconfident in certain operations. However, he is very skilled in other operations, and is particularly strong in troubleshooting. In future training sessions, we plan to create a program to strengthen A's weak points."

[0541] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0542] Step 1:

[0543] The server reads the audio file and obtains the audio data. Specifically, an audio file recorded using a microphone is uploaded to the server, and the audio data is obtained using the SpeechRecognition library. The input is the audio file, and the output is the audio data.

[0544] Step 2:

[0545] The server converts the acquired voice data into text. Specifically, it converts the voice data into text using the SpeechRecognition library. The input is voice data, and the output is text data.

[0546] Step 3:

[0547] The server uses a generative AI model to summarize the text data. Specifically, it uses a summarization model from the Transformers library to summarize the text data. The input is the text data, and the output is the summarized text.

[0548] Step 4:

[0549] The server uses an emotion engine to analyze emotions from text data. Specifically, it uses the EmotionRecognition library to extract emotions from text data. The input is text data and the output is emotion data.

[0550] Step 5:

[0551] The server creates a meeting record based on the summarized text and emotion data. Specifically, it combines the summarized text and emotion data to generate a meeting record in JSON format. The input is the summarized text and emotion data, and the output is the meeting record.

[0552] Step 6:

[0553] The server stores the created interview record in the personnel management system. Specifically, it saves the generated interview record in JSON format in the database. The input is the interview record, and the output is the record saved in the database.

[0554] Step 7:

[0555] The server automatically updates the ToDo list and calendar based on the contents of the interview record. Specifically, tasks and schedules extracted from the interview record are added to the ToDo list and calendar. The input is the interview record, and the output is the updated ToDo list and calendar.

[0556] Step 8:

[0557] The user records interviews and training sessions conducted by factory robot operators, and supports the operator's skill development and career path planning. Specifically, the user creates individual training plans based on the interview records, and aims to improve the operator's skills. The input is the interview records, and the output is the training plans.

[0558] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0559] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0560] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0561] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0562] [Second embodiment]

[0563] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0564] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0565] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0566] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0567] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0568] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0569] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0570] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0571] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0572] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0573] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0574] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0575] "Example 1"

[0576] As an embodiment of the present invention, an interview implementation support tool that utilizes generative AI is developed and implemented in a personnel management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript. When the interview ends, the generative AI summarizes the verbatim transcript and stores the summary in the personnel management system. It also has a function that automatically reflects the contents of the interview in a to-do list or calendar. This reduces the burden on line managers of creating interview records, allowing them to concentrate on observing nonverbal information during the interview.

[0577] "Example 2"

[0578] In another embodiment of the present invention, the implementation records are utilized to create an environment for continuous employee development. Specifically, interview records are stored in a personnel management system, and based on that, employee skills are improved and career paths are designed. For example, the strengths and weaknesses of employees are extracted from the interview records, and based on that, individual development plans are created. In addition, the interests and inclinations of employees are understood from the interview records, and career paths are proposed based on that. This makes it possible to provide development tailored to each employee.

[0579] The process flow of each embodiment will be described below.

[0580] "Example 1"

[0581] Step 1: When the interview begins, the generative AI starts creating a transcript.

[0582] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the summary in the HR management system.

[0583] Step 3: It also has a function that automatically updates your to-do list and calendar based on the contents of the interview.

[0584] "Example 2"

[0585] Step 1: Store the interview records in a human resources management system and use them to improve employee skills and design career paths.

[0586] Step 2: Extract the employee's strengths and weaknesses from the interview records and create an individual development plan based on that.

[0587] Step 3: Understand the employee's interests and aspirations from the interview records and use that to suggest a career path.

[0588] Example 1

[0589] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0590] Traditional interview records are often created manually, placing a heavy burden on line managers and supervisors. In addition, there is no time to observe non-verbal information during the interview, which can lead to a decline in the quality of the interview. Furthermore, it is difficult to efficiently manage the interview content and reflect it in a to-do list or calendar, which can lead to insufficient follow-up after the interview.

[0591] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0592] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for utilizing the generative AI to create and summarize a verbatim record of an interview and store the interview record in the personnel management system, a means for a terminal to collect voice data at the start of an interview and send it to the server, a means for the server to convert the voice data into text data using voice recognition software, a means for summarizing the verbatim record using a generative AI model, a means for storing the summarized verbatim record in a database, a means for automatically updating a to-do list or a calendar based on the interview content, and a means for the terminal to notify the user of the summarized verbatim record and update information. This reduces the burden of creating interview records, making it possible to concentrate on observing non-verbal information during interviews. In addition, the contents of the interview can be efficiently managed and automatically reflected in a to-do list or a calendar, making it easier to follow up after the interview.

[0593] "Generative AI" is a system that uses artificial intelligence technology to generate and process data.

[0594] An "interview implementation support tool" is a combination of software and / or hardware that assists in the conduct of interviews and the creation of records.

[0595] A "human resources management system" is an integrated system for managing employee information, evaluating, and training.

[0596] A verbatim transcript is a recording of an interview, meeting, etc., transcribed verbatim.

[0597] A "summary" is a concise summary of detailed information, such as a verbatim transcript.

[0598] "Speech recognition software" refers to software for converting voice data into text data.

[0599] "Generative AI model" refers to the core algorithm or trained model of generative AI.

[0600] A "database" is a system for efficiently storing, managing, and searching data.

[0601] A "ToDo list" is a list for managing tasks and schedules that need to be done in a list format.

[0602] A "calendar" is a tool for managing schedules and events based on dates and times.

[0603] A "terminal" is a device such as a computer or smartphone that is directly operated by a user.

[0604] A "server" is a computer system for processing and storing data over a network.

[0605] "Nonverbal information" refers to information that is conveyed by means other than words, such as facial expressions and gestures.

[0606] A "notification" is a message or alert sent from the system to inform the user of information.

[0607] This invention involves developing an interview support tool that utilizes generative AI and implementing it in a personnel management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript, and when the interview ends, it summarizes the verbatim transcript and stores the summary in the personnel management system. It also has a function that automatically reflects the contents of the interview in a to-do list or calendar.

[0608] Hardware and Software Used

[0609] server

[0610] The server provides the computational resources to run the generative AI model, converts the voice data into text using speech recognition software, and summarizes the transcripts using a generative AI model (e.g., OpenAI's GPT-4). The server also stores the summarized transcripts in the HR management system's database and automatically updates to-do lists and calendars based on the interview content.

[0611] Terminal

[0612] The terminal is a device on which the user conducts the interview. When the interview begins, the terminal collects voice data using a built-in microphone and transmits it to the server in real time. When the interview ends, the terminal receives a summarized transcript and updated to-do list and calendar information from the server and notifies the user.

[0613] User

[0614] The user is the person conducting the interview. The user initiates the interview through the device and does not need to do anything during the interview. Once the interview is over, the user can view the summarized transcript and manage to-do lists and calendar updates through the device.

[0615] Examples

[0616] Examples of prompt statements

[0617] "Please summarize the interview below. Interview content: 'Hello, today I would like to discuss your work progress. First, please tell me how the project went last week.'"

[0618] Processing flow

[0619] 1. Starting the interview: The user launches the dedicated application on the device and clicks the "Start interview" button.

[0620] 2. Voice data collection: The device uses the built-in microphone to collect voice data and transmits it to the server in real time.

[0621] 3. Transcription: The server uses voice recognition software to convert the audio data into text.

[0622] 4. Summary generation: The server summarizes the transcript using a generative AI model.

[0623] 5. Data storage: The server stores the summarized transcripts in the HR management system database.

[0624] 6. ToDo list / calendar update: The server automatically updates the toDo list and calendar based on the meeting contents.

[0625] 7. User Notification: The terminal notifies the user with the summarized transcript and updates.

[0626] This system reduces the burden of creating interview records and allows you to concentrate on observing non-verbal information during the interview. It also makes it easy to follow up after an interview, as the contents of the interview can be efficiently managed and automatically reflected in a to-do list or calendar.

[0627] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0628] Step 1:

[0629] Interview begins

[0630] Subject: User

[0631] The user starts the dedicated application on the terminal and clicks the "Start interview" button. This causes the terminal to send a request to start the interview to the server. The input is the user's operation, and the output is the request to start the interview. As a specific operation, the terminal displays "Interview has started."

[0632] Step 2:

[0633] Audio data collection

[0634] Subject: Terminal

[0635] As soon as the interview begins, the terminal starts collecting voice data using the built-in microphone. The collected voice data is sent to the server in real time. The input is the user's voice, and the output is the voice data sent to the server. In concrete terms, the terminal displays "collecting voice data" and shows the collection status in real time.

[0636] Step 3:

[0637] Transcription

[0638] Subject: Server

[0639] The server receives the voice data sent from the terminal and converts it into text data using voice recognition software. The input is voice data and the output is text data. In concrete terms, the server displays "Creating a transcript" and updates the progress in real time.

[0640] Step 4:

[0641] Summary Generation

[0642] Subject: Server

[0643] The server uses a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The input is the text data of the verbatim transcript, and the output is the summarized text. The server displays "Generating summary" and provides real-time updates on the progress.

[0644] Step 5:

[0645] Data storage

[0646] Subject: Server

[0647] The server stores the summarized transcript in the database of the human resources management system. The input is the summarized text and the output is the record stored in the database. In concrete terms, the server displays "Saving data" and, when storage is complete, displays "Data storage completed."

[0648] Step 6:

[0649] ToDo list / Calendar update

[0650] Subject: Server

[0651] The server automatically updates the to-do list and calendar based on the contents of the meeting. The input is a summarized verbatim transcript, and the output is an updated to-do list and calendar. Specifically, the server displays "Updating to-do list and calendar" and then displays "Update completed" when the update is complete.

[0652] Step 7:

[0653] User Notifications

[0654] Subject: Terminal

[0655] The terminal receives the summarized verbatim transcript and updated ToDo list / calendar information from the server and notifies the user. The input is the notification information from the server, and the output is the notification to the user. As a specific operation, the terminal displays "Summary and updated information received" and notifies the user.

[0656] (Application example 1)

[0657] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0658] Conventional interview support tools have the problem that tasks such as creating interview records, summarizing, and updating to-do lists and calendars are often done manually, which takes time and effort. Similar problems exist in interviews between workers and robots in factories, making it difficult to provide efficient work instructions and manage schedules.

[0659] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and storing the interview record in the personnel management system, a means for automatically reflecting the results in a ToDo list or a calendar, a means for supporting interviews between workers and robots in a factory, and a means for automatically updating work instructions and schedules based on the interview contents. This reduces the burden of creating interview records, and enables efficient work instructions and schedule management.

[0660] "Generative AI" is a system that uses artificial intelligence technology to generate data and automate certain tasks.

[0661] An "interview implementation support tool" is a software or hardware system that supports the progress of interviews and the creation of records.

[0662] A "human resources management system" is a system used to manage employee information in a company or organization, and to manage labor and develop human resources.

[0663] A verbatim transcript is a document that records exactly what was said during an interview or meeting.

[0664] A "summary" is a document that extracts important points from a detailed record such as a verbatim transcript and summarizes them concisely.

[0665] A "ToDo list" is a list of tasks or work items that need to be done.

[0666] A "calendar" is a tool for managing schedules and events based on dates and times.

[0667] "Workers" are employees who actually perform work in a factory or on-site.

[0668] A "robot" is a mechanical device that performs tasks automatically according to programmed instructions.

[0669] A "work instruction" is a specific instruction or command for carrying out a specific task.

[0670] A "schedule" is a plan of tasks or events to be performed within a specific period of time.

[0671] As an embodiment of the present invention, a system for supporting interviews between workers and robots in a factory is constructed. A specific implementation method thereof is shown below.

[0672] System Configuration

[0673] 1. Hardware Configuration

[0674] Microphone: Used to collect interview audio.

[0675] Computer: Used to process the audio data and run generative AI models.

[0676] Robot: An automated machine that performs tasks in a factory.

[0677] 2. Software Configuration

[0678] OpenAI API: Used for audio transcription and summary generation.

[0679] Python: A programming language used for data processing and file manipulation.

[0680] Human Resources Management System: A system for storing interview records and summaries, and reflecting them in to-do lists and calendars.

[0681] Data processing and calculation

[0682] 1. Collecting audio data and transcribing

[0683] The server collects the interview audio through a microphone.

[0684] The collected audio data is converted into text using the OpenAI API, and a verbatim transcript is created.

[0685] 2. Summary of the Verbatim

[0686] The server summarizes the generated transcripts using the OpenAI API.

[0687] The summarized text is stored in the personnel management system.

[0688] 3. Adding to your ToDo list and calendar

[0689] The server extracts important tasks from the summarized text and adds them to a to-do list.

[0690] In addition, the schedule will be updated based on the contents of the interview and reflected on the calendar.

[0691] Examples

[0692] For example, if during an interview someone says, "We need to check the next maintenance schedule," that content is recorded in the transcript. After the interview ends, the server summarizes the transcript and generates a summary called "Check the next maintenance schedule." This summary is stored in the personnel management system, and "Check the next maintenance schedule" is automatically added to the to-do list.

[0693] Examples of prompt statements

[0694] An example prompt for summarizing a transcript is as follows:

[0695] Please summarize the following transcript:

[0696] [Transcript text]

[0697] summary:

[0698] An example prompt for retrieving to-do list items is:

[0699] Extract to-do list items from the summary below:

[0700] [Summary text]

[0701] To-do list:

[0702] In this way, it is possible to efficiently support meetings between workers and robots within the factory and automate work instructions and schedule management.

[0703] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0704] Step 1:

[0705] The server collects the interview voice through a microphone. The input is the interview voice, and the output is the voice data. Specifically, the microphone captures the voice during the interview in real time and transmits the voice data to the server.

[0706] Step 2:

[0707] The server converts the collected voice data into text using the OpenAI API and creates a transcript. The input is the voice data, and the output is the transcript text. Specifically, the voice data is input into OpenAI's speech recognition model, and the generated text is saved as a transcript.

[0708] Step 3:

[0709] The server summarizes the generated transcripts using the OpenAI API. The input is the transcript text, and the output is the summary text. Specifically, a prompt for summarizing the transcript text is input to the OpenAI text generation model, and the generated summary is obtained.

[0710] Step 4:

[0711] The server stores the summarized text in the human resources management system. The input is the summarized text, and the output is the summary data stored in the human resources management system. Specifically, the server performs an operation to store the summarized text in the database of the human resources management system.

[0712] Step 5:

[0713] The server extracts important tasks from the summarized text and adds them to the to-do list. The input is the summarized text, and the output is an updated to-do list. Specifically, the summary text is input to OpenAI's text generation model as a prompt for task extraction, and the generated tasks are added to the to-do list.

[0714] Step 6:

[0715] The server updates the schedule based on the meeting contents and reflects it in the calendar. The input is the summary text and the output is the updated calendar. Specifically, the server extracts schedule-related information from the summary text and adds that information to the calendar system.

[0716] Step 7:

[0717] The user checks the updated ToDo list and calendar and performs the necessary tasks. The input is the updated ToDo list and calendar, and the output is the user's execution of the tasks. Specifically, the user logs into the system, checks the ToDo list and calendar, and performs the tasks instructed.

[0718] Example 2

[0719] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0720] In conventional personnel management systems, the creation and storage of interview records and the creation of employee training plans were often done manually, which was time-consuming and labor-intensive. In addition, there was a lack of sufficient extraction of employees' strengths and weaknesses based on interview records, creation of individual training plans, and career path suggestions, making it difficult to provide effective training tailored to each employee.

[0721] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0722] In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and store the interview record in the personnel management system, a means for analyzing the interview record and extracting the strengths and weaknesses of employees, a means for creating an individual training plan based on the extracted data, a means for proposing a career path based on the interests and inclinations of employees, and a means for automatically reflecting the results in a to-do list or a calendar. This automates the creation and storage of interview records, and makes it possible to efficiently extract the strengths and weaknesses of employees, create individual training plans, and propose career paths, enabling effective training tailored to each employee.

[0723] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[0724] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[0725] A "human resources management system" is software or a system for managing employee information and supporting human resources tasks such as training and evaluation.

[0726] "Verbatim transcription" is the process of recording the exact verbatim content of an interview.

[0727] "Summarizing" is the process of concisely summarizing the contents of an interview.

[0728] "Interview records" are data that record the contents of an interview.

[0729] "Storage" is the process of storing data and making it accessible when needed.

[0730] "Analysis" is the process of analyzing data and extracting useful information.

[0731] "Strengths" are the skills and abilities that employees excel at.

[0732] "Weaknesses" are skills or abilities that an employee is not good at.

[0733] A "development plan" is a plan to support employees in improving their skills and growing their careers.

[0734] "Interests" are areas or activities that interest employees.

[0735] "Orientation" refers to the direction or goal that an employee aims for.

[0736] A "career path" is the path of the job or role that an employee will pursue in the future.

[0737] A "ToDo list" is a list of tasks that need to be done.

[0738] A "calendar" is a tool for managing appointments and schedules.

[0739] This invention is a system that develops an interview implementation support tool that utilizes generative AI and implements it in a personnel management system to efficiently create and store interview records, create employee training plans, and propose career paths. A specific embodiment of this system is described below.

[0740] Hardware and Software Used

[0741] Hardware: Servers, terminals (PCs, tablets, etc.)

[0742] Software: Human resource management systems, generative AI models (e.g. OpenAI GPT-4)

[0743] Program Processing

[0744] 1. Entering interview records

[0745] After the interview, the user uses a terminal (PC or tablet) to enter the interview record into the human resources management system.

[0746] Specifically, the user accesses the input form of the personnel management system and inputs the details of the interview in text format.

[0747] Example: "A user uses a terminal to enter interview records for employee A. His strengths are project management and his weaknesses are presentation skills."

[0748] 2. Keeping records of interviews

[0749] The server receives the interview records sent by the users and stores them in the database of the personnel management system.

[0750] Specifically, the server converts the input data into an appropriate format and stores it in a database.

[0751] Example: "The server receives the interview notes and stores them in a database."

[0752] 3. Data extraction and analysis

[0753] The server analyzes the stored interview records and extracts employees' strengths and weaknesses.

[0754] Specifically, the server inputs the interview records into a generative AI model (e.g., OpenAI GPT-4) and obtains the analysis results.

[0755] Example: "The server inputs the interview records into the generative AI model and extracts the strengths and weaknesses of employee A."

[0756] 4. Creating a training plan

[0757] The server creates an individual training plan based on the extracted data.

[0758] Specifically, the server will suggest the training and education necessary to improve employee skills based on the analysis results of the generative AI model.

[0759] Example: "The server will propose training in project management to utilize Employee A's strengths, and to address Employee A's weaknesses, he will plan training to improve presentation skills."

[0760] 5. Career path suggestions

[0761] The server suggests appropriate career paths based on employees' interests and aspirations.

[0762] Specifically, the server designs future career paths for employees based on the analysis results of the generative AI model and reflects the suggestions in the human resources management system.

[0763] Example: "Since Employee A is interested in project management, the server will suggest that Employee A should become a project manager in the future."

[0764] Examples

[0765] Example of interview record entry:

[0766] "The user uses the terminal to input the interview record of employee B. The strength is data analysis, and the weakness is teamwork."

[0767] Example prompt for a generative AI model:

[0768] "Based on the following interview record, please identify Employee B's strengths and weaknesses and propose an appropriate training plan. Interview record: Strengths are data analysis, and weaknesses are teamwork."

[0769] This system automates the creation and storage of interview records, making it possible to efficiently identify employees' strengths and weaknesses, create individual training plans, and suggest career paths, enabling effective training tailored to each individual employee.

[0770] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0771] Step 1:

[0772] Enter interview records

[0773] After the interview, the user uses a terminal (PC or tablet) to enter the interview record into the human resources management system.

[0774] Input: Interview details (in text format), employee strengths and weaknesses, and other interview information.

[0775] Specific operation: The user accesses the input form of the personnel management system and inputs the interview details in text format. For example, the user inputs, "Enter the interview record of employee A. His strengths are project management and his weaknesses are presentation skills."

[0776] Output: The entered interview record is sent to the HR management system.

[0777] Step 2:

[0778] Keeping records of interviews

[0779] The server receives the interview records sent by the users and stores them in the database of the personnel management system.

[0780] Input: Interview notes submitted by user.

[0781] Specific operation: The server converts the input data into an appropriate format and stores it in the database. For example, the server receives the interview record and stores "Employee A's interview record" in the database.

[0782] Output: Interview notes stored in a database.

[0783] Step 3:

[0784] Data extraction and analysis

[0785] The server analyzes the stored interview records and extracts employees' strengths and weaknesses.

[0786] Input: Interview notes stored in database.

[0787] Specific operation: The server inputs the interview record into the generative AI model (e.g., OpenAI GPT-4) and obtains the analysis results. For example, the server inputs "Employee A's interview record" into the generative AI model and extracts strengths and weaknesses.

[0788] Output: Employee strengths and weaknesses as analyzed results.

[0789] Step 4:

[0790] Creating a training plan

[0791] The server creates an individual training plan based on the extracted data.

[0792] Input: Employee strengths and weaknesses as analyzed results.

[0793] Specific operation: Based on the analysis results of the generative AI model, the server proposes the training and education necessary to improve the skills of employees. For example, the server creates a training plan such as, "Utilizing employee A's strengths, we propose training in project management. To compensate for his weaknesses, we plan training to improve presentation skills."

[0794] Output: An individual development plan.

[0795] Step 5:

[0796] Career path suggestions

[0797] The server suggests appropriate career paths based on employees' interests and aspirations.

[0798] Input: Employee interests and inclinations as analyzed results.

[0799] Specific operation: Based on the analysis results of the generative AI model, the server designs future career paths for employees and reflects the proposed content in the personnel management system. For example, the server may suggest a career path such as "Employee A is interested in project management, so we propose a career path to become a project manager in the future."

[0800] Output: A proposed career path.

[0801] (Application example 2)

[0802] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0803] In conventional personnel management systems, creating and managing interview records was a hassle, and line managers had no time to observe non-verbal information during interviews. In addition, there was a lack of a system that efficiently collected and analyzed robot work records and proposed optimal work placements and improvement plans. This made it difficult to improve the skills of employees and robots and to place them in efficient work positions.

[0804] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0805] In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and store the interview record in the personnel management system, a means for automatically reflecting it in a to-do list or a calendar, a means for collecting the robot's work record and analyzing its efficiency and error rate, a means for grasping the robot's good and bad tasks and proposing optimal work allocation, and a means for creating individual improvement plans. This reduces the burden of creating interview records, allowing the line manager to focus on non-verbal information, and makes it possible to analyze the robot's work efficiency and error rate and propose optimal work allocation and improvement plans.

[0806] "Generative AI" is a system that uses artificial intelligence techniques to generate data to automate or assist with specific tasks.

[0807] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[0808] A "human resources management system" is a system for managing employee information and streamlining human resources tasks such as evaluation, training, and placement.

[0809] "Verbatim transcription" is the process of recording the exact words of a conversation, such as an interview or meeting.

[0810] A "summary" is a short summary of a longer piece of text or conversation.

[0811] A "ToDo list" is a tool that allows you to manage tasks and schedules in list format.

[0812] "Auto-add to calendar" is a feature that automatically adds specific events and tasks to your calendar.

[0813] "Robot work record" refers to a detailed record of the work performed by a robot.

[0814] "Efficiency" is a measure of how effectively a task or process is carried out.

[0815] An "error rate" is an indicator that indicates the frequency with which errors occur in a task or process.

[0816] A "good task" refers to a task that a particular robot can perform more efficiently than other tasks.

[0817] A "weak task" refers to a task that a particular robot performs less efficiently than other tasks.

[0818] "Optimal work placement" refers to placing robots and employees so that they can perform their work most efficiently.

[0819] "Improvement plan" refers to a specific plan for improving work efficiency and error rate.

[0820] The system for implementing this invention is to develop an interview support tool that utilizes generative AI and implement it in a personnel management system. Specifically, generative AI is used to create and summarize interview transcripts, and the interview records are stored in the personnel management system. It also includes a function to automatically reflect the results in a to-do list or calendar.

[0821] In addition, the system also includes a function to collect the robot's work records and analyze its efficiency and error rate. This allows the system to understand the tasks that the robot is good at and bad at, and to propose optimal work arrangements. It also includes a function to create individual improvement plans.

[0822] Hardware and Software Used

[0823] Hardware: Servers, factory robots, user terminals

[0824] Software: Python, JSON, generative AI models

[0825] Data processing and calculation

[0826] The server creates and summarizes interview transcripts using generative AI models. These data are stored in the HR management system in JSON format. In addition, information extracted from the interview records is automatically updated in to-do lists and calendars.

[0827] The robot's work records are collected to calculate efficiency and error rates. These data are analyzed on the server to identify tasks that the robot is good at and bad at. Based on the analysis results, optimal work placement and individual improvement plans are generated.

[0828] Examples

[0829] For example, consider a scenario in which a factory robot saves records of its assembly work and creates an improvement plan based on those records. The robot's work records are collected as follows:

[0830] Example prompt:

[0831] Robot ID: robot_1

[0832] Operation: Assembly

[0833] Efficiency: 0.9

[0834] Error rate: 0.05

[0835] By inputting this prompt sentence into the generative AI model, the process of saving and analyzing the robot's work records and creating improvement plans is automated. This reduces the burden of creating interview records and allows line managers to focus on non-verbal information, while also enabling them to analyze the robot's work efficiency and error rate, and propose optimal work placement and improvement plans.

[0836] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0837] Step 1:

[0838] The server creates a transcript of the interview using a generative AI model. As input, it receives the audio data of the interview and generates a transcript using the generative AI model. As output, the transcript is obtained in text format.

[0839] Step 2:

[0840] The server summarizes the generated transcripts: as input it takes the text data of the transcript and applies a summarization algorithm; as output it obtains the summarized text.

[0841] Step 3:

[0842] The server stores the transcripts and summaries in JSON format in the HR management system. As input, it receives the text data of the transcripts and summaries and converts them into JSON format. As output, the JSON format data is stored in the HR management system.

[0843] Step 4:

[0844] The server automatically updates the to-do list and calendar based on the information extracted from the interview record. As input, it receives the JSON data of the interview record and converts it into the format of the to-do list and calendar. As output, new tasks and events are added to the to-do list and calendar.

[0845] Step 5:

[0846] The server collects the robot's work records. As input, it receives the work data (such as work ID, efficiency, error rate, etc.) sent from the robot. As output, the collected work records are stored in a database.

[0847] Step 6:

[0848] The server analyzes the collected work records. As input, it receives the work records stored in the database, calculates the efficiency and error rate, and as output, it obtains the analysis results.

[0849] Step 7:

[0850] The server knows what tasks the robot is good at and bad at. As input, it takes the analysis results and applies an algorithm to identify what tasks the robot is good at and bad at. As output, it gets a list of what tasks the robot is good at and bad at.

[0851] Step 8:

[0852] The server proposes an optimal task placement. As input, it receives a list of good and bad tasks and applies an algorithm to calculate the optimal task placement. As output, it obtains a proposal for the optimal task placement.

[0853] Step 9:

[0854] The server creates an individualized improvement plan. As input, it receives the analysis results and the optimal work placement proposals, and applies an algorithm to generate the improvement plan. As output, it obtains the individualized improvement plan.

[0855] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0856] "Example 1"

[0857] As an example 1 of the present invention, there is a system that combines an emotion engine. In this system, an interview implementation support tool that utilizes a generation AI is developed and implemented in a personnel management system. The generation AI creates and summarizes a verbatim record of the interview, and stores the results in the personnel management system. In addition, it also has a function that automatically reflects the contents of the interview in a to-do list and a calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This makes it possible to grasp not only the contents of the interview but also the user's emotional state at that time, enabling a deeper understanding and response.

[0858] "Example 2"

[0859] As a second embodiment of the present invention, there is a system that reduces the burden on line managers in creating interview records and allows them to concentrate on observing nonverbal information during interviews. In this system, a generation AI creates and summarizes interviews verbatim, and stores the results in a personnel management system. In addition, it also has a function that automatically reflects the contents of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the emotions of the user during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This reduces the burden on line managers in creating interview records and allows them to concentrate on observing nonverbal information during the interview.

[0860] "Example 4"

[0861] As a fourth embodiment of the present invention, there is a system that utilizes implementation records to create an environment for continuous employee development. In this system, a generation AI creates and summarizes interviews verbatim, and stores the results in a personnel management system. In addition, it also has a function to automatically reflect the contents of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the emotions of the user during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This makes it possible to create an environment for continuous employee development by utilizing interview records.

[0862] The process flow of each embodiment will be described below.

[0863] "Example 1"

[0864] Step 1: When the interview begins, the generative AI starts creating a transcript.

[0865] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[0866] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[0867] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[0868] Step 5: The results of the emotion engine analysis are reflected in the interview transcript.

[0869] "Example 2"

[0870] Step 1: When the interview begins, the generative AI starts creating a transcript.

[0871] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[0872] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[0873] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[0874] Step 5: The analysis results of the emotion engine are reflected in the interview record, reducing the line manager's burden of creating interview records and allowing him or her to focus on observing non-verbal information during the interview.

[0875] "Example 4"

[0876] Step 1: When the interview begins, the generative AI starts creating a transcript.

[0877] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[0878] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[0879] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[0880] Step 5: The analysis results of the emotion engine are reflected in the interview records, and the interview records can be used to create an environment for continuous employee development.

[0881] Example 1

[0882] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0883] Previously, interview records were created manually, placing a heavy burden on line managers and administrators. It was also difficult to properly record non-verbal information and emotional states during interviews, resulting in a lack of information to improve the quality of interviews. Furthermore, the task of reflecting the contents of interviews in to-do lists and calendars was time-consuming, and efficient management was required.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0885] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create a verbatim transcript and summarize the interview and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or a calendar, a means for recognizing an emotional state during the interview using an emotion engine and reflecting the information in the interview record, and a means for automating a series of processes from the start to the end of the interview. This reduces the burden of creating interview records, enables detailed interview records including non-verbal information and emotional states, and makes it possible to efficiently manage the contents of the interview.

[0886] "Generative AI" is a system that uses artificial intelligence technology to generate text and data.

[0887] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[0888] A "human resources management system" is an integrated system for managing employee information, evaluating, and training.

[0889] A verbatim transcript is a document that records exactly what was said during an interview or meeting.

[0890] A "summary" is a short document that extracts the important points from a detailed record such as a verbatim transcript.

[0891] A "ToDo List" is a list that displays tasks or action items that need to be completed.

[0892] A "calendar" is a tool that displays dates and times for schedule management.

[0893] An "emotion engine" is a technology or system for analyzing emotional states from voice, facial expressions, choice of words, etc.

[0894] "Nonverbal information" is information that is conveyed by means other than words, and includes facial expressions, gestures, and tone of voice.

[0895] "Process automation" refers to the automatic execution of a series of tasks that were previously performed manually by a system.

[0896] This invention relates to developing an interview implementation support tool that utilizes generative AI and implementing it in a personnel management system. A specific embodiment of this system is described below.

[0897] System Configuration

[0898] The system consists of three main components: a server, a terminal, and a user. The server is equipped with a generative AI model and an emotion engine, and is responsible for creating transcripts of interviews, summarizing them, recognizing emotional states, and automatically updating to-do lists and calendars. The terminal is responsible for collecting voice data from interviews and sending it to the server. The user conducts the interviews and operates the system.

[0899] Hardware and Software Used

[0900] Server: A server with high-performance computing power (e.g. cloud server)

[0901] Generative AI models: Natural language processing models such as OpenAI's GPT-4

[0902] Emotion engine: Emotion recognition API, etc.

[0903] Human Resources Management System: Integrated System

[0904] To-do list and calendar: schedule management tools

[0905] Device: A computer or tablet equipped with a microphone to collect audio data.

[0906] Data processing and calculation

[0907] When a user starts an interview, the device collects audio data and transmits it in real time to the server, which then inputs the received audio data into a generative AI model to generate a transcript, which is then temporarily stored on the server.

[0908] Once the interview is over, the server uses the generative AI model again to summarize the transcript, which is then stored in the personnel management system. The server also analyzes the interview content and automatically updates it on to-do lists and calendars.

[0909] Using an emotion engine, the server analyzes the user's tone of voice, facial expressions and word choice to recognize their emotional state. This emotion information is also added to the transcript and stored in the personnel management system.

[0910] Examples of concrete examples and prompts

[0911] As a concrete example, consider the following scenario.

[0912] scenario:

[0913] A user has a meeting with a subordinate. During the meeting, the subordinate reports the progress of the project and sets some tasks as next steps. The subordinate also expresses concerns about the project.

[0914] Example prompt:

[0915] "Start a meeting with your subordinate. Transcribe and summarize the conversation. Analyze the emotional state of the meeting and reflect it on your to-do list and calendar."

[0916] This system reduces the burden of creating interview records and allows users to focus on observing non-verbal information during interviews. It also enables users to efficiently manage interview content and keep detailed records, including emotional states.

[0917] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0918] Step 1:

[0919] The user starts an interview. The user clicks the interview start button on the terminal to notify the system that he or she wants to start the interview. The input is the user's operation, and the output is the signal to start the interview. The terminal displays the message "Starting the interview."

[0920] Step 2:

[0921] The terminal collects voice data during the interview in real time. The input is the user's voice, and the output is the collected voice data. The terminal collects voice data using a built-in microphone and transmits it to the server in real time. The server stores the received voice data in a buffer.

[0922] Step 3:

[0923] The server inputs the received voice data into a generative AI model to generate a verbatim transcript. The input is the voice data and the output is the generated verbatim transcript. The server inputs the buffered voice data into a generative AI model to generate a verbatim transcript. The generated verbatim transcript is stored in the server's temporary storage.

[0924] Step 4:

[0925] Once the interview is over, the server uses the generative AI model again to summarize the transcript. The input is the verbatim transcript and the output is the summarized verbatim transcript. The server re-inputs the verbatim transcript into the generative AI model and generates a summary. The summarized content is saved in temporary storage.

[0926] Step 5:

[0927] The server stores the summarized verbatim transcript in the HRMS. The input is the summarized verbatim transcript and the output is the data stored in the HRMS. The server saves the summarized verbatim transcript in the HRMS database. Once saved, the user is notified "Summary saved."

[0928] Step 6:

[0929] The server parses the condensed verbatim transcript and adds the necessary information to the to-do list or calendar. The input is the condensed verbatim transcript and the output is an updated to-do list or calendar. The server parses the condensed verbatim transcript and adds the necessary information to the to-do list or calendar. For example, action items such as "schedule next meeting" or "complete specific task" are automatically added.

[0930] Step 7:

[0931] The server uses an emotion engine to analyze the user's emotional state from the voice data or the verbatim transcript. The input is the voice data or the verbatim transcript and the output is the recognized emotion information. The server uses an emotion engine to analyze the user's emotional state from the voice data or the verbatim transcript. For example, if the user expresses "anxiety", that information is recognized by the emotion engine.

[0932] Step 8:

[0933] The server adds the recognized emotion information to the transcript and stores it in the HR management system. The input is the recognized emotion information and the output is the updated transcript. The server adds the recognized emotion information to the transcript and stores it in the HR management system database. Once the saving is complete, the user is notified that "emotion information has been saved."

[0934] (Application example 1)

[0935] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0936] In conventional interview support systems, tasks such as creating verbatim transcripts of interviews, summarizing, and recognizing emotions were often done manually, which led to the problem of a high workload for creating interview records. In addition, it was difficult to concentrate on observing non-verbal information during the interview, which could lead to a decline in the quality of the interview. Furthermore, there was a lack of functionality to automatically reflect the contents of the interview in a to-do list or calendar, which meant that follow-up after the interview was insufficient. In order to solve these issues, an efficient interview support system that utilizes generative AI was needed.

[0937] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0938] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create and summarize a verbatim record of an interview and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or calendar, a means for combining an emotion engine to recognize the user's emotions during the interview and reflecting the information in the interview record, a means for converting the voice of the interview into text as a verbatim record using speech recognition technology, a means for summarizing the verbatim record and analyzing emotions using a generative AI model, and a means for saving the results of the summary and emotion analysis as a calendar or interview record. This reduces the burden of creating interview records and allows users to concentrate on observing non-verbal information during the interview. Furthermore, follow-up after the interview is performed efficiently, improving the quality of the interview.

[0939] "Generative AI" is a system that uses artificial intelligence techniques to generate, analyze, and summarize data.

[0940] An "interview support tool" is a combination of software and hardware that supports the progress of interviews and efficiently creates records.

[0941] A "human resources management system" is a system used to manage employee information and evaluate and develop employees in a company or organization.

[0942] A verbatim transcript is a recording of an interview, meeting, etc., transcribed verbatim.

[0943] A "summary" is a short summary of long data such as a verbatim transcript, extracting the important points.

[0944] An "interview record" is a document or data that records the contents of an interview, including a verbatim transcript, a summary, or the results of sentiment analysis.

[0945] A "ToDo list" is a list of tasks that need to be done.

[0946] A "calendar" is a tool for managing appointments and schedules.

[0947] An "emotion engine" is a system that analyzes emotions from voice and text data and outputs the results.

[0948] "Speech recognition technology" is a technology that converts speech into text.

[0949] A “generative AI model” is a specific implementation of generative AI: a machine learning model trained to perform a specific task.

[0950] "Nonverbal information" is information that is conveyed by means other than words, and includes facial expressions, tone of voice, gestures, etc.

[0951] As an embodiment of the present invention, a factory robot interview support system will be described as an example. This system utilizes generative AI to create verbatim records of interviews, summarize them, recognize emotions, and automatically reflect them in to-do lists and calendars.

[0952] Sabato will develop an interview support tool that utilizes generative AI and implement it in the personnel management system. Specifically, it will use voice recognition technology to convert the audio of the interview into a verbatim transcript, and then use a generative AI model to summarize the transcript and analyze emotions. This will reduce the burden of creating interview records, and allow employees to focus on observing non-verbal information during the interview.

[0953] The server uses a generative AI model to summarize the transcript and analyze emotions. For example, Library A is used as the generative AI model. For speech recognition technology, Library B is used. This allows the interview transcript to be automatically created and summarized.

[0954] The server recognizes the user's emotions during the interview by combining the emotion engine and reflects the information in the interview record. A specified pipeline is used as the emotion engine. This allows the server to grasp not only the contents of the interview but also the user's emotional state at that time.

[0955] The server saves the summary and the results of the sentiment analysis as a calendar or interview record. The saved data is automatically reflected in the to-do list and calendar. This makes follow-up after the interview more efficient and improves the quality of the interview.

[0956] As a concrete example, a summary can be obtained by inputting the following prompt sentence into a generative AI model:

[0957] Example prompt:

[0958] "Today I would like to discuss the efficiency of our new production line. Although the current production rate is not up to the target, there are some areas for improvement. First, we need to review the maintenance schedule of our machines. Also, we need to update the training program for our workers."

[0959] By inputting this prompt into a generative AI model, a summary can be obtained, allowing the content of the interview to be efficiently understood and necessary actions to be taken quickly.

[0960] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0961] Step 1:

[0962] The server uses speech recognition technology to convert the interview audio into text, verbatim.

[0963] Input: Audio data from the interview

[0964] Output: Verbatim transcription (text data)

[0965] Specific operation: The server converts the voice data acquired from the microphone into text, which records the contents of the interview verbatim.

[0966] Step 2:

[0967] The server summarizes the transcript using a generative AI model.

[0968] Input: Verbatim transcription (text data)

[0969] Output: Summary (text data)

[0970] Specific Actions: The server summarizes the transcript, specifically extracting the key points of the transcript and summarizing them in a concise format.

[0971] Step 3:

[0972] The server analyzes the sentiment of the verbatim transcript using a sentiment engine.

[0973] Input: Verbatim transcription (text data)

[0974] Output: Sentiment analysis results (text data)

[0975] What it does: The server analyzes the emotions in the transcript, specifically recognizing emotions from the user's tone of voice and choice of words, and outputs the results.

[0976] Step 4:

[0977] The server stores the summaries and the results of sentiment analysis as calendars and interview records.

[0978] Input: Summary (text data), sentiment analysis results (text data)

[0979] Output: Calendar entries, meeting notes (data in JSON format)

[0980] How it works: The server saves the summary and the results of sentiment analysis in JSON format and automatically updates the calendar and to-do list, allowing for efficient follow-up after the interview.

[0981] Step 5:

[0982] The user reviews the interview record and takes necessary actions.

[0983] Input: Calendar entries, interview notes (data in JSON format)

[0984] Output: Action plan (text data)

[0985] Specific Actions: The user checks the meeting records and calendar entries provided by the server and plans the necessary actions, allowing specific responses based on the contents of the meeting.

[0986] Example 2

[0987] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0988] In the conventional interview record preparation method, line managers had to prepare interview records manually, which was a heavy burden. It was also difficult to properly observe non-verbal information during interviews (tone of voice, facial expressions, etc.) and reflect them in the records. Furthermore, the environment for utilizing interview records for continuous employee development was not in place.

[0989] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in the personnel management system, a means for using the generation AI to create a verbatim record of the interview and summarize it, and storing the interview record in the personnel management system, a means for collecting the tone of voice and facial expression of the user during the interview, analyzing emotions using an emotion engine, and reflecting the results in the interview record, a means for automatically reflecting the interview content in a ToDo list or a calendar, and a means for storing the results of the verbatim record, summary, and emotion analysis in the personnel management system. This reduces the burden on the line manager in creating interview records, allowing him to concentrate on observing non-verbal information during the interview, and further makes it possible to prepare an environment for continuous employee development by utilizing the interview records.

[0990] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[0991] "Interview implementation support tool" is a general term for software or hardware used to assist in the progress of interviews and the creation of records.

[0992] A "human resources management system" is an integrated software system for managing employee information, evaluating, and training.

[0993] A "verbatim transcript" is text data that records the content of an interview or conversation verbatim.

[0994] A "summary" is text data that is created by extracting important points and keywords from a verbatim transcript and summarizing them concisely.

[0995] An "emotion engine" is software or hardware that analyzes data such as voice and facial expressions and recognizes the user's emotions.

[0996] "Nonverbal information" is information that is conveyed by means other than words, and specifically includes tone of voice, facial expressions, gestures, etc.

[0997] A "ToDo list" is a tool for managing tasks and schedules in list format.

[0998] A "calendar" is a tool for managing schedules and events based on dates and times.

[0999] "Verbatim transcription" is the process of converting audio data into text and recording the verbatim content of an interview or conversation.

[1000] "Interview records" are data that record the contents of an interview, including verbatim transcripts, summaries, and the results of sentiment analysis.

[1001] "Sentiment analysis" is the process of analyzing data such as voice and facial expressions to recognize a user's emotions.

[1002] "Auto-reflection" is the process of automatically reflecting certain data or information in other systems or tools.

[1003] This invention is a system that uses an interview implementation support tool that utilizes generative AI to create a verbatim transcript of an interview, summarize it, and analyze emotions, and then stores the data in a personnel management system and automatically reflects it in a to-do list or calendar. A specific embodiment of this system is described below.

[1004] Hardware and Software Used

[1005] Hardware: Servers, terminals (PCs, tablets, smartphones)

[1006] Software: Generative AI models (e.g. GPT-4), emotion engines, human resource management systems

[1007] Data processing and calculation flow

[1008] 1. Collecting audio data from interviews

[1009] The user starts the interview.

[1010] The device collects audio data from interviews in real time, specifically by recording audio using the microphone on a PC, tablet, or smartphone.

[1011] The terminal transmits the collected voice data to a server.

[1012] 2. Transcription of audio data

[1013] The server sends the received voice data to the generative AI model.

[1014] A generative AI model converts the audio data into text and creates a transcript, using speech recognition technology to convert speech to text.

[1015] 3. Summary of the Transcript

[1016] The generative AI model summarizes the transcript by extracting key points and keywords and generating a concise summary.

[1017] The server receives the summarized text and proceeds to the next processing step.

[1018] 4. Sentiment analysis

[1019] The device collects the user's tone of voice and facial expressions during the interview, specifically by using a camera and microphone to record changes in facial expressions and voice.

[1020] The server sends the collected data to the emotion engine.

[1021] The emotion engine analyzes the user's emotions by recognizing their tone of voice, facial expressions, and choice of words.

[1022] The server adds the results of the sentiment analysis to the transcript.

[1023] 5. Data Retention

[1024] The server stores the transcripts, summaries, and sentiment analysis results in the personnel management system, specifically in a database.

[1025] The HR management system manages the stored data and ensures that it can be accessed when needed.

[1026] 6. Adding to your ToDo list and calendar

[1027] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar.

[1028] The HR management system automatically adds the generated tasks to to-do lists and calendars, and in particular, reflects them in employees' schedules and task management tools.

[1029] Examples

[1030] For example, when employee A has an interview with line manager B, the following specific actions are taken:

[1031] 1. The users (employee A and line manager B) start an interview, and the terminal (PC) records the audio.

[1032] 2. The device transmits the recorded audio data to the server in real time.

[1033] 3. The server sends the audio data to a generative AI model to create a transcript.

[1034] 4. A generative AI model summarizes the transcript and extracts key takeaways.

[1035] 5. The terminal collects Employee A’s facial expressions and tone of voice during the interview, and the server sends them to the emotion engine.

[1036] 6. The sentiment engine analyzes the sentiment and adds the results to the transcript.

[1037] 7. The server stores the transcripts, summaries, and sentiment analysis results in the human resources management system.

[1038] 8. The HR management system analyzes the interview content and generates tasks to be reflected in to-do lists and calendars.

[1039] Examples of prompt statements

[1040] "Please prepare and summarize a transcript of the meeting between Employee A and Line Manager B. Also, please analyze Employee A's emotions during the meeting and add the results to the transcript."

[1041] This system reduces the burden on line managers of creating interview records, allowing them to focus on observing non-verbal information during interviews. It also creates an environment for continuous employee development by utilizing interview records.

[1042] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1043] Step 1:

[1044] The user starts the interview. The device collects the voice data of the interview in real time. Specifically, the device records the voice using the microphone of a PC, tablet, or smartphone. The device sends the collected voice data to the server. The input is the voice data of the interview, and the output is the voice data sent to the server.

[1045] Step 2:

[1046] The server sends the received voice data to the generative AI model. The generative AI model converts the voice data into text and creates a transcript. Specifically, it uses speech recognition technology to convert speech to text. The input is the voice data sent to the server, and the output is the generated transcript.

[1047] Step 3:

[1048] The generative AI model summarizes the transcript by extracting key points and keywords and generating a concise summary. The server receives the summarized text and proceeds to the next processing step. The input is the generated transcript and the output is the summarized text.

[1049] Step 4:

[1050] The device collects the user's tone of voice and facial expressions during the interview. Specifically, it uses a camera and microphone to record facial expressions and changes in voice. The server sends the collected data to the emotion engine. The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from tone of voice, facial expressions, choice of words, etc. The input is the collected tone of voice and facial expression data, and the output is the result of the emotion analysis.

[1051] Step 5:

[1052] The server adds the results of the sentiment analysis to the transcript, i.e. integrates the results of the sentiment analysis into the transcript in text form. The input is the results of the sentiment analysis and the verbatim transcript, and the output is the verbatim transcript with the sentiment analysis added.

[1053] Step 6:

[1054] The server stores the verbatim transcripts, summaries, and the results of sentiment analysis in the HRMS, specifically in a database. The HRMS manages the stored data and makes it accessible when needed. The input is the verbatim transcripts with added sentiment analysis, summaries, and the results of sentiment analysis, and the output is the data stored in the HRMS.

[1055] Step 7:

[1056] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar. The personnel management system automatically adds the generated tasks to the to-do list and calendar. Specifically, it reflects them in the employee's schedule and task management tool. The input is the analyzed interview content, and the output is the tasks reflected in the to-do list and calendar.

[1057] (Application example 2)

[1058] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1059] In conventional interview record creation systems, the creation of verbatim records of interviews, summarization, and emotion recognition were done manually, placing a heavy burden on line managers and administrators. In addition, interview records were not fully utilized to improve employees' skills or design career paths, making it difficult to create individual training plans. Furthermore, there was a problem in that training and skill improvement support for factory robot operators was not effectively provided.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1061] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create a verbatim transcript of an interview and summarize it, and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or a calendar, a means for recognizing emotions during an interview using an emotion engine and reflecting the information in the interview record, and a means for recording interviews and training sessions conducted by operators of factory robots and supporting the improvement of the operators' skills and the design of their career paths. This reduces the burden of creating interview records, making it possible to concentrate on observing non-verbal information during interviews, and further makes it possible to create an environment for continuous employee development by utilizing the implementation records.

[1062] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[1063] An "interview implementation support tool" is software or a system that supports the recording and analysis of interviews.

[1064] A "human resources management system" is a system that manages employee information and supports skill development and career path planning.

[1065] "Verbatim transcription" is the process of recording the verbatim contents of an interview or conversation.

[1066] "Summarizing" is the process of making a shorter summary of a longer piece of text or conversation.

[1067] A "ToDo List" is a tool that allows you to manage tasks that need to be done in list format.

[1068] A "calendar" is a tool for managing schedules and appointments.

[1069] An "emotion engine" is a system for analyzing and recognizing emotions from voice and text.

[1070] An "interview record" is a document or data that records the contents of an interview.

[1071] A "factory robot" is an automated machine used to perform tasks in a factory.

[1072] An "operator" is a person who operates a machine or system.

[1073] A "Training Session" is a training session designed to improve a particular skill or knowledge.

[1074] "Upskilling" is the process of improving a particular skill or ability.

[1075] A "career path" is the path to the occupation or position that an employee will pursue in the future.

[1076] In order to implement the present invention, the following system configuration and program are required.

[1077] System Configuration

[1078] The server will develop an interview support tool that utilizes generative AI and implement it in the personnel management system. Using generative AI, it will transcribe and summarize interviews verbatim, and store the interview records in the personnel management system. It will also have a function to automatically reflect these in to-do lists and calendars. It will use an emotion engine to recognize emotions during interviews, and reflect this information in the interview records. It will record interviews and training sessions conducted by factory robot operators, and support the operators in improving their skills and designing their career paths.

[1079] Hardware and Software Used

[1080] Hardware: Microphone, Computer

[1081] Software: Python, SpeechRecognition library, Transformers library, EmotionRecognition library

[1082] Program processing explanation

[1083] The server reads the audio file and obtains the audio data. It then converts the audio data into text. It uses a generative AI model to summarize the text and uses an emotion engine to analyze emotions from the text. Based on this information, it creates an interview record and saves it in JSON format. The interview record is stored in the HR management system and automatically reflected in the to-do list and calendar.

[1084] Examples

[1085] An operator is interviewed about the operation of a factory robot, and the audio is recorded. The recorded audio file is input to a server, which generates a verbatim transcript and summary of the interview, and also analyzes the operator's emotions. This makes it easier to improve the operator's skills and design career paths.

[1086] Examples of prompt statements

[1087] Summarize the following text:

[1088] "Today we discussed the operation of a factory robot. It seems that operator A feels unconfident in certain operations. However, he is very skilled in other operations, and is particularly strong in troubleshooting. In future training sessions, we plan to create a program to strengthen A's weak points."

[1089] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1090] Step 1:

[1091] The server reads the audio file and obtains the audio data. Specifically, an audio file recorded using a microphone is uploaded to the server, and the audio data is obtained using the SpeechRecognition library. The input is the audio file, and the output is the audio data.

[1092] Step 2:

[1093] The server converts the acquired voice data into text. Specifically, it converts the voice data into text using the SpeechRecognition library. The input is voice data, and the output is text data.

[1094] Step 3:

[1095] The server uses a generative AI model to summarize the text data. Specifically, it uses a summarization model from the Transformers library to summarize the text data. The input is the text data, and the output is the summarized text.

[1096] Step 4:

[1097] The server uses an emotion engine to analyze emotions from text data. Specifically, it uses the EmotionRecognition library to extract emotions from text data. The input is text data and the output is emotion data.

[1098] Step 5:

[1099] The server creates a meeting record based on the summarized text and emotion data. Specifically, it combines the summarized text and emotion data to generate a meeting record in JSON format. The input is the summarized text and emotion data, and the output is the meeting record.

[1100] Step 6:

[1101] The server stores the created interview record in the personnel management system. Specifically, it saves the generated interview record in JSON format in the database. The input is the interview record, and the output is the record saved in the database.

[1102] Step 7:

[1103] The server automatically updates the ToDo list and calendar based on the contents of the interview record. Specifically, tasks and schedules extracted from the interview record are added to the ToDo list and calendar. The input is the interview record, and the output is the updated ToDo list and calendar.

[1104] Step 8:

[1105] The user records interviews and training sessions conducted by factory robot operators, and supports the operator's skill development and career path planning. Specifically, the user creates individual training plans based on the interview records, and aims to improve the operator's skills. The input is the interview records, and the output is the training plans.

[1106] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1107] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1108] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1109] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1110] [Third embodiment]

[1111] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1112] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1113] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1114] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1115] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1116] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1117] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1118] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1119] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1120] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1121] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1122] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1123] "Example 1"

[1124] As an embodiment of the present invention, an interview implementation support tool that utilizes generative AI is developed and implemented in a personnel management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript. When the interview ends, the generative AI summarizes the verbatim transcript and stores the summary in the personnel management system. It also has a function that automatically reflects the contents of the interview in a to-do list or calendar. This reduces the burden on line managers of creating interview records, allowing them to concentrate on observing nonverbal information during the interview.

[1125] "Example 2"

[1126] In another embodiment of the present invention, the implementation records are utilized to create an environment for continuous employee development. Specifically, interview records are stored in a personnel management system, and based on that, employee skills are improved and career paths are designed. For example, the strengths and weaknesses of employees are extracted from the interview records, and based on that, individual development plans are created. In addition, the interests and inclinations of employees are understood from the interview records, and career paths are proposed based on that. This makes it possible to provide development tailored to each individual employee.

[1127] The process flow of each embodiment will be described below.

[1128] "Example 1"

[1129] Step 1: When the interview begins, the generative AI starts creating a transcript.

[1130] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the summary in the HR management system.

[1131] Step 3: It also has a function that automatically updates your to-do list and calendar based on the contents of the interview.

[1132] "Example 2"

[1133] Step 1: Store the interview records in a human resources management system and use them to improve employee skills and design career paths.

[1134] Step 2: Extract the employee's strengths and weaknesses from the interview records and create an individual development plan based on that.

[1135] Step 3: Understand the employee's interests and aspirations from the interview records and use that to suggest a career path.

[1136] Example 1

[1137] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1138] Traditional interview records are often created manually, placing a heavy burden on line managers and supervisors. In addition, there is no time to observe non-verbal information during the interview, which can lead to a decline in the quality of the interview. Furthermore, it is difficult to efficiently manage the interview content and reflect it in a to-do list or calendar, which can lead to insufficient follow-up after the interview.

[1139] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1140] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for utilizing the generative AI to create and summarize a verbatim record of an interview and store the interview record in the personnel management system, a means for a terminal to collect voice data at the start of an interview and send it to the server, a means for the server to convert the voice data into text data using voice recognition software, a means for summarizing the verbatim record using a generative AI model, a means for storing the summarized verbatim record in a database, a means for automatically updating a to-do list or a calendar based on the interview content, and a means for the terminal to notify the user of the summarized verbatim record and update information. This reduces the burden of creating interview records, making it possible to concentrate on observing non-verbal information during interviews. In addition, the contents of the interview can be efficiently managed and automatically reflected in a to-do list or a calendar, making it easier to follow up after the interview.

[1141] "Generative AI" is a system that uses artificial intelligence technology to generate and process data.

[1142] An "interview implementation support tool" is a combination of software and / or hardware that assists in the conduct of interviews and the creation of records.

[1143] A "human resources management system" is an integrated system for managing employee information, evaluating, and training.

[1144] A verbatim transcript is a recording of an interview, meeting, etc., transcribed verbatim.

[1145] A "summary" is a concise summary of detailed information, such as a verbatim transcript.

[1146] "Speech recognition software" refers to software for converting voice data into text data.

[1147] "Generative AI model" refers to the core algorithm or trained model of generative AI.

[1148] A "database" is a system for efficiently storing, managing, and searching data.

[1149] A "ToDo list" is a list for managing tasks and schedules that need to be done in a list format.

[1150] A "calendar" is a tool for managing schedules and events based on dates and times.

[1151] A "terminal" is a device such as a computer or smartphone that is directly operated by a user.

[1152] A "server" is a computer system for processing and storing data over a network.

[1153] "Nonverbal information" refers to information that is conveyed by means other than words, such as facial expressions and gestures.

[1154] A "notification" is a message or alert sent from the system to inform the user of information.

[1155] This invention involves developing an interview support tool that utilizes generative AI and implementing it in a personnel management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript, and when the interview ends, it summarizes the verbatim transcript and stores the summary in the personnel management system. It also has a function that automatically reflects the contents of the interview in a to-do list or calendar.

[1156] Hardware and Software Used

[1157] server

[1158] The server provides the computational resources to run the generative AI model, converts the voice data into text using speech recognition software, and summarizes the transcripts using a generative AI model (e.g., OpenAI's GPT-4). The server also stores the summarized transcripts in the HR management system's database and automatically updates to-do lists and calendars based on the interview content.

[1159] Terminal

[1160] The terminal is a device on which the user conducts the interview. When the interview begins, the terminal collects voice data using a built-in microphone and transmits it to the server in real time. When the interview ends, the terminal receives a summarized transcript and updated to-do list and calendar information from the server and notifies the user.

[1161] User

[1162] The user is the person conducting the interview. The user initiates the interview through the device and does not need to do anything during the interview. Once the interview is over, the user can view the summarized transcript and manage to-do lists and calendar updates through the device.

[1163] Examples

[1164] Examples of prompt statements

[1165] "Please summarize the interview below. Interview content: 'Hello, today I would like to discuss your work progress. First, please tell me how the project went last week.'"

[1166] Processing flow

[1167] 1. Starting the interview: The user launches the dedicated application on the device and clicks the "Start interview" button.

[1168] 2. Voice data collection: The device uses the built-in microphone to collect voice data and transmits it to the server in real time.

[1169] 3. Transcription: The server uses voice recognition software to convert the audio data into text.

[1170] 4. Summary generation: The server summarizes the transcript using a generative AI model.

[1171] 5. Data storage: The server stores the summarized transcripts in the HR management system database.

[1172] 6. ToDo list / calendar update: The server automatically updates the toDo list and calendar based on the meeting contents.

[1173] 7. User Notification: The terminal notifies the user with the summarized transcript and updates.

[1174] This system reduces the burden of creating interview records and allows you to concentrate on observing nonverbal information during interviews. It also makes it easy to follow up after interviews, as the contents of interviews can be efficiently managed and automatically reflected in to-do lists and calendars.

[1175] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1176] Step 1:

[1177] Interview begins

[1178] Subject: User

[1179] The user starts the dedicated application on the terminal and clicks the "Start interview" button. This causes the terminal to send a request to start the interview to the server. The input is the user's operation, and the output is the request to start the interview. As a specific operation, the terminal displays "Interview has started."

[1180] Step 2:

[1181] Audio data collection

[1182] Subject: Terminal

[1183] As soon as the interview begins, the terminal starts collecting voice data using the built-in microphone. The collected voice data is sent to the server in real time. The input is the user's voice, and the output is the voice data sent to the server. In concrete terms, the terminal displays "collecting voice data" and shows the collection status in real time.

[1184] Step 3:

[1185] Transcription

[1186] Subject: Server

[1187] The server receives the voice data sent from the terminal and converts it into text data using voice recognition software. The input is voice data and the output is text data. In concrete terms, the server displays "Creating a transcript" and updates the progress in real time.

[1188] Step 4:

[1189] Summary Generation

[1190] Subject: Server

[1191] The server uses a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The input is the text data of the verbatim transcript, and the output is the summarized text. The server displays "Generating summary" and provides real-time updates on the progress.

[1192] Step 5:

[1193] Data storage

[1194] Subject: Server

[1195] The server stores the summarized transcript in the database of the human resources management system. The input is the summarized text and the output is the record stored in the database. In concrete terms, the server displays "Saving data" and, when storage is complete, displays "Data storage completed."

[1196] Step 6:

[1197] ToDo list / Calendar update

[1198] Subject: Server

[1199] The server automatically updates the to-do list and calendar based on the contents of the meeting. The input is a summarized verbatim transcript, and the output is an updated to-do list and calendar. Specifically, the server displays "Updating to-do list and calendar" and then displays "Update completed" when the update is complete.

[1200] Step 7:

[1201] User Notifications

[1202] Subject: Terminal

[1203] The terminal receives the summarized verbatim transcript and updated ToDo list / calendar information from the server and notifies the user. The input is the notification information from the server, and the output is the notification to the user. As a specific operation, the terminal displays "Summary and updated information received" and notifies the user.

[1204] (Application example 1)

[1205] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1206] Conventional interview support tools have the problem that tasks such as creating interview records, summarizing, and updating to-do lists and calendars are often done manually, which takes time and effort. Similar problems exist in interviews between workers and robots in factories, making it difficult to provide efficient work instructions and manage schedules.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and storing the interview record in the personnel management system, a means for automatically reflecting the results in a ToDo list or a calendar, a means for supporting interviews between workers and robots in a factory, and a means for automatically updating work instructions and schedules based on the interview contents. This reduces the burden of creating interview records, and enables efficient work instructions and schedule management.

[1208] "Generative AI" is a system that uses artificial intelligence technology to generate data and automate certain tasks.

[1209] An "interview implementation support tool" is a software or hardware system that supports the progress of interviews and the creation of records.

[1210] A "human resources management system" is a system used to manage employee information in a company or organization, and to manage labor and develop human resources.

[1211] A verbatim transcript is a document that records exactly what was said during an interview or meeting.

[1212] A "summary" is a document that extracts important points from a detailed record such as a verbatim transcript and summarizes them concisely.

[1213] A "ToDo list" is a list of tasks or work items that need to be done.

[1214] A "calendar" is a tool for managing schedules and events based on dates and times.

[1215] "Workers" are employees who actually perform work in a factory or on-site.

[1216] A "robot" is a mechanical device that performs tasks automatically according to programmed instructions.

[1217] A "work instruction" is a specific instruction or command for carrying out a specific task.

[1218] A "schedule" is a plan of tasks or events to be performed within a specific period of time.

[1219] As an embodiment of the present invention, a system for supporting interviews between workers and robots in a factory is constructed. A specific implementation method thereof is shown below.

[1220] System Configuration

[1221] 1. Hardware Configuration

[1222] Microphone: Used to collect interview audio.

[1223] Computer: Used to process the audio data and run generative AI models.

[1224] Robot: An automated machine that performs tasks in a factory.

[1225] 2. Software Configuration

[1226] OpenAI API: Used for audio transcription and summary generation.

[1227] Python: A programming language used for data processing and file manipulation.

[1228] Human Resources Management System: A system for storing interview records and summaries, and reflecting them in to-do lists and calendars.

[1229] Data processing and calculation

[1230] 1. Collecting audio data and transcribing

[1231] The server collects the interview audio through a microphone.

[1232] The collected audio data is converted into text using the OpenAI API, and a verbatim transcript is created.

[1233] 2. Summary of the Verbatim

[1234] The server summarizes the generated transcripts using the OpenAI API.

[1235] The summarized text is stored in the personnel management system.

[1236] 3. Adding to your ToDo list and calendar

[1237] The server extracts important tasks from the summarized text and adds them to a to-do list.

[1238] In addition, the schedule will be updated based on the contents of the interview and reflected on the calendar.

[1239] Examples

[1240] For example, if during an interview someone says, "We need to check the next maintenance schedule," that content is recorded in the transcript. After the interview ends, the server summarizes the transcript and generates a summary called "Check the next maintenance schedule." This summary is stored in the personnel management system, and "Check the next maintenance schedule" is automatically added to the to-do list.

[1241] Examples of prompt statements

[1242] An example prompt for summarizing a transcript is as follows:

[1243] Please summarize the following transcript:

[1244] [Transcript text]

[1245] summary:

[1246] An example prompt for retrieving to-do list items is:

[1247] Extract to-do list items from the summary below:

[1248] [Summary text]

[1249] To-do list:

[1250] In this way, it is possible to efficiently support meetings between workers and robots within the factory and automate work instructions and schedule management.

[1251] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1252] Step 1:

[1253] The server collects the interview voice through a microphone. The input is the interview voice, and the output is the voice data. Specifically, the microphone captures the voice during the interview in real time and transmits the voice data to the server.

[1254] Step 2:

[1255] The server converts the collected voice data into text using the OpenAI API and creates a transcript. The input is the voice data, and the output is the transcript text. Specifically, the voice data is input into OpenAI's speech recognition model, and the generated text is saved as a transcript.

[1256] Step 3:

[1257] The server summarizes the generated transcripts using the OpenAI API. The input is the transcript text, and the output is the summary text. Specifically, a prompt for summarizing the transcript text is input to the OpenAI text generation model, and the generated summary is obtained.

[1258] Step 4:

[1259] The server stores the summarized text in the human resources management system. The input is the summarized text, and the output is the summary data stored in the human resources management system. Specifically, the server performs an operation to store the summarized text in the database of the human resources management system.

[1260] Step 5:

[1261] The server extracts important tasks from the summarized text and adds them to the to-do list. The input is the summarized text, and the output is an updated to-do list. Specifically, the summary text is input to OpenAI's text generation model as a prompt for task extraction, and the generated tasks are added to the to-do list.

[1262] Step 6:

[1263] The server updates the schedule based on the meeting contents and reflects it in the calendar. The input is the summary text and the output is the updated calendar. Specifically, the server extracts schedule-related information from the summary text and adds that information to the calendar system.

[1264] Step 7:

[1265] The user checks the updated ToDo list and calendar and performs the necessary tasks. The input is the updated ToDo list and calendar, and the output is the user's execution of the tasks. Specifically, the user logs into the system, checks the ToDo list and calendar, and performs the tasks instructed.

[1266] Example 2

[1267] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1268] In conventional personnel management systems, the creation and storage of interview records and the creation of employee training plans were often done manually, which was time-consuming and labor-intensive. In addition, there was a lack of sufficient extraction of employees' strengths and weaknesses based on interview records, creation of individual training plans, and career path suggestions, making it difficult to provide effective training tailored to each employee.

[1269] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1270] In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and store the interview record in the personnel management system, a means for analyzing the interview record and extracting the strengths and weaknesses of employees, a means for creating an individual training plan based on the extracted data, a means for proposing a career path based on the interests and inclinations of employees, and a means for automatically reflecting the results in a to-do list or a calendar. This automates the creation and storage of interview records, and makes it possible to efficiently extract the strengths and weaknesses of employees, create individual training plans, and propose career paths, enabling effective training tailored to each employee.

[1271] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[1272] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[1273] A "human resources management system" is software or a system for managing employee information and supporting human resources tasks such as training and evaluation.

[1274] "Verbatim transcription" is the process of recording the exact verbatim content of an interview.

[1275] "Summarizing" is the process of concisely summarizing the contents of an interview.

[1276] "Interview records" are data that record the contents of an interview.

[1277] "Storage" is the process of storing data and making it accessible when needed.

[1278] "Analysis" is the process of analyzing data and extracting useful information.

[1279] "Strengths" are the skills and abilities that employees excel at.

[1280] "Weaknesses" are skills or abilities that an employee is not good at.

[1281] A "development plan" is a plan to support employees in improving their skills and growing their careers.

[1282] "Interests" are areas or activities that interest employees.

[1283] "Orientation" refers to the direction or goal that an employee aims for.

[1284] A "career path" is the path of the job or role that an employee will pursue in the future.

[1285] A "ToDo list" is a list of tasks that need to be done.

[1286] A "calendar" is a tool for managing appointments and schedules.

[1287] This invention is a system that develops an interview implementation support tool that utilizes generative AI and implements it in a personnel management system to efficiently create and store interview records, create employee training plans, and propose career paths. A specific embodiment of this system is described below.

[1288] Hardware and Software Used

[1289] Hardware: Servers, terminals (PCs, tablets, etc.)

[1290] Software: Human resource management systems, generative AI models (e.g. OpenAI GPT-4)

[1291] Program Processing

[1292] 1. Entering interview records

[1293] After the interview, the user uses a terminal (PC or tablet) to enter the interview record into the human resources management system.

[1294] Specifically, the user accesses the input form of the personnel management system and inputs the details of the interview in text format.

[1295] Example: "A user uses a terminal to enter interview records for employee A. His strengths are project management and his weaknesses are presentation skills."

[1296] 2. Keeping records of interviews

[1297] The server receives the interview records sent by the users and stores them in the database of the personnel management system.

[1298] Specifically, the server converts the input data into an appropriate format and stores it in a database.

[1299] Example: "The server receives the interview notes and stores them in a database."

[1300] 3. Data extraction and analysis

[1301] The server analyzes the stored interview records and extracts employees' strengths and weaknesses.

[1302] Specifically, the server inputs the interview records into a generative AI model (e.g., OpenAI GPT-4) and obtains the analysis results.

[1303] Example: "The server inputs the interview records into the generative AI model and extracts the strengths and weaknesses of employee A."

[1304] 4. Creating a training plan

[1305] The server creates an individual training plan based on the extracted data.

[1306] Specifically, the server will suggest the training and education necessary to improve employee skills based on the analysis results of the generative AI model.

[1307] Example: "The server will propose training in project management to utilize Employee A's strengths, and to address Employee A's weaknesses, he will plan training to improve presentation skills."

[1308] 5. Career path suggestions

[1309] The server suggests appropriate career paths based on employees' interests and aspirations.

[1310] Specifically, the server designs future career paths for employees based on the analysis results of the generative AI model and reflects the suggestions in the human resources management system.

[1311] Example: "Since Employee A is interested in project management, the server will suggest that Employee A should become a project manager in the future."

[1312] Examples

[1313] Example of interview record entry:

[1314] "The user uses the terminal to input the interview record of employee B. The strength is data analysis, and the weakness is teamwork."

[1315] Example prompt for a generative AI model:

[1316] "Based on the following interview record, please identify Employee B's strengths and weaknesses and propose an appropriate training plan. Interview record: Strengths are data analysis, and weaknesses are teamwork."

[1317] This system automates the creation and storage of interview records, making it possible to efficiently identify employees' strengths and weaknesses, create individual training plans, and suggest career paths, enabling effective training tailored to each individual employee.

[1318] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1319] Step 1:

[1320] Enter interview records

[1321] After the interview, the user uses a terminal (PC or tablet) to enter the interview record into the human resources management system.

[1322] Input: Interview details (in text format), employee strengths and weaknesses, and other interview information.

[1323] Specific operation: The user accesses the input form of the personnel management system and inputs the interview details in text format. For example, the user inputs, "Enter the interview record of employee A. His strengths are project management and his weaknesses are presentation skills."

[1324] Output: The entered interview record is sent to the HR management system.

[1325] Step 2:

[1326] Keeping records of interviews

[1327] The server receives the interview records sent by the users and stores them in the database of the personnel management system.

[1328] Input: Interview notes submitted by user.

[1329] Specific operation: The server converts the input data into an appropriate format and stores it in the database. For example, the server receives the interview record and stores "Employee A's interview record" in the database.

[1330] Output: Interview notes stored in a database.

[1331] Step 3:

[1332] Data extraction and analysis

[1333] The server analyzes the stored interview records and extracts employees' strengths and weaknesses.

[1334] Input: Interview notes stored in database.

[1335] Specific operation: The server inputs the interview record into the generative AI model (e.g., OpenAI GPT-4) and obtains the analysis results. For example, the server inputs "Employee A's interview record" into the generative AI model and extracts strengths and weaknesses.

[1336] Output: Employee strengths and weaknesses as analyzed results.

[1337] Step 4:

[1338] Creating a training plan

[1339] The server creates an individual training plan based on the extracted data.

[1340] Input: Employee strengths and weaknesses as analyzed results.

[1341] Specific operation: Based on the analysis results of the generative AI model, the server proposes the training and education necessary to improve the skills of employees. For example, the server creates a training plan such as, "Utilizing employee A's strengths, we propose training in project management. To compensate for his weaknesses, we plan training to improve presentation skills."

[1342] Output: An individual development plan.

[1343] Step 5:

[1344] Career path suggestions

[1345] The server suggests appropriate career paths based on employees' interests and aspirations.

[1346] Input: Employee interests and inclinations as analyzed results.

[1347] Specific operation: Based on the analysis results of the generative AI model, the server designs future career paths for employees and reflects the proposed content in the personnel management system. For example, the server may suggest a career path such as "Employee A is interested in project management, so we propose a career path to become a project manager in the future."

[1348] Output: A proposed career path.

[1349] (Application example 2)

[1350] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1351] In conventional personnel management systems, creating and managing interview records was a hassle, and line managers had no time to observe non-verbal information during interviews. In addition, there was a lack of a system that efficiently collected and analyzed robot work records and proposed optimal work placements and improvement plans. This made it difficult to improve the skills of employees and robots and to place them in efficient work positions.

[1352] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1353] In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and store the interview record in the personnel management system, a means for automatically reflecting it in a to-do list or a calendar, a means for collecting the robot's work record and analyzing its efficiency and error rate, a means for grasping the robot's good and bad tasks and proposing optimal work allocation, and a means for creating individual improvement plans. This reduces the burden of creating interview records, allowing the line manager to focus on non-verbal information, and makes it possible to analyze the robot's work efficiency and error rate and propose optimal work allocation and improvement plans.

[1354] "Generative AI" is a system that uses artificial intelligence techniques to generate data to automate or assist with specific tasks.

[1355] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[1356] A "human resources management system" is a system for managing employee information and streamlining human resources tasks such as evaluation, training, and placement.

[1357] "Verbatim transcription" is the process of recording the exact words of a conversation, such as an interview or meeting.

[1358] A "summary" is a short summary of a longer piece of text or conversation.

[1359] A "ToDo list" is a tool that allows you to manage tasks and schedules in list format.

[1360] "Auto-add to calendar" is a feature that automatically adds specific events and tasks to your calendar.

[1361] "Robot work record" refers to a detailed record of the work performed by a robot.

[1362] "Efficiency" is a measure of how effectively a task or process is carried out.

[1363] An "error rate" is an indicator that indicates the frequency with which errors occur in a task or process.

[1364] A "good task" refers to a task that a particular robot can perform more efficiently than other tasks.

[1365] A "weak task" refers to a task that a particular robot performs less efficiently than other tasks.

[1366] "Optimal work placement" refers to placing robots and employees so that they can perform their work most efficiently.

[1367] "Improvement plan" refers to a specific plan for improving work efficiency and error rate.

[1368] The system for implementing this invention is to develop an interview support tool that utilizes generative AI and implement it in a personnel management system. Specifically, generative AI is used to create and summarize interview transcripts, and the interview records are stored in the personnel management system. It also includes a function to automatically reflect the results in a to-do list or calendar.

[1369] In addition, the system also includes a function to collect the robot's work records and analyze its efficiency and error rate. This allows the system to understand the tasks that the robot is good at and bad at, and to propose optimal work arrangements. It also includes a function to create individual improvement plans.

[1370] Hardware and Software Used

[1371] Hardware: Servers, factory robots, user terminals

[1372] Software: Python, JSON, generative AI models

[1373] Data processing and calculation

[1374] The server creates and summarizes interview transcripts using generative AI models. These data are stored in the HR management system in JSON format. In addition, information extracted from the interview records is automatically updated in to-do lists and calendars.

[1375] The robot's work records are collected to calculate efficiency and error rates. These data are analyzed on the server to identify tasks that the robot is good at and bad at. Based on the analysis results, optimal work placement and individual improvement plans are generated.

[1376] Examples

[1377] For example, consider a scenario in which a factory robot saves records of its assembly work and creates an improvement plan based on those records. The robot's work records are collected as follows:

[1378] Example prompt:

[1379] Robot ID: robot_1

[1380] Operation: Assembly

[1381] Efficiency: 0.9

[1382] Error rate: 0.05

[1383] By inputting this prompt sentence into the generative AI model, the process of saving and analyzing the robot's work records and creating improvement plans is automated. This reduces the burden of creating interview records and allows line managers to focus on non-verbal information, while also enabling them to analyze the robot's work efficiency and error rate, and propose optimal work placement and improvement plans.

[1384] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1385] Step 1:

[1386] The server creates a transcript of the interview using a generative AI model. As input, it receives the audio data of the interview and generates a transcript using the generative AI model. As output, the transcript is obtained in text format.

[1387] Step 2:

[1388] The server summarizes the generated transcripts: as input it takes the text data of the transcript and applies a summarization algorithm; as output it obtains the summarized text.

[1389] Step 3:

[1390] The server stores the transcripts and summaries in JSON format in the HR management system. As input, it receives the text data of the transcripts and summaries and converts them into JSON format. As output, the JSON format data is stored in the HR management system.

[1391] Step 4:

[1392] The server automatically updates the to-do list and calendar based on the information extracted from the interview record. As input, it receives the JSON data of the interview record and converts it into the format of the to-do list and calendar. As output, new tasks and events are added to the to-do list and calendar.

[1393] Step 5:

[1394] The server collects the robot's work records. As input, it receives the work data (such as work ID, efficiency, error rate, etc.) sent from the robot. As output, the collected work records are stored in a database.

[1395] Step 6:

[1396] The server analyzes the collected work records. As input, it receives the work records stored in the database, calculates the efficiency and error rate, and as output, it obtains the analysis results.

[1397] Step 7:

[1398] The server knows what tasks the robot is good at and bad at. As input, it takes the analysis results and applies an algorithm to identify what tasks the robot is good at and bad at. As output, it gets a list of what tasks the robot is good at and bad at.

[1399] Step 8:

[1400] The server proposes an optimal task placement. As input, it receives a list of good and bad tasks and applies an algorithm to calculate the optimal task placement. As output, it obtains a proposal for the optimal task placement.

[1401] Step 9:

[1402] The server creates an individualized improvement plan. As input, it receives the analysis results and the optimal work placement proposals, and applies an algorithm to generate the improvement plan. As output, it obtains the individualized improvement plan.

[1403] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[1404] "Example 1"

[1405] As an example 1 of the present invention, there is a system that combines an emotion engine. In this system, an interview implementation support tool that utilizes a generation AI is developed and implemented in a personnel management system. The generation AI creates and summarizes a verbatim record of the interview, and stores the results in the personnel management system. In addition, it also has a function that automatically reflects the contents of the interview in a to-do list and a calendar. By combining this system with an emotion engine, it becomes possible to recognize the user's emotions during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This makes it possible to grasp not only the contents of the interview but also the user's emotional state at that time, enabling a deeper understanding and response.

[1406] "Example 2"

[1407] As a second embodiment of the present invention, there is a system that reduces the burden on line managers in creating interview records and allows them to concentrate on observing nonverbal information during interviews. In this system, a generation AI creates and summarizes interviews verbatim, and stores the results in a personnel management system. In addition, it also has a function that automatically reflects the contents of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the emotions of the user during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This reduces the burden on line managers in creating interview records and allows them to concentrate on observing nonverbal information during the interview.

[1408] "Example 4"

[1409] As a fourth embodiment of the present invention, there is a system that utilizes implementation records to create an environment for continuous employee development. In this system, a generation AI creates and summarizes interviews verbatim, and stores the results in a personnel management system. In addition, it also has a function to automatically reflect the contents of the interview in a to-do list or calendar. By combining this system with an emotion engine, it becomes possible to recognize the emotions of the user during the interview and reflect that information in the interview record. Specifically, emotions are analyzed from the user's tone of voice, facial expressions, choice of words, etc., and the results are added to the interview record. This makes it possible to create an environment for continuous employee development by utilizing interview records.

[1410] The process flow of each embodiment will be described below.

[1411] "Example 1"

[1412] Step 1: When the interview begins, the generative AI starts creating a transcript.

[1413] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[1414] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[1415] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[1416] Step 5: The results of the emotion engine analysis are reflected in the interview transcript.

[1417] "Example 2"

[1418] Step 1: When the interview begins, the generative AI starts creating a transcript.

[1419] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[1420] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[1421] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[1422] Step 5: The analysis results of the emotion engine are reflected in the interview record, reducing the line manager's burden of creating interview records and allowing him or her to focus on observing non-verbal information during the interview.

[1423] "Example 4"

[1424] Step 1: When the interview begins, the generative AI starts creating a transcript.

[1425] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the results in the HR management system.

[1426] Step 3: The contents of the interview are automatically reflected in your to-do list and calendar.

[1427] Step 4: The emotion engine analyzes the user's emotions based on their tone of voice, facial expressions, choice of words, etc.

[1428] Step 5: The analysis results of the emotion engine are reflected in the interview records, and the interview records can be used to create an environment for continuous employee development.

[1429] Example 1

[1430] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1431] Previously, interview records were created manually, placing a heavy burden on line managers and administrators. It was also difficult to properly record non-verbal information and emotional states during interviews, resulting in a lack of information to improve the quality of interviews. Furthermore, the task of reflecting the contents of interviews in to-do lists and calendars was time-consuming, and efficient management was required.

[1432] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1433] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create a verbatim transcript and summarize the interview and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or a calendar, a means for recognizing an emotional state during the interview using an emotion engine and reflecting the information in the interview record, and a means for automating a series of processes from the start to the end of the interview. This reduces the burden of creating interview records, enables detailed interview records including non-verbal information and emotional states, and makes it possible to efficiently manage the contents of the interview.

[1434] "Generative AI" is a system that uses artificial intelligence technology to generate text and data.

[1435] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[1436] A "human resources management system" is an integrated system for managing employee information, evaluating, and training.

[1437] A verbatim transcript is a document that records exactly what was said during an interview or meeting.

[1438] A "summary" is a short document that extracts the important points from a detailed record such as a verbatim transcript.

[1439] A "ToDo List" is a list that displays tasks or action items that need to be completed.

[1440] A "calendar" is a tool that displays dates and times for schedule management.

[1441] An "emotion engine" is a technology or system for analyzing emotional states from voice, facial expressions, choice of words, etc.

[1442] "Nonverbal information" is information that is conveyed by means other than words, and includes facial expressions, gestures, and tone of voice.

[1443] "Process automation" refers to the automatic execution of a series of tasks that were previously performed manually by a system.

[1444] This invention relates to developing an interview implementation support tool that utilizes generative AI and implementing it in a personnel management system. A specific embodiment of this system is described below.

[1445] System Configuration

[1446] The system consists of three main components: a server, a terminal, and a user. The server is equipped with a generative AI model and an emotion engine, and is responsible for creating transcripts of interviews, summarizing them, recognizing emotional states, and automatically updating to-do lists and calendars. The terminal is responsible for collecting voice data from interviews and sending it to the server. The user conducts the interviews and operates the system.

[1447] Hardware and Software Used

[1448] Server: A server with high-performance computing power (e.g. cloud server)

[1449] Generative AI models: Natural language processing models such as OpenAI's GPT-4

[1450] Emotion engine: Emotion recognition API, etc.

[1451] Human Resources Management System: Integrated System

[1452] To-do list and calendar: schedule management tools

[1453] Device: A computer or tablet equipped with a microphone to collect audio data.

[1454] Data processing and calculation

[1455] When a user starts an interview, the device collects audio data and transmits it in real time to the server, which then inputs the received audio data into a generative AI model to generate a transcript, which is then temporarily stored on the server.

[1456] Once the interview is over, the server uses the generative AI model again to summarize the transcript, which is then stored in the personnel management system. The server also analyzes the interview content and automatically updates it on to-do lists and calendars.

[1457] Using an emotion engine, the server analyzes the user's tone of voice, facial expressions and word choice to recognize their emotional state. This emotion information is also added to the transcript and stored in the personnel management system.

[1458] Examples of concrete examples and prompts

[1459] As a concrete example, consider the following scenario.

[1460] scenario:

[1461] A user has a meeting with a subordinate. During the meeting, the subordinate reports the progress of the project and sets some tasks as next steps. The subordinate also expresses concerns about the project.

[1462] Example prompt:

[1463] "Start a meeting with your subordinate. Transcribe and summarize the conversation. Analyze the emotional state of the meeting and reflect it on your to-do list and calendar."

[1464] This system reduces the burden of creating interview records and allows users to focus on observing non-verbal information during interviews. It also enables users to efficiently manage interview content and keep detailed records, including emotional states.

[1465] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1466] Step 1:

[1467] The user starts an interview. The user clicks the interview start button on the terminal to notify the system that he or she wants to start the interview. The input is the user's operation, and the output is the signal to start the interview. The terminal displays the message "Starting the interview."

[1468] Step 2:

[1469] The terminal collects voice data during the interview in real time. The input is the user's voice, and the output is the collected voice data. The terminal collects voice data using a built-in microphone and transmits it to the server in real time. The server stores the received voice data in a buffer.

[1470] Step 3:

[1471] The server inputs the received voice data into a generative AI model to generate a verbatim transcript. The input is the voice data and the output is the generated verbatim transcript. The server inputs the buffered voice data into a generative AI model to generate a verbatim transcript. The generated verbatim transcript is stored in the server's temporary storage.

[1472] Step 4:

[1473] Once the interview is over, the server uses the generative AI model again to summarize the transcript. The input is the verbatim transcript and the output is the summarized verbatim transcript. The server re-inputs the verbatim transcript into the generative AI model and generates a summary. The summarized content is saved in temporary storage.

[1474] Step 5:

[1475] The server stores the summarized verbatim transcript in the HRMS. The input is the summarized verbatim transcript and the output is the data stored in the HRMS. The server saves the summarized verbatim transcript in the HRMS database. Once saved, the user is notified "Summary saved."

[1476] Step 6:

[1477] The server parses the condensed verbatim transcript and adds the necessary information to the to-do list or calendar. The input is the condensed verbatim transcript and the output is an updated to-do list or calendar. The server parses the condensed verbatim transcript and adds the necessary information to the to-do list or calendar. For example, action items such as "schedule next meeting" or "complete specific task" are automatically added.

[1478] Step 7:

[1479] The server uses an emotion engine to analyze the user's emotional state from the voice data or the verbatim transcript. The input is the voice data or the verbatim transcript and the output is the recognized emotion information. The server uses an emotion engine to analyze the user's emotional state from the voice data or the verbatim transcript. For example, if the user expresses "anxiety", that information is recognized by the emotion engine.

[1480] Step 8:

[1481] The server adds the recognized emotion information to the transcript and stores it in the HR management system. The input is the recognized emotion information and the output is the updated transcript. The server adds the recognized emotion information to the transcript and stores it in the HR management system database. Once the saving is complete, the user is notified that "emotion information has been saved."

[1482] (Application example 1)

[1483] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1484] In conventional interview support systems, tasks such as creating verbatim transcripts of interviews, summarizing, and recognizing emotions were often done manually, which led to the problem of a high workload for creating interview records. In addition, it was difficult to concentrate on observing non-verbal information during the interview, which could lead to a decline in the quality of the interview. Furthermore, there was a lack of functionality to automatically reflect the contents of the interview in a to-do list or calendar, which meant that follow-up after the interview was insufficient. In order to solve these issues, an efficient interview support system that utilizes generative AI was needed.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1486] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create and summarize a verbatim record of an interview and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or calendar, a means for combining an emotion engine to recognize the user's emotions during the interview and reflecting the information in the interview record, a means for converting the voice of the interview into text as a verbatim record using speech recognition technology, a means for summarizing the verbatim record and analyzing emotions using a generative AI model, and a means for saving the results of the summary and emotion analysis as a calendar or interview record. This reduces the burden of creating interview records and allows users to concentrate on observing non-verbal information during the interview. Furthermore, follow-up after the interview is performed efficiently, improving the quality of the interview.

[1487] "Generative AI" is a system that uses artificial intelligence techniques to generate, analyze, and summarize data.

[1488] An "interview support tool" is a combination of software and hardware that supports the progress of interviews and efficiently creates records.

[1489] A "human resources management system" is a system used to manage employee information and evaluate and develop employees in a company or organization.

[1490] A verbatim transcript is a recording of an interview, meeting, etc., transcribed verbatim.

[1491] A "summary" is a short summary of long data such as a verbatim transcript, extracting the important points.

[1492] An "interview record" is a document or data that records the contents of an interview, including a verbatim transcript, a summary, or the results of sentiment analysis.

[1493] A "ToDo list" is a list of tasks that need to be done.

[1494] A "calendar" is a tool for managing appointments and schedules.

[1495] An "emotion engine" is a system that analyzes emotions from voice and text data and outputs the results.

[1496] "Speech recognition technology" is a technology that converts speech into text.

[1497] A “generative AI model” is a specific implementation of generative AI: a machine learning model trained to perform a specific task.

[1498] "Nonverbal information" is information that is conveyed by means other than words, and includes facial expressions, tone of voice, gestures, etc.

[1499] As an embodiment of the present invention, a factory robot interview support system will be described as an example. This system utilizes generative AI to create verbatim records of interviews, summarize them, recognize emotions, and automatically reflect them in to-do lists and calendars.

[1500] Sabato will develop an interview support tool that utilizes generative AI and implement it in the personnel management system. Specifically, it will use voice recognition technology to convert the audio of the interview into a verbatim transcript, and then use a generative AI model to summarize the transcript and analyze emotions. This will reduce the burden of creating interview records, and allow employees to focus on observing non-verbal information during the interview.

[1501] The server uses a generative AI model to summarize the transcript and analyze emotions. For example, Library A is used as the generative AI model. For speech recognition technology, Library B is used. This allows the interview transcript to be automatically created and summarized.

[1502] The server recognizes the user's emotions during the interview by combining the emotion engine and reflects the information in the interview record. A specified pipeline is used as the emotion engine. This allows the server to grasp not only the contents of the interview but also the user's emotional state at that time.

[1503] The server saves the summary and the results of the sentiment analysis as a calendar or interview record. The saved data is automatically reflected in the to-do list and calendar. This makes follow-up after the interview more efficient and improves the quality of the interview.

[1504] As a concrete example, a summary can be obtained by inputting the following prompt sentence into a generative AI model:

[1505] Example prompt:

[1506] "Today I would like to discuss the efficiency of our new production line. Although the current production rate is not up to the target, there are some areas for improvement. First, we need to review the maintenance schedule of our machines. Also, we need to update the training program for our workers."

[1507] By inputting this prompt into a generative AI model, a summary can be obtained, allowing the content of the interview to be efficiently understood and necessary actions to be taken quickly.

[1508] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1509] Step 1:

[1510] The server uses speech recognition technology to convert the interview audio into text, verbatim.

[1511] Input: Audio data from the interview

[1512] Output: Verbatim transcription (text data)

[1513] Specific operation: The server converts the voice data acquired from the microphone into text, which records the contents of the interview verbatim.

[1514] Step 2:

[1515] The server summarizes the transcript using a generative AI model.

[1516] Input: Verbatim transcription (text data)

[1517] Output: Summary (text data)

[1518] Specific Actions: The server summarizes the transcript, specifically extracting the key points of the transcript and summarizing them in a concise format.

[1519] Step 3:

[1520] The server analyzes the sentiment of the verbatim transcript using a sentiment engine.

[1521] Input: Verbatim transcription (text data)

[1522] Output: Sentiment analysis results (text data)

[1523] What it does: The server analyzes the emotions in the transcript, specifically recognizing emotions from the user's tone of voice and choice of words, and outputs the results.

[1524] Step 4:

[1525] The server stores the summaries and the results of sentiment analysis as calendars and interview records.

[1526] Input: Summary (text data), sentiment analysis results (text data)

[1527] Output: Calendar entries, meeting notes (data in JSON format)

[1528] How it works: The server saves the summary and the results of sentiment analysis in JSON format and automatically updates the calendar and to-do list, allowing for efficient follow-up after the interview.

[1529] Step 5:

[1530] The user reviews the interview record and takes necessary actions.

[1531] Input: Calendar entries, interview notes (data in JSON format)

[1532] Output: Action plan (text data)

[1533] Specific Actions: The user checks the meeting records and calendar entries provided by the server and plans the necessary actions, allowing specific responses based on the contents of the meeting.

[1534] Example 2

[1535] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1536] In the conventional interview record preparation method, line managers had to prepare interview records manually, which was a heavy burden. It was also difficult to properly observe non-verbal information during interviews (tone of voice, facial expressions, etc.) and reflect them in the records. Furthermore, the environment for utilizing interview records for continuous employee development was not in place.

[1537] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in the personnel management system, a means for using the generation AI to create a verbatim record of the interview and summarize it, and storing the interview record in the personnel management system, a means for collecting the tone of voice and facial expression of the user during the interview, analyzing emotions using an emotion engine, and reflecting the results in the interview record, a means for automatically reflecting the interview content in a ToDo list or a calendar, and a means for storing the results of the verbatim record, summary, and emotion analysis in the personnel management system. This reduces the burden on the line manager in creating interview records, allowing him to concentrate on observing non-verbal information during the interview, and further makes it possible to prepare an environment for continuous employee development by utilizing the interview records.

[1538] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[1539] "Interview implementation support tool" is a general term for software or hardware used to assist in the progress of interviews and the creation of records.

[1540] A "human resources management system" is an integrated software system for managing employee information, evaluating, and training.

[1541] A "verbatim transcript" is text data that records the content of an interview or conversation verbatim.

[1542] A "summary" is text data that is created by extracting important points and keywords from a verbatim transcript and summarizing them concisely.

[1543] An "emotion engine" is software or hardware that analyzes data such as voice and facial expressions and recognizes the user's emotions.

[1544] "Nonverbal information" is information that is conveyed by means other than words, and specifically includes tone of voice, facial expressions, gestures, etc.

[1545] A "ToDo list" is a tool for managing tasks and schedules in list format.

[1546] A "calendar" is a tool for managing schedules and events based on dates and times.

[1547] "Verbatim transcription" is the process of converting audio data into text and recording the verbatim content of an interview or conversation.

[1548] "Interview records" are data that record the contents of an interview, including verbatim transcripts, summaries, and the results of sentiment analysis.

[1549] "Sentiment analysis" is the process of analyzing data such as voice and facial expressions to recognize a user's emotions.

[1550] "Auto-reflection" is the process of automatically reflecting certain data or information in other systems or tools.

[1551] This invention is a system that uses an interview implementation support tool that utilizes generative AI to create verbatim records of interviews, summarize them, and analyze their emotions, and then stores the data in a personnel management system and automatically reflects them in to-do lists and calendars. A specific embodiment of this system is described below.

[1552] Hardware and Software Used

[1553] Hardware: Servers, terminals (PCs, tablets, smartphones)

[1554] Software: Generative AI models (e.g. GPT-4), emotion engines, human resource management systems

[1555] Data processing and calculation flow

[1556] 1. Collecting audio data from interviews

[1557] The user starts the interview.

[1558] The device collects audio data from interviews in real time, specifically by recording audio using the microphone on a PC, tablet, or smartphone.

[1559] The terminal transmits the collected voice data to a server.

[1560] 2. Transcription of audio data

[1561] The server sends the received voice data to the generative AI model.

[1562] A generative AI model converts the audio data into text and creates a transcript, using speech recognition technology to convert speech to text.

[1563] 3. Summary of the Transcript

[1564] The generative AI model summarizes the transcript by extracting key points and keywords and generating a concise summary.

[1565] The server receives the summarized text and proceeds to the next processing step.

[1566] 4. Sentiment analysis

[1567] The device collects the user's tone of voice and facial expressions during the interview, specifically by using a camera and microphone to record changes in facial expressions and voice.

[1568] The server sends the collected data to the emotion engine.

[1569] The emotion engine analyzes the user's emotions by recognizing their tone of voice, facial expressions, and choice of words.

[1570] The server adds the results of the sentiment analysis to the transcript.

[1571] 5. Data Retention

[1572] The server stores the transcripts, summaries, and sentiment analysis results in the personnel management system, specifically in a database.

[1573] The HR management system manages the stored data and ensures that it can be accessed when needed.

[1574] 6. Adding to your ToDo list and calendar

[1575] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar.

[1576] The HR management system automatically adds the generated tasks to to-do lists and calendars, and in particular, reflects them in employees' schedules and task management tools.

[1577] Examples

[1578] For example, when employee A has an interview with line manager B, the following specific actions are taken:

[1579] 1. The users (employee A and line manager B) start an interview, and the terminal (PC) records the audio.

[1580] 2. The device transmits the recorded audio data to the server in real time.

[1581] 3. The server sends the audio data to a generative AI model to create a transcript.

[1582] 4. A generative AI model summarizes the transcript and extracts key takeaways.

[1583] 5. The terminal collects Employee A’s facial expressions and tone of voice during the interview, and the server sends them to the emotion engine.

[1584] 6. The sentiment engine analyzes the sentiment and adds the results to the transcript.

[1585] 7. The server stores the transcripts, summaries, and sentiment analysis results in the human resources management system.

[1586] 8. The HR management system analyzes the interview content and generates tasks to be reflected in to-do lists and calendars.

[1587] Examples of prompt statements

[1588] "Please prepare and summarize a transcript of the meeting between Employee A and Line Manager B. Also, please analyze Employee A's emotions during the meeting and add the results to the transcript."

[1589] This system reduces the burden on line managers of creating interview records, allowing them to focus on observing non-verbal information during interviews. It also creates an environment for continuous employee development by utilizing interview records.

[1590] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1591] Step 1:

[1592] The user starts the interview. The device collects the voice data of the interview in real time. Specifically, the device records the voice using the microphone of a PC, tablet, or smartphone. The device sends the collected voice data to the server. The input is the voice data of the interview, and the output is the voice data sent to the server.

[1593] Step 2:

[1594] The server sends the received voice data to the generative AI model. The generative AI model converts the voice data into text and creates a transcript. Specifically, it uses speech recognition technology to convert speech to text. The input is the voice data sent to the server, and the output is the generated transcript.

[1595] Step 3:

[1596] The generative AI model summarizes the transcript by extracting key points and keywords and generating a concise summary. The server receives the summarized text and proceeds to the next processing step. The input is the generated transcript and the output is the summarized text.

[1597] Step 4:

[1598] The device collects the user's tone of voice and facial expressions during the interview. Specifically, it uses a camera and microphone to record facial expressions and changes in voice. The server sends the collected data to the emotion engine. The emotion engine analyzes the user's emotions. Specifically, it recognizes emotions from tone of voice, facial expressions, choice of words, etc. The input is the collected tone of voice and facial expression data, and the output is the result of the emotion analysis.

[1599] Step 5:

[1600] The server adds the results of the sentiment analysis to the transcript, i.e. integrates the results of the sentiment analysis into the transcript in text form. The input is the results of the sentiment analysis and the verbatim transcript, and the output is the verbatim transcript with the sentiment analysis added.

[1601] Step 6:

[1602] The server stores the verbatim transcripts, summaries, and the results of sentiment analysis in the HRMS, specifically in a database. The HRMS manages the stored data and makes it accessible when needed. The input is the verbatim transcripts with added sentiment analysis, summaries, and the results of sentiment analysis, and the output is the data stored in the HRMS.

[1603] Step 7:

[1604] The server analyzes the interview content and generates tasks to be reflected in the to-do list and calendar. The personnel management system automatically adds the generated tasks to the to-do list and calendar. Specifically, it reflects them in the employee's schedule and task management tool. The input is the analyzed interview content, and the output is the tasks reflected in the to-do list and calendar.

[1605] (Application example 2)

[1606] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1607] In conventional interview record creation systems, the creation of verbatim records of interviews, summarization, and emotion recognition were done manually, placing a heavy burden on line managers and administrators. In addition, interview records were not fully utilized to improve employees' skills or design career paths, making it difficult to create individual training plans. Furthermore, there was a problem in that training and skill improvement support for factory robot operators was not effectively provided.

[1608] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1609] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for using generative AI to create a verbatim transcript of an interview and summarize it, and store the interview record in the personnel management system, a means for automatically reflecting the results in a to-do list or a calendar, a means for recognizing emotions during an interview using an emotion engine and reflecting the information in the interview record, and a means for recording interviews and training sessions conducted by operators of factory robots and supporting the improvement of the operators' skills and the design of their career paths. This reduces the burden of creating interview records, making it possible to concentrate on observing non-verbal information during interviews, and further makes it possible to create an environment for continuous employee development by utilizing the implementation records.

[1610] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[1611] An "interview implementation support tool" is software or a system that supports the recording and analysis of interviews.

[1612] A "human resources management system" is a system that manages employee information and supports skill development and career path planning.

[1613] "Verbatim transcription" is the process of recording the verbatim contents of an interview or conversation.

[1614] "Summarizing" is the process of making a shorter summary of a longer piece of text or conversation.

[1615] A "ToDo List" is a tool that allows you to manage tasks that need to be done in list format.

[1616] A "calendar" is a tool for managing schedules and appointments.

[1617] An "emotion engine" is a system for analyzing and recognizing emotions from voice and text.

[1618] An "interview record" is a document or data that records the contents of an interview.

[1619] A "factory robot" is an automated machine used to perform tasks in a factory.

[1620] An "operator" is a person who operates a machine or system.

[1621] A "Training Session" is a training session designed to improve a particular skill or knowledge.

[1622] "Upskilling" is the process of improving a particular skill or ability.

[1623] A "career path" is the path to the occupation or position that an employee will pursue in the future.

[1624] In order to implement the present invention, the following system configuration and program are required.

[1625] System Configuration

[1626] The server will develop an interview support tool that utilizes generative AI and implement it in the personnel management system. Using generative AI, it will transcribe and summarize interviews verbatim, and store the interview records in the personnel management system. It will also have a function to automatically reflect these in to-do lists and calendars. It will use an emotion engine to recognize emotions during interviews, and reflect this information in the interview records. It will record interviews and training sessions conducted by factory robot operators, and support the operators in improving their skills and designing their career paths.

[1627] Hardware and Software Used

[1628] Hardware: Microphone, Computer

[1629] Software: Python, SpeechRecognition library, Transformers library, EmotionRecognition library

[1630] Program processing explanation

[1631] The server reads the audio file and obtains the audio data. It then converts the audio data into text. It uses a generative AI model to summarize the text and uses an emotion engine to analyze emotions from the text. Based on this information, it creates an interview record and saves it in JSON format. The interview record is stored in the HR management system and automatically reflected in the to-do list and calendar.

[1632] Examples

[1633] An operator is interviewed about the operation of a factory robot, and the audio is recorded. The recorded audio file is input to a server, which generates a verbatim transcript and summary of the interview, and also analyzes the operator's emotions. This makes it easier to improve the operator's skills and design career paths.

[1634] Examples of prompt statements

[1635] Summarize the following text:

[1636] "Today we discussed the operation of a factory robot. It seems that operator A feels unconfident in certain operations. However, he is very skilled in other operations, and is particularly strong in troubleshooting. In future training sessions, we plan to create a program to strengthen A's weak points."

[1637] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1638] Step 1:

[1639] The server reads the audio file and obtains the audio data. Specifically, an audio file recorded using a microphone is uploaded to the server, and the audio data is obtained using the SpeechRecognition library. The input is the audio file, and the output is the audio data.

[1640] Step 2:

[1641] The server converts the acquired voice data into text. Specifically, it converts the voice data into text using the SpeechRecognition library. The input is voice data, and the output is text data.

[1642] Step 3:

[1643] The server uses a generative AI model to summarize the text data. Specifically, it uses a summarization model from the Transformers library to summarize the text data. The input is the text data, and the output is the summarized text.

[1644] Step 4:

[1645] The server uses an emotion engine to analyze emotions from text data. Specifically, it uses the EmotionRecognition library to extract emotions from text data. The input is text data and the output is emotion data.

[1646] Step 5:

[1647] The server creates a meeting record based on the summarized text and emotion data. Specifically, it combines the summarized text and emotion data to generate a meeting record in JSON format. The input is the summarized text and emotion data, and the output is the meeting record.

[1648] Step 6:

[1649] The server stores the created interview record in the personnel management system. Specifically, it saves the generated interview record in JSON format in the database. The input is the interview record, and the output is the record saved in the database.

[1650] Step 7:

[1651] The server automatically updates the ToDo list and calendar based on the contents of the interview record. Specifically, tasks and schedules extracted from the interview record are added to the ToDo list and calendar. The input is the interview record, and the output is the updated ToDo list and calendar.

[1652] Step 8:

[1653] The user records interviews and training sessions conducted by factory robot operators, and supports the operator's skill development and career path planning. Specifically, the user creates individual training plans based on the interview records, and aims to improve the operator's skills. The input is the interview records, and the output is the training plans.

[1654] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1655] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1656] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1657] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1658] [Fourth embodiment]

[1659] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1660] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1661] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1662] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1663] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1664] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1665] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1666] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1667] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1668] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1669] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1670] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1671] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1672] "Example 1"

[1673] As an embodiment of the present invention, an interview implementation support tool that utilizes generative AI is developed and implemented in a personnel management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript. When the interview ends, the generative AI summarizes the verbatim transcript and stores the summary in the personnel management system. It also has a function that automatically reflects the contents of the interview in a to-do list or calendar. This reduces the burden on line managers of creating interview records, allowing them to concentrate on observing nonverbal information during the interview.

[1674] "Example 2"

[1675] In another embodiment of the present invention, the implementation records are utilized to create an environment for continuous employee development. Specifically, interview records are stored in a personnel management system, and based on that, employee skills are improved and career paths are designed. For example, the strengths and weaknesses of employees are extracted from the interview records, and based on that, individual development plans are created. In addition, the interests and inclinations of employees are understood from the interview records, and career paths are proposed based on that. This makes it possible to provide development tailored to each employee.

[1676] The process flow of each embodiment will be described below.

[1677] "Example 1"

[1678] Step 1: When the interview begins, the generative AI starts creating a transcript.

[1679] Step 2: Once the interview is over, the generative AI summarizes the transcript and stores the summary in the HR management system.

[1680] Step 3: It also has a function that automatically updates your to-do list and calendar based on the contents of the interview.

[1681] "Example 2"

[1682] Step 1: Store the interview records in a human resources management system and use them to improve employee skills and design career paths.

[1683] Step 2: Extract the employee's strengths and weaknesses from the interview records and create an individual development plan based on that.

[1684] Step 3: Understand the employee's interests and aspirations from the interview records and use that to suggest a career path.

[1685] Example 1

[1686] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1687] Traditional interview records are often created manually, placing a heavy burden on line managers and supervisors. In addition, there is no time to observe non-verbal information during the interview, which can lead to a decline in the quality of the interview. Furthermore, it is difficult to efficiently manage the interview content and reflect it in a to-do list or calendar, which can lead to insufficient follow-up after the interview.

[1688] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1689] In this invention, the server includes a means for developing an interview implementation support tool using a generative AI and implementing it in a personnel management system, a means for utilizing the generative AI to create and summarize a verbatim record of an interview and store the interview record in the personnel management system, a means for a terminal to collect voice data at the start of an interview and send it to the server, a means for the server to convert the voice data into text data using voice recognition software, a means for summarizing the verbatim record using a generative AI model, a means for storing the summarized verbatim record in a database, a means for automatically updating a to-do list or a calendar based on the interview content, and a means for the terminal to notify the user of the summarized verbatim record and update information. This reduces the burden of creating interview records, making it possible to concentrate on observing non-verbal information during interviews. In addition, the contents of the interview can be efficiently managed and automatically reflected in a to-do list or a calendar, making it easier to follow up after the interview.

[1690] "Generative AI" is a system that uses artificial intelligence technology to generate and process data.

[1691] An "interview implementation support tool" is a combination of software and / or hardware that assists in the conduct of interviews and the creation of records.

[1692] A "human resources management system" is an integrated system for managing employee information, evaluating, and training.

[1693] A verbatim transcript is a recording of an interview, meeting, etc., transcribed verbatim.

[1694] A "summary" is a concise summary of detailed information, such as a verbatim transcript.

[1695] "Speech recognition software" refers to software for converting voice data into text data.

[1696] "Generative AI model" refers to the core algorithm or trained model of generative AI.

[1697] A "database" is a system for efficiently storing, managing, and searching data.

[1698] A "ToDo list" is a list for managing tasks and schedules that need to be done in a list format.

[1699] A "calendar" is a tool for managing schedules and events based on dates and times.

[1700] A "terminal" is a device such as a computer or smartphone that is directly operated by a user.

[1701] A "server" is a computer system for processing and storing data over a network.

[1702] "Nonverbal information" refers to information that is conveyed by means other than words, such as facial expressions and gestures.

[1703] A "notification" is a message or alert sent from the system to inform the user of information.

[1704] This invention involves developing an interview support tool that utilizes generative AI and implementing it in a personnel management system. Specifically, when an interview begins, the generative AI starts creating a verbatim transcript, and when the interview ends, it summarizes the verbatim transcript and stores the summary in the personnel management system. It also has a function that automatically reflects the contents of the interview in a to-do list or calendar.

[1705] Hardware and Software Used

[1706] server

[1707] The server provides the computational resources to run the generative AI model, converts the voice data into text using speech recognition software, and summarizes the transcripts using a generative AI model (e.g., OpenAI's GPT-4). The server also stores the summarized transcripts in the HR management system's database and automatically updates to-do lists and calendars based on the interview content.

[1708] Terminal

[1709] The terminal is a device on which the user conducts the interview. When the interview begins, the terminal collects voice data using a built-in microphone and transmits it to the server in real time. When the interview ends, the terminal receives a summarized transcript and updated to-do list and calendar information from the server and notifies the user.

[1710] User

[1711] The user is the person conducting the interview. The user initiates the interview through the device and does not need to do anything during the interview. Once the interview is over, the user can view the summarized transcript and manage to-do lists and calendar updates through the device.

[1712] Examples

[1713] Examples of prompt statements

[1714] "Please summarize the interview below. Interview content: 'Hello, today I would like to discuss your work progress. First, please tell me how the project went last week.'"

[1715] Processing flow

[1716] 1. Starting the interview: The user launches the dedicated application on the device and clicks the "Start interview" button.

[1717] 2. Voice data collection: The device uses the built-in microphone to collect voice data and transmits it to the server in real time.

[1718] 3. Transcription: The server uses voice recognition software to convert the audio data into text.

[1719] 4. Summary generation: The server summarizes the transcript using a generative AI model.

[1720] 5. Data storage: The server stores the summarized transcripts in the HR management system database.

[1721] 6. ToDo list / calendar update: The server automatically updates the toDo list and calendar based on the meeting contents.

[1722] 7. User Notification: The terminal notifies the user with the summarized transcript and updates.

[1723] This system reduces the burden of creating interview records and allows you to concentrate on observing non-verbal information during the interview. It also makes it easy to follow up after an interview, as the contents of the interview can be efficiently managed and automatically reflected in a to-do list or calendar.

[1724] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1725] Step 1:

[1726] Interview begins

[1727] Subject: User

[1728] The user starts the dedicated application on the terminal and clicks the "Start interview" button. This causes the terminal to send a request to start the interview to the server. The input is the user's operation, and the output is the request to start the interview. As a specific operation, the terminal displays "Interview has started."

[1729] Step 2:

[1730] Audio data collection

[1731] Subject: Terminal

[1732] As soon as the interview begins, the terminal starts collecting voice data using the built-in microphone. The collected voice data is sent to the server in real time. The input is the user's voice, and the output is the voice data sent to the server. In concrete terms, the terminal displays "collecting voice data" and shows the collection status in real time.

[1733] Step 3:

[1734] Transcription

[1735] Subject: Server

[1736] The server receives the voice data sent from the terminal and converts it into text data using voice recognition software. The input is voice data and the output is text data. In concrete terms, the server displays "Creating a transcript" and updates the progress in real time.

[1737] Step 4:

[1738] Summary Generation

[1739] Subject: Server

[1740] The server uses a generative AI model (e.g., OpenAI's GPT-4) to summarize the verbatim transcript. The input is the text data of the verbatim transcript, and the output is the summarized text. The server displays "Generating summary" and provides real-time updates on the progress.

[1741] Step 5:

[1742] Data storage

[1743] Subject: Server

[1744] The server stores the summarized transcript in the database of the human resources management system. The input is the summarized text and the output is the record stored in the database. In concrete terms, the server displays "Saving data" and, when storage is complete, displays "Data storage completed."

[1745] Step 6:

[1746] ToDo list / Calendar update

[1747] Subject: Server

[1748] The server automatically updates the to-do list and calendar based on the contents of the meeting. The input is a summarized verbatim transcript, and the output is an updated to-do list and calendar. Specifically, the server displays "Updating to-do list and calendar" and then displays "Update completed" when the update is complete.

[1749] Step 7:

[1750] User Notifications

[1751] Subject: Terminal

[1752] The terminal receives the summarized verbatim transcript and updated ToDo list / calendar information from the server and notifies the user. The input is the notification information from the server, and the output is the notification to the user. As a specific operation, the terminal displays "Summary and updated information received" and notifies the user.

[1753] (Application example 1)

[1754] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1755] Conventional interview support tools have the problem that tasks such as creating interview records, summarizing, and updating to-do lists and calendars are often done manually, which takes time and effort. Similar problems exist in interviews between workers and robots in factories, making it difficult to provide efficient work instructions and manage schedules.

[1756] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and storing the interview record in the personnel management system, a means for automatically reflecting the results in a ToDo list or a calendar, a means for supporting interviews between workers and robots in a factory, and a means for automatically updating work instructions and schedules based on the interview contents. This reduces the burden of creating interview records, and enables efficient work instructions and schedule management.

[1757] "Generative AI" is a system that uses artificial intelligence technology to generate data and automate certain tasks.

[1758] An "interview implementation support tool" is a software or hardware system that supports the progress of interviews and the creation of records.

[1759] A "human resources management system" is a system used to manage employee information in a company or organization, and to manage labor and develop human resources.

[1760] A verbatim transcript is a document that records exactly what was said during an interview or meeting.

[1761] A "summary" is a document that extracts important points from a detailed record such as a verbatim transcript and summarizes them concisely.

[1762] A "ToDo list" is a list of tasks or work items that need to be done.

[1763] A "calendar" is a tool for managing schedules and events based on dates and times.

[1764] "Workers" are employees who actually perform work in a factory or on-site.

[1765] A "robot" is a mechanical device that performs tasks automatically according to programmed instructions.

[1766] A "work instruction" is a specific instruction or command for carrying out a specific task.

[1767] A "schedule" is a plan of tasks or events to be performed within a specific period of time.

[1768] As an embodiment of the present invention, a system for supporting interviews between workers and robots in a factory is constructed. A specific implementation method thereof is shown below.

[1769] System Configuration

[1770] 1. Hardware Configuration

[1771] Microphone: Used to collect interview audio.

[1772] Computer: Used to process the audio data and run generative AI models.

[1773] Robot: An automated machine that performs tasks in a factory.

[1774] 2. Software Configuration

[1775] OpenAI API: Used for audio transcription and summary generation.

[1776] Python: A programming language used for data processing and file manipulation.

[1777] Human Resources Management System: A system for storing interview records and summaries, and reflecting them in to-do lists and calendars.

[1778] Data processing and calculation

[1779] 1. Collecting audio data and transcribing

[1780] The server collects the interview audio through a microphone.

[1781] The collected audio data is converted into text using the OpenAI API, and a verbatim transcript is created.

[1782] 2. Summary of the Verbatim

[1783] The server summarizes the generated transcripts using the OpenAI API.

[1784] The summarized text is stored in the personnel management system.

[1785] 3. Adding to your ToDo list and calendar

[1786] The server extracts important tasks from the summarized text and adds them to a to-do list.

[1787] In addition, the schedule will be updated based on the contents of the interview and reflected on the calendar.

[1788] Examples

[1789] For example, if during an interview someone says, "We need to check the next maintenance schedule," that content is recorded in the transcript. After the interview ends, the server summarizes the transcript and generates a summary called "Check the next maintenance schedule." This summary is stored in the personnel management system, and "Check the next maintenance schedule" is automatically added to the to-do list.

[1790] Examples of prompt statements

[1791] An example prompt for summarizing a transcript is as follows:

[1792] Please summarize the following transcript:

[1793] [Transcript text]

[1794] summary:

[1795] An example prompt for retrieving to-do list items is:

[1796] Extract to-do list items from the summary below:

[1797] [Summary text]

[1798] To-do list:

[1799] In this way, it is possible to efficiently support meetings between workers and robots within the factory and automate work instructions and schedule management.

[1800] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1801] Step 1:

[1802] The server collects the interview voice through a microphone. The input is the interview voice, and the output is the voice data. Specifically, the microphone captures the voice during the interview in real time and transmits the voice data to the server.

[1803] Step 2:

[1804] The server converts the collected voice data into text using the OpenAI API and creates a transcript. The input is the voice data, and the output is the transcript text. Specifically, the voice data is input into OpenAI's speech recognition model, and the generated text is saved as a transcript.

[1805] Step 3:

[1806] The server summarizes the generated transcripts using the OpenAI API. The input is the transcript text, and the output is the summary text. Specifically, a prompt for summarizing the transcript text is input to the OpenAI text generation model, and the generated summary is obtained.

[1807] Step 4:

[1808] The server stores the summarized text in the human resources management system. The input is the summarized text, and the output is the summary data stored in the human resources management system. Specifically, the server performs an operation to store the summarized text in the database of the human resources management system.

[1809] Step 5:

[1810] The server extracts important tasks from the summarized text and adds them to the to-do list. The input is the summarized text, and the output is an updated to-do list. Specifically, the summary text is input to OpenAI's text generation model as a prompt for task extraction, and the generated tasks are added to the to-do list.

[1811] Step 6:

[1812] The server updates the schedule based on the meeting contents and reflects it in the calendar. The input is the summary text and the output is the updated calendar. Specifically, the server extracts schedule-related information from the summary text and adds that information to the calendar system.

[1813] Step 7:

[1814] The user checks the updated ToDo list and calendar and performs the necessary tasks. The input is the updated ToDo list and calendar, and the output is the user's execution of the tasks. Specifically, the user logs into the system, checks the ToDo list and calendar, and performs the tasks instructed.

[1815] Example 2

[1816] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1817] In conventional personnel management systems, the creation and storage of interview records and the creation of employee training plans were often done manually, which was time-consuming and labor-intensive. In addition, there was a lack of sufficient extraction of employees' strengths and weaknesses based on interview records, creation of individual training plans, and career path suggestions, making it difficult to provide effective training tailored to each employee.

[1818] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1819] In this invention, the server includes a means for developing an interview implementation support tool using a generation AI and implementing it in a personnel management system, a means for using the generation AI to create a verbatim transcript of the interview and summarize it, and store the interview record in the personnel management system, a means for analyzing the interview record and extracting the strengths and weaknesses of employees, a means for creating an individual training plan based on the extracted data, a means for proposing a career path based on the interests and inclinations of employees, and a means for automatically reflecting the results in a to-do list or a calendar. This automates the creation and storage of interview records, and makes it possible to efficiently extract the strengths and weaknesses of employees, create individual training plans, and propose career paths, enabling effective training tailored to each employee.

[1820] "Generative AI" is a system that uses artificial intelligence technology to generate and analyze data.

[1821] An "interview implementation support tool" is software or a system that supports the progress of interviews and the creation of records.

[1822] A "human resources management system" is software or a system for managing employee information and supporting human resources tasks such as training and evaluation.

[1823] "Verbatim transcription" is the process of recording the exact verbatim content of an interview.

[1824] "Summarizing" is the process of concisely ...

Claims

1. A first means for collecting voice data during an interview with a user; a second means for generating a prompt sentence for outputting a summary of a verbatim transcript of the user's interview based on text data obtained by converting the voice data; a third means for inputting the generated prompt sentences into a generative AI model to generate a summary of the verbatim transcript; a fourth means for notifying the user of the generated summary of the transcript; A system including:

2. A fifth means for analyzing an interview record of the interview with the user and extracting strengths and weaknesses of the user; A sixth means for creating an individual training plan for the user based on the strengths and weaknesses of the user; The system of claim 1 , comprising:

3. a seventh means for recognizing an emotional state of the user from the tone of voice, facial expression, and choice of words of the user during the interview using an emotion engine; an eighth means for storing the emotional state of the user during the interview in a database; 3. The system according to claim 1 or 2, comprising:

Images