System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving recruitment guidelines and job information; means for automatically generating interview questions based on the received recruitment guidelines and job information; means for displaying the interview questions and recording audio and video of an applicant; means for analyzing the recorded audio and video and generating an evaluation of the initial screening; and means for outputting the evaluation of the initial screening.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The modern recruitment process faces several challenges, including the following. First, arranging interviewers and interview venues takes a great deal of time and money. Second, it is difficult to maintain consistent interview quality, which can result in the risk of mismatches in recruitment. Additionally, for applicants, the tension and time constraints of interviewing large numbers of people are major issues. To solve these issues, an efficient and high-quality recruitment process is required. [Means for solving the problem]

[0005] The present invention is a system that includes a means for receiving job descriptions and job information, a means for automatically generating interview questions based on the received information, a means for displaying interview questions and recording the applicant's audio and video, a means for analyzing the recorded data to generate an initial selection evaluation, and a means for outputting the evaluation results. Specifically, the job descriptions and job information are received by a server, and an AI model generates appropriate interview questions based on the received information. Interviews are conducted by a virtual human AI, and the applicant's audio and video are recorded. The server analyzes this data and generates an initial selection evaluation based on audio analysis, facial expression analysis, etc. The evaluation results are provided to human resources personnel, making the final selection process more efficient and effective. This system eliminates the need to arrange interviewers and interview venues, ensuring uniform interview quality and reducing applicant tension and time constraints.

[0006] A "job description" is a document that describes the qualifications and skills required by a company or organization for a particular position or job.

[0007] "Job Information" is data that includes detailed descriptions, job duties, required skills, responsibilities, and other information related to a specific job title or job function.

[0008] "Interview questions" are a series of questions designed to assess an applicant's aptitude, skills, experience, and personality.

[0009] A "means" refers to a method, device, or process designed to achieve a specific purpose.

[0010] A "server" is a computer system that provides data and executes processes in response to requests from clients on a network.

[0011] A "terminal" is a type of device that a user accesses and uses to input and display data.

[0012] An "AI model" is a collection of algorithms trained to perform a specific task using artificial intelligence techniques.

[0013] A "virtual human" is a virtual being that uses computer graphics and artificial intelligence to behave like a human.

[0014] "Voice analysis" is a technology that analyzes recorded voice data and extracts information.

[0015] "Facial expression analysis" is a technology that analyzes facial expressions from video data and evaluates emotions and reactions.

[0016] "Initial selection" is the process of evaluating and selecting candidates that takes place at an early stage in the recruitment process.

[0017] "Evaluation Results" means the results of the evaluation of a Candidate obtained through the analysis and evaluation process. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] To implement the present invention, three main entities, the server, the terminal, and the user, must work together. The specific roles and operations of each entity are described below.

[0040] Enter and save the recruitment information

[0041] First, the user (human resources officer) uses a terminal to input the job requirements and job information. This information includes the job type, required skills, years of experience, etc. Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a notification to the terminal that the information has been saved.

[0042] Automatic generation of interview scenarios

[0043] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, if the job description mentions "Java experience," the model generates questions such as "Tell us about a Java project you recently worked on."

[0044] Interview schedule notification

[0045] The automatically generated interview scenario is sent from the server to the terminal. The terminal then notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time, 24 hours a day.

[0046] Virtual Human AI Interview

[0047] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0048] First selection and evaluation

[0049] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. For example, the user's (job seeker's) voice is used to evaluate the appropriateness of their responses to questions and their ability to accurately understand the other person's intentions. Meanwhile, facial expression analysis can detect emotions such as nervousness or confidence.

[0050] As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," etc. The generated evaluation results are sent to the terminal and notified to the user (human resources officer).

[0051] Final selection and hiring decision

[0052] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made on whether to hire the candidate. The results of the final selection are entered into the terminal and sent to the server. The server then automatically generates and sends a notification to the job seeker who has been selected for employment.

[0053] Specific examples

[0054] For example, if a company posts a job posting for a software engineer, the user (human resources officer) enters the requirements "Java project experience" and "more than three years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time, and an initial selection evaluation is conducted. Finally, the human resources officer conducts a final selection based on the evaluation results and the recording, and a hiring decision is made.

[0055] This system eliminates the need to arrange interviewers and interview venues, improving the quality and efficiency of interviews, reducing stress for job seekers, and allowing for flexible interview schedules.

[0056] The processing flow will be explained below.

[0057] Step 1: Enter the job description

[0058] The user enters the job description and job information into the terminal.

[0059] The terminal transmits the input data to the server.

[0060] Step 2: Save the job description

[0061] The server stores the received job posting and job information in a database.

[0062] The server sends a notification to the terminal that "recruitment requirements have been saved."

[0063] Step 3: Auto-generate interview questions

[0064] An AI model on the server generates an appropriate set of interview questions based on stored job descriptions and job information.

[0065] Step 4: Communicate the interview scenario

[0066] The server transmits the generated interview scenario to the terminal.

[0067] The terminal notifies the user (job seeker) that "You are ready for the interview."

[0068] Step 5: Start the interview

[0069] The user (job seeker) presses the interview start button on the terminal.

[0070] The device communicates with the server and activates the virtual human AI.

[0071] Step 6: Presenting the interview questions

[0072] The server controls the virtual human AI and sequentially displays pre-generated interview questions.

[0073] The terminal records the user's (job seeker's) answers.

[0074] Step 7: Recording audio and video

[0075] The terminal records the user's (job seeker's) voice and video in real time.

[0076] The recorded data is sent to the server.

[0077] Step 8: Analyze the data

[0078] The server analyzes the recorded audio and video.

[0079] Voice and facial expression analysis is performed to generate an evaluation for the first selection.

[0080] Step 9: Output of first selection results

[0081] The server sends the evaluation results of the first selection to the terminal.

[0082] The terminal notifies the user (human resources officer).

[0083] Step 10: Final selection

[0084] The user (human resources officer) checks the initial selection results and interview recording on the terminal.

[0085] The results of the final selection are entered into the terminal and sent to the server.

[0086] Step 11: Notification of hiring decision

[0087] The server will decide whether to hire the candidate based on the final selection results.

[0088] The server sends a job offer to the job seeker.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] In the traditional recruitment process, it is necessary to arrange interviewers and interview venues, which is time-consuming and costly. In addition, it is difficult for job seekers to flexibly schedule interviews, which often increases tension. There is a need to solve these problems and improve the quality and efficiency of interviews.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions using a generative AI model based on the received job requirements and job information, means for displaying the automatically generated interview questions and recording audio and video of applicants, means for analyzing the recorded audio and video data in real time to generate an evaluation score for the first selection, and means for outputting the generated evaluation score. This eliminates the need to arrange interviewers and interview venues, provides job seekers with flexible interview schedules, and makes it possible to improve the quality and efficiency of interviews.

[0094] A "recruitment requirement" is a document that lists the job description and application conditions that a company presents when recruiting new employees.

[0095] "Job Information" means information that details the skills, experience, and other requirements needed for a particular job.

[0096] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate interview questions and other information based on specific rules and patterns.

[0097] "Interview questions" are questions that companies ask job seekers to assess their skills, experience, and aptitude.

[0098] "Audio and video recording means" refers to devices and software that store audio and video recordings of job seekers as digital data.

[0099] "Means for real-time analysis" refers to technology for instantly analyzing and processing recorded data and generating an evaluation.

[0100] The "first selection evaluation score" is a numerical value obtained by evaluating the skills and aptitude of a job seeker, and is used as the result of the first selection process.

[0101] "Facial expression analysis" is a technology that reads the emotions and reactions of job applicants from their facial expressions and reflects this in their evaluation.

[0102] "Voice analysis" is a technology that analyzes the speaking style and content of a job seeker's voice, and uses this information to evaluate their skills and aptitude.

[0103] "Output means" means a device or software for displaying or reporting the generated evaluation scores or other analytical results.

[0104] To implement the invention, three main entities must work together: the server, the terminal, and the user. Below we will explain in detail how these entities work together to implement the invention.

[0105] Enter and save job postings and job information

[0106] First, the user (human resources staff) uses a terminal to input the job requirements and job information. The terminal can be a general computer device such as a PC or tablet. The user inputs information such as "Java project experience" or "3+ years of development experience." Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a notification to the terminal that the information has been saved.

[0107] Automatic generation of interview questions

[0108] The server uses a generative AI model to automatically generate interview questions based on the received job description and job information. The generative AI model uses dedicated software equipped with machine learning algorithms. For example, if the job description mentions "Java experience," the generative AI model automatically generates specific questions such as "Tell us about a Java project you recently worked on." These questions are stored in a database on the server.

[0109] Notification and preparation of interview scenarios

[0110] The automatically generated interview scenario is sent from the server to the device. Based on this, the device sends the user (job seeker) a notification that "the interview is ready." This notification is sent via email or push notification. After receiving the notification, the job seeker can access the interview system and begin the interview within 24 hours.

[0111] Virtual Human AI Interview

[0112] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, which the job seeker answers. At this time, the job seeker's answers are recorded as audio and video on the device and sent to the server in real time. The virtual human AI uses advanced artificial intelligence technology to enable natural dialogue.

[0113] Real-time analysis of responses and initial selection evaluation

[0114] The server analyzes the audio and video data it receives in real time. This analysis includes voice recognition and facial expression analysis technologies. For example, voice analysis evaluates the content of responses and speaking style, while facial expression analysis reads emotions such as nervousness or confidence. Based on the analysis results, the server generates a first-stage selection evaluation score, including "technical skills," "communication ability," and "adaptability." This evaluation score is sent to a terminal as digital data and notified to the user (human resources officer).

[0115] Final selection and hiring decision

[0116] The user (human resources staff) receives the evaluation results via the device and checks the evaluation scores from the first selection and the interview recording data. Based on this information, the final selection is made and the decision on whether to hire is entered into the device. The server then automatically generates and sends a hiring decision notice to the job seeker based on the final selection results. This notification is also sent via email or push notification.

[0117] Specific examples

[0118] For example, if a company posts a job posting for a software engineer, the user (human resources officer) enters the requirements "Java project experience" and "3+ years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The generated questions include a specific prompt, such as "Tell us about a Java project you recently worked on." The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time to evaluate the candidate for the first selection stage. Finally, the human resources officer conducts a final selection based on the evaluation results and recording, and the candidate is hired. This system eliminates the need to arrange interviewers and interview venues, improving the quality and efficiency of interviews. It also reduces stress for job seekers and allows for flexible interview schedules.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] The flow of this system's program processing

[0121] Step 1:

[0122] The user (human resources staff) uses a terminal to input the job requirements and job information. For example, they input specific items such as "Java project experience" or "3+ years of development experience." The job type, required skills, years of experience, etc. are input into the terminal as input data. The output is the entered job requirements and job information.

[0123] Step 2:

[0124] The terminal sends the job description and job information data entered by the user to the server. The secure HTTP(S) protocol is used to send the data. The input is the job description and job information entered by the user into the terminal, and the output is the data sent to the server.

[0125] Step 3:

[0126] The server saves the received data in a database. After saving is complete, the server notifies the terminal that "data saving is complete." The input is the job description and job information data received by the server, and the output is a notification that saving is complete.

[0127] Step 4:

[0128] The server uses a generative AI model to automatically generate interview questions based on the saved job description and job information. For example, it generates specific questions such as "Tell me about a Java project you recently worked on." The generative AI model analyzes the received data and generates prompts. The input is the job description and job information saved on the server, and the output is a set of interview questions.

[0129] Step 5:

[0130] The server sends automatically generated interview questions to the terminal. The input is the interview questions generated by the server, and the output is the set of questions sent to the terminal.

[0131] Step 6:

[0132] Based on the interview questions received, the device sends a notification to the user (job seeker) saying "Your interview is ready." This notification is sent via email or push notification. The input is a set of interview questions from the server, and the output is a notification to the job seeker.

[0133] Step 7:

[0134] After receiving the notification, the user (job seeker) clicks a button on their device to start the interview. The input is the notification, and the output is the start of the interview.

[0135] Step 8:

[0136] The terminal communicates with the server and starts the virtual human AI, which causes the virtual human to appear on the screen. The input is the start of the interview, and the output is the start of the virtual human AI.

[0137] Step 9:

[0138] The virtual human AI sequentially presents interview questions received from the server to the user. The user's answers are recorded as audio and video on the device and sent to the server in real time. The input is the set of questions from the server, and the output is the recorded audio and video data.

[0139] Step 10:

[0140] The server analyzes the received audio and video data in real time. This analysis uses voice recognition and facial expression analysis technologies. For example, it can detect a user's level of tension from their tone of voice and facial expression. The input is audio and video data, and the output is the analysis results.

[0141] Step 11:

[0142] Based on the analysis results, the server generates a first-stage selection evaluation score for "technical skills," "communication skills," "adaptability," etc. The input is the analysis results, and the output is the first-stage selection evaluation score.

[0143] Step 12:

[0144] The server sends the generated evaluation score to the terminal, and the terminal notifies the user (human resources officer). The input is the evaluation score, and the output is the notification.

[0145] Step 13:

[0146] The user (human resources staff) checks the evaluation scores from the first selection and the interview recording data. Based on this information, the final selection is made and the decision on whether to hire is entered into the terminal. The input is the evaluation scores and the recording data, and the output is the decision on whether to hire.

[0147] Step 14:

[0148] The server receives the final selection results from the device and automatically generates and sends a hiring decision notice to the job seeker. This notification is also sent via email or push notification. The input is the final selection results, and the output is the hiring decision notice.

[0149] (Application example 1)

[0150] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0151] Conventional interview systems and shopping assistant systems face the problem of difficulty in collecting and evaluating appropriate information from users. Interviews, in particular, require the arrangement of interviewers and interview venues, which is inefficient. Furthermore, when it comes to shopping, it is difficult to recommend products that match the preferences of individual buyers. Furthermore, there is no system that can analyze user responses in real time and change responses based on that analysis, which makes it difficult to improve the user experience.

[0152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0153] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions based on the received job requirements and job information, means for displaying interview questions and recording audio and video of applicants, means for analyzing the recorded audio and video and generating an evaluation of the initial selection, means for outputting the evaluation of the initial selection, means for receiving behavioral information of purchasers, means for automatically generating product recommendations based on the received behavioral information, means for displaying product recommendations and recording purchaser responses, means for analyzing the recorded materials and re-evaluating the product recommendations, and means for generating evaluations and updating the display in real time. This enables improved efficiency and quality of interviews, appropriate product recommendations for individual purchasers, and optimization of the user experience in real time.

[0154] "Job Description and Job Information" means detailed information about a job opening, including the skills and years of experience required for a particular position.

[0155] The "means for automatically generating interview questions" refers to a function that enables an AI model to automatically generate questions to be used in interviews based on the job description and job information.

[0156] "Means for displaying interview questions and recording audio and video of applicants" refers to a device or application that displays automatically generated interview questions to an applicant and records audio and video of the applicant's responses.

[0157] "Means for analyzing recorded audio and video and generating an evaluation for the first selection" refers to a system that analyzes recorded audio and video data of applicants and generates a numerical or textual evaluation for the first selection based on that data.

[0158] The "means for outputting the evaluation of the first selection" has the function of notifying or displaying the generated evaluation results of the first selection to the user.

[0159] "Means for receiving customer behavior information" refers to a device or application that collects customer behavior on a store or online platform (e.g., browsing history, purchase history, click behavior, etc.) and sends it to a server.

[0160] "Means for automatically generating product recommendations based on received behavioral information" refers to a system in which an AI model automatically recommends products suitable for a purchaser based on collected behavioral information of the purchaser.

[0161] "Means for displaying product recommendations and recording buyer responses" refers to a device or application that displays recommended products to a buyer and records the buyer's responses (e.g., audio, video, interaction) at that time.

[0162] The "means for analyzing recorded data and reevaluating product recommendations" is a system that analyzes buyer responses, reviews product recommendations based on the results, and makes new recommendations as necessary.

[0163] The "means for generating ratings in real time and updating the display" is a function that analyzes buyer response data in real time and instantly updates the display content based on the results.

[0164] To implement the present invention, three main entities, the server, the terminal, and the user, must work together. The specific roles and operations of each entity are described below.

[0165] Enter and save the recruitment information

[0166] The server first assists the user (human resources staff) in the process of entering job requirements and job information using a terminal. This information includes job type, required skills, years of experience, etc. Once the information has been entered, the terminal sends this information to the server, which then stores the received information in a database. The server then notifies the terminal that the information has been saved.

[0167] Automatic generation of interview scenarios

[0168] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, if the job description mentions "Java experience," the model generates questions such as "Tell us about a Java project you recently worked on."

[0169] Interview schedule notification

[0170] The automatically generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time, 24 hours a day.

[0171] Virtual Human AI Interview

[0172] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0173] First selection and evaluation

[0174] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. For example, the user's (job seeker's) voice is used to evaluate the appropriate response to questions and the ability to accurately understand the other person's intentions. Facial expression analysis can detect emotions such as nervousness and confidence. As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," and other areas. The generated evaluation results are sent to a terminal and notified to the user (human resources officer).

[0175] Product recommendations and real-time ratings

[0176] The server has a means for receiving buyer behavioral information and a means for automatically generating product recommendations based on the received behavioral information. When a buyer uses a smartphone app to input behavioral information (browsing history, purchase history, click behavior, etc.), the server analyzes it and uses an AI model to recommend products. It also displays product recommendations and records the buyer's responses (audio and video). The recorded data is analyzed to reevaluate the product recommendations, and the display is updated in real time.

[0177] Examples of specific examples and prompts

[0178] A concrete example is a scenario for a smart shopping assistant app. A user launches the smartphone app and enters their profile (preferences and albums). The shopping assistant starts recommending products and displays a message saying, "AI has prepared some recommended products for you." When the user asks the virtual assistant, "How do I use this product?", the virtual human AI provides detailed explanations, saying, "This product can also be used for commercial purposes, allowing you to cook more efficiently." The process continues by analyzing the user's facial expressions and voice, and further customizing the product.

[0179] Example prompt sentence:

[0180] 1. [User]: What are your recommended fashion items for summer?

[0181] 2. [Shopping Assistant]: Looking for recommended fashion items.

[0182] 3. [AI Model]: Recommends the following items based on the user's profile and purchase history:

[0183] 1. Summer dress

[0184] 2. Sunglasses

[0185] 3. Flip-flops

[0186] 4. [User]: How do I use a summer dress?

[0187] 5. [Virtual Human AI]: Summer dresses are versatile and can be worn for casual everyday wear or beach photoshoots.

[0188] In this way, cooperation between the server, terminal, and user allows both the interview and shopping systems to operate effectively, improving the user experience.

[0189] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0190] Step 1:

[0191] A user (human resources officer) uses a terminal to input job requirements and job information. The input includes details such as job type, required skills, years of experience, etc. The input data is sent from the terminal to the server, which stores it in a database. The input data is processed and converted into a structured format and stored appropriately in the database.

[0192] Step 2:

[0193] The server automatically generates interview questions using an AI model based on the saved job description and job information. The AI ​​model uses a generative AI model to create appropriate questions based on past data and algorithmic prompts. This question dataset is saved on the server and structured as an interview scenario. The generated questions are saved on the server as structured data.

[0194] Step 3:

[0195] The server sends the automatically generated interview scenario to the terminal. The terminal displays a notification to the user (job seeker) informing them that "the interview is ready." Once this notification is displayed, the user (job seeker) can start the interview within 24 hours.

[0196] Step 4:

[0197] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) records their answers as audio and video. The recorded data is sent to the server in real time.

[0198] Step 5:

[0199] The server analyzes the recorded audio and video in real time. The analysis involves using speech recognition APIs such as Google Cloud Speech-to-Text and Microsoft Azure Face API to perform voice and facial expression analysis. Data analysis confirms appropriate responses to questions from the voice and evaluates emotions such as nervousness or confidence from facial expressions. The analysis results are generated as a score or evaluation and stored on the server.

[0200] Step 6:

[0201] The evaluation data generated as a result of the analysis is sent from the server to the terminal and notified to the user (human resources officer). The terminal displays the evaluation results in an easy-to-read format, including the score and feedback for each evaluation item.

[0202] Step 7:

[0203] When the user (job seeker) finishes the interview, the terminal finalizes the recorded data and stores it on the server, which can be used as a reference later.

[0204] Step 8:

[0205] The user (human resources officer) will conduct the final selection based on the evaluation results and the interview recording. This selection result is also entered into the server from the terminal and saved as the final pass / fail data. The server will generate a result notification based on this data and send it to the job seeker.

[0206] Step 9:

[0207] The server operates a means for receiving purchaser behavioral information and uses a smartphone app to collect behavioral information (browsing history, purchase history, click behavior, etc.) The collected data is sent back to the server and used for analysis.

[0208] Step 10:

[0209] The server uses a generative AI model based on the received behavioral information to automatically generate product recommendations. The generated recommendations are displayed to the buyer via a smartphone app. The buyer then records their reactions to the recommended products via audio and video.

[0210] Step 11:

[0211] The server analyzes the recorded responses and re-evaluates the product recommendations. The server generates ratings and updates the recommendations in real time, resulting in more accurate recommendations. Real-time display updates improve the user experience.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] To implement this invention, three entities, the server, the terminal, and the user, work together, and an emotion engine is combined to recognize the user's emotions. The specific roles and operations of each entity, as well as how the emotion engine is integrated, are explained below.

[0214] Enter and save the recruitment information

[0215] First, the user (human resources officer) uses a terminal to input the job requirements and job information. For example, this includes items such as "job type," "required skills," and "years of experience." Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a save completion notification to the terminal.

[0216] Automatic generation of interview scenarios

[0217] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, for a requirement such as "Java experience," it generates questions such as "Tell us about a Java project you recently worked on."

[0218] Interview schedule notification

[0219] The generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." Upon receiving this notification, the user (job seeker) can start the interview at any time, 24 hours a day.

[0220] Virtual Human AI Interview

[0221] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0222] First selection and evaluation

[0223] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. The emotion engine also uses this data to analyze the user's (job seeker's) emotions. For example, voice data can identify an applicant's nervousness or confidence, while video data can identify emotions such as joy or surprise.

[0224] As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," "emotional stability," etc. The generated evaluation results are sent to the terminal and notified to the user (human resources officer).

[0225] Final selection and hiring decision

[0226] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made as to whether or not to hire the candidate. The final selection results are entered into the terminal and sent to the server. The server then decides whether or not to hire the candidate based on the final selection results and sends a notification to the job seeker.

[0227] Specific examples

[0228] For example, if a company posts a job posting for a "software engineer wanted," the user (human resources officer) would enter the requirements "Java project experience" and "more than three years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time using an emotion engine, and an initial selection evaluation is made. Finally, the human resources officer conducts a final selection based on the evaluation results and the recording, and a hiring decision is made.

[0229] This system eliminates the need to arrange interviewers and interview venues, significantly improves the quality and efficiency of interviews, and also enables more accurate understanding of applicants' true reactions and feelings through emotion analysis.

[0230] The processing flow will be explained below.

[0231] Step 1: Enter the job description

[0232] The user (human resources officer) enters the job requirements and job information into the terminal.

[0233] The terminal transmits the input data to the server.

[0234] Step 2: Save the job description

[0235] The server stores the received job posting and job information in a database.

[0236] The server sends a notification to the terminal that "recruitment requirements have been saved."

[0237] Step 3: Auto-generate interview questions

[0238] An AI model on the server generates an appropriate set of interview questions based on stored job descriptions and job information.

[0239] For example, if the job description states "Java experience," it will generate a question such as "Tell us about a Java project you've recently worked on."

[0240] Step 4: Communicate the interview scenario

[0241] The server transmits the generated interview scenario to the terminal.

[0242] The terminal notifies the user (job seeker) that "You are ready for the interview."

[0243] Step 5: Start the interview

[0244] The user (job seeker) presses the interview start button on the terminal.

[0245] The device communicates with the server and activates the virtual human AI.

[0246] Step 6: Presenting the interview questions

[0247] The server controls the virtual human AI and sequentially displays pre-generated interview questions.

[0248] The terminal records the user's (job seeker's) answers.

[0249] Step 7: Recording audio and video

[0250] The terminal records the user's (job seeker's) voice and video in real time.

[0251] The recorded data is sent to the server.

[0252] Step 8: Analyze the data

[0253] The server analyzes the recorded audio and video.

[0254] Voice analysis, facial expression analysis, and emotion recognition using an emotion engine are performed to generate an evaluation for the first selection round.

[0255] For example, the emotion engine analyzes the user's (job seeker's) nervousness and confidence from voice, and facial expression data such as smiles and surprise from video.

[0256] Step 9: Output of first selection results

[0257] The server generates the evaluation results of the first selection and sends them to the terminal.

[0258] The terminal displays the "first selection results" to the user (human resources officer).

[0259] Step 10: Final selection

[0260] The user (human resources officer) checks the initial selection results and interview recording on the terminal.

[0261] Human resources personnel will conduct the final selection based on the interview recording and evaluation results, and enter the final selection results into the terminal.

[0262] Step 11: Notification of hiring decision

[0263] The terminal transmits the final selection results to the server.

[0264] The server determines whether or not to hire the job seeker based on the final selection results and sends a notice of employment to the job seeker.

[0265] Through these steps, the server, terminal, user, and emotion engine work together to realize an efficient and sophisticated interview process. This system not only improves the quality and efficiency of interviews, but also enables a deeper understanding of the applicant's true suitability through emotion analysis.

[0266] Example 2

[0267] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0268] Traditional hiring processes require arranging interviewers and interview locations, which is time-consuming and costly. Furthermore, interviewers' subjective assessments of applicants can be inconsistent. Furthermore, it is difficult to accurately grasp applicants' emotions and true reactions. Therefore, there is a need for a system that can improve the efficiency and accuracy of the hiring process.

[0269] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0270] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions based on the received job requirements and job information, means for displaying the interview questions and recording the applicant's voice and video, means for analyzing the recorded voice and video and generating an evaluation of the first selection, means for analyzing the applicant's emotions based on the analyzed content, and means for outputting the evaluation of the first selection. This eliminates the need to arrange interviewers and interview venues, improves the quality and efficiency of interviews, and makes it possible to more accurately grasp the applicant's true reactions and emotions.

[0271] "Recruitment requirements" are information about the job, specifically including the job type, required skills, years of experience, etc.

[0272] "Job information" is detailed information about a specific job, specifically including job content, roles, scope of responsibilities, and the like.

[0273] The "receiving means" refers to the technical means by which the server receives information input by the user, and includes a communication interface, a data receiving module, and the like.

[0274] "Automatic generation means" refers to means for automatically creating new content based on specified information using technologies such as AI models.

[0275] The "display means" is a device such as a display or monitor for visually presenting the generated questions and information to the user.

[0276] "Recording means" refers to technical means for saving data such as audio and video, and includes, for example, cameras, microphones, and recording devices.

[0277] "Means for analyzing" refers to technical means for analyzing data such as audio and video to derive specific identification information or scores, and includes voice analysis algorithms and facial expression analysis algorithms.

[0278] "Means for analyzing emotions" refers to technical means for identifying an applicant's emotions from audio or video, including emotion engines and machine learning models.

[0279] The "means for generating an evaluation" is a means for generating a score or evaluation based on specific criteria based on the analysis results.

[0280] "Means for outputting evaluation" refers to technical means for communicating the generated evaluation results to the user, including, for example, a screen display or a notification system.

[0281] To implement this invention, three entities - the server, the terminal, and the user - must work together, and an emotion engine must be combined to recognize the user's emotions. The specific roles and operations of each entity, as well as how the emotion engine is integrated, are explained below.

[0282] Enter and save the recruitment information

[0283] First, the user (human resources officer) uses a terminal to input the job requirements and job information. The input fields include "job type," "required skills," and "years of experience." Once the user has completed their input, the terminal sends this information to the server. The server saves the received information in a database, and once the saving is complete, it sends a save completion notification to the terminal. This ensures that the information is stored securely.

[0284] Automatic generation of interview scenarios

[0285] The server passes the saved job description and job information to the AI ​​model. A generative AI model, such as TensorFlow or PyTorch, is used to automatically generate interview questions appropriate to the requirements. For example, for the requirement "Java experience," a question such as "Tell us about a Java project you recently worked on" is generated. The server then stores the generated questions in a database.

[0286] Interview schedule notification

[0287] The generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time within 24 hours.

[0288] Virtual Human AI Interview

[0289] When a user (job seeker) starts an interview on their device, the device sends a request to start the interview to the server. The server then starts a virtual human AI and displays interview questions in sequence. The user (job seeker) responds via voice and video, and the data is recorded on the device and sent to the server in real time. Core software used for recording the audio and video includes OpenCV and FFmpeg.

[0290] First selection and evaluation

[0291] The server analyzes the received audio and video data. This analysis includes voice and facial expression analysis, using libraries such as Librosa and Dlib. An emotion engine also uses this data to analyze the user's (job seeker's) emotions. For example, the voice data can identify an applicant's nervousness or confidence, while the video data can identify emotions such as joy or surprise. As a result of the analysis, the server generates scores for such criteria as "technical skills," "communication ability," "adaptability," and "emotional stability." The generated evaluation results are sent to the terminal and notified to the user (human resources staff).

[0292] Final selection and hiring decision

[0293] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made on whether to hire the candidate. The final selection results are entered into the terminal and sent to the server. The server then sends a notification to the job seeker based on the final selection results.

[0294] Specific examples

[0295] For example, if a company posts a job posting for a software engineer, the user (a human resources officer) inputs the following requirements into the terminal: "Java project experience" and "3 or more years of development experience." The server receives this information and generates a set of interview questions.

[0296] Examples of specific prompts include:

[0297] "Generate interview questions suitable for candidates with at least 3 years of Java project experience. For example, questions about recent projects they worked on."

[0298] Interviews are conducted based on this information. Job seekers use a 24-hour interview system to interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time using an emotion engine, and an initial evaluation is conducted. Finally, a human resources staff member conducts the final selection based on the evaluation results and recording, and the candidate is hired. This system eliminates the need to arrange interviewers and interview venues, significantly improving the quality and efficiency of interviews, and also makes it possible to more accurately grasp the applicant's true reactions and emotions through emotion analysis.

[0299] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0300] Step 1:

[0301] The user (human resources staff) uses a terminal to enter job requirements and job information. Specifically, they enter information such as "job type," "required skills," and "years of experience" into the terminal's input form. This information is sent to the server through the terminal's user interface. Based on the input data, the terminal sends JSON format data to the server, which then stores it in a database on the server side.

[0302] Input: Job posting and job information entered on the terminal

[0303] Output: Information sent to the server and stored in the database

[0304] Step 2:

[0305] The server passes the received job description and job information to the AI ​​model. During this process, the server sends the information to the generative AI model as prompts. For example, for a requirement such as "Java project experience," a question such as "Tell us about a Java project you recently worked on" is generated. The AI ​​model generates questions using the TensorFlow and PyTorch libraries.

[0306] Input: Job postings and job information stored in the database

[0307] Output: Generated interview questions

[0308] Step 3:

[0309] The server saves the generated interview questions in a database. Once saving is complete, the server sends the interview questions to the terminal, and the terminal notifies the user (job seeker) that "the interview is ready."

[0310] Input: Generated interview questions

[0311] Output: Notification of interview settings to the terminal

[0312] Step 4:

[0313] When a user (job seeker) starts an interview on their device, the device sends a request to start the interview to the server. The server launches a virtual human AI and connects it to the device. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers using audio and video. Specifically, the device uses a camera and microphone to record audio and video and transmits them to the server in real time. OpenCV and FFmpeg libraries are used for this.

[0314] Input: Interview start request from the terminal and applicant's audio and video data

[0315] Output: Real-time video and audio data transmission to the server

[0316] Step 5:

[0317] The server analyzes the received audio and video data. Using the Librosa and Dlib libraries, voice and facial analysis is performed. An emotion engine also analyzes the applicant's emotions, such as nervousness and confidence. The analysis generates scores for factors such as technical skills, communication skills, adaptability, and emotional stability.

[0318] Input: Real-time video and audio data of applicant

[0319] Output: Evaluation scores for technical skills, communication skills, etc.

[0320] Step 6:

[0321] The server stores the generated evaluation results in a database and sends them to the terminal, which displays the evaluation results of the first selection and the interview recording so that the user (human resources staff) can check them.

[0322] Input: Rating score

[0323] Output: Evaluation results of the first selection and recorded data displayed on the terminal

[0324] Step 7:

[0325] The user (human resources officer) checks the evaluation results of the first selection on the terminal and conducts the final selection. The final selection results are entered into the terminal and sent to the server. Based on the final selection results, the server notifies the job seeker whether they have been hired or not.

[0326] Input: Final selection results by user (HR person)

[0327] Output: Notification of job offer to job seeker

[0328] (Application example 2)

[0329] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] Existing interview systems require preparation for interviewers, arrangement of interview venues, and adjustments to maintain the quality and efficiency of interviews, which leads to inefficient use of resources. Existing systems also make it difficult to accurately grasp applicants' emotions and reactions during interviews, making objective evaluations difficult. Furthermore, there is no system in place that can monitor and analyze the work status of robots in factories and the emotional health of workers in real time and provide immediate feedback, making quality control and work efficiency improvements a challenge.

[0331] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the job description and job information, means for automatically generating interview questions based on the received job description and job information, means for displaying the interview questions and recording the applicant's voice and video, means for analyzing the recorded voice and video in real time using an emotion analysis engine and generating an evaluation, means for outputting the initial selection evaluation, and means for issuing a warning if a certain evaluation standard is not met based on the analysis results. This enables more efficient interview preparation, accurate understanding of the applicant's emotions and reactions, and real-time monitoring of quality control and the emotional health of workers in factories.

[0332] A "job description" is a document that contains information about a job, such as job duties, required skills, and years of experience.

[0333] "Job information" is data that includes detailed information related to a particular job, such as job responsibilities, required qualifications, and location.

[0334] "Interview questions" are questions asked of applicants to assess their aptitude and skills.

[0335] "Audio and video" refers to digital data used to record the voices, facial expressions, and movements of applicants and workers.

[0336] An "emotion analysis engine" is an algorithm and software that analyzes data such as audio and video to evaluate and determine the emotional state of a subject.

[0337] "First selection" refers to the initial selection process, which is primarily a procedure for evaluating the applicant's basic aptitude and skills.

[0338] "Metrics" are standards or guidelines established for evaluating data or actions.

[0339] "Means for issuing warnings" refers to the ability to send alerts or notifications when evaluation results do not meet set standards.

[0340] To implement this invention, three entities - a server, a terminal, and a user - work together, and an emotion analysis engine is integrated to determine the user's emotion. The specific roles and operations of each entity, as well as how the emotion analysis engine is integrated, are explained below.

[0341] Overall structure

[0342] Hardware and Software Configuration

[0343] Hardware:

[0344] Device: User (HR personnel and job seeker) operation device (PC or tablet)

[0345] Server: Server for data analysis and automatic generation of interview questions

[0346] Factory robots: robots with built-in cameras and microphones

[0347] software:

[0348] Server side: Sentiment analysis AI using Python / TensorFlow / Keras

[0349] Frontend: User interface with React.js

[0350] Database: MySQL or PostgreSQL

[0351] Communication: Real-time communication via WebSocket

[0352] System Overview

[0353] 1. Enter and save the recruitment information

[0354] First, the user (human resources staff) uses a terminal to enter the job requirements and job information. The entered information is sent to the server and stored in the database. This procedure allows the user to easily register the necessary requirements.

[0355] 2. Automatic generation of interview scenarios

[0356] After the server receives and stores the job description and job information, the generative AI model automatically generates interview questions based on this information. For example, a request for "Java experience" generates a question such as "Tell us about a Java project you recently worked on."

[0357] 3. Notification of interview schedule

[0358] The generated interview scenario is sent from the server to the terminal, and the user (job seeker) is notified that "the interview is ready." This allows job seekers to start an interview at any time, 24 hours a day.

[0359] 4. Virtual Human AI Interviews

[0360] When a user (job seeker) starts an interview on their device, it communicates with the server and activates the virtual human AI, which sequentially displays pre-generated interview questions, and the recorded audio and video are sent to the server in real time.

[0361] 5. First selection and evaluation

[0362] The server analyzes the recorded audio and video. This analysis uses emotion analysis AI with TensorFlow and Keras. For example, it can recognize an applicant's nervousness or confidence from audio data, and emotions such as joy or surprise from video data. This generates scores for such things as "technical skills," "communication ability," "adaptability," and "emotional stability."

[0363] 6. Notification of evaluation results and feedback

[0364] The generated evaluation results are sent to the terminal and notified to the user (human resources personnel). Furthermore, if the analysis results do not meet certain evaluation standards, a warning is issued. This feedback loop improves the quality and efficiency of interviews.

[0365] Examples of specific examples and prompts

[0366] For example, when a worker is performing "assembly work" in a factory, a factory robot monitors the worker's movements in real time and sends the data collected by a camera and microphone to an emotion analysis engine. Based on the analysis results, if the worker is feeling excessively stressed, a warning will be issued and appropriate feedback will be provided.

[0367] Prompt Sentence Examples

[0368] plain

[0369] "Analyze video and audio data of workers as they assemble parts, and analyze emotions such as stress and satisfaction in real time."

[0370] This will enable quality control in factories and improve the emotional well-being of workers.

[0371] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0372] Step 1: Enter and save job description and job information

[0373] The user (human resources officer) uses a terminal to input the job requirements and job information. The input information is sent from the terminal to the server and saved in a database. The input data includes the job type, required skills, years of experience, etc. The server then sends a notification to the terminal that the data has been saved to the database.

[0374] Input: Job requirements and job information

[0375] Output: Information saved in the database, notification of save completion

[0376] Step 2: Auto-generate interview questions

[0377] The server uses a generative AI model to automatically generate interview questions based on the saved job description and job information. For example, if Java experience is required, the generative model generates questions such as, "Tell me about a Java project you recently worked on."

[0378] Input: Saved job postings and job information

[0379] Output: Auto-generated interview questions

[0380] Step 3: Notification of interview arrangement

[0381] The automatically generated interview questions are sent from the server to the terminal, and the terminal notifies the user (job seeker) that "the interview is ready." This notification allows the job seeker to prepare for the interview and start the interview at any time, 24 hours a day.

[0382] Input: Auto-generated interview questions

[0383] Output: Interview preparation notice

[0384] Step 4: Virtual Human AI Interview

[0385] When a user (job seeker) starts an interview through their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays the generated interview questions and records the applicant's voice and video. The audio and video data is sent to the server in real time.

[0386] Input: Interview questions, audio and video of job seeker

[0387] Output: Recorded audio and video data

[0388] Step 5: First selection and evaluation

[0389] The server analyzes the received audio and video data in real time. It uses an emotion analysis engine based on TensorFlow and Keras to recognize the applicant's nervousness or confidence from the audio data, and emotions such as joy or surprise from the video data. This generates an evaluation score based on factors such as technical skills, communication ability, adaptability, and emotional stability.

[0390] Input: Recorded audio and video data

[0391] Output: Evaluation score

[0392] Step 6: Notification of evaluation results and feedback

[0393] The server sends the generated evaluation score to the terminal and notifies the user (human resources officer). Furthermore, if the analysis results indicate that a certain evaluation standard has not been met, a warning is issued. At this step, feedback based on the evaluation results is provided, improving the quality and efficiency of the interview.

[0394] Input: Rating score

[0395] Output: Evaluation result notification, warning notification

[0396] Step 7: Emotion analysis and quality control for factory robots

[0397] In factories, robots use cameras and microphones to monitor the movements and emotions of workers in real time, and send the collected data to a server, which uses an emotion analysis engine to analyze the worker's stress, satisfaction, and other factors, and issue warnings as necessary.

[0398] Input: Video and audio data of the worker

[0399] Output: Sentiment analysis data, quality control assessment, warning notifications

[0400] The above processing steps enable efficient and objective evaluation, which is expected to have a significant effect on personnel and factory quality control.

[0401] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0402] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0403] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0404] [Second embodiment]

[0405] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0406] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0407] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0408] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0409] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0410] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0411] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0412] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0413] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0414] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0415] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0416] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0417] To implement the present invention, three main entities, the server, the terminal, and the user, must work together. The specific roles and operations of each entity are described below.

[0418] Enter and save the recruitment information

[0419] First, the user (human resources officer) uses a terminal to input the job requirements and job information. This information includes the job type, required skills, years of experience, etc. Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a notification to the terminal that the information has been saved.

[0420] Automatic generation of interview scenarios

[0421] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, if the job description mentions "Java experience," the model generates questions such as "Tell us about a Java project you recently worked on."

[0422] Interview schedule notification

[0423] The automatically generated interview scenario is sent from the server to the terminal. The terminal then notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time, 24 hours a day.

[0424] Virtual Human AI Interview

[0425] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0426] First selection and evaluation

[0427] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. For example, the user's (job seeker's) voice is used to evaluate the appropriateness of their responses to questions and their ability to accurately understand the other person's intentions. Meanwhile, facial expression analysis can detect emotions such as nervousness or confidence.

[0428] As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," etc. The generated evaluation results are sent to the terminal and notified to the user (human resources officer).

[0429] Final selection and hiring decision

[0430] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made on whether to hire the candidate. The results of the final selection are entered into the terminal and sent to the server. The server then automatically generates and sends a notification to the job seeker who has been selected for employment.

[0431] Specific examples

[0432] For example, if a company posts a job posting for a software engineer, the user (human resources officer) enters the requirements "Java project experience" and "more than three years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time, and an initial selection evaluation is conducted. Finally, the human resources officer conducts a final selection based on the evaluation results and the recording, and a hiring decision is made.

[0433] This system eliminates the need to arrange interviewers and interview venues, improving the quality and efficiency of interviews, reducing stress for job seekers, and allowing for flexible interview schedules.

[0434] The processing flow will be explained below.

[0435] Step 1: Enter the job description

[0436] The user enters the job description and job information into the terminal.

[0437] The terminal transmits the input data to the server.

[0438] Step 2: Save the job description

[0439] The server stores the received job posting and job information in a database.

[0440] The server sends a notification to the terminal that "recruitment requirements have been saved."

[0441] Step 3: Auto-generate interview questions

[0442] An AI model on the server generates an appropriate set of interview questions based on stored job descriptions and job information.

[0443] Step 4: Communicate the interview scenario

[0444] The server transmits the generated interview scenario to the terminal.

[0445] The terminal notifies the user (job seeker) that "You are ready for the interview."

[0446] Step 5: Start the interview

[0447] The user (job seeker) presses the interview start button on the terminal.

[0448] The device communicates with the server and activates the virtual human AI.

[0449] Step 6: Presenting the interview questions

[0450] The server controls the virtual human AI and sequentially displays pre-generated interview questions.

[0451] The terminal records the user's (job seeker's) answers.

[0452] Step 7: Recording audio and video

[0453] The terminal records the user's (job seeker's) voice and video in real time.

[0454] The recorded data is sent to the server.

[0455] Step 8: Analyze the data

[0456] The server analyzes the recorded audio and video.

[0457] Voice and facial expression analysis is performed to generate an evaluation for the first selection.

[0458] Step 9: Output of first selection results

[0459] The server sends the evaluation results of the first selection to the terminal.

[0460] The terminal notifies the user (human resources officer).

[0461] Step 10: Final selection

[0462] The user (human resources officer) checks the initial selection results and interview recording on the terminal.

[0463] The results of the final selection are entered into the terminal and sent to the server.

[0464] Step 11: Notification of hiring decision

[0465] The server will decide whether to hire the candidate based on the final selection results.

[0466] The server sends a job offer to the job seeker.

[0467] Example 1

[0468] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0469] In the traditional recruitment process, it is necessary to arrange interviewers and interview venues, which is time-consuming and costly. In addition, it is difficult for job seekers to flexibly schedule interviews, which often increases tension. There is a need to solve these problems and improve the quality and efficiency of interviews.

[0470] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0471] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions using a generative AI model based on the received job requirements and job information, means for displaying the automatically generated interview questions and recording audio and video of applicants, means for analyzing the recorded audio and video data in real time to generate an evaluation score for the first selection, and means for outputting the generated evaluation score. This eliminates the need to arrange interviewers and interview venues, provides job seekers with flexible interview schedules, and makes it possible to improve the quality and efficiency of interviews.

[0472] A "recruitment requirement" is a document that lists the job description and application conditions that a company presents when recruiting new employees.

[0473] "Job Information" means information that details the skills, experience, and other requirements needed for a particular job.

[0474] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate interview questions and other information based on specific rules and patterns.

[0475] "Interview questions" are questions that companies ask job seekers to assess their skills, experience, and aptitude.

[0476] "Audio and video recording means" refers to devices and software that store audio and video recordings of job seekers as digital data.

[0477] "Means for real-time analysis" refers to technology for instantly analyzing and processing recorded data and generating an evaluation.

[0478] The "first selection evaluation score" is a numerical value obtained by evaluating the skills and aptitude of a job seeker, and is used as the result of the first selection process.

[0479] "Facial expression analysis" is a technology that reads the emotions and reactions of job applicants from their facial expressions and reflects this in their evaluation.

[0480] "Voice analysis" is a technology that analyzes the speaking style and content of a job seeker's voice, and uses this information to evaluate their skills and aptitude.

[0481] "Output means" means a device or software for displaying or reporting the generated evaluation scores or other analytical results.

[0482] To implement the invention, three main entities must work together: the server, the terminal, and the user. Below we will explain in detail how these entities work together to implement the invention.

[0483] Enter and save job postings and job information

[0484] First, the user (human resources staff) uses a terminal to input the job requirements and job information. The terminal can be a general computer device such as a PC or tablet. The user inputs information such as "Java project experience" or "3+ years of development experience." Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a notification to the terminal that the information has been saved.

[0485] Automatic generation of interview questions

[0486] The server uses a generative AI model to automatically generate interview questions based on the received job description and job information. The generative AI model uses dedicated software equipped with machine learning algorithms. For example, if the job description mentions "Java experience," the generative AI model automatically generates specific questions such as "Tell us about a Java project you recently worked on." These questions are stored in a database on the server.

[0487] Notification and preparation of interview scenarios

[0488] The automatically generated interview scenario is sent from the server to the device. Based on this, the device sends the user (job seeker) a notification that "the interview is ready." This notification is sent via email or push notification. After receiving the notification, the job seeker can access the interview system and begin the interview within 24 hours.

[0489] Virtual Human AI Interview

[0490] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, which the job seeker answers. At this time, the job seeker's answers are recorded as audio and video on the device and sent to the server in real time. The virtual human AI uses advanced artificial intelligence technology to enable natural dialogue.

[0491] Real-time analysis of responses and initial selection evaluation

[0492] The server analyzes the audio and video data it receives in real time. This analysis includes voice recognition and facial expression analysis technologies. For example, voice analysis evaluates the content of responses and speaking style, while facial expression analysis reads emotions such as nervousness or confidence. Based on the analysis results, the server generates a first-stage selection evaluation score, including "technical skills," "communication ability," and "adaptability." This evaluation score is sent to a terminal as digital data and notified to the user (human resources officer).

[0493] Final selection and hiring decision

[0494] The user (human resources staff) receives the evaluation results via the device and checks the evaluation scores from the first selection and the interview recording data. Based on this information, the final selection is made and the decision on whether to hire is entered into the device. The server then automatically generates and sends a hiring decision notice to the job seeker based on the final selection results. This notification is also sent via email or push notification.

[0495] Specific examples

[0496] For example, if a company posts a job posting for a software engineer, the user (human resources officer) enters the requirements "Java project experience" and "3+ years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The generated questions include a specific prompt, such as "Tell us about a Java project you recently worked on." The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time to evaluate the candidate for the first selection stage. Finally, the human resources officer conducts a final selection based on the evaluation results and recording, and the candidate is hired. This system eliminates the need to arrange interviewers and interview venues, improving the quality and efficiency of interviews. It also reduces stress for job seekers and allows for flexible interview schedules.

[0497] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0498] The flow of this system's program processing

[0499] Step 1:

[0500] The user (human resources staff) uses a terminal to input the job requirements and job information. For example, they input specific items such as "Java project experience" or "3+ years of development experience." The job type, required skills, years of experience, etc. are input into the terminal as input data. The output is the entered job requirements and job information.

[0501] Step 2:

[0502] The terminal sends the job description and job information data entered by the user to the server. The secure HTTP(S) protocol is used to send the data. The input is the job description and job information entered by the user into the terminal, and the output is the data sent to the server.

[0503] Step 3:

[0504] The server saves the received data in a database. After saving is complete, the server notifies the terminal that "data saving is complete." The input is the job description and job information data received by the server, and the output is a notification that saving is complete.

[0505] Step 4:

[0506] The server uses a generative AI model to automatically generate interview questions based on the saved job description and job information. For example, it generates specific questions such as "Tell me about a Java project you recently worked on." The generative AI model analyzes the received data and generates prompts. The input is the job description and job information saved on the server, and the output is a set of interview questions.

[0507] Step 5:

[0508] The server sends automatically generated interview questions to the terminal. The input is the interview questions generated by the server, and the output is the set of questions sent to the terminal.

[0509] Step 6:

[0510] Based on the interview questions received, the device sends a notification to the user (job seeker) saying "Your interview is ready." This notification is sent via email or push notification. The input is a set of interview questions from the server, and the output is a notification to the job seeker.

[0511] Step 7:

[0512] After receiving the notification, the user (job seeker) clicks a button on their device to start the interview. The input is the notification, and the output is the start of the interview.

[0513] Step 8:

[0514] The terminal communicates with the server and starts the virtual human AI, which causes the virtual human to appear on the screen. The input is the start of the interview, and the output is the start of the virtual human AI.

[0515] Step 9:

[0516] The virtual human AI sequentially presents interview questions received from the server to the user. The user's answers are recorded as audio and video on the device and sent to the server in real time. The input is the set of questions from the server, and the output is the recorded audio and video data.

[0517] Step 10:

[0518] The server analyzes the received audio and video data in real time. This analysis uses voice recognition and facial expression analysis technologies. For example, it can detect a user's level of tension from their tone of voice and facial expression. The input is audio and video data, and the output is the analysis results.

[0519] Step 11:

[0520] Based on the analysis results, the server generates a first-stage selection evaluation score for "technical skills," "communication skills," "adaptability," etc. The input is the analysis results, and the output is the first-stage selection evaluation score.

[0521] Step 12:

[0522] The server sends the generated evaluation score to the terminal, and the terminal notifies the user (human resources officer). The input is the evaluation score, and the output is the notification.

[0523] Step 13:

[0524] The user (human resources staff) checks the evaluation scores from the first selection and the interview recording data. Based on this information, the final selection is made and the decision on whether to hire is entered into the terminal. The input is the evaluation scores and the recording data, and the output is the decision on whether to hire.

[0525] Step 14:

[0526] The server receives the final selection results from the device and automatically generates and sends a hiring decision notice to the job seeker. This notification is also sent via email or push notification. The input is the final selection results, and the output is the hiring decision notice.

[0527] (Application example 1)

[0528] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0529] Conventional interview systems and shopping assistant systems face the problem of difficulty in collecting and evaluating appropriate information from users. Interviews, in particular, require the arrangement of interviewers and interview venues, which is inefficient. Furthermore, when it comes to shopping, it is difficult to recommend products that match the preferences of individual buyers. Furthermore, there is no system that can analyze user responses in real time and change responses based on that analysis, which makes it difficult to improve the user experience.

[0530] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0531] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions based on the received job requirements and job information, means for displaying interview questions and recording audio and video of applicants, means for analyzing the recorded audio and video and generating an evaluation of the initial selection, means for outputting the evaluation of the initial selection, means for receiving behavioral information of purchasers, means for automatically generating product recommendations based on the received behavioral information, means for displaying product recommendations and recording purchaser responses, means for analyzing the recorded materials and re-evaluating the product recommendations, and means for generating evaluations and updating the display in real time. This enables improved efficiency and quality of interviews, appropriate product recommendations for individual purchasers, and optimization of the user experience in real time.

[0532] "Job Description and Job Information" means detailed information about a job opening, including the skills and years of experience required for a particular position.

[0533] The "means for automatically generating interview questions" refers to a function that enables an AI model to automatically generate questions to be used in interviews based on the job description and job information.

[0534] "Means for displaying interview questions and recording audio and video of applicants" refers to a device or application that displays automatically generated interview questions to an applicant and records audio and video of the applicant's responses.

[0535] "Means for analyzing recorded audio and video and generating an evaluation for the first selection" refers to a system that analyzes recorded audio and video data of applicants and generates a numerical or textual evaluation for the first selection based on that data.

[0536] The "means for outputting the evaluation of the first selection" has the function of notifying or displaying the generated evaluation results of the first selection to the user.

[0537] "Means for receiving customer behavior information" refers to a device or application that collects customer behavior on a store or online platform (e.g., browsing history, purchase history, click behavior, etc.) and sends it to a server.

[0538] "Means for automatically generating product recommendations based on received behavioral information" refers to a system in which an AI model automatically recommends products suitable for a purchaser based on collected behavioral information of the purchaser.

[0539] "Means for displaying product recommendations and recording buyer responses" refers to a device or application that displays recommended products to a buyer and records the buyer's responses (e.g., audio, video, interaction) at that time.

[0540] The "means for analyzing recorded data and reevaluating product recommendations" is a system that analyzes buyer responses, reviews product recommendations based on the results, and makes new recommendations as necessary.

[0541] The "means for generating ratings in real time and updating the display" is a function that analyzes buyer response data in real time and instantly updates the display content based on the results.

[0542] To implement the present invention, three main entities, the server, the terminal, and the user, must work together. The specific roles and operations of each entity are described below.

[0543] Enter and save the recruitment information

[0544] The server first assists the user (human resources staff) in the process of entering job requirements and job information using a terminal. This information includes job type, required skills, years of experience, etc. Once the information has been entered, the terminal sends this information to the server, which then stores the received information in a database. The server then notifies the terminal that the information has been saved.

[0545] Automatic generation of interview scenarios

[0546] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, if the job description mentions "Java experience," the model generates questions such as "Tell us about a Java project you recently worked on."

[0547] Interview schedule notification

[0548] The automatically generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time, 24 hours a day.

[0549] Virtual Human AI Interview

[0550] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0551] First selection and evaluation

[0552] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. For example, the user's (job seeker's) voice is used to evaluate the appropriate response to questions and the ability to accurately understand the other person's intentions. Facial expression analysis can detect emotions such as nervousness and confidence. As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," and other areas. The generated evaluation results are sent to a terminal and notified to the user (human resources officer).

[0553] Product recommendations and real-time ratings

[0554] The server has a means for receiving buyer behavioral information and a means for automatically generating product recommendations based on the received behavioral information. When a buyer uses a smartphone app to input behavioral information (browsing history, purchase history, click behavior, etc.), the server analyzes it and uses an AI model to recommend products. It also displays product recommendations and records the buyer's responses (audio and video). The recorded data is analyzed to reevaluate the product recommendations, and the display is updated in real time.

[0555] Examples of specific examples and prompts

[0556] A concrete example is a scenario for a smart shopping assistant app. A user launches the smartphone app and enters their profile (preferences and albums). The shopping assistant starts recommending products and displays a message saying, "AI has prepared some recommended products for you." When the user asks the virtual assistant, "How do I use this product?", the virtual human AI provides detailed explanations, saying, "This product can also be used for commercial purposes, allowing you to cook more efficiently." The process continues by analyzing the user's facial expressions and voice, and further customizing the product.

[0557] Example prompt sentence:

[0558] 1. [User]: What are your recommended fashion items for summer?

[0559] 2. [Shopping Assistant]: Looking for recommended fashion items.

[0560] 3. [AI Model]: Recommends the following items based on the user's profile and purchase history:

[0561] 1. Summer dress

[0562] 2. Sunglasses

[0563] 3. Flip-flops

[0564] 4. [User]: How do I use a summer dress?

[0565] 5. [Virtual Human AI]: Summer dresses are versatile and can be worn for casual everyday wear or beach photoshoots.

[0566] In this way, cooperation between the server, terminal, and user allows both the interview and shopping systems to operate effectively, improving the user experience.

[0567] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0568] Step 1:

[0569] A user (human resources officer) uses a terminal to input job requirements and job information. The input includes details such as job type, required skills, years of experience, etc. The input data is sent from the terminal to the server, which stores it in a database. The input data is processed and converted into a structured format and stored appropriately in the database.

[0570] Step 2:

[0571] The server automatically generates interview questions using an AI model based on the saved job description and job information. The AI ​​model uses a generative AI model to create appropriate questions based on past data and algorithmic prompts. This question dataset is saved on the server and structured as an interview scenario. The generated questions are saved on the server as structured data.

[0572] Step 3:

[0573] The server sends the automatically generated interview scenario to the terminal. The terminal displays a notification to the user (job seeker) informing them that "the interview is ready." Once this notification is displayed, the user (job seeker) can start the interview within 24 hours.

[0574] Step 4:

[0575] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) records their answers as audio and video. The recorded data is sent to the server in real time.

[0576] Step 5:

[0577] The server analyzes the recorded audio and video in real time. The analysis involves using speech recognition APIs such as Google Cloud Speech-to-Text and Microsoft Azure Face API to perform voice and facial expression analysis. Data analysis confirms appropriate responses to questions from the voice and evaluates emotions such as nervousness or confidence from facial expressions. The analysis results are generated as a score or evaluation and stored on the server.

[0578] Step 6:

[0579] The evaluation data generated as a result of the analysis is sent from the server to the terminal and notified to the user (human resources officer). The terminal displays the evaluation results in an easy-to-read format, including the score and feedback for each evaluation item.

[0580] Step 7:

[0581] When the user (job seeker) finishes the interview, the terminal finalizes the recorded data and stores it on the server, which can be used as a reference later.

[0582] Step 8:

[0583] The user (human resources officer) will conduct the final selection based on the evaluation results and the interview recording. This selection result is also entered into the server from the terminal and saved as the final pass / fail data. The server will generate a result notification based on this data and send it to the job seeker.

[0584] Step 9:

[0585] The server operates a means for receiving purchaser behavioral information and uses a smartphone app to collect behavioral information (browsing history, purchase history, click behavior, etc.) The collected data is sent back to the server and used for analysis.

[0586] Step 10:

[0587] The server uses a generative AI model based on the received behavioral information to automatically generate product recommendations. The generated recommendations are displayed to the buyer via a smartphone app. The buyer then records their reactions to the recommended products via audio and video.

[0588] Step 11:

[0589] The server analyzes the recorded responses and re-evaluates the product recommendations. The server generates ratings and updates the recommendations in real time, resulting in more accurate recommendations. Real-time display updates improve the user experience.

[0590] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0591] To implement this invention, three entities, the server, the terminal, and the user, work together, and an emotion engine is combined to recognize the user's emotions. The specific roles and operations of each entity, as well as how the emotion engine is integrated, are explained below.

[0592] Enter and save the recruitment information

[0593] First, the user (human resources officer) uses a terminal to input the job requirements and job information. For example, this includes items such as "job type," "required skills," and "years of experience." Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a save completion notification to the terminal.

[0594] Automatic generation of interview scenarios

[0595] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, for a requirement such as "Java experience," it generates questions such as "Tell us about a Java project you recently worked on."

[0596] Interview schedule notification

[0597] The generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." Upon receiving this notification, the user (job seeker) can start the interview at any time, 24 hours a day.

[0598] Virtual Human AI Interview

[0599] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0600] First selection and evaluation

[0601] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. The emotion engine also uses this data to analyze the user's (job seeker's) emotions. For example, voice data can identify an applicant's nervousness or confidence, while video data can identify emotions such as joy or surprise.

[0602] As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," "emotional stability," etc. The generated evaluation results are sent to the terminal and notified to the user (human resources officer).

[0603] Final selection and hiring decision

[0604] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made as to whether or not to hire the candidate. The final selection results are entered into the terminal and sent to the server. The server then decides whether or not to hire the candidate based on the final selection results and sends a notification to the job seeker.

[0605] Specific examples

[0606] For example, if a company posts a job posting for a "software engineer wanted," the user (human resources officer) would enter the requirements "Java project experience" and "more than three years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time using an emotion engine, and an initial selection evaluation is made. Finally, the human resources officer conducts a final selection based on the evaluation results and the recording, and a hiring decision is made.

[0607] This system eliminates the need to arrange interviewers and interview venues, significantly improves the quality and efficiency of interviews, and also enables more accurate understanding of applicants' true reactions and feelings through emotion analysis.

[0608] The processing flow will be explained below.

[0609] Step 1: Enter the job description

[0610] The user (human resources officer) enters the job requirements and job information into the terminal.

[0611] The terminal transmits the input data to the server.

[0612] Step 2: Save the job description

[0613] The server stores the received job posting and job information in a database.

[0614] The server sends a notification to the terminal that "recruitment requirements have been saved."

[0615] Step 3: Auto-generate interview questions

[0616] An AI model on the server generates an appropriate set of interview questions based on stored job descriptions and job information.

[0617] For example, if the job description states "Java experience," it will generate a question such as "Tell us about a Java project you've recently worked on."

[0618] Step 4: Communicate the interview scenario

[0619] The server transmits the generated interview scenario to the terminal.

[0620] The terminal notifies the user (job seeker) that "You are ready for the interview."

[0621] Step 5: Start the interview

[0622] The user (job seeker) presses the interview start button on the terminal.

[0623] The device communicates with the server and activates the virtual human AI.

[0624] Step 6: Presenting the interview questions

[0625] The server controls the virtual human AI and sequentially displays pre-generated interview questions.

[0626] The terminal records the user's (job seeker's) answers.

[0627] Step 7: Recording audio and video

[0628] The terminal records the user's (job seeker's) voice and video in real time.

[0629] The recorded data is sent to the server.

[0630] Step 8: Analyze the data

[0631] The server analyzes the recorded audio and video.

[0632] Voice analysis, facial expression analysis, and emotion recognition using an emotion engine are performed to generate an evaluation for the first selection round.

[0633] For example, the emotion engine analyzes the user's (job seeker's) nervousness and confidence from voice, and facial expression data such as smiles and surprise from video.

[0634] Step 9: Output of first selection results

[0635] The server generates the evaluation results of the first selection and sends them to the terminal.

[0636] The terminal displays the "first selection results" to the user (human resources officer).

[0637] Step 10: Final selection

[0638] The user (human resources officer) checks the initial selection results and interview recording on the terminal.

[0639] Human resources personnel will conduct the final selection based on the interview recording and evaluation results, and enter the final selection results into the terminal.

[0640] Step 11: Notification of hiring decision

[0641] The terminal transmits the final selection results to the server.

[0642] The server determines whether or not to hire the job seeker based on the final selection results and sends a notice of employment to the job seeker.

[0643] Through these steps, the server, terminal, user, and emotion engine work together to realize an efficient and sophisticated interview process. This system not only improves the quality and efficiency of interviews, but also enables a deeper understanding of the applicant's true suitability through emotion analysis.

[0644] Example 2

[0645] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0646] Traditional hiring processes require arranging interviewers and interview locations, which is time-consuming and costly. Furthermore, interviewers' subjective assessments of applicants can be inconsistent. Furthermore, it is difficult to accurately grasp applicants' emotions and true reactions. Therefore, there is a need for a system that can improve the efficiency and accuracy of the hiring process.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0648] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions based on the received job requirements and job information, means for displaying the interview questions and recording the applicant's voice and video, means for analyzing the recorded voice and video and generating an evaluation of the first selection, means for analyzing the applicant's emotions based on the analyzed content, and means for outputting the evaluation of the first selection. This eliminates the need to arrange interviewers and interview venues, improves the quality and efficiency of interviews, and makes it possible to more accurately grasp the applicant's true reactions and emotions.

[0649] "Recruitment requirements" are information about the job, specifically including the job type, required skills, years of experience, etc.

[0650] "Job information" is detailed information about a specific job, specifically including job content, roles, scope of responsibilities, and the like.

[0651] The "receiving means" refers to the technical means by which the server receives information input by the user, and includes a communication interface, a data receiving module, and the like.

[0652] "Automatic generation means" refers to means for automatically creating new content based on specified information using technologies such as AI models.

[0653] The "display means" is a device such as a display or monitor for visually presenting the generated questions and information to the user.

[0654] "Recording means" refers to technical means for saving data such as audio and video, and includes, for example, cameras, microphones, and recording devices.

[0655] "Means for analyzing" refers to technical means for analyzing data such as audio and video to derive specific identification information or scores, and includes voice analysis algorithms and facial expression analysis algorithms.

[0656] "Means for analyzing emotions" refers to technical means for identifying an applicant's emotions from audio or video, including emotion engines and machine learning models.

[0657] The "means for generating an evaluation" is a means for generating a score or evaluation based on specific criteria based on the analysis results.

[0658] "Means for outputting evaluation" refers to technical means for communicating the generated evaluation results to the user, including, for example, a screen display or a notification system.

[0659] To implement this invention, three entities - the server, the terminal, and the user - must work together, and an emotion engine must be combined to recognize the user's emotions. The specific roles and operations of each entity, as well as how the emotion engine is integrated, are explained below.

[0660] Enter and save the recruitment information

[0661] First, the user (human resources officer) uses a terminal to input the job requirements and job information. The input fields include "job type," "required skills," and "years of experience." Once the user has completed their input, the terminal sends this information to the server. The server saves the received information in a database, and once the saving is complete, it sends a save completion notification to the terminal. This ensures that the information is stored securely.

[0662] Automatic generation of interview scenarios

[0663] The server passes the saved job description and job information to the AI ​​model. A generative AI model, such as TensorFlow or PyTorch, is used to automatically generate interview questions appropriate to the requirements. For example, for the requirement "Java experience," a question such as "Tell us about a Java project you recently worked on" is generated. The server then stores the generated questions in a database.

[0664] Interview schedule notification

[0665] The generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time within 24 hours.

[0666] Virtual Human AI Interview

[0667] When a user (job seeker) starts an interview on their device, the device sends a request to start the interview to the server. The server then starts a virtual human AI and displays interview questions in sequence. The user (job seeker) responds via voice and video, and the data is recorded on the device and sent to the server in real time. Core software used for recording the audio and video includes OpenCV and FFmpeg.

[0668] First selection and evaluation

[0669] The server analyzes the received audio and video data. This analysis includes voice and facial expression analysis, using libraries such as Librosa and Dlib. An emotion engine also uses this data to analyze the user's (job seeker's) emotions. For example, the voice data can identify an applicant's nervousness or confidence, while the video data can identify emotions such as joy or surprise. As a result of the analysis, the server generates scores for such criteria as "technical skills," "communication ability," "adaptability," and "emotional stability." The generated evaluation results are sent to the terminal and notified to the user (human resources staff).

[0670] Final selection and hiring decision

[0671] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made on whether to hire the candidate. The final selection results are entered into the terminal and sent to the server. The server then sends a notification to the job seeker based on the final selection results.

[0672] Specific examples

[0673] For example, if a company posts a job posting for a software engineer, the user (a human resources officer) inputs the following requirements into the terminal: "Java project experience" and "3 or more years of development experience." The server receives this information and generates a set of interview questions.

[0674] Examples of specific prompts include:

[0675] "Generate interview questions suitable for candidates with at least 3 years of Java project experience. For example, questions about recent projects they worked on."

[0676] Interviews are conducted based on this information. Job seekers use a 24-hour interview system to interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time using an emotion engine, and an initial evaluation is conducted. Finally, a human resources staff member conducts the final selection based on the evaluation results and recording, and the candidate is hired. This system eliminates the need to arrange interviewers and interview venues, significantly improving the quality and efficiency of interviews, and also makes it possible to more accurately grasp the applicant's true reactions and emotions through emotion analysis.

[0677] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0678] Step 1:

[0679] The user (human resources staff) uses a terminal to enter job requirements and job information. Specifically, they enter information such as "job type," "required skills," and "years of experience" into the terminal's input form. This information is sent to the server through the terminal's user interface. Based on the input data, the terminal sends JSON format data to the server, which then stores it in a database on the server side.

[0680] Input: Job posting and job information entered on the terminal

[0681] Output: Information sent to the server and stored in the database

[0682] Step 2:

[0683] The server passes the received job description and job information to the AI ​​model. During this process, the server sends the information to the generative AI model as prompts. For example, for a requirement such as "Java project experience," a question such as "Tell us about a Java project you recently worked on" is generated. The AI ​​model generates questions using the TensorFlow and PyTorch libraries.

[0684] Input: Job postings and job information stored in the database

[0685] Output: Generated interview questions

[0686] Step 3:

[0687] The server saves the generated interview questions in a database. Once saving is complete, the server sends the interview questions to the terminal, and the terminal notifies the user (job seeker) that "the interview is ready."

[0688] Input: Generated interview questions

[0689] Output: Notification of interview settings to the terminal

[0690] Step 4:

[0691] When a user (job seeker) starts an interview on their device, the device sends a request to start the interview to the server. The server launches a virtual human AI and connects it to the device. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers using audio and video. Specifically, the device uses a camera and microphone to record audio and video and transmits them to the server in real time. OpenCV and FFmpeg libraries are used for this.

[0692] Input: Interview start request from the terminal and applicant's audio and video data

[0693] Output: Real-time video and audio data transmission to the server

[0694] Step 5:

[0695] The server analyzes the received audio and video data. Using the Librosa and Dlib libraries, voice and facial analysis is performed. An emotion engine also analyzes the applicant's emotions, such as nervousness and confidence. The analysis generates scores for factors such as technical skills, communication skills, adaptability, and emotional stability.

[0696] Input: Real-time video and audio data of applicant

[0697] Output: Evaluation scores for technical skills, communication skills, etc.

[0698] Step 6:

[0699] The server stores the generated evaluation results in a database and sends them to the terminal, which displays the evaluation results of the first selection and the interview recording so that the user (human resources staff) can check them.

[0700] Input: Rating score

[0701] Output: Evaluation results of the first selection and recorded data displayed on the terminal

[0702] Step 7:

[0703] The user (human resources officer) checks the evaluation results of the first selection on the terminal and conducts the final selection. The final selection results are entered into the terminal and sent to the server. Based on the final selection results, the server notifies the job seeker whether they have been hired or not.

[0704] Input: Final selection results by user (HR person)

[0705] Output: Notification of job offer to job seeker

[0706] (Application example 2)

[0707] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0708] Existing interview systems require preparation for interviewers, arrangement of interview venues, and adjustments to maintain the quality and efficiency of interviews, which leads to inefficient use of resources. Existing systems also make it difficult to accurately grasp applicants' emotions and reactions during interviews, making objective evaluations difficult. Furthermore, there is no system in place that can monitor and analyze the work status of robots in factories and the emotional health of workers in real time and provide immediate feedback, making quality control and work efficiency improvements a challenge.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the job description and job information, means for automatically generating interview questions based on the received job description and job information, means for displaying the interview questions and recording the applicant's voice and video, means for analyzing the recorded voice and video in real time using an emotion analysis engine and generating an evaluation, means for outputting the initial selection evaluation, and means for issuing a warning if a certain evaluation standard is not met based on the analysis results. This enables more efficient interview preparation, accurate understanding of the applicant's emotions and reactions, and real-time monitoring of quality control and the emotional health of workers in factories.

[0710] A "job description" is a document that contains information about a job, such as job duties, required skills, and years of experience.

[0711] "Job information" is data that includes detailed information related to a particular job, such as job responsibilities, required qualifications, and location.

[0712] "Interview questions" are questions asked of applicants to assess their aptitude and skills.

[0713] "Audio and video" refers to digital data used to record the voices, facial expressions, and movements of applicants and workers.

[0714] An "emotion analysis engine" is an algorithm and software that analyzes data such as audio and video to evaluate and determine the emotional state of a subject.

[0715] "First selection" refers to the initial selection process, which is primarily a procedure for evaluating the applicant's basic aptitude and skills.

[0716] "Metrics" are standards or guidelines established for evaluating data or actions.

[0717] "Means for issuing warnings" refers to the ability to send alerts or notifications when evaluation results do not meet set standards.

[0718] To implement this invention, three entities - a server, a terminal, and a user - work together, and an emotion analysis engine is integrated to determine the user's emotion. The specific roles and operations of each entity, as well as how the emotion analysis engine is integrated, are explained below.

[0719] Overall structure

[0720] Hardware and Software Configuration

[0721] Hardware:

[0722] Device: User (HR personnel and job seeker) operation device (PC or tablet)

[0723] Server: Server for data analysis and automatic generation of interview questions

[0724] Factory robots: robots with built-in cameras and microphones

[0725] software:

[0726] Server side: Sentiment analysis AI using Python / TensorFlow / Keras

[0727] Frontend: User interface with React.js

[0728] Database: MySQL or PostgreSQL

[0729] Communication: Real-time communication via WebSocket

[0730] System Overview

[0731] 1. Enter and save the recruitment information

[0732] First, the user (human resources staff) uses a terminal to enter the job requirements and job information. The entered information is sent to the server and stored in the database. This procedure allows the user to easily register the necessary requirements.

[0733] 2. Automatic generation of interview scenarios

[0734] After the server receives and stores the job description and job information, the generative AI model automatically generates interview questions based on this information. For example, a request for "Java experience" generates a question such as "Tell us about a Java project you recently worked on."

[0735] 3. Notification of interview schedule

[0736] The generated interview scenario is sent from the server to the terminal, and the user (job seeker) is notified that "the interview is ready." This allows job seekers to start an interview at any time, 24 hours a day.

[0737] 4. Virtual Human AI Interviews

[0738] When a user (job seeker) starts an interview on their device, it communicates with the server and activates the virtual human AI, which sequentially displays pre-generated interview questions, and the recorded audio and video are sent to the server in real time.

[0739] 5. First selection and evaluation

[0740] The server analyzes the recorded audio and video. This analysis uses emotion analysis AI with TensorFlow and Keras. For example, it can recognize an applicant's nervousness or confidence from audio data, and emotions such as joy or surprise from video data. This generates scores for such things as "technical skills," "communication ability," "adaptability," and "emotional stability."

[0741] 6. Notification of evaluation results and feedback

[0742] The generated evaluation results are sent to the terminal and notified to the user (human resources personnel). Furthermore, if the analysis results do not meet certain evaluation standards, a warning is issued. This feedback loop improves the quality and efficiency of interviews.

[0743] Examples of specific examples and prompts

[0744] For example, when a worker is performing "assembly work" in a factory, a factory robot monitors the worker's movements in real time and sends the data collected by a camera and microphone to an emotion analysis engine. Based on the analysis results, if the worker is feeling excessively stressed, a warning will be issued and appropriate feedback will be provided.

[0745] Prompt Sentence Examples

[0746] plain

[0747] "Analyze video and audio data of workers as they assemble parts, and analyze emotions such as stress and satisfaction in real time."

[0748] This will enable quality control in factories and improve the emotional well-being of workers.

[0749] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0750] Step 1: Enter and save job description and job information

[0751] The user (human resources officer) uses a terminal to input the job requirements and job information. The input information is sent from the terminal to the server and saved in a database. The input data includes the job type, required skills, years of experience, etc. The server then sends a notification to the terminal that the data has been saved to the database.

[0752] Input: Job requirements and job information

[0753] Output: Information saved in the database, notification of save completion

[0754] Step 2: Auto-generate interview questions

[0755] The server uses a generative AI model to automatically generate interview questions based on the saved job description and job information. For example, if Java experience is required, the generative model generates questions such as, "Tell me about a Java project you recently worked on."

[0756] Input: Saved job postings and job information

[0757] Output: Auto-generated interview questions

[0758] Step 3: Notification of interview arrangement

[0759] The automatically generated interview questions are sent from the server to the terminal, and the terminal notifies the user (job seeker) that "the interview is ready." This notification allows the job seeker to prepare for the interview and start the interview at any time, 24 hours a day.

[0760] Input: Auto-generated interview questions

[0761] Output: Interview preparation notice

[0762] Step 4: Virtual Human AI Interview

[0763] When a user (job seeker) starts an interview through their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays the generated interview questions and records the applicant's voice and video. The audio and video data is sent to the server in real time.

[0764] Input: Interview questions, audio and video of job seeker

[0765] Output: Recorded audio and video data

[0766] Step 5: First selection and evaluation

[0767] The server analyzes the received audio and video data in real time. It uses an emotion analysis engine based on TensorFlow and Keras to recognize the applicant's nervousness or confidence from the audio data, and emotions such as joy or surprise from the video data. This generates an evaluation score based on factors such as technical skills, communication ability, adaptability, and emotional stability.

[0768] Input: Recorded audio and video data

[0769] Output: Evaluation score

[0770] Step 6: Notification of evaluation results and feedback

[0771] The server sends the generated evaluation score to the terminal and notifies the user (human resources officer). Furthermore, if the analysis results indicate that a certain evaluation standard has not been met, a warning is issued. At this step, feedback based on the evaluation results is provided, improving the quality and efficiency of the interview.

[0772] Input: Rating score

[0773] Output: Evaluation result notification, warning notification

[0774] Step 7: Emotion analysis and quality control for factory robots

[0775] In factories, robots use cameras and microphones to monitor the movements and emotions of workers in real time, and send the collected data to a server, which uses an emotion analysis engine to analyze the worker's stress, satisfaction, and other factors, and issue warnings as necessary.

[0776] Input: Video and audio data of the worker

[0777] Output: Sentiment analysis data, quality control assessment, warning notifications

[0778] The above processing steps enable efficient and objective evaluation, which is expected to have a significant effect on personnel and factory quality control.

[0779] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0780] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0781] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0782] [Third embodiment]

[0783] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0784] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0785] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0786] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0787] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0788] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0789] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0790] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0791] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0792] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0793] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0794] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0795] To implement the present invention, three main entities, the server, the terminal, and the user, must work together. The specific roles and operations of each entity are described below.

[0796] Enter and save the recruitment information

[0797] First, the user (human resources officer) uses a terminal to input the job requirements and job information. This information includes the job type, required skills, years of experience, etc. Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a notification to the terminal that the information has been saved.

[0798] Automatic generation of interview scenarios

[0799] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, if the job description mentions "Java experience," the model generates questions such as "Tell us about a Java project you recently worked on."

[0800] Interview schedule notification

[0801] The automatically generated interview scenario is sent from the server to the terminal. The terminal then notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time, 24 hours a day.

[0802] Virtual Human AI Interview

[0803] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0804] First selection and evaluation

[0805] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. For example, the user's (job seeker's) voice is used to evaluate the appropriateness of their responses to questions and their ability to accurately understand the other person's intentions. Meanwhile, facial expression analysis can detect emotions such as nervousness or confidence.

[0806] As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," etc. The generated evaluation results are sent to the terminal and notified to the user (human resources officer).

[0807] Final selection and hiring decision

[0808] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made on whether to hire the candidate. The results of the final selection are entered into the terminal and sent to the server. The server then automatically generates and sends a notification to the job seeker who has been selected for employment.

[0809] Specific examples

[0810] For example, if a company posts a job posting for a software engineer, the user (human resources officer) enters the requirements "Java project experience" and "more than three years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time, and an initial selection evaluation is conducted. Finally, the human resources officer conducts a final selection based on the evaluation results and the recording, and a hiring decision is made.

[0811] This system eliminates the need to arrange interviewers and interview venues, improving the quality and efficiency of interviews, reducing stress for job seekers, and allowing for flexible interview schedules.

[0812] The processing flow will be explained below.

[0813] Step 1: Enter the job description

[0814] The user enters the job description and job information into the terminal.

[0815] The terminal transmits the input data to the server.

[0816] Step 2: Save the job description

[0817] The server stores the received job posting and job information in a database.

[0818] The server sends a notification to the terminal that "recruitment requirements have been saved."

[0819] Step 3: Auto-generate interview questions

[0820] An AI model on the server generates an appropriate set of interview questions based on stored job descriptions and job information.

[0821] Step 4: Communicate the interview scenario

[0822] The server transmits the generated interview scenario to the terminal.

[0823] The terminal notifies the user (job seeker) that "You are ready for the interview."

[0824] Step 5: Start the interview

[0825] The user (job seeker) presses the interview start button on the terminal.

[0826] The device communicates with the server and activates the virtual human AI.

[0827] Step 6: Presenting the interview questions

[0828] The server controls the virtual human AI and sequentially displays pre-generated interview questions.

[0829] The terminal records the user's (job seeker's) answers.

[0830] Step 7: Recording audio and video

[0831] The terminal records the user's (job seeker's) voice and video in real time.

[0832] The recorded data is sent to the server.

[0833] Step 8: Analyze the data

[0834] The server analyzes the recorded audio and video.

[0835] Voice and facial expression analysis is performed to generate an evaluation for the first selection.

[0836] Step 9: Output of first selection results

[0837] The server sends the evaluation results of the first selection to the terminal.

[0838] The terminal notifies the user (human resources officer).

[0839] Step 10: Final selection

[0840] The user (human resources officer) checks the initial selection results and interview recording on the terminal.

[0841] The results of the final selection are entered into the terminal and sent to the server.

[0842] Step 11: Notification of hiring decision

[0843] The server will decide whether to hire the candidate based on the final selection results.

[0844] The server sends a job offer to the job seeker.

[0845] Example 1

[0846] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0847] In the traditional recruitment process, it is necessary to arrange interviewers and interview venues, which is time-consuming and costly. In addition, it is difficult for job seekers to flexibly schedule interviews, which often increases tension. There is a need to solve these problems and improve the quality and efficiency of interviews.

[0848] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0849] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions using a generative AI model based on the received job requirements and job information, means for displaying the automatically generated interview questions and recording audio and video of applicants, means for analyzing the recorded audio and video data in real time to generate an evaluation score for the first selection, and means for outputting the generated evaluation score. This eliminates the need to arrange interviewers and interview venues, provides job seekers with flexible interview schedules, and makes it possible to improve the quality and efficiency of interviews.

[0850] A "recruitment requirement" is a document that lists the job description and application conditions that a company presents when recruiting new employees.

[0851] "Job Information" means information that details the skills, experience, and other requirements needed for a particular job.

[0852] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate interview questions and other information based on specific rules and patterns.

[0853] "Interview questions" are questions that companies ask job seekers to assess their skills, experience, and aptitude.

[0854] "Audio and video recording means" refers to devices and software that store audio and video recordings of job seekers as digital data.

[0855] "Means for real-time analysis" refers to technology for instantly analyzing and processing recorded data and generating an evaluation.

[0856] The "first selection evaluation score" is a numerical value obtained by evaluating the skills and aptitude of a job seeker, and is used as the result of the first selection process.

[0857] "Facial expression analysis" is a technology that reads the emotions and reactions of job applicants from their facial expressions and reflects this in their evaluation.

[0858] "Voice analysis" is a technology that analyzes the speaking style and content of a job seeker's voice, and uses this information to evaluate their skills and aptitude.

[0859] "Output means" means a device or software for displaying or reporting the generated evaluation scores or other analytical results.

[0860] To implement the invention, three main entities must work together: the server, the terminal, and the user. Below we will explain in detail how these entities work together to implement the invention.

[0861] Enter and save job postings and job information

[0862] First, the user (human resources staff) uses a terminal to input the job requirements and job information. The terminal can be a general computer device such as a PC or tablet. The user inputs information such as "Java project experience" or "3+ years of development experience." Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a notification to the terminal that the information has been saved.

[0863] Automatic generation of interview questions

[0864] The server uses a generative AI model to automatically generate interview questions based on the received job description and job information. The generative AI model uses dedicated software equipped with machine learning algorithms. For example, if the job description mentions "Java experience," the generative AI model automatically generates specific questions such as "Tell us about a Java project you recently worked on." These questions are stored in a database on the server.

[0865] Notification and preparation of interview scenarios

[0866] The automatically generated interview scenario is sent from the server to the device. Based on this, the device sends the user (job seeker) a notification that "the interview is ready." This notification is sent via email or push notification. After receiving the notification, the job seeker can access the interview system and begin the interview within 24 hours.

[0867] Virtual Human AI Interview

[0868] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, which the job seeker answers. At this time, the job seeker's answers are recorded as audio and video on the device and sent to the server in real time. The virtual human AI uses advanced artificial intelligence technology to enable natural dialogue.

[0869] Real-time analysis of responses and initial selection evaluation

[0870] The server analyzes the audio and video data it receives in real time. This analysis includes voice recognition and facial expression analysis technologies. For example, voice analysis evaluates the content of responses and speaking style, while facial expression analysis reads emotions such as nervousness or confidence. Based on the analysis results, the server generates a first-stage selection evaluation score, including "technical skills," "communication ability," and "adaptability." This evaluation score is sent to a terminal as digital data and notified to the user (human resources officer).

[0871] Final selection and hiring decision

[0872] The user (human resources staff) receives the evaluation results via the device and checks the evaluation scores from the first selection and the interview recording data. Based on this information, the final selection is made and the decision on whether to hire is entered into the device. The server then automatically generates and sends a hiring decision notice to the job seeker based on the final selection results. This notification is also sent via email or push notification.

[0873] Specific examples

[0874] For example, if a company posts a job posting for a software engineer, the user (human resources officer) enters the requirements "Java project experience" and "3+ years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The generated questions include a specific prompt, such as "Tell us about a Java project you recently worked on." The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time to evaluate the candidate for the first selection stage. Finally, the human resources officer conducts a final selection based on the evaluation results and recording, and the candidate is hired. This system eliminates the need to arrange interviewers and interview venues, improving the quality and efficiency of interviews. It also reduces stress for job seekers and allows for flexible interview schedules.

[0875] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0876] The flow of this system's program processing

[0877] Step 1:

[0878] The user (human resources staff) uses a terminal to input the job requirements and job information. For example, they input specific items such as "Java project experience" or "3+ years of development experience." The job type, required skills, years of experience, etc. are input into the terminal as input data. The output is the entered job requirements and job information.

[0879] Step 2:

[0880] The terminal sends the job description and job information data entered by the user to the server. The secure HTTP(S) protocol is used to send the data. The input is the job description and job information entered by the user into the terminal, and the output is the data sent to the server.

[0881] Step 3:

[0882] The server saves the received data in a database. After saving is complete, the server notifies the terminal that "data saving is complete." The input is the job description and job information data received by the server, and the output is a notification that saving is complete.

[0883] Step 4:

[0884] The server uses a generative AI model to automatically generate interview questions based on the saved job description and job information. For example, it generates specific questions such as "Tell me about a Java project you recently worked on." The generative AI model analyzes the received data and generates prompts. The input is the job description and job information saved on the server, and the output is a set of interview questions.

[0885] Step 5:

[0886] The server sends automatically generated interview questions to the terminal. The input is the interview questions generated by the server, and the output is the set of questions sent to the terminal.

[0887] Step 6:

[0888] Based on the interview questions received, the device sends a notification to the user (job seeker) saying "Your interview is ready." This notification is sent via email or push notification. The input is a set of interview questions from the server, and the output is a notification to the job seeker.

[0889] Step 7:

[0890] After receiving the notification, the user (job seeker) clicks a button on their device to start the interview. The input is the notification, and the output is the start of the interview.

[0891] Step 8:

[0892] The terminal communicates with the server and starts the virtual human AI, which causes the virtual human to appear on the screen. The input is the start of the interview, and the output is the start of the virtual human AI.

[0893] Step 9:

[0894] The virtual human AI sequentially presents interview questions received from the server to the user. The user's answers are recorded as audio and video on the device and sent to the server in real time. The input is the set of questions from the server, and the output is the recorded audio and video data.

[0895] Step 10:

[0896] The server analyzes the received audio and video data in real time. This analysis uses voice recognition and facial expression analysis technologies. For example, it can detect a user's level of tension from their tone of voice and facial expression. The input is audio and video data, and the output is the analysis results.

[0897] Step 11:

[0898] Based on the analysis results, the server generates a first-stage selection evaluation score for "technical skills," "communication skills," "adaptability," etc. The input is the analysis results, and the output is the first-stage selection evaluation score.

[0899] Step 12:

[0900] The server sends the generated evaluation score to the terminal, and the terminal notifies the user (human resources officer). The input is the evaluation score, and the output is the notification.

[0901] Step 13:

[0902] The user (human resources staff) checks the evaluation scores from the first selection and the interview recording data. Based on this information, the final selection is made and the decision on whether to hire is entered into the terminal. The input is the evaluation scores and the recording data, and the output is the decision on whether to hire.

[0903] Step 14:

[0904] The server receives the final selection results from the device and automatically generates and sends a hiring decision notice to the job seeker. This notification is also sent via email or push notification. The input is the final selection results, and the output is the hiring decision notice.

[0905] (Application example 1)

[0906] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0907] Conventional interview systems and shopping assistant systems face the problem of difficulty in collecting and evaluating appropriate information from users. Interviews, in particular, require the arrangement of interviewers and interview venues, which is inefficient. Furthermore, when it comes to shopping, it is difficult to recommend products that match the preferences of individual buyers. Furthermore, there is no system that can analyze user responses in real time and change responses based on that analysis, which makes it difficult to improve the user experience.

[0908] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0909] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions based on the received job requirements and job information, means for displaying interview questions and recording audio and video of applicants, means for analyzing the recorded audio and video and generating an evaluation of the initial selection, means for outputting the evaluation of the initial selection, means for receiving behavioral information of purchasers, means for automatically generating product recommendations based on the received behavioral information, means for displaying product recommendations and recording purchaser responses, means for analyzing the recorded materials and re-evaluating the product recommendations, and means for generating evaluations and updating the display in real time. This enables improved efficiency and quality of interviews, appropriate product recommendations for individual purchasers, and optimization of the user experience in real time.

[0910] "Job Description and Job Information" means detailed information about a job opening, including the skills and years of experience required for a particular position.

[0911] The "means for automatically generating interview questions" refers to a function that enables an AI model to automatically generate questions to be used in interviews based on the job description and job information.

[0912] "Means for displaying interview questions and recording audio and video of applicants" refers to a device or application that displays automatically generated interview questions to an applicant and records audio and video of the applicant's responses.

[0913] "Means for analyzing recorded audio and video and generating an evaluation for the first selection" refers to a system that analyzes recorded audio and video data of applicants and generates a numerical or textual evaluation for the first selection based on that data.

[0914] The "means for outputting the evaluation of the first selection" has the function of notifying or displaying the generated evaluation results of the first selection to the user.

[0915] "Means for receiving customer behavior information" refers to a device or application that collects customer behavior on a store or online platform (e.g., browsing history, purchase history, click behavior, etc.) and sends it to a server.

[0916] "Means for automatically generating product recommendations based on received behavioral information" refers to a system in which an AI model automatically recommends products suitable for a purchaser based on collected behavioral information of the purchaser.

[0917] "Means for displaying product recommendations and recording buyer responses" refers to a device or application that displays recommended products to a buyer and records the buyer's responses (e.g., audio, video, interaction) at that time.

[0918] The "means for analyzing recorded data and reevaluating product recommendations" is a system that analyzes buyer responses, reviews product recommendations based on the results, and makes new recommendations as necessary.

[0919] The "means for generating ratings in real time and updating the display" is a function that analyzes buyer response data in real time and instantly updates the display content based on the results.

[0920] To implement the present invention, three main entities, the server, the terminal, and the user, must work together. The specific roles and operations of each entity are described below.

[0921] Enter and save the recruitment information

[0922] The server first assists the user (human resources staff) in the process of entering job requirements and job information using a terminal. This information includes job type, required skills, years of experience, etc. Once the information has been entered, the terminal sends this information to the server, which then stores the received information in a database. The server then notifies the terminal that the information has been saved.

[0923] Automatic generation of interview scenarios

[0924] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, if the job description mentions "Java experience," the model generates questions such as "Tell us about a Java project you recently worked on."

[0925] Interview schedule notification

[0926] The automatically generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time, 24 hours a day.

[0927] Virtual Human AI Interview

[0928] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0929] First selection and evaluation

[0930] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. For example, the user's (job seeker's) voice is used to evaluate the appropriate response to questions and the ability to accurately understand the other person's intentions. Facial expression analysis can detect emotions such as nervousness and confidence. As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," and other areas. The generated evaluation results are sent to a terminal and notified to the user (human resources officer).

[0931] Product recommendations and real-time ratings

[0932] The server has a means for receiving buyer behavioral information and a means for automatically generating product recommendations based on the received behavioral information. When a buyer uses a smartphone app to input behavioral information (browsing history, purchase history, click behavior, etc.), the server analyzes it and uses an AI model to recommend products. It also displays product recommendations and records the buyer's responses (audio and video). The recorded data is analyzed to reevaluate the product recommendations, and the display is updated in real time.

[0933] Examples of specific examples and prompts

[0934] A concrete example is a scenario for a smart shopping assistant app. A user launches the smartphone app and enters their profile (preferences and albums). The shopping assistant starts recommending products and displays a message saying, "AI has prepared some recommended products for you." When the user asks the virtual assistant, "How do I use this product?", the virtual human AI provides detailed explanations, saying, "This product can also be used for commercial purposes, allowing you to cook more efficiently." The process continues by analyzing the user's facial expressions and voice, and further customizing the product.

[0935] Example prompt sentence:

[0936] 1. [User]: What are your recommended fashion items for summer?

[0937] 2. [Shopping Assistant]: Looking for recommended fashion items.

[0938] 3. [AI Model]: Recommends the following items based on the user's profile and purchase history:

[0939] 1. Summer dress

[0940] 2. Sunglasses

[0941] 3. Flip-flops

[0942] 4. [User]: How do I use a summer dress?

[0943] 5. [Virtual Human AI]: Summer dresses are versatile and can be worn for casual everyday wear or beach photoshoots.

[0944] In this way, cooperation between the server, terminal, and user allows both the interview and shopping systems to operate effectively, improving the user experience.

[0945] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0946] Step 1:

[0947] A user (human resources officer) uses a terminal to input job requirements and job information. The input includes details such as job type, required skills, years of experience, etc. The input data is sent from the terminal to the server, which stores it in a database. The input data is processed and converted into a structured format and stored appropriately in the database.

[0948] Step 2:

[0949] The server automatically generates interview questions using an AI model based on the saved job description and job information. The AI ​​model uses a generative AI model to create appropriate questions based on past data and algorithmic prompts. This question dataset is saved on the server and structured as an interview scenario. The generated questions are saved on the server as structured data.

[0950] Step 3:

[0951] The server sends the automatically generated interview scenario to the terminal. The terminal displays a notification to the user (job seeker) informing them that "the interview is ready." Once this notification is displayed, the user (job seeker) can start the interview within 24 hours.

[0952] Step 4:

[0953] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) records their answers as audio and video. The recorded data is sent to the server in real time.

[0954] Step 5:

[0955] The server analyzes the recorded audio and video in real time. The analysis involves using speech recognition APIs such as Google Cloud Speech-to-Text and Microsoft Azure Face API to perform voice and facial expression analysis. Data analysis confirms appropriate responses to questions from the voice and evaluates emotions such as nervousness or confidence from facial expressions. The analysis results are generated as a score or evaluation and stored on the server.

[0956] Step 6:

[0957] The evaluation data generated as a result of the analysis is sent from the server to the terminal and notified to the user (human resources officer). The terminal displays the evaluation results in an easy-to-read format, including the score and feedback for each evaluation item.

[0958] Step 7:

[0959] When the user (job seeker) finishes the interview, the terminal finalizes the recorded data and stores it on the server, which can be used as a reference later.

[0960] Step 8:

[0961] The user (human resources officer) will conduct the final selection based on the evaluation results and the interview recording. This selection result is also entered into the server from the terminal and saved as the final pass / fail data. The server will generate a result notification based on this data and send it to the job seeker.

[0962] Step 9:

[0963] The server operates a means for receiving purchaser behavioral information and uses a smartphone app to collect behavioral information (browsing history, purchase history, click behavior, etc.) The collected data is sent back to the server and used for analysis.

[0964] Step 10:

[0965] The server uses a generative AI model based on the received behavioral information to automatically generate product recommendations. The generated recommendations are displayed to the buyer via a smartphone app. The buyer then records their reactions to the recommended products via audio and video.

[0966] Step 11:

[0967] The server analyzes the recorded responses and re-evaluates the product recommendations. The server generates ratings and updates the recommendations in real time, resulting in more accurate recommendations. Real-time display updates improve the user experience.

[0968] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0969] To implement this invention, three entities, the server, the terminal, and the user, work together, and an emotion engine is combined to recognize the user's emotions. The specific roles and operations of each entity, as well as how the emotion engine is integrated, are explained below.

[0970] Enter and save the recruitment information

[0971] First, the user (human resources officer) uses a terminal to input the job requirements and job information. For example, this includes items such as "job type," "required skills," and "years of experience." Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a save completion notification to the terminal.

[0972] Automatic generation of interview scenarios

[0973] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, for a requirement such as "Java experience," it generates questions such as "Tell us about a Java project you recently worked on."

[0974] Interview schedule notification

[0975] The generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." Upon receiving this notification, the user (job seeker) can start the interview at any time, 24 hours a day.

[0976] Virtual Human AI Interview

[0977] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[0978] First selection and evaluation

[0979] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. The emotion engine also uses this data to analyze the user's (job seeker's) emotions. For example, voice data can identify an applicant's nervousness or confidence, while video data can identify emotions such as joy or surprise.

[0980] As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," "emotional stability," etc. The generated evaluation results are sent to the terminal and notified to the user (human resources officer).

[0981] Final selection and hiring decision

[0982] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made as to whether or not to hire the candidate. The final selection results are entered into the terminal and sent to the server. The server then decides whether or not to hire the candidate based on the final selection results and sends a notification to the job seeker.

[0983] Specific examples

[0984] For example, if a company posts a job posting for a "software engineer wanted," the user (human resources officer) would enter the requirements "Java project experience" and "more than three years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time using an emotion engine, and an initial selection evaluation is made. Finally, the human resources officer conducts a final selection based on the evaluation results and the recording, and a hiring decision is made.

[0985] This system eliminates the need to arrange interviewers and interview venues, significantly improves the quality and efficiency of interviews, and also enables more accurate understanding of applicants' true reactions and feelings through emotion analysis.

[0986] The processing flow will be explained below.

[0987] Step 1: Enter the job description

[0988] The user (human resources officer) enters the job requirements and job information into the terminal.

[0989] The terminal transmits the input data to the server.

[0990] Step 2: Save the job description

[0991] The server stores the received job posting and job information in a database.

[0992] The server sends a notification to the terminal that "recruitment requirements have been saved."

[0993] Step 3: Auto-generate interview questions

[0994] An AI model on the server generates an appropriate set of interview questions based on stored job descriptions and job information.

[0995] For example, if the job description states "Java experience," it will generate a question such as "Tell us about a Java project you've recently worked on."

[0996] Step 4: Communicate the interview scenario

[0997] The server transmits the generated interview scenario to the terminal.

[0998] The terminal notifies the user (job seeker) that "You are ready for the interview."

[0999] Step 5: Start the interview

[1000] The user (job seeker) presses the interview start button on the terminal.

[1001] The device communicates with the server and activates the virtual human AI.

[1002] Step 6: Presenting the interview questions

[1003] The server controls the virtual human AI and sequentially displays pre-generated interview questions.

[1004] The terminal records the user's (job seeker's) answers.

[1005] Step 7: Recording audio and video

[1006] The terminal records the user's (job seeker's) voice and video in real time.

[1007] The recorded data is sent to the server.

[1008] Step 8: Analyze the data

[1009] The server analyzes the recorded audio and video.

[1010] Voice analysis, facial expression analysis, and emotion recognition using an emotion engine are performed to generate an evaluation for the first selection round.

[1011] For example, the emotion engine analyzes the user's (job seeker's) nervousness and confidence from voice, and facial expression data such as smiles and surprise from video.

[1012] Step 9: Output of first selection results

[1013] The server generates the evaluation results of the first selection and sends them to the terminal.

[1014] The terminal displays the "first selection results" to the user (human resources officer).

[1015] Step 10: Final selection

[1016] The user (human resources officer) checks the initial selection results and interview recording on the terminal.

[1017] Human resources personnel will conduct the final selection based on the interview recording and evaluation results, and enter the final selection results into the terminal.

[1018] Step 11: Notification of hiring decision

[1019] The terminal transmits the final selection results to the server.

[1020] The server determines whether or not to hire the job seeker based on the final selection results and sends a notice of employment to the job seeker.

[1021] Through these steps, the server, terminal, user, and emotion engine work together to realize an efficient and sophisticated interview process. This system not only improves the quality and efficiency of interviews, but also enables a deeper understanding of the applicant's true suitability through emotion analysis.

[1022] Example 2

[1023] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1024] Traditional hiring processes require arranging interviewers and interview locations, which is time-consuming and costly. Furthermore, interviewers' subjective assessments of applicants can be inconsistent. Furthermore, it is difficult to accurately grasp applicants' emotions and true reactions. Therefore, there is a need for a system that can improve the efficiency and accuracy of the hiring process.

[1025] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1026] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions based on the received job requirements and job information, means for displaying the interview questions and recording the applicant's voice and video, means for analyzing the recorded voice and video and generating an evaluation of the first selection, means for analyzing the applicant's emotions based on the analyzed content, and means for outputting the evaluation of the first selection. This eliminates the need to arrange interviewers and interview venues, improves the quality and efficiency of interviews, and makes it possible to more accurately grasp the applicant's true reactions and emotions.

[1027] "Recruitment requirements" are information about the job, specifically including the job type, required skills, years of experience, etc.

[1028] "Job information" is detailed information about a specific job, specifically including job content, roles, scope of responsibilities, and the like.

[1029] The "receiving means" refers to the technical means by which the server receives information input by the user, and includes a communication interface, a data receiving module, and the like.

[1030] "Automatic generation means" refers to means for automatically creating new content based on specified information using technologies such as AI models.

[1031] The "display means" is a device such as a display or monitor for visually presenting the generated questions and information to the user.

[1032] "Recording means" refers to technical means for saving data such as audio and video, and includes, for example, cameras, microphones, and recording devices.

[1033] "Means for analyzing" refers to technical means for analyzing data such as audio and video to derive specific identification information or scores, and includes voice analysis algorithms and facial expression analysis algorithms.

[1034] "Means for analyzing emotions" refers to technical means for identifying an applicant's emotions from audio or video, including emotion engines and machine learning models.

[1035] The "means for generating an evaluation" is a means for generating a score or evaluation based on specific criteria based on the analysis results.

[1036] "Means for outputting evaluation" refers to technical means for communicating the generated evaluation results to the user, including, for example, a screen display or a notification system.

[1037] To implement this invention, three entities - the server, the terminal, and the user - must work together, and an emotion engine must be combined to recognize the user's emotions. The specific roles and operations of each entity, as well as how the emotion engine is integrated, are explained below.

[1038] Enter and save the recruitment information

[1039] First, the user (human resources officer) uses a terminal to input the job requirements and job information. The input fields include "job type," "required skills," and "years of experience." Once the user has completed their input, the terminal sends this information to the server. The server saves the received information in a database, and once the saving is complete, it sends a save completion notification to the terminal. This ensures that the information is stored securely.

[1040] Automatic generation of interview scenarios

[1041] The server passes the saved job description and job information to the AI ​​model. A generative AI model, such as TensorFlow or PyTorch, is used to automatically generate interview questions appropriate to the requirements. For example, for the requirement "Java experience," a question such as "Tell us about a Java project you recently worked on" is generated. The server then stores the generated questions in a database.

[1042] Interview schedule notification

[1043] The generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time within 24 hours.

[1044] Virtual Human AI Interview

[1045] When a user (job seeker) starts an interview on their device, the device sends a request to start the interview to the server. The server then starts a virtual human AI and displays interview questions in sequence. The user (job seeker) responds via voice and video, and the data is recorded on the device and sent to the server in real time. Core software used for recording the audio and video includes OpenCV and FFmpeg.

[1046] First selection and evaluation

[1047] The server analyzes the received audio and video data. This analysis includes voice and facial expression analysis, using libraries such as Librosa and Dlib. An emotion engine also uses this data to analyze the user's (job seeker's) emotions. For example, the voice data can identify an applicant's nervousness or confidence, while the video data can identify emotions such as joy or surprise. As a result of the analysis, the server generates scores for such criteria as "technical skills," "communication ability," "adaptability," and "emotional stability." The generated evaluation results are sent to the terminal and notified to the user (human resources staff).

[1048] Final selection and hiring decision

[1049] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made on whether to hire the candidate. The final selection results are entered into the terminal and sent to the server. The server then sends a notification to the job seeker based on the final selection results.

[1050] Specific examples

[1051] For example, if a company posts a job posting for a software engineer, the user (a human resources officer) inputs the following requirements into the terminal: "Java project experience" and "3 or more years of development experience." The server receives this information and generates a set of interview questions.

[1052] Examples of specific prompts include:

[1053] "Generate interview questions suitable for candidates with at least 3 years of Java project experience. For example, questions about recent projects they worked on."

[1054] Interviews are conducted based on this information. Job seekers use a 24-hour interview system to interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time using an emotion engine, and an initial evaluation is conducted. Finally, a human resources staff member conducts the final selection based on the evaluation results and recording, and the candidate is hired. This system eliminates the need to arrange interviewers and interview venues, significantly improving the quality and efficiency of interviews, and also makes it possible to more accurately grasp the applicant's true reactions and emotions through emotion analysis.

[1055] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1056] Step 1:

[1057] The user (human resources staff) uses a terminal to enter job requirements and job information. Specifically, they enter information such as "job type," "required skills," and "years of experience" into the terminal's input form. This information is sent to the server through the terminal's user interface. Based on the input data, the terminal sends JSON format data to the server, which then stores it in a database on the server side.

[1058] Input: Job posting and job information entered on the terminal

[1059] Output: Information sent to the server and stored in the database

[1060] Step 2:

[1061] The server passes the received job description and job information to the AI ​​model. During this process, the server sends the information to the generative AI model as prompts. For example, for a requirement such as "Java project experience," a question such as "Tell us about a Java project you recently worked on" is generated. The AI ​​model generates questions using the TensorFlow and PyTorch libraries.

[1062] Input: Job postings and job information stored in the database

[1063] Output: Generated interview questions

[1064] Step 3:

[1065] The server saves the generated interview questions in a database. Once saving is complete, the server sends the interview questions to the terminal, and the terminal notifies the user (job seeker) that "the interview is ready."

[1066] Input: Generated interview questions

[1067] Output: Notification of interview settings to the terminal

[1068] Step 4:

[1069] When a user (job seeker) starts an interview on their device, the device sends a request to start the interview to the server. The server launches a virtual human AI and connects it to the device. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers using audio and video. Specifically, the device uses a camera and microphone to record audio and video and transmits them to the server in real time. OpenCV and FFmpeg libraries are used for this.

[1070] Input: Interview start request from the terminal and applicant's audio and video data

[1071] Output: Real-time video and audio data transmission to the server

[1072] Step 5:

[1073] The server analyzes the received audio and video data. Using the Librosa and Dlib libraries, voice and facial analysis is performed. An emotion engine also analyzes the applicant's emotions, such as nervousness and confidence. The analysis generates scores for factors such as technical skills, communication skills, adaptability, and emotional stability.

[1074] Input: Real-time video and audio data of applicant

[1075] Output: Evaluation scores for technical skills, communication skills, etc.

[1076] Step 6:

[1077] The server stores the generated evaluation results in a database and sends them to the terminal, which displays the evaluation results of the first selection and the interview recording so that the user (human resources staff) can check them.

[1078] Input: Rating score

[1079] Output: Evaluation results of the first selection and recorded data displayed on the terminal

[1080] Step 7:

[1081] The user (human resources officer) checks the evaluation results of the first selection on the terminal and conducts the final selection. The final selection results are entered into the terminal and sent to the server. Based on the final selection results, the server notifies the job seeker whether they have been hired or not.

[1082] Input: Final selection results by user (HR person)

[1083] Output: Notification of job offer to job seeker

[1084] (Application example 2)

[1085] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1086] Existing interview systems require preparation for interviewers, arrangement of interview venues, and adjustments to maintain the quality and efficiency of interviews, which leads to inefficient use of resources. Existing systems also make it difficult to accurately grasp applicants' emotions and reactions during interviews, making objective evaluations difficult. Furthermore, there is no system in place that can monitor and analyze the work status of robots in factories and the emotional health of workers in real time and provide immediate feedback, making quality control and work efficiency improvements a challenge.

[1087] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the job description and job information, means for automatically generating interview questions based on the received job description and job information, means for displaying the interview questions and recording the applicant's voice and video, means for analyzing the recorded voice and video in real time using an emotion analysis engine and generating an evaluation, means for outputting the initial selection evaluation, and means for issuing a warning if a certain evaluation standard is not met based on the analysis results. This enables more efficient interview preparation, accurate understanding of the applicant's emotions and reactions, and real-time monitoring of quality control and the emotional health of workers in factories.

[1088] A "job description" is a document that contains information about a job, such as job duties, required skills, and years of experience.

[1089] "Job information" is data that includes detailed information related to a particular job, such as job responsibilities, required qualifications, and location.

[1090] "Interview questions" are questions asked of applicants to assess their aptitude and skills.

[1091] "Audio and video" refers to digital data used to record the voices, facial expressions, and movements of applicants and workers.

[1092] An "emotion analysis engine" is an algorithm and software that analyzes data such as audio and video to evaluate and determine the emotional state of a subject.

[1093] "First selection" refers to the initial selection process, which is primarily a procedure for evaluating the applicant's basic aptitude and skills.

[1094] "Metrics" are standards or guidelines established for evaluating data or actions.

[1095] "Means for issuing warnings" refers to the ability to send alerts or notifications when evaluation results do not meet set standards.

[1096] To implement this invention, three entities - a server, a terminal, and a user - work together, and an emotion analysis engine is integrated to determine the user's emotion. The specific roles and operations of each entity, as well as how the emotion analysis engine is integrated, are explained below.

[1097] Overall structure

[1098] Hardware and Software Configuration

[1099] Hardware:

[1100] Device: User (HR personnel and job seeker) operation device (PC or tablet)

[1101] Server: Server for data analysis and automatic generation of interview questions

[1102] Factory robots: robots with built-in cameras and microphones

[1103] software:

[1104] Server side: Sentiment analysis AI using Python / TensorFlow / Keras

[1105] Frontend: User interface with React.js

[1106] Database: MySQL or PostgreSQL

[1107] Communication: Real-time communication via WebSocket

[1108] System Overview

[1109] 1. Enter and save the recruitment information

[1110] First, the user (human resources staff) uses a terminal to enter the job requirements and job information. The entered information is sent to the server and stored in the database. This procedure allows the user to easily register the necessary requirements.

[1111] 2. Automatic generation of interview scenarios

[1112] After the server receives and stores the job description and job information, the generative AI model automatically generates interview questions based on this information. For example, a request for "Java experience" generates a question such as "Tell us about a Java project you recently worked on."

[1113] 3. Notification of interview schedule

[1114] The generated interview scenario is sent from the server to the terminal, and the user (job seeker) is notified that "the interview is ready." This allows job seekers to start an interview at any time, 24 hours a day.

[1115] 4. Virtual Human AI Interviews

[1116] When a user (job seeker) starts an interview on their device, it communicates with the server and activates the virtual human AI, which sequentially displays pre-generated interview questions, and the recorded audio and video are sent to the server in real time.

[1117] 5. First selection and evaluation

[1118] The server analyzes the recorded audio and video. This analysis uses emotion analysis AI with TensorFlow and Keras. For example, it can recognize an applicant's nervousness or confidence from audio data, and emotions such as joy or surprise from video data. This generates scores for such things as "technical skills," "communication ability," "adaptability," and "emotional stability."

[1119] 6. Notification of evaluation results and feedback

[1120] The generated evaluation results are sent to the terminal and notified to the user (human resources personnel). Furthermore, if the analysis results do not meet certain evaluation standards, a warning is issued. This feedback loop improves the quality and efficiency of interviews.

[1121] Examples of specific examples and prompts

[1122] For example, when a worker is performing "assembly work" in a factory, a factory robot monitors the worker's movements in real time and sends the data collected by a camera and microphone to an emotion analysis engine. Based on the analysis results, if the worker is feeling excessively stressed, a warning will be issued and appropriate feedback will be provided.

[1123] Prompt Sentence Examples

[1124] plain

[1125] "Analyze video and audio data of workers as they assemble parts, and analyze emotions such as stress and satisfaction in real time."

[1126] This will enable quality control in factories and improve the emotional well-being of workers.

[1127] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1128] Step 1: Enter and save job description and job information

[1129] The user (human resources officer) uses a terminal to input the job requirements and job information. The input information is sent from the terminal to the server and saved in a database. The input data includes the job type, required skills, years of experience, etc. The server then sends a notification to the terminal that the data has been saved to the database.

[1130] Input: Job requirements and job information

[1131] Output: Information saved in the database, notification of save completion

[1132] Step 2: Auto-generate interview questions

[1133] The server uses a generative AI model to automatically generate interview questions based on the saved job description and job information. For example, if Java experience is required, the generative model generates questions such as, "Tell me about a Java project you recently worked on."

[1134] Input: Saved job postings and job information

[1135] Output: Auto-generated interview questions

[1136] Step 3: Notification of interview arrangement

[1137] The automatically generated interview questions are sent from the server to the terminal, and the terminal notifies the user (job seeker) that "the interview is ready." This notification allows the job seeker to prepare for the interview and start the interview at any time, 24 hours a day.

[1138] Input: Auto-generated interview questions

[1139] Output: Interview preparation notice

[1140] Step 4: Virtual Human AI Interview

[1141] When a user (job seeker) starts an interview through their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays the generated interview questions and records the applicant's voice and video. The audio and video data is sent to the server in real time.

[1142] Input: Interview questions, audio and video of job seeker

[1143] Output: Recorded audio and video data

[1144] Step 5: First selection and evaluation

[1145] The server analyzes the received audio and video data in real time. It uses an emotion analysis engine based on TensorFlow and Keras to recognize the applicant's nervousness or confidence from the audio data, and emotions such as joy or surprise from the video data. This generates an evaluation score based on factors such as technical skills, communication ability, adaptability, and emotional stability.

[1146] Input: Recorded audio and video data

[1147] Output: Evaluation score

[1148] Step 6: Notification of evaluation results and feedback

[1149] The server sends the generated evaluation score to the terminal and notifies the user (human resources officer). Furthermore, if the analysis results indicate that a certain evaluation standard has not been met, a warning is issued. At this step, feedback based on the evaluation results is provided, improving the quality and efficiency of the interview.

[1150] Input: Rating score

[1151] Output: Evaluation result notification, warning notification

[1152] Step 7: Emotion analysis and quality control for factory robots

[1153] In factories, robots use cameras and microphones to monitor the movements and emotions of workers in real time, and send the collected data to a server, which uses an emotion analysis engine to analyze the worker's stress, satisfaction, and other factors, and issue warnings as necessary.

[1154] Input: Video and audio data of the worker

[1155] Output: Sentiment analysis data, quality control assessment, warning notifications

[1156] The above processing steps enable efficient and objective evaluation, which is expected to have a significant effect on personnel and factory quality control.

[1157] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1158] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1159] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1160] [Fourth embodiment]

[1161] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1162] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1163] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1164] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1165] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1166] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1167] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1168] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1169] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1170] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1171] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1172] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1173] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1174] To implement the present invention, three main entities, the server, the terminal, and the user, must work together. The specific roles and operations of each entity are described below.

[1175] Enter and save the recruitment information

[1176] First, the user (human resources officer) uses a terminal to input the job requirements and job information. This information includes the job type, required skills, years of experience, etc. Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a notification to the terminal that the information has been saved.

[1177] Automatic generation of interview scenarios

[1178] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, if the job description mentions "Java experience," the model generates questions such as "Tell us about a Java project you recently worked on."

[1179] Interview schedule notification

[1180] The automatically generated interview scenario is sent from the server to the terminal. The terminal then notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time, 24 hours a day.

[1181] Virtual Human AI Interview

[1182] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[1183] First selection and evaluation

[1184] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. For example, the user's (job seeker's) voice is used to evaluate the appropriateness of their responses to questions and their ability to accurately understand the other person's intentions. Meanwhile, facial expression analysis can detect emotions such as nervousness or confidence.

[1185] As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," etc. The generated evaluation results are sent to the terminal and notified to the user (human resources officer).

[1186] Final selection and hiring decision

[1187] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made on whether to hire the candidate. The results of the final selection are entered into the terminal and sent to the server. The server then automatically generates and sends a notification to the job seeker who has been selected for employment.

[1188] Specific examples

[1189] For example, if a company posts a job posting for a software engineer, the user (human resources officer) enters the requirements "Java project experience" and "more than three years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time, and an initial selection evaluation is conducted. Finally, the human resources officer conducts a final selection based on the evaluation results and the recording, and a hiring decision is made.

[1190] This system eliminates the need to arrange interviewers and interview venues, improving the quality and efficiency of interviews, reducing stress for job seekers, and allowing for flexible interview schedules.

[1191] The processing flow will be explained below.

[1192] Step 1: Enter the job description

[1193] The user enters the job description and job information into the terminal.

[1194] The terminal transmits the input data to the server.

[1195] Step 2: Save the job description

[1196] The server stores the received job posting and job information in a database.

[1197] The server sends a notification to the terminal that "recruitment requirements have been saved."

[1198] Step 3: Auto-generate interview questions

[1199] An AI model on the server generates an appropriate set of interview questions based on stored job descriptions and job information.

[1200] Step 4: Communicate the interview scenario

[1201] The server transmits the generated interview scenario to the terminal.

[1202] The terminal notifies the user (job seeker) that "You are ready for the interview."

[1203] Step 5: Start the interview

[1204] The user (job seeker) presses the interview start button on the terminal.

[1205] The device communicates with the server and activates the virtual human AI.

[1206] Step 6: Presenting the interview questions

[1207] The server controls the virtual human AI and sequentially displays pre-generated interview questions.

[1208] The terminal records the user's (job seeker's) answers.

[1209] Step 7: Recording audio and video

[1210] The terminal records the user's (job seeker's) voice and video in real time.

[1211] The recorded data is sent to the server.

[1212] Step 8: Analyze the data

[1213] The server analyzes the recorded audio and video.

[1214] Voice and facial expression analysis is performed to generate an evaluation for the first selection.

[1215] Step 9: Output of first selection results

[1216] The server sends the evaluation results of the first selection to the terminal.

[1217] The terminal notifies the user (human resources officer).

[1218] Step 10: Final selection

[1219] The user (human resources officer) checks the initial selection results and interview recording on the terminal.

[1220] The results of the final selection are entered into the terminal and sent to the server.

[1221] Step 11: Notification of hiring decision

[1222] The server will decide whether to hire the candidate based on the final selection results.

[1223] The server sends a job offer to the job seeker.

[1224] Example 1

[1225] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1226] In the traditional recruitment process, it is necessary to arrange interviewers and interview venues, which is time-consuming and costly. In addition, it is difficult for job seekers to flexibly schedule interviews, which often increases tension. There is a need to solve these problems and improve the quality and efficiency of interviews.

[1227] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1228] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions using a generative AI model based on the received job requirements and job information, means for displaying the automatically generated interview questions and recording audio and video of applicants, means for analyzing the recorded audio and video data in real time to generate an evaluation score for the first selection, and means for outputting the generated evaluation score. This eliminates the need to arrange interviewers and interview venues, provides job seekers with flexible interview schedules, and makes it possible to improve the quality and efficiency of interviews.

[1229] A "recruitment requirement" is a document that lists the job description and application conditions that a company presents when recruiting new employees.

[1230] "Job Information" means information that details the skills, experience, and other requirements needed for a particular job.

[1231] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate interview questions and other information based on specific rules and patterns.

[1232] "Interview questions" are questions that companies ask job seekers to assess their skills, experience, and aptitude.

[1233] "Audio and video recording means" refers to devices and software that store audio and video recordings of job seekers as digital data.

[1234] "Means for real-time analysis" refers to technology for instantly analyzing and processing recorded data and generating an evaluation.

[1235] The "first selection evaluation score" is a numerical value obtained by evaluating the skills and aptitude of a job seeker, and is used as the result of the first selection process.

[1236] "Facial expression analysis" is a technology that reads the emotions and reactions of job applicants from their facial expressions and reflects this in their evaluation.

[1237] "Voice analysis" is a technology that analyzes the speaking style and content of a job seeker's voice, and uses this information to evaluate their skills and aptitude.

[1238] "Output means" means a device or software for displaying or reporting the generated evaluation scores or other analytical results.

[1239] To implement the invention, three main entities must work together: the server, the terminal, and the user. Below we will explain in detail how these entities work together to implement the invention.

[1240] Enter and save job postings and job information

[1241] First, the user (human resources staff) uses a terminal to input the job requirements and job information. The terminal can be a general computer device such as a PC or tablet. The user inputs information such as "Java project experience" or "3+ years of development experience." Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a notification to the terminal that the information has been saved.

[1242] Automatic generation of interview questions

[1243] The server uses a generative AI model to automatically generate interview questions based on the received job description and job information. The generative AI model uses dedicated software equipped with machine learning algorithms. For example, if the job description mentions "Java experience," the generative AI model automatically generates specific questions such as "Tell us about a Java project you recently worked on." These questions are stored in a database on the server.

[1244] Notification and preparation of interview scenarios

[1245] The automatically generated interview scenario is sent from the server to the device. Based on this, the device sends the user (job seeker) a notification that "the interview is ready." This notification is sent via email or push notification. After receiving the notification, the job seeker can access the interview system and begin the interview within 24 hours.

[1246] Virtual Human AI Interview

[1247] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, which the job seeker answers. At this time, the job seeker's answers are recorded as audio and video on the device and sent to the server in real time. The virtual human AI uses advanced artificial intelligence technology to enable natural dialogue.

[1248] Real-time analysis of responses and initial selection evaluation

[1249] The server analyzes the audio and video data it receives in real time. This analysis includes voice recognition and facial expression analysis technologies. For example, voice analysis evaluates the content of responses and speaking style, while facial expression analysis reads emotions such as nervousness or confidence. Based on the analysis results, the server generates a first-stage selection evaluation score, including "technical skills," "communication ability," and "adaptability." This evaluation score is sent to a terminal as digital data and notified to the user (human resources officer).

[1250] Final selection and hiring decision

[1251] The user (human resources staff) receives the evaluation results via the device and checks the evaluation scores from the first selection and the interview recording data. Based on this information, the final selection is made and the decision on whether to hire is entered into the device. The server then automatically generates and sends a hiring decision notice to the job seeker based on the final selection results. This notification is also sent via email or push notification.

[1252] Specific examples

[1253] For example, if a company posts a job posting for a software engineer, the user (human resources officer) enters the requirements "Java project experience" and "3+ years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The generated questions include a specific prompt, such as "Tell us about a Java project you recently worked on." The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time to evaluate the candidate for the first selection stage. Finally, the human resources officer conducts a final selection based on the evaluation results and recording, and the candidate is hired. This system eliminates the need to arrange interviewers and interview venues, improving the quality and efficiency of interviews. It also reduces stress for job seekers and allows for flexible interview schedules.

[1254] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1255] The flow of this system's program processing

[1256] Step 1:

[1257] The user (human resources staff) uses a terminal to input the job requirements and job information. For example, they input specific items such as "Java project experience" or "3+ years of development experience." The job type, required skills, years of experience, etc. are input into the terminal as input data. The output is the entered job requirements and job information.

[1258] Step 2:

[1259] The terminal sends the job description and job information data entered by the user to the server. The secure HTTP(S) protocol is used to send the data. The input is the job description and job information entered by the user into the terminal, and the output is the data sent to the server.

[1260] Step 3:

[1261] The server saves the received data in a database. After saving is complete, the server notifies the terminal that "data saving is complete." The input is the job description and job information data received by the server, and the output is a notification that saving is complete.

[1262] Step 4:

[1263] The server uses a generative AI model to automatically generate interview questions based on the saved job description and job information. For example, it generates specific questions such as "Tell me about a Java project you recently worked on." The generative AI model analyzes the received data and generates prompts. The input is the job description and job information saved on the server, and the output is a set of interview questions.

[1264] Step 5:

[1265] The server sends automatically generated interview questions to the terminal. The input is the interview questions generated by the server, and the output is the set of questions sent to the terminal.

[1266] Step 6:

[1267] Based on the interview questions received, the device sends a notification to the user (job seeker) saying "Your interview is ready." This notification is sent via email or push notification. The input is a set of interview questions from the server, and the output is a notification to the job seeker.

[1268] Step 7:

[1269] After receiving the notification, the user (job seeker) clicks a button on their device to start the interview. The input is the notification, and the output is the start of the interview.

[1270] Step 8:

[1271] The terminal communicates with the server and starts the virtual human AI, which causes the virtual human to appear on the screen. The input is the start of the interview, and the output is the start of the virtual human AI.

[1272] Step 9:

[1273] The virtual human AI sequentially presents interview questions received from the server to the user. The user's answers are recorded as audio and video on the device and sent to the server in real time. The input is the set of questions from the server, and the output is the recorded audio and video data.

[1274] Step 10:

[1275] The server analyzes the received audio and video data in real time. This analysis uses voice recognition and facial expression analysis technologies. For example, it can detect a user's level of tension from their tone of voice and facial expression. The input is audio and video data, and the output is the analysis results.

[1276] Step 11:

[1277] Based on the analysis results, the server generates a first-stage selection evaluation score for "technical skills," "communication skills," "adaptability," etc. The input is the analysis results, and the output is the first-stage selection evaluation score.

[1278] Step 12:

[1279] The server sends the generated evaluation score to the terminal, and the terminal notifies the user (human resources officer). The input is the evaluation score, and the output is the notification.

[1280] Step 13:

[1281] The user (human resources staff) checks the evaluation scores from the first selection and the interview recording data. Based on this information, the final selection is made and the decision on whether to hire is entered into the terminal. The input is the evaluation scores and the recording data, and the output is the decision on whether to hire.

[1282] Step 14:

[1283] The server receives the final selection results from the device and automatically generates and sends a hiring decision notice to the job seeker. This notification is also sent via email or push notification. The input is the final selection results, and the output is the hiring decision notice.

[1284] (Application example 1)

[1285] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] Conventional interview systems and shopping assistant systems face the problem of difficulty in collecting and evaluating appropriate information from users. Interviews, in particular, require the arrangement of interviewers and interview venues, which is inefficient. Furthermore, when it comes to shopping, it is difficult to recommend products that match the preferences of individual buyers. Furthermore, there is no system that can analyze user responses in real time and change responses based on that analysis, which makes it difficult to improve the user experience.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1288] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions based on the received job requirements and job information, means for displaying interview questions and recording audio and video of applicants, means for analyzing the recorded audio and video and generating an evaluation of the initial selection, means for outputting the evaluation of the initial selection, means for receiving behavioral information of purchasers, means for automatically generating product recommendations based on the received behavioral information, means for displaying product recommendations and recording purchaser responses, means for analyzing the recorded materials and re-evaluating the product recommendations, and means for generating evaluations and updating the display in real time. This enables improved efficiency and quality of interviews, appropriate product recommendations for individual purchasers, and optimization of the user experience in real time.

[1289] "Job Description and Job Information" means detailed information about a job opening, including the skills and years of experience required for a particular position.

[1290] The "means for automatically generating interview questions" refers to a function that enables an AI model to automatically generate questions to be used in interviews based on the job description and job information.

[1291] "Means for displaying interview questions and recording audio and video of applicants" refers to a device or application that displays automatically generated interview questions to an applicant and records audio and video of the applicant's responses.

[1292] "Means for analyzing recorded audio and video and generating an evaluation for the first selection" refers to a system that analyzes recorded audio and video data of applicants and generates a numerical or textual evaluation for the first selection based on that data.

[1293] The "means for outputting the evaluation of the first selection" has the function of notifying or displaying the generated evaluation results of the first selection to the user.

[1294] "Means for receiving customer behavior information" refers to a device or application that collects customer behavior on a store or online platform (e.g., browsing history, purchase history, click behavior, etc.) and sends it to a server.

[1295] "Means for automatically generating product recommendations based on received behavioral information" refers to a system in which an AI model automatically recommends products suitable for a purchaser based on collected behavioral information of the purchaser.

[1296] "Means for displaying product recommendations and recording buyer responses" refers to a device or application that displays recommended products to a buyer and records the buyer's responses (e.g., audio, video, interaction) at that time.

[1297] The "means for analyzing recorded data and reevaluating product recommendations" is a system that analyzes buyer responses, reviews product recommendations based on the results, and makes new recommendations as necessary.

[1298] The "means for generating ratings in real time and updating the display" is a function that analyzes buyer response data in real time and instantly updates the display content based on the results.

[1299] To implement the present invention, three main entities, the server, the terminal, and the user, must work together. The specific roles and operations of each entity are described below.

[1300] Enter and save the recruitment information

[1301] The server first assists the user (human resources staff) in the process of entering job requirements and job information using a terminal. This information includes job type, required skills, years of experience, etc. Once the information has been entered, the terminal sends this information to the server, which then stores the received information in a database. The server then notifies the terminal that the information has been saved.

[1302] Automatic generation of interview scenarios

[1303] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, if the job description mentions "Java experience," the model generates questions such as "Tell us about a Java project you recently worked on."

[1304] Interview schedule notification

[1305] The automatically generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time, 24 hours a day.

[1306] Virtual Human AI Interview

[1307] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[1308] First selection and evaluation

[1309] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. For example, the user's (job seeker's) voice is used to evaluate the appropriate response to questions and the ability to accurately understand the other person's intentions. Facial expression analysis can detect emotions such as nervousness and confidence. As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," and other areas. The generated evaluation results are sent to a terminal and notified to the user (human resources officer).

[1310] Product recommendations and real-time ratings

[1311] The server has a means for receiving buyer behavioral information and a means for automatically generating product recommendations based on the received behavioral information. When a buyer uses a smartphone app to input behavioral information (browsing history, purchase history, click behavior, etc.), the server analyzes it and uses an AI model to recommend products. It also displays product recommendations and records the buyer's responses (audio and video). The recorded data is analyzed to reevaluate the product recommendations, and the display is updated in real time.

[1312] Examples of specific examples and prompts

[1313] A concrete example is a scenario for a smart shopping assistant app. A user launches the smartphone app and enters their profile (preferences and albums). The shopping assistant starts recommending products and displays a message saying, "AI has prepared some recommended products for you." When the user asks the virtual assistant, "How do I use this product?", the virtual human AI provides detailed explanations, saying, "This product can also be used for commercial purposes, allowing you to cook more efficiently." The process continues by analyzing the user's facial expressions and voice, and further customizing the product.

[1314] Example prompt sentence:

[1315] 1. [User]: What are your recommended fashion items for summer?

[1316] 2. [Shopping Assistant]: Looking for recommended fashion items.

[1317] 3. [AI Model]: Recommends the following items based on the user's profile and purchase history:

[1318] 1. Summer dress

[1319] 2. Sunglasses

[1320] 3. Flip-flops

[1321] 4. [User]: How do I use a summer dress?

[1322] 5. [Virtual Human AI]: Summer dresses are versatile and can be worn for casual everyday wear or beach photoshoots.

[1323] In this way, cooperation between the server, terminal, and user allows both the interview and shopping systems to operate effectively, improving the user experience.

[1324] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1325] Step 1:

[1326] A user (human resources officer) uses a terminal to input job requirements and job information. The input includes details such as job type, required skills, years of experience, etc. The input data is sent from the terminal to the server, which stores it in a database. The input data is processed and converted into a structured format and stored appropriately in the database.

[1327] Step 2:

[1328] The server automatically generates interview questions using an AI model based on the saved job description and job information. The AI ​​model uses a generative AI model to create appropriate questions based on past data and algorithmic prompts. This question dataset is saved on the server and structured as an interview scenario. The generated questions are saved on the server as structured data.

[1329] Step 3:

[1330] The server sends the automatically generated interview scenario to the terminal. The terminal displays a notification to the user (job seeker) informing them that "the interview is ready." Once this notification is displayed, the user (job seeker) can start the interview within 24 hours.

[1331] Step 4:

[1332] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) records their answers as audio and video. The recorded data is sent to the server in real time.

[1333] Step 5:

[1334] The server analyzes the recorded audio and video in real time. The analysis involves using speech recognition APIs such as Google Cloud Speech-to-Text and Microsoft Azure Face API to perform voice and facial expression analysis. Data analysis confirms appropriate responses to questions from the voice and evaluates emotions such as nervousness or confidence from facial expressions. The analysis results are generated as a score or evaluation and stored on the server.

[1335] Step 6:

[1336] The evaluation data generated as a result of the analysis is sent from the server to the terminal and notified to the user (human resources officer). The terminal displays the evaluation results in an easy-to-read format, including the score and feedback for each evaluation item.

[1337] Step 7:

[1338] When the user (job seeker) finishes the interview, the terminal finalizes the recorded data and stores it on the server, which can be used as a reference later.

[1339] Step 8:

[1340] The user (human resources officer) will conduct the final selection based on the evaluation results and the interview recording. This selection result is also entered into the server from the terminal and saved as the final pass / fail data. The server will generate a result notification based on this data and send it to the job seeker.

[1341] Step 9:

[1342] The server operates a means for receiving purchaser behavioral information and uses a smartphone app to collect behavioral information (browsing history, purchase history, click behavior, etc.) The collected data is sent back to the server and used for analysis.

[1343] Step 10:

[1344] The server uses a generative AI model based on the received behavioral information to automatically generate product recommendations. The generated recommendations are displayed to the buyer via a smartphone app. The buyer then records their reactions to the recommended products via audio and video.

[1345] Step 11:

[1346] The server analyzes the recorded responses and re-evaluates the product recommendations. The server generates ratings and updates the recommendations in real time, resulting in more accurate recommendations. Real-time display updates improve the user experience.

[1347] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1348] To implement this invention, three entities, the server, the terminal, and the user, work together, and an emotion engine is combined to recognize the user's emotions. The specific roles and operations of each entity, as well as how the emotion engine is integrated, are explained below.

[1349] Enter and save the recruitment information

[1350] First, the user (human resources officer) uses a terminal to input the job requirements and job information. For example, this includes items such as "job type," "required skills," and "years of experience." Once the input is complete, the terminal sends this information to the server. The server saves the received information in a database and sends a save completion notification to the terminal.

[1351] Automatic generation of interview scenarios

[1352] The server receives and stores the job description and job information, and then the AI ​​model automatically generates interview questions based on this information. For example, for a requirement such as "Java experience," it generates questions such as "Tell us about a Java project you recently worked on."

[1353] Interview schedule notification

[1354] The generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." Upon receiving this notification, the user (job seeker) can start the interview at any time, 24 hours a day.

[1355] Virtual Human AI Interview

[1356] When a user (job seeker) starts an interview on their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers them. The answers are recorded as audio and video on the device and sent to the server in real time.

[1357] First selection and evaluation

[1358] The server analyzes the recorded audio and video in real time. This analysis includes voice analysis and facial expression analysis. The emotion engine also uses this data to analyze the user's (job seeker's) emotions. For example, voice data can identify an applicant's nervousness or confidence, while video data can identify emotions such as joy or surprise.

[1359] As a result of the analysis, the server generates scores for "technical skills," "communication skills," "adaptability," "emotional stability," etc. The generated evaluation results are sent to the terminal and notified to the user (human resources officer).

[1360] Final selection and hiring decision

[1361] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made as to whether or not to hire the candidate. The final selection results are entered into the terminal and sent to the server. The server then decides whether or not to hire the candidate based on the final selection results and sends a notification to the job seeker.

[1362] Specific examples

[1363] For example, if a company posts a job posting for a "software engineer wanted," the user (human resources officer) would enter the requirements "Java project experience" and "more than three years of development experience" into their terminal. The server receives this information and generates a set of interview questions. The job seeker then uses a 24-hour interview system to conduct an interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time using an emotion engine, and an initial selection evaluation is made. Finally, the human resources officer conducts a final selection based on the evaluation results and the recording, and a hiring decision is made.

[1364] This system eliminates the need to arrange interviewers and interview venues, significantly improves the quality and efficiency of interviews, and also enables more accurate understanding of applicants' true reactions and feelings through emotion analysis.

[1365] The processing flow will be explained below.

[1366] Step 1: Enter the job description

[1367] The user (human resources officer) enters the job requirements and job information into the terminal.

[1368] The terminal transmits the input data to the server.

[1369] Step 2: Save the job description

[1370] The server stores the received job posting and job information in a database.

[1371] The server sends a notification to the terminal that "recruitment requirements have been saved."

[1372] Step 3: Auto-generate interview questions

[1373] An AI model on the server generates an appropriate set of interview questions based on stored job descriptions and job information.

[1374] For example, if the job description states "Java experience," it will generate a question such as "Tell us about a Java project you've recently worked on."

[1375] Step 4: Communicate the interview scenario

[1376] The server transmits the generated interview scenario to the terminal.

[1377] The terminal notifies the user (job seeker) that "You are ready for the interview."

[1378] Step 5: Start the interview

[1379] The user (job seeker) presses the interview start button on the terminal.

[1380] The device communicates with the server and activates the virtual human AI.

[1381] Step 6: Presenting the interview questions

[1382] The server controls the virtual human AI and sequentially displays pre-generated interview questions.

[1383] The terminal records the user's (job seeker's) answers.

[1384] Step 7: Recording audio and video

[1385] The terminal records the user's (job seeker's) voice and video in real time.

[1386] The recorded data is sent to the server.

[1387] Step 8: Analyze the data

[1388] The server analyzes the recorded audio and video.

[1389] Voice analysis, facial expression analysis, and emotion recognition using an emotion engine are performed to generate an evaluation for the first selection round.

[1390] For example, the emotion engine analyzes the user's (job seeker's) nervousness and confidence from voice, and facial expression data such as smiles and surprise from video.

[1391] Step 9: Output of first selection results

[1392] The server generates the evaluation results of the first selection and sends them to the terminal.

[1393] The terminal displays the "first selection results" to the user (human resources officer).

[1394] Step 10: Final selection

[1395] The user (human resources officer) checks the initial selection results and interview recording on the terminal.

[1396] Human resources personnel will conduct the final selection based on the interview recording and evaluation results, and enter the final selection results into the terminal.

[1397] Step 11: Notification of hiring decision

[1398] The terminal transmits the final selection results to the server.

[1399] The server determines whether or not to hire the job seeker based on the final selection results and sends a notice of employment to the job seeker.

[1400] Through these steps, the server, terminal, user, and emotion engine work together to realize an efficient and sophisticated interview process. This system not only improves the quality and efficiency of interviews, but also enables a deeper understanding of the applicant's true suitability through emotion analysis.

[1401] Example 2

[1402] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1403] Traditional hiring processes require arranging interviewers and interview locations, which is time-consuming and costly. Furthermore, interviewers' subjective assessments of applicants can be inconsistent. Furthermore, it is difficult to accurately grasp applicants' emotions and true reactions. Therefore, there is a need for a system that can improve the efficiency and accuracy of the hiring process.

[1404] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1405] In this invention, the server includes means for receiving job requirements and job information, means for automatically generating interview questions based on the received job requirements and job information, means for displaying the interview questions and recording the applicant's voice and video, means for analyzing the recorded voice and video and generating an evaluation of the first selection, means for analyzing the applicant's emotions based on the analyzed content, and means for outputting the evaluation of the first selection. This eliminates the need to arrange interviewers and interview venues, improves the quality and efficiency of interviews, and makes it possible to more accurately grasp the applicant's true reactions and emotions.

[1406] "Recruitment requirements" are information about the job, specifically including the job type, required skills, years of experience, etc.

[1407] "Job information" is detailed information about a specific job, specifically including job content, roles, scope of responsibilities, and the like.

[1408] The "receiving means" refers to the technical means by which the server receives information input by the user, and includes a communication interface, a data receiving module, and the like.

[1409] "Automatic generation means" refers to means for automatically creating new content based on specified information using technologies such as AI models.

[1410] The "display means" is a device such as a display or monitor for visually presenting the generated questions and information to the user.

[1411] "Recording means" refers to technical means for saving data such as audio and video, and includes, for example, cameras, microphones, and recording devices.

[1412] "Means for analyzing" refers to technical means for analyzing data such as audio and video to derive specific identification information or scores, and includes voice analysis algorithms and facial expression analysis algorithms.

[1413] "Means for analyzing emotions" refers to technical means for identifying an applicant's emotions from audio or video, including emotion engines and machine learning models.

[1414] The "means for generating an evaluation" is a means for generating a score or evaluation based on specific criteria based on the analysis results.

[1415] "Means for outputting evaluation" refers to technical means for communicating the generated evaluation results to the user, including, for example, a screen display or a notification system.

[1416] To implement this invention, three entities - the server, the terminal, and the user - must work together, and an emotion engine must be combined to recognize the user's emotions. The specific roles and operations of each entity, as well as how the emotion engine is integrated, are explained below.

[1417] Enter and save the recruitment information

[1418] First, the user (human resources officer) uses a terminal to input the job requirements and job information. The input fields include "job type," "required skills," and "years of experience." Once the user has completed their input, the terminal sends this information to the server. The server saves the received information in a database, and once the saving is complete, it sends a save completion notification to the terminal. This ensures that the information is stored securely.

[1419] Automatic generation of interview scenarios

[1420] The server passes the saved job description and job information to the AI ​​model. A generative AI model, such as TensorFlow or PyTorch, is used to automatically generate interview questions appropriate to the requirements. For example, for the requirement "Java experience," a question such as "Tell us about a Java project you recently worked on" is generated. The server then stores the generated questions in a database.

[1421] Interview schedule notification

[1422] The generated interview scenario is sent from the server to the terminal. The terminal notifies the user (job seeker) that "the interview is ready." The user (job seeker) who receives this notification can start the interview at any time within 24 hours.

[1423] Virtual Human AI Interview

[1424] When a user (job seeker) starts an interview on their device, the device sends a request to start the interview to the server. The server then starts a virtual human AI and displays interview questions in sequence. The user (job seeker) responds via voice and video, and the data is recorded on the device and sent to the server in real time. Core software used for recording the audio and video includes OpenCV and FFmpeg.

[1425] First selection and evaluation

[1426] The server analyzes the received audio and video data. This analysis includes voice and facial expression analysis, using libraries such as Librosa and Dlib. An emotion engine also uses this data to analyze the user's (job seeker's) emotions. For example, the voice data can identify an applicant's nervousness or confidence, while the video data can identify emotions such as joy or surprise. As a result of the analysis, the server generates scores for such criteria as "technical skills," "communication ability," "adaptability," and "emotional stability." The generated evaluation results are sent to the terminal and notified to the user (human resources staff).

[1427] Final selection and hiring decision

[1428] The user (human resources officer) receives the notification via the terminal and checks the evaluation results of the first selection and the video recording of the interview. Based on this information, the final selection is conducted and a decision is made on whether to hire the candidate. The final selection results are entered into the terminal and sent to the server. The server then sends a notification to the job seeker based on the final selection results.

[1429] Specific examples

[1430] For example, if a company posts a job posting for a software engineer, the user (a human resources officer) inputs the following requirements into the terminal: "Java project experience" and "3 or more years of development experience." The server receives this information and generates a set of interview questions.

[1431] Examples of specific prompts include:

[1432] "Generate interview questions suitable for candidates with at least 3 years of Java project experience. For example, questions about recent projects they worked on."

[1433] Interviews are conducted based on this information. Job seekers use a 24-hour interview system to interview with a virtual human AI. The audio and video recorded during the interview are analyzed in real time using an emotion engine, and an initial evaluation is conducted. Finally, a human resources staff member conducts the final selection based on the evaluation results and recording, and the candidate is hired. This system eliminates the need to arrange interviewers and interview venues, significantly improving the quality and efficiency of interviews, and also makes it possible to more accurately grasp the applicant's true reactions and emotions through emotion analysis.

[1434] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1435] Step 1:

[1436] The user (human resources staff) uses a terminal to enter job requirements and job information. Specifically, they enter information such as "job type," "required skills," and "years of experience" into the terminal's input form. This information is sent to the server through the terminal's user interface. Based on the input data, the terminal sends JSON format data to the server, which then stores it in a database on the server side.

[1437] Input: Job posting and job information entered on the terminal

[1438] Output: Information sent to the server and stored in the database

[1439] Step 2:

[1440] The server passes the received job description and job information to the AI ​​model. During this process, the server sends the information to the generative AI model as prompts. For example, for a requirement such as "Java project experience," a question such as "Tell us about a Java project you recently worked on" is generated. The AI ​​model generates questions using the TensorFlow and PyTorch libraries.

[1441] Input: Job postings and job information stored in the database

[1442] Output: Generated interview questions

[1443] Step 3:

[1444] The server saves the generated interview questions in a database. Once saving is complete, the server sends the interview questions to the terminal, and the terminal notifies the user (job seeker) that "the interview is ready."

[1445] Input: Generated interview questions

[1446] Output: Notification of interview settings to the terminal

[1447] Step 4:

[1448] When a user (job seeker) starts an interview on their device, the device sends a request to start the interview to the server. The server launches a virtual human AI and connects it to the device. The virtual human AI sequentially displays pre-generated interview questions, and the user (job seeker) answers using audio and video. Specifically, the device uses a camera and microphone to record audio and video and transmits them to the server in real time. OpenCV and FFmpeg libraries are used for this.

[1449] Input: Interview start request from the terminal and applicant's audio and video data

[1450] Output: Real-time video and audio data transmission to the server

[1451] Step 5:

[1452] The server analyzes the received audio and video data. Using the Librosa and Dlib libraries, voice and facial analysis is performed. An emotion engine also analyzes the applicant's emotions, such as nervousness and confidence. The analysis generates scores for factors such as technical skills, communication skills, adaptability, and emotional stability.

[1453] Input: Real-time video and audio data of applicant

[1454] Output: Evaluation scores for technical skills, communication skills, etc.

[1455] Step 6:

[1456] The server stores the generated evaluation results in a database and sends them to the terminal, which displays the evaluation results of the first selection and the interview recording so that the user (human resources staff) can check them.

[1457] Input: Rating score

[1458] Output: Evaluation results of the first selection and recorded data displayed on the terminal

[1459] Step 7:

[1460] The user (human resources officer) checks the evaluation results of the first selection on the terminal and conducts the final selection. The final selection results are entered into the terminal and sent to the server. Based on the final selection results, the server notifies the job seeker whether they have been hired or not.

[1461] Input: Final selection results by user (HR person)

[1462] Output: Notification of job offer to job seeker

[1463] (Application example 2)

[1464] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1465] Existing interview systems require preparation for interviewers, arrangement of interview venues, and adjustments to maintain the quality and efficiency of interviews, which leads to inefficient use of resources. Existing systems also make it difficult to accurately grasp applicants' emotions and reactions during interviews, making objective evaluations difficult. Furthermore, there is no system in place that can monitor and analyze the work status of robots in factories and the emotional health of workers in real time and provide immediate feedback, making quality control and work efficiency improvements a challenge.

[1466] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the job description and job information, means for automatically generating interview questions based on the received job description and job information, means for displaying the interview questions and recording the applicant's voice and video, means for analyzing the recorded voice and video in real time using an emotion analysis engine and generating an evaluation, means for outputting the initial selection evaluation, and means for issuing a warning if a certain evaluation standard is not met based on the analysis results. This enables more efficient interview preparation, accurate understanding of the applicant's emotions and reactions, and real-time monitoring of quality control and the emotional health of workers in factories.

[1467] A "job description" is a document that contains information about a job, such as job duties, required skills, and years of experience.

[1468] "Job information" is data that includes detailed information related to a particular job, such as job responsibilities, required qualifications, and location.

[1469] "Interview questions" are questions asked of applicants to assess their aptitude and skills.

[1470] "Audio and video" refers to digital data used to record the voices, facial expressions, and movements of applicants and workers.

[1471] An "emotion analysis engine" is an algorithm and software that analyzes data such as audio and video to evaluate and determine the emotional state of a subject.

[1472] "First selection" refers to the initial selection process, which is primarily a procedure for evaluating the applicant's basic aptitude and skills.

[1473] "Metrics" are standards or guidelines established for evaluating data or actions.

[1474] "Means for issuing warnings" refers to the ability to send alerts or notifications when evaluation results do not meet set standards.

[1475] To implement this invention, three entities - a server, a terminal, and a user - work together, and an emotion analysis engine is integrated to determine the user's emotion. The specific roles and operations of each entity, as well as how the emotion analysis engine is integrated, are explained below.

[1476] Overall structure

[1477] Hardware and Software Configuration

[1478] Hardware:

[1479] Device: User (HR personnel and job seeker) operation device (PC or tablet)

[1480] Server: Server for data analysis and automatic generation of interview questions

[1481] Factory robots: robots with built-in cameras and microphones

[1482] software:

[1483] Server side: Sentiment analysis AI using Python / TensorFlow / Keras

[1484] Frontend: User interface with React.js

[1485] Database: MySQL or PostgreSQL

[1486] Communication: Real-time communication via WebSocket

[1487] System Overview

[1488] 1. Enter and save the recruitment information

[1489] First, the user (human resources staff) uses a terminal to enter the job requirements and job information. The entered information is sent to the server and stored in the database. This procedure allows the user to easily register the necessary requirements.

[1490] 2. Automatic generation of interview scenarios

[1491] After the server receives and stores the job description and job information, the generative AI model automatically generates interview questions based on this information. For example, a request for "Java experience" generates a question such as "Tell us about a Java project you recently worked on."

[1492] 3. Notification of interview schedule

[1493] The generated interview scenario is sent from the server to the terminal, and the user (job seeker) is notified that "the interview is ready." This allows job seekers to start an interview at any time, 24 hours a day.

[1494] 4. Virtual Human AI Interviews

[1495] When a user (job seeker) starts an interview on their device, it communicates with the server and activates the virtual human AI, which sequentially displays pre-generated interview questions, and the recorded audio and video are sent to the server in real time.

[1496] 5. First selection and evaluation

[1497] The server analyzes the recorded audio and video. This analysis uses emotion analysis AI with TensorFlow and Keras. For example, it can recognize an applicant's nervousness or confidence from audio data, and emotions such as joy or surprise from video data. This generates scores for such things as "technical skills," "communication ability," "adaptability," and "emotional stability."

[1498] 6. Notification of evaluation results and feedback

[1499] The generated evaluation results are sent to the terminal and notified to the user (human resources personnel). Furthermore, if the analysis results do not meet certain evaluation standards, a warning is issued. This feedback loop improves the quality and efficiency of interviews.

[1500] Examples of specific examples and prompts

[1501] For example, when a worker is performing "assembly work" in a factory, a factory robot monitors the worker's movements in real time and sends the data collected by a camera and microphone to an emotion analysis engine. Based on the analysis results, if the worker is feeling excessively stressed, a warning will be issued and appropriate feedback will be provided.

[1502] Prompt Sentence Examples

[1503] plain

[1504] "Analyze video and audio data of workers as they assemble parts, and analyze emotions such as stress and satisfaction in real time."

[1505] This will enable quality control in factories and improve the emotional well-being of workers.

[1506] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1507] Step 1: Enter and save job description and job information

[1508] The user (human resources officer) uses a terminal to input the job requirements and job information. The input information is sent from the terminal to the server and saved in a database. The input data includes the job type, required skills, years of experience, etc. The server then sends a notification to the terminal that the data has been saved to the database.

[1509] Input: Job requirements and job information

[1510] Output: Information saved in the database, notification of save completion

[1511] Step 2: Auto-generate interview questions

[1512] The server uses a generative AI model to automatically generate interview questions based on the saved job description and job information. For example, if Java experience is required, the generative model generates questions such as, "Tell me about a Java project you recently worked on."

[1513] Input: Saved job postings and job information

[1514] Output: Auto-generated interview questions

[1515] Step 3: Notification of interview arrangement

[1516] The automatically generated interview questions are sent from the server to the terminal, and the terminal notifies the user (job seeker) that "the interview is ready." This notification allows the job seeker to prepare for the interview and start the interview at any time, 24 hours a day.

[1517] Input: Auto-generated interview questions

[1518] Output: Interview preparation notice

[1519] Step 4: Virtual Human AI Interview

[1520] When a user (job seeker) starts an interview through their device, the device communicates with the server and activates the virtual human AI. The virtual human AI sequentially displays the generated interview questions and records the applicant's voice and video. The audio and video data is sent to the server in real time.

[1521] Input: Interview questions, audio and video of job seeker

[1522] Output: Recorded audio and video data

[1523] Step 5: First selection and evaluation

[1524] The server analyzes the received audio and video data in real time. It uses an emotion analysis engine based on TensorFlow and Keras to recognize the applicant's nervousness or confidence from the audio data, and emotions such as joy or surprise from the video data. This generates an evaluation score based on factors such as technical skills, communication ability, adaptability, and emotional stability.

[1525] Input: Recorded audio and video data

[1526] Output: Evaluation score

[1527] Step 6: Notification of evaluation results and feedback

[1528] The server sends the generated evaluation score to the terminal and notifies the user (human resources officer). Furthermore, if the analysis results indicate that a certain evaluation standard has not been met, a warning is issued. At this step, feedback based on the evaluation results is provided, improving the quality and efficiency of the interview.

[1529] Input: Rating score

[1530] Output: Evaluation result notification, warning notification

[1531] Step 7: Emotion analysis and quality control for factory robots

[1532] In factories, robots use cameras and microphones to monitor the movements and emotions of workers in real time, and send the collected data to a server, which uses an emotion analysis engine to analyze the worker's stress, satisfaction, and other factors, and issue warnings as necessary.

[1533] Input: Video and audio data of the worker

[1534] Output: Sentiment analysis data, quality control assessment, warning notifications

[1535] The above processing steps enable efficient and objective evaluation, which is expected to have a significant effect on personnel and factory quality control.

[1536] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1537] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1538] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1539] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1540] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1541] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1542] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1543] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1544] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1545] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1546] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1547] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1548] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1549] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1550] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1551] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1552] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1553] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1554] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1555] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1556] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1557] The following is further disclosed regarding the above embodiment.

[1558] (Claim 1)

[1559] a means for receiving job postings and job information;

[1560] means for automatically generating interview questions based on the received job description and job information;

[1561] means for displaying interview questions and recording audio and video of the applicant;

[1562] means for analyzing the recorded audio and video to generate an initial screening evaluation;

[1563] A means for outputting an evaluation of the first selection;

[1564] A system including:

[1565] (Claim 2)

[1566] 10. The system of claim 1, wherein the means for analyzing the recorded audio and video includes an algorithm for performing facial expression analysis of the applicant to generate a rating.

[1567] (Claim 3)

[1568] 10. The system of claim 1, wherein the means for analyzing the recorded audio and video includes an algorithm for performing an audio analysis of the applicant and generating a rating.

[1569]

[1570] "Example 1"

[1571] (Claim 1)

[1572] a means for receiving job postings and job information;

[1573] A means for automatically generating interview questions using a generative AI model based on the received job description and job information;

[1574] means for displaying automatically generated interview questions and recording audio and video of the applicant;

[1575] a means for analyzing the recorded audio and video data in real time and generating an evaluation score for the first selection;

[1576] means for outputting the generated evaluation score;

[1577] A system including:

[1578] (Claim 2)

[1579] 10. The system of claim 1, wherein the means for analyzing the recorded audio and video data includes an algorithm for analyzing facial expressions of the applicant and generating an evaluation score.

[1580] (Claim 3)

[1581] 10. The system of claim 1, wherein the means for analyzing the recorded audio and video data includes an algorithm for performing an audio analysis of the applicant and generating an evaluation score.

[1582] "Application Example 1"

[1583] (Claim 1)

[1584] a means for receiving job postings and job information;

[1585] means for automatically generating interview questions based on the received job description and job information;

[1586] means for displaying interview questions and recording audio and video of the applicant;

[1587] means for analyzing the recorded audio and video to generate an initial screening evaluation;

[1588] A means for outputting an evaluation of the first selection;

[1589] a means for receiving purchaser behavior information;

[1590] means for automatically generating product recommendations based on the received behavioral information;

[1591] a means for displaying product recommendations and recording buyer responses;

[1592] a means for analyzing the recorded material and re-evaluating product recommendations;

[1593] means for generating and displaying ratings in real time;

[1594] A system including:

[1595] (Claim 2)

[1596] 10. The system of c...

Claims

1. a means for receiving job postings and job information; means for automatically generating interview questions based on the received job description and job information; means for displaying interview questions and recording audio and video of the applicant; means for analyzing the recorded audio and video to generate an initial screening evaluation; A means for outputting an evaluation of the first selection; A system including:

2. 10. The system of claim 1, wherein the means for analyzing the recorded audio and video includes an algorithm for performing facial expression analysis of the applicant to generate a rating.

3. 10. The system of claim 1, wherein the means for analyzing the recorded audio and video includes an algorithm for performing an audio analysis of the applicant and generating a rating.

Images