System

Abstract

To provide a system.SOLUTION: A system includes: means for collecting direct information and indirect information about an applicant; means for storing the information collected by the means for collecting in a database; means for analyzing the information stored by the means for storing by using a generative AI model; means for recommending an optimal section or department to the applicant on the basis of an analysis result by the means for analyzing; and means for providing a recommendation result by the means for recommending via a management screen.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When recruiting personnel, direct decisions and specific judgments about the right person for the right job cannot be made based on one-sided information from the applicant, and there is a problem that even if someone is hired, they do not last long. [Means for solving the problem]

[0005] Using generative AI, the system makes decisions based on direct and indirect information about applicants, and suggests departments and divisions where they can best utilize their skills based on information about their credibility from facial expressions and voice data during interviews, enabling more appropriate hiring decisions. [Brief explanation of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, the processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), Examples include a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention is a recruitment support system that uses generative AI. This system collects direct information such as resumes and application forms submitted by applicants, as well as indirect information such as social media posts and blog articles, and analyzes this information using generative AI. The generative AI uses this information to evaluate the applicant's skills and aptitude, and recommends the most suitable department or division for the job.

[0029] "Example 2"

[0030] Another embodiment of the present invention includes a function to analyze facial expressions and voice data during interviews. During interviews, facial expressions and voice data of applicants are collected via a webcam and microphone. The generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in the hiring decision.

[0031] "Example 3"

[0032] As a specific example, when considering hiring an engineer, generative AI collects direct information such as the applicant's programming portfolio and GitHub commit history, as well as indirect information such as technical blog posts and activity history on Stack Overflow, and analyzes these. Furthermore, during interviews, the applicant's facial expressions and voice data are collected via webcam and microphone, which are then analyzed by generative AI. Based on this information, the system recommends departments and projects where the applicant is likely to perform best.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: Collect direct information such as resumes and application forms submitted by applicants, as well as indirect information such as social media posts and blog articles.

[0036] Step 2: Input the collected information into the generative AI.

[0037] Step 3: Generative AI uses this information to evaluate the applicant's skills and aptitude, and recommends the most suitable department or division.

[0038] "Example 2"

[0039] Step 1: Collect facial expressions and audio data from applicants during the interview via webcam and microphone.

[0040] Step 2: Input the collected facial expressions and voice data into the generative AI.

[0041] Step 3: Generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in hiring decisions.

[0042] "Example 3"

[0043] Step 1: Collect direct information such as the applicant's programming portfolio and GitHub commit history, as well as indirect information such as technical blog posts and Stack Overflow activity.

[0044] Step 2: Input the collected information into the generative AI.

[0045] Step 3: Generative AI analyzes this information and evaluates the applicant's skills and aptitude.

[0046] Step 4: During the interview, facial expressions and audio data from the applicant are collected via a webcam and microphone.

[0047] Step 5: Input the collected facial expressions and voice data into the generative AI.

[0048] Step 6: Generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in hiring decisions.

[0049] Step 7: Based on this information, we recommend the department and project where the applicant would be most likely to excel.

[0050] Example 1

[0051] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0052] Conventional recruitment systems have difficulty accurately evaluating applicants' skills and aptitudes and recommending the most suitable department or division. Furthermore, they are unable to take into account not only direct information about applicants but also indirect information, resulting in a problem of reduced evaluation accuracy. Furthermore, there was a lack of a way to effectively provide evaluation results to human resources personnel.

[0053] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for collecting direct and indirect information about the applicant, a means for storing the collected information in a database, a means for analyzing the stored information using a generative AI model, a means for recommending an optimal department or division based on the analysis results, and a means for providing the recommendation results through a management screen. This makes it possible to evaluate the skills and aptitude of the applicant with high accuracy and recommend an optimal department or division.

[0054] "Direct information from the applicant" refers to information provided by the applicant themselves, such as in a resume or application form.

[0055] "Indirect information about applicants" refers to information that applicants have made public on the Internet, such as social media posts or blog articles.

[0056] A "database" is a system for organizing and storing collected information, and examples include MySQL (registered trademark) and MongoDB.

[0057] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to analyze applicant information and evaluate skills and aptitude.

[0058] The "analysis results" are the results of the skills and aptitudes evaluated by the generative AI model based on the applicant's information.

[0059] "Department or division recommendation" refers to suggesting the most suitable job or role for an applicant based on the analysis results.

[0060] The "administration screen" is an interface for providing recommendation results to human resources personnel, and is built using a front-end framework.

[0061] The present invention relates to a recruitment support system that uses generative AI. A specific embodiment of this system will be described below.

[0062] The server first collects direct and indirect information about applicants. Direct information includes resumes and application forms, which applicants provide themselves. Indirect information includes social media posts and blog articles, which applicants make public online. This information is collected using web scraping tools (e.g., BeautifulSoup and Scrapy) and APIs (e.g., Twitter API and Facebook Graph API).

[0063] Next, the server stores the collected information in a database (e.g., MySQL or MongoDB). When storing the information, it assigns a unique ID to each applicant and organizes the information. For example, the resume information for applicant A is stored in a table called "applicant_A_resume."

[0064] The stored information is analyzed using a generative AI model (e.g., OpenAI's GPT-4®). The server inputs a prompt like the following into the generative AI model:

[0065] The applicant's resume states that he has "5 years of Python programming experience," and his social media posts state that he is "interested in data science." Use this information to evaluate the applicant's skills and aptitude, and recommend the department or division that would be the best fit.

[0066] The generative AI model evaluates the applicant's skills and suitability based on this prompt and returns the results.

[0067] The server recommends the most suitable department or division for each applicant based on the evaluation results obtained from the generative AI model. For example, if the evaluation result shows that "Applicant A is suitable for the Data Science Division," the server will recommend Applicant A to the Data Science Division.

[0068] Finally, the server provides the recommendation results to the human resources personnel via an administration screen. The administration screen is built using a front-end framework (e.g., React or Vue.js) and can be accessed by the user via a browser. The user can check the recommendation results for applicant A on the administration screen and make a decision on assigning the candidate to an appropriate department.

[0069] In this way, the server, terminals, and users work together to create a system that provides efficient support for recruiting personnel.

[0070] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0071] Step 1: Gather applicant information

[0072] The server receives resumes and application forms submitted by applicants. This direct information is often uploaded in PDF or Word format. The server uses an OCR (optical character recognition) tool (e.g., Tesseract) to convert these files into text data. It also uses web scraping tools (e.g., BeautifulSoup or Scrapy) or APIs (e.g., Twitter API or Facebook Graph API) to collect indirect information such as applicants' social media posts and blog articles. The input is the resume or social media URL, and the output is text data.

[0073] Step 2: Save your applicant information

[0074] The server saves the collected text data in a database (e.g., MySQL or MongoDB). When saving, a unique ID is assigned to each applicant and the information is organized. For example, the resume information for applicant A is saved in a table called "applicant_A_resume." The input is text data, and the output is the information saved in the database.

[0075] Step 3: Analyze applicant information

[0076] The server inputs the stored information into a generative AI model (e.g., OpenAI's GPT-4) for analysis. Specifically, it inputs the following prompt sentence into the generative AI model:

[0077] The applicant's resume states that he has "5 years of Python programming experience," and his social media posts state that he is "interested in data science." Use this information to evaluate the applicant's skills and aptitude, and recommend the department or division that would be the best fit.

[0078] The generative AI model evaluates the applicant's skills and aptitude based on this prompt and returns the results. The input is the prompt and the output is the evaluation result.

[0079] Step 4: Department / Division Recommendations

[0080] The server recommends the most suitable department or division for the applicant based on the evaluation results obtained from the generative AI model. For example, if the evaluation result shows that "Applicant A is suitable for the Data Science Division," the server will recommend Applicant A to the Data Science Division. The input is the evaluation result, and the output is the recommendation result.

[0081] Step 5: Providing recommendations

[0082] The server provides the recommendation results to the human resources personnel via an administration screen. The administration screen is built using a front-end framework (e.g., React or Vue.js) and can be accessed by the user via a browser. The user checks the recommendation results for applicant A on the administration screen and makes a decision on assigning the applicant to an appropriate department. The input is the recommendation results, and the output is the information displayed on the administration screen.

[0083] (Application example 1)

[0084] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] Conventional recruitment systems have typically made decisions based on direct information such as applicants' resumes and application forms, but this makes it difficult to fully evaluate the applicant's aptitude and skills. Furthermore, credibility is not assessed based on facial expressions or voice data during interviews, making it difficult to assign candidates to the appropriate department or division. Furthermore, there is a lack of concrete methods for assessing suitability as a driver of an autonomous vehicle.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0087] In this invention, the server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for collecting the applicant's resume, application form, social media posts, blog articles, etc. and for the generative AI to analyze this information to evaluate the applicant's skills and aptitude, and means for evaluating the applicant's aptitude as a driver of an autonomous vehicle and recommending the most suitable position. This makes it possible to comprehensively evaluate the aptitude and skills of the applicant and assign them to the most suitable department or division.

[0088] "Generative AI" is an artificial intelligence technology that collects and analyzes information about applicants and evaluates their aptitude and skills based on the results.

[0089] "Direct information" refers to information contained in official documents submitted by applicants, such as resumes and application forms.

[0090] "Indirect information" refers to information obtained from sources other than official documents, such as applicants' social media posts or blog articles.

[0091] "Facial expressions during an interview" refers to the facial expressions and expressions shown by applicants during an interview.

[0092] "Audio data" refers to data related to the voice of the applicant during the interview, such as the tone of voice and speaking style.

[0093] "Credibility" is a measure of how trustworthy the information provided by the applicant and the expressions given during the interview are.

[0094] "Skills" refers to the specialized knowledge, techniques, and abilities possessed by the applicant.

[0095] "Qualities" are characteristics that indicate how well an applicant is suited to a particular job or role.

[0096] A "department" is a division within a company or organization that is responsible for a specific task or role.

[0097] A "department" refers to a large business unit within a company or organization, and may include multiple departments.

[0098] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0099] "Driver" refers to the person who operates and monitors the autonomous vehicle.

[0100] A "position" refers to a specific role or job within a company or organization.

[0101] The system for implementing this invention is composed of three main elements: a server, a terminal, and a user. The specific roles and processes of each element are explained below.

[0102] Server Roles and Operations

[0103] The server uses the following hardware and software to collect, analyze, and evaluate data:

[0104] Hardware used:

[0105] Server (data collection and execution of AI models)

[0106] Software used:

[0107] Python (data processing and running AI models)

[0108] TENSORFLOW (registered trademark) (implementation of generative AI models)

[0109] Flask (Web application framework)

[0110] BeautifulSoup (web scraping library)

[0111] Data processing and calculation:

[0112] 1. Data Collection:

[0113] The server uses BeautifulSoup to scrape applicants' social media posts and blog posts, and converts the applicants' submitted resumes and application forms from PDFs and images to text.

[0114] 2. Data Preprocessing:

[0115] The server cleans the collected text data, removing unnecessary information, and then uses natural language processing (NLP) techniques to tokenize the text data and extract features.

[0116] 3. Analysis using generative AI models:

[0117] The server uses TensorFlow to build a generative AI model and uses the applicant's text data as input to evaluate their skills and aptitude.

[0118] 4. Displaying the results:

[0119] The server uses Flask to build a web application and display the results.

[0120] Terminal roles and processing

[0121] The terminal provides an interface for the user to enter applicant information and check the results.

[0122] Hardware used:

[0123] Smartphone (entering applicant information and displaying results)

[0124] Software used:

[0125] A web browser (accessing the Flask application)

[0126] User Roles and Actions

[0127] The user enters applicant information and checks the evaluation results generated by the generative AI model.

[0128] Examples:

[0129] Applicant A's resume states that he has "5 years of experience driving self-driving vehicles."

[0130] Applicant A's social media posts state that he "frequently shares articles about the latest autonomous driving technology."

[0131] The generative AI model analyzes this information and assesses that Applicant A has high suitability as a driver of an autonomous vehicle.

[0132] Example prompts to input to a generative AI model:

[0133] The applicant's resume states that he has "5 years of experience driving autonomous vehicles." His social media posts state that he "frequently shares articles about the latest autonomous driving technology." Please rate this applicant's suitability as a driver.

[0134] In this way, a system can be constructed in which the server, terminals, and users work together to comprehensively evaluate the aptitude and skills of applicants and assign them to the most suitable department or division.

[0135] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0136] Step 1:

[0137] The server collects applicants' social media posts and blog articles. Specifically, it uses BeautifulSoup to perform web scraping to obtain text data from applicants' social media accounts and blogs. The input is the URL of the applicants' social media accounts and blogs, and the output is the collected text data.

[0138] Step 2:

[0139] The server converts the resumes and application forms submitted by applicants from PDF or image to text. Specifically, it uses OCR (optical character recognition) technology to extract text data from PDF or image files. The input is the PDF or image file, and the output is the converted text data.

[0140] Step 3:

[0141] The server preprocesses the collected text data. Specifically, it cleans the text data and removes unnecessary information. Then, it uses natural language processing (NLP) techniques to tokenize the text data and extract features. The input is the collected text data, and the output is the preprocessed text data.

[0142] Step 4:

[0143] The server uses TensorFlow to build a generative AI model and evaluates skills and aptitude using preprocessed text data as input. Specifically, the generative AI model inputs text data and outputs a score that evaluates the applicant's skills and aptitude. The input is preprocessed text data, and the output is an evaluation score for skills and aptitude.

[0144] Step 5:

[0145] The server uses Flask to build a web application and display the evaluation results. Specifically, based on the evaluation scores, it generates recommendations for the most suitable department or division for the applicant and displays them through the web application. The input is the evaluation scores for skills and aptitude, and the output is the recommendation results.

[0146] Step 6:

[0147] The terminal provides an interface for users to enter applicant information and check the results. Specifically, users use a smartphone's web browser to access the Flask application and enter applicant information. The input is applicant information, and the output is the recommendation results.

[0148] Step 7:

[0149] The user checks the evaluation results of the generative AI model. Specifically, the user checks the recommendation results on their smartphone screen and inputs additional information as needed. The input is the recommendation results, and the output is the user's feedback.

[0150] In this way, a system can be constructed in which the server, terminals, and users work together to comprehensively evaluate the aptitude and skills of applicants and assign them to the most suitable department or division.

[0151] Example 2

[0152] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] With conventional interview systems, it was difficult to objectively evaluate the credibility and seriousness of applicants, and hiring decisions often involved subjective factors. It was also difficult to properly recommend departments or divisions that would best utilize an applicant's skills.

[0154] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for activating the webcam and microphone and starting data collection at the same time as the start of the interview, means for dividing the collected video data into frames and extracting facial regions, means for removing noise from the collected audio data and extracting audio features, means for integrating facial expression data and audio data and inputting the data into a generative AI model, means for the generative AI model to evaluate the credibility and seriousness based on the facial expression data and audio data, and means for providing the evaluation results to the hiring manager. This makes it possible to objectively evaluate the credibility and seriousness of the applicant and make an appropriate hiring decision.

[0155] A "webcam" is a digital camera for capturing video.

[0156] A "microphone" is an acoustic device for capturing sound.

[0157] "Data collection" is the process of capturing video and audio data using a webcam and microphone.

[0158] "Video Data" is a collection of video frames captured by a webcam.

[0159] A "frame" is an individual still image that makes up video data.

[0160] The "face area" refers to the part of the video data where a face is present.

[0161] "Audio Data" is a recording of sound captured by a microphone.

[0162] "Noise reduction" is the process of removing unwanted background sounds and noise from audio data.

[0163] "Voice features" are characteristics such as pitch, tone, and speed contained in the voice data.

[0164] "Facial expression data" is information about facial expressions extracted from video data.

[0165] A "generative AI model" is an algorithm that uses artificial intelligence to analyze data and make specific assessments or predictions.

[0166] "Credibility" refers to the reliability of whether an applicant's statements and actions are true.

[0167] "Seriousness" refers to how seriously the applicant takes the interview.

[0168] The "evaluation results" are scores and judgments of credibility and seriousness calculated by the generative AI model.

[0169] A "recruiter" is a person who makes hiring decisions based on the results of an applicant's interview.

[0170] This invention is a system for analyzing facial expressions and voice data of an applicant during an interview to evaluate the applicant's credibility and seriousness. A specific embodiment of this system will be described below.

[0171] Hardware and software used

[0172] Hardware: Webcam, microphone

[0173] Software: Generative AI models (e.g., general-purpose natural language processing models), facial expression recognition software (e.g., OpenFace), and speech analysis software (e.g., Praat)

[0174] Data collection

[0175] The device activates the webcam and microphone and begins collecting data as soon as the interview begins. The webcam captures video data, and the microphone captures audio data.

[0176] Data Preprocessing

[0177] The device splits the video data it collects into frames and extracts facial regions using facial expression recognition software, while it uses voice analysis software to remove noise and extract voice characteristics from the audio data.

[0178] Analyzing the data

[0179] The server receives the facial expression and voice data sent from the device and integrates them. The integrated data is then input into a generative AI model, which evaluates the authenticity and seriousness of the user based on the facial expression and voice data.

[0180] Output of evaluation results

[0181] The server receives the evaluation results from the generative AI model and provides them to recruiters. The evaluation results are displayed in the form of a dashboard or report, clearly indicating scores for credibility and seriousness.

[0182] Specific examples

[0183] For example, when an applicant introduces themselves, the device collects data via a webcam and microphone. The collected data is sent to a server and analyzed by a generative AI model. The resulting evaluation result may be something like, "The applicant's facial expression indicates nervousness, but their voice tone is stable, and they are evaluated as being highly credible."

[0184] Prompt Sentence Examples

[0185] "Please rate the applicant's credibility and seriousness based on their facial and voice data. Enter the following data: [Facial Expression Data], [Voice Data]"

[0186] In this way, the system analyzes the applicant's facial expressions and voice data during the interview to evaluate their credibility and seriousness, and the results of this evaluation are reflected in the hiring decision.

[0187] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0188] Step 1:

[0189] The device activates the webcam and microphone and starts collecting data at the same time as the interview begins. The input is the start signal for the interview, and the output is the video and audio data captured in real time. Specifically, the device captures the video stream from the webcam and the audio stream from the microphone in real time.

[0190] Step 2:

[0191] The device divides the collected video data into frames and extracts facial regions. The input is the captured video data, and the output is facial region data extracted from each frame. Specifically, the device uses facial expression recognition software (e.g., OpenFace) to extract facial feature points from each frame.

[0192] Step 3:

[0193] The device removes noise from the collected voice data and extracts voice features. The input is the captured voice data, and the output is the voice data with noise removed and features extracted. Specifically, the device uses voice analysis software (e.g., Praat) to extract voice features such as pitch, tone, and speed.

[0194] Step 4:

[0195] The server receives the facial expression data and voice data sent from the device and integrates them. The input is the facial expression data and voice data sent from the device, and the output is the integrated data. Specifically, the server combines the facial expression data and voice data into a single data set.

[0196] Step 5:

[0197] The server inputs the integrated data into the generative AI model to evaluate its credibility and seriousness. The input is the integrated data, and the output is the evaluation result of its credibility and seriousness. Specifically, the server inputs the integrated data as a prompt sentence to the generative AI model (e.g., GPT-4) and performs an evaluation.

[0198] Step 6:

[0199] The server receives the evaluation results from the generative AI model and provides them to recruiters. The input is the evaluation results from the generative AI model, and the output is the evaluation results provided to recruiters. Specifically, the server displays the evaluation results in the form of a dashboard or report, clearly indicating scores for credibility and seriousness.

[0200] (Application example 2)

[0201] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Conventional interview systems make it difficult to accurately assess applicants' credibility and seriousness, and rely heavily on the interviewer's subjective judgment. Furthermore, there was a lack of a way to analyze participants' facial expressions and voices in real time during interviews and meetings to assess their credibility and seriousness, making it difficult to make appropriate personnel assignments and decisions. This hinders companies' efficient use of human resources and the improvement of the quality of important meetings.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0204] In this invention, the server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for analyzing the facial expressions and voice of participants in real time during interviews and important meetings using a smartphone camera and microphone, and means for the generative AI to analyze the data, evaluate the credibility and seriousness of the participants, and display the results in real time. This enables objective evaluation of the credibility and seriousness of applicants and participants, enabling appropriate personnel placement and decision-making.

[0205] "Generative AI" is an artificial intelligence technology that generates new data and information based on input data.

[0206] "Applicant" refers to a person applying for a particular job or role.

[0207] "Direct information" refers to explicit data provided by applicants, such as resumes and curriculum vitae.

[0208] "Indirect information" refers to data obtained from the applicant's behavior and reactions, such as facial expressions and voice data during the interview.

[0209] "Credibility" is a measure of whether an applicant's statements and actions are true.

[0210] "Seriousness" is a measure of how seriously an applicant takes an interview or meeting.

[0211] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and use applications.

[0212] A "camera" is a device for taking images or videos.

[0213] A "microphone" is a device for recording sound.

[0214] "Real-time" refers to data collection and processing occurring immediately.

[0215] A "department" is a group responsible for a specific task or function within a company or organization.

[0216] A "department" is a small group responsible for a specific task or role within a company or organization.

[0217] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[0218] "Display" refers to the visual presentation of data or information.

[0219] A system for carrying out this invention has the following configuration: The server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for analyzing the facial expressions and voices of participants in real time during interviews and important meetings using a smartphone camera and microphone, and means for the generative AI to analyze the data, evaluate the credibility and seriousness of the participants, and display the results in real time.

[0220] Hardware and software used

[0221] Hardware: Smartphone camera, smartphone microphone

[0222] Software: OpenCV (video capture and processing), SoundDevice (audio capture), TensorFlow (AI model for facial expression analysis), Transformers (generative AI model for audio data)

[0223] Data processing and calculation

[0224] 1. Video data capture and processing:

[0225] Use your smartphone camera to capture real-time footage of participants during interviews and meetings.

[0226] The video data is processed using OpenCV to extract the facial area.

[0227] The extracted facial area is subjected to facial expression analysis using TensorFlow to evaluate credibility and seriousness.

[0228] 2. Audio data capture and processing:

[0229] Use your smartphone's microphone to capture participants' voices in real time during interviews and meetings.

[0230] Audio data is recorded using SoundDevice and analyzed using Transformers.

[0231] The analysis results are used to evaluate the participants' credibility and seriousness.

[0232] Specific examples

[0233] For example, to evaluate in real time whether an applicant's facial expression is nervous during an interview, the applicant's face is captured with a smartphone camera and the facial region is extracted using OpenCV. The extracted facial region is analyzed using TensorFlow to evaluate the level of nervousness. Similarly, to analyze the seriousness of participants' comments during a meeting from audio data, the audio is captured with a smartphone microphone and recorded using SoundDevice. The recorded audio data is analyzed using Transformers to evaluate the level of seriousness.

[0234] Prompt Sentence Examples

[0235] "Analyze the applicant's facial expression data and assess their level of nervousness."

[0236] "Please analyze the seriousness of statements made during meetings from audio data."

[0237] In this way, the system can objectively assess the credibility and seriousness of applicants and participants, and support appropriate staffing and decision-making.

[0238] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0239] Step 1:

[0240] The server uses the smartphone camera to capture video of participants in interviews or meetings in real time. The input is video data from the smartphone camera, and the output is the captured video data. Specifically, the camera starts up and continuously acquires video frames.

[0241] Step 2:

[0242] The server processes the captured video data using OpenCV and extracts the face region. The input is the video data obtained in step 1, and the output is the extracted face region data. Specifically, it applies a face detection algorithm to the video frame to identify the position of the face.

[0243] Step 3:

[0244] The server performs facial expression analysis on the extracted facial region using TensorFlow. The input is the facial region data obtained in step 2, and the output is the result of the facial expression analysis. Specifically, the facial region data is preprocessed and input into an AI model to analyze facial expression features.

[0245] Step 4:

[0246] The server uses the smartphone's microphone to capture the voice of participants in an interview or meeting in real time. The input is the voice data from the smartphone's microphone, and the output is the captured voice data. Specifically, the microphone is turned on and continuously records voice.

[0247] Step 5:

[0248] The server records the audio data captured using SoundDevice and analyzes it using Transformers. The input is the audio data obtained in step 4, and the output is the result of the audio analysis. Specifically, the audio data is preprocessed and input into the AI ​​model to analyze the audio features.

[0249] Step 6:

[0250] The server integrates the results of the facial expression analysis and voice analysis obtained in steps 3 and 5 to evaluate the participants' credibility and seriousness. The inputs are the results of the facial expression analysis and voice analysis, and the output is the evaluation results of credibility and seriousness. Specifically, the server inputs both analysis results into an integration algorithm to perform a comprehensive evaluation.

[0251] Step 7:

[0252] The server displays the evaluation results in real time and provides feedback to the interviewer and the conference organizer. The input is the evaluation results obtained in step 6, and the output is the displayed feedback information. Specific operations include updating the interface to visually display the evaluation results.

[0253] Example 3

[0254] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0255] In the traditional recruitment process for engineers, it was difficult to accurately evaluate applicants' technical skills and experience and assign them to the most appropriate departments and projects. Furthermore, there was a lack of methods to evaluate applicants' emotions and communication skills using facial expressions and voice data during interviews, which led to a problem of reduced accuracy in recruitment.

[0256] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for collecting information about the applicant; means for transmitting the collected information to the server; means for the server to analyze the applicant's technical skills and experience using a generative AI model; means for the terminal to collect facial expression and voice data about the applicant via a webcam and microphone during the interview; means for transmitting the collected interview data to the server; means for the server to analyze the interview data using the generative AI model and evaluate the applicant's emotions and communication skills; means for the server to recommend a department or project that is most suitable for the applicant based on the analysis results; and means for notifying the user of the recommendation results. This makes it possible to comprehensively evaluate the applicant's technical skills, experience, emotions, and communication skills and assign them to the most suitable department or project.

[0257] "Means for collecting applicant information" refers to devices or software for obtaining data such as the applicant's programming portfolio, online platform history, technical blog posts, and activity history on Q&A sites.

[0258] The "means for transmitting collected information to the server" refers to a communication device or software for transferring applicant information from the terminal to the server.

[0259] "Means for analyzing applicants' technical skills and experience using generative AI models" refers to the AI ​​model and its execution environment used to analyze collected applicant information and evaluate technical skills and experience.

[0260] "Means for the device to collect facial expressions and voice data of the applicant through a webcam and microphone during the interview" refers to hardware and software for capturing the facial expressions and voice of the applicant in real time during the interview.

[0261] The "means for transmitting collected interview data to the server" refers to a communication device or software for transferring facial expression and voice data collected during the interview to the server.

[0262] "Means for analyzing interview data using a generative AI model to evaluate applicants' emotions and communication skills" refers to an AI model and its execution environment used to analyze collected interview data and evaluate applicants' emotions and communication skills.

[0263] "The means by which the server recommends the department or project that is most suitable for the applicant based on the analysis results" refers to the algorithm and its execution environment for determining and recommending the most suitable department or project for the applicant based on the analysis results of the AI ​​model.

[0264] The "means for notifying the user of the recommendation results" refers to a communication device or software for transmitting the recommendation results from the server to the user.

[0265] This invention is a system that streamlines the engineer recruitment process and comprehensively evaluates applicants' technical skills, experience, emotions, and communication skills. Specific embodiments of this system are described below.

[0266] First, the user initiates the engineer recruitment process. The terminal provides an interface for collecting information about the applicant. The terminal uses web scraping technology and APIs to obtain data such as the applicant's programming portfolio, online platform history, technical blog posts, and activity history on Q&A sites.

[0267] The collected information is sent from the device to a server, which then uses a generative AI model (such as OpenAI's GPT-4) to analyze the applicant's technical skills and experience. Specifically, the system evaluates the quality of the code, the frequency of commits, and the content of technical blog posts.

[0268] Next, as the user conducts the interview, the device collects the candidate's facial expressions and voice data through a webcam and microphone. Specifically, data is acquired using a Logitech webcam and a Blue Yeti microphone. The collected interview data is then sent from the device to a server.

[0269] The server then inputs the collected interview data back into the generative AI model, which uses facial recognition and voice analysis technologies to evaluate the applicant's emotions and communication skills. For example, it analyzes whether the applicant spoke confidently during the interview.

[0270] Based on the results of these analyses, the server recommends the department and project that best suits the applicant. It then comprehensively assesses the results of the evaluation of technical and communication skills to determine the most suitable assignment. The server notifies the user of the recommendation results. The user can then check the recommendation results on the system's dashboard and make the final hiring decision.

[0271] As a concrete example, a user starts a hiring process for an engineer, collects the commit history from Applicant A's GitHub account, and also retrieves technical blog posts. The server inputs the collected data into GPT-4 to analyze Applicant A's technical skills. For example, the result may be "Applicant A is strong in JavaScript (registered trademark) and React."

[0272] During the interview, the user interviews Applicant A, and the device uses the Logitech webcam and Blue Yeti microphone to collect Applicant A's facial expressions and voice. The server inputs the collected interview data into GPT-4 to analyze Applicant A's communication skills. For example, the result may be, "Applicant A speaks confidently during the interview."

[0273] Finally, based on these analysis results, the server recommends Applicant A to the front-end team. The user can check the recommendation results on the system dashboard and make the final hiring decision.

[0274] An example of a prompt sentence is, "Please analyze applicant A's GitHub commit history and technical blog articles to evaluate his / her technical skills. Also, please evaluate his / her communication skills based on facial expressions and voice data during the interview and recommend the most suitable department for him / her." The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0275] Step 1:

[0276] The user initiates the hiring process for an engineer. The device displays a form for entering applicant information. The user enters information such as the applicant's programming portfolio, history on online platforms, technical blog posts, and activity history on Q&A sites. The entered information is saved on the device.

[0277] Step 2:

[0278] The device collects the information entered by the applicant. Specifically, it obtains data through links or files provided by the applicant. For example, it downloads the commit history from a GitHub URL and obtains technical blog posts as text data. The collected information is stored on the device.

[0279] Step 3:

[0280] The device sends the collected information to the server. Specifically, it transfers the data to the server using an HTTP request. The server stores the received data in a database. The input is the applicant's information, and the output is the data stored on the server.

[0281] Step 4:

[0282] The server retrieves applicant information from a database and inputs it into a generative AI model. Specifically, a Python script is used to pass the data to the AI ​​model. The AI ​​model analyzes the applicant's technical skills and experience. For example, it evaluates the quality of the code, the frequency of commits, and the content of technical blog posts. The input is the applicant's information, and the output is the evaluation of their technical skills and experience.

[0283] Step 5:

[0284] When a user conducts an interview, the device collects facial expression and voice data from the applicant through a webcam and microphone. Specifically, data is acquired using a Logitech webcam and a Blue Yeti microphone. The collected data is stored on the device. The input is facial expression and voice data from the interview, and the output is the data stored on the device.

[0285] Step 6:

[0286] The device sends the collected interview data to the server. Specifically, it transfers the data to the server using an HTTP request. The server stores the received data in a database. The input is the interview data, and the output is the data stored on the server.

[0287] Step 7:

[0288] The server retrieves interview data from the database and inputs it into the generative AI model. Specifically, a Python script is used to pass the data to the AI ​​model. The AI ​​model analyzes the applicant's emotions and communication skills from their facial expressions and voice. For example, it uses facial recognition technology and voice analysis technology to make the evaluation. The input is the interview data, and the output is the evaluation results of emotions and communication skills.

[0289] Step 8:

[0290] The server recommends the department and project that best suits the applicant based on the evaluation results of technical and communication skills. Specifically, it uses an algorithm to comprehensively judge the evaluation results and determine the most suitable assignment. The input is the evaluation results of technical and communication skills, and the output is the recommendation results.

[0291] Step 9:

[0292] The server notifies the user of the recommendation results. Specifically, the results are communicated via email or the system dashboard. The user checks the recommendation results and makes the final adoption decision. The input is the recommendation results, and the output is the results notified to the user.

[0293] (Application example 3)

[0294] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0295] Conventional engineer recruitment systems have difficulty accurately evaluating applicants' skills and aptitudes and recommending the most suitable departments and work content. Furthermore, there was no system that could assess the aptitude of robot operators working in factories and recommend the most suitable departments and work content, making it difficult to assign the right person to the right job.

[0296] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: a means for using generative AI to make a judgment based on direct and indirect information about the applicant and suggesting a department or division where the applicant's skills can be maximized based on information about credibility from facial expressions and voice data during the interview; a means for diagnosing the aptitude of robot operators working in factories and recommending the most appropriate department and work content; a means for capturing facial expressions using a webcam and analyzing emotions with a generative AI model; and a means for capturing voice data using a microphone, converting it into text with a generative AI model, and analyzing emotions and content. This makes it possible to accurately evaluate the aptitude of applicants and robot operators and recommend the most appropriate department and work content.

[0297] "Generative AI" is an artificial intelligence technology that generates new information and results based on input data.

[0298] "Direct information" refers to specific data provided by the applicant themselves, such as a programming portfolio submitted by the applicant or a GitHub commit history.

[0299] "Indirect information" refers to supplementary data obtained from an applicant's activities and behavior, such as technical blog posts or activity history on Stack Overflow.

[0300] "Facial expressions" refers to the applicant's facial movements and changes in expression captured via webcam during the interview.

[0301] "Audio data" refers to audio information such as the applicant's statements and tone of voice collected through a microphone during an interview.

[0302] "Credibility" refers to the reliability of the information provided by the applicant and the facial and voice data during the interview.

[0303] A "department or division" is a group or section within a company or organization that is responsible for a specific task or role.

[0304] A "robot operator working in a factory" is a person who operates robots in a factory and performs tasks such as manufacturing and assembly.

[0305] "Aptitude" refers to the ability and adaptability of an applicant or robot operator for a particular task or role.

[0306] A "webcam" is a camera device that connects to a computer or other device to capture video.

[0307] A "microphone" is a device for capturing sound and converting it into a digital signal.

[0308] "Analyzing emotions" means assessing the emotional state of applicants based on collected facial expressions and voice data.

[0309] "Convert to text" is the process of converting audio data into written information.

[0310] "Analyzing emotions and content" means evaluating the emotions and content of statements based on audio data converted into text.

[0311] The system for implementing this invention consists of three main elements: a server, a terminal, and a user. The following explains how each element functions.

[0312] server

[0313] The server uses generative AI to analyze direct and indirect information about applicants. Specifically, it collects information such as programming portfolios submitted by applicants, GitHub commit histories, technical blog posts, and activity history on Stack Overflow, and analyzes this data using a generative AI model. Furthermore, during interviews, facial expressions and voice data collected via a webcam and microphone are analyzed to assess credibility. This allows the server to recommend departments and divisions that can best utilize the applicant's skills and aptitudes.

[0314] Terminal

[0315] The devices are computers or smart devices equipped with webcams and microphones. These devices are used to capture the applicant's facial expressions and voice data during the interview. The webcam uses the OpenCV library to capture facial expressions and analyze emotions using a generative AI model. The microphone uses the SpeechRecognition library to capture voice data and convert it into text. The text is then analyzed for sentiment and content using a generative AI model.

[0316] User

[0317] Users are recruiters and factory managers at companies using the system. Based on the analysis results generated by the server, users evaluate the suitability of applicants and robot operators and determine the most suitable departments and work tasks. This makes it possible to assign the right people to the right jobs.

[0318] Specific examples

[0319] For example, when a new robot operator for a factory is interviewed, facial expressions and voice are collected via the device's webcam and microphone, and a generative AI on the server analyzes these to assess suitability. If the applicant says, "I'm good at detailed work," the generative AI analyzes the emotion and content of the statement and recommends a department that requires detailed work.

[0320] Prompt Sentence Examples

[0321] "Collect facial expressions and voice data from applicants and use generative AI to analyze their emotions and content. Based on the analysis results, recommend the most suitable department and work content."

[0322] In this way, a system is realized in which the server, terminals, and users work together to accurately evaluate the suitability of applicants and robot operators and recommend the most suitable departments and work contents.

[0323] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0324] Step 1:

[0325] The server collects direct information about applicants (e.g., programming portfolios, GitHub commit histories) and indirect information (e.g., technical blog posts, Stack Overflow activity histories). This information is obtained from data submitted online by applicants and publicly available databases. The input is various data about applicants, and the output is a dataset of the collected information.

[0326] Step 2:

[0327] The server inputs the collected information into a generative AI model to analyze the applicant's skills and aptitude. The generative AI model uses natural language processing and machine learning algorithms to evaluate the applicant's technical capabilities and past performance. The input is a dataset of collected information, and the output is an evaluation of the applicant's skills and aptitude.

[0328] Step 3:

[0329] The device captures the applicant's facial expressions using a webcam during the interview. It uses the OpenCV library to detect faces and analyze facial expressions. The input is the video data from the webcam, and the output is the analyzed facial expression data.

[0330] Step 4:

[0331] The device captures the applicant's voice data using a microphone during the interview. It converts the voice data into text using the SpeechRecognition library. The input is the voice data from the microphone, and the output is the converted text data.

[0332] Step 5:

[0333] The server inputs the captured facial expression data and converted text data into a generative AI model to analyze emotions and content. The generative AI model uses an emotion analysis algorithm to evaluate the applicant's emotional state and the content of their comments. The input is facial expression data and text data, and the output is an evaluation result regarding emotions and content.

[0334] Step 6:

[0335] The server integrates the evaluation results of the applicant's skills, aptitude, and emotions and content during the interview, and recommends the most suitable department or division. Based on these evaluation results, the generative AI model identifies the department or project where the applicant will be most effective. The input is the various evaluation results, and the output is a list of recommended departments and divisions.

[0336] Step 7:

[0337] The user checks the recommendation results generated by the server and makes the final decision. Based on the recommendation results, the user assigns the applicant to the appropriate department or project. The input is the recommendation results, and the output is the final assignment decision.

[0338] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0339] "Example 1"

[0340] One embodiment of the present invention is a system that incorporates an emotion engine. This system uses an emotion engine that recognizes the user's emotions, and this emotion information is included in the generative AI's decision-making. Specifically, the emotion engine recognizes the user's emotional state (joy, anger, sadness, surprise, etc.) in real time when using the system and provides this information to the generative AI. The generative AI takes this emotion information into consideration and recommends a department or division that will maximize the user's skills.

[0341] "Example 2"

[0342] Another embodiment of the present invention is a system that includes a function in which an emotion engine provides a user's emotional information to a generative AI and, based on that information, recommends the most appropriate department or division. Specifically, the emotion engine recognizes the user's emotional state in real time when using the system and provides that information to the generative AI. The generative AI then takes into account this emotional information and the user's skills and aptitudes to recommend the most appropriate department or division.

[0343] "Example 3"

[0344] In a further embodiment of the present invention, a system includes a function in which an emotion engine provides a user's emotional information to a generative AI and evaluates the user's credibility during an interview based on that information. Specifically, the emotion engine recognizes the user's emotional state during an interview in real time and provides that information to the generative AI. The generative AI evaluates the user's credibility by taking into account this emotional information, as well as facial expressions and voice data during the interview.

[0345] The processing flow of each embodiment will be described below.

[0346] "Example 1"

[0347] Step 1: When a user uses the system, the emotion engine recognizes the user's emotional state in real time.

[0348] Step 2: The emotion engine provides the recognized emotion information to the generative AI.

[0349] Step 3: The generative AI takes into account the provided emotional information and recommends departments or divisions that would best utilize the user's skills.

[0350] "Example 2"

[0351] Step 1: When a user uses the system, the emotion engine recognizes the user's emotional state in real time.

[0352] Step 2: The emotion engine provides the recognized emotion information to the generative AI.

[0353] Step 3: The generative AI takes into account the provided emotional information and the user's skills and aptitudes to recommend the most suitable department or division.

[0354] "Example 3"

[0355] Step 1: When a user is interviewed, the emotion engine recognizes the user's emotional state in real time.

[0356] Step 2: The emotion engine provides the recognized emotion information to the generative AI.

[0357] Step 3: The generative AI evaluates the user's credibility, taking into account the provided emotional information and facial expressions and voice data from the interview.

[0358] Example 1

[0359] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] Conventional recruitment systems have typically made decisions based on direct information such as applicants' resumes and application forms. However, this information alone makes it difficult to fully evaluate an applicant's skills and aptitude, making it difficult to assign them to the appropriate department or division. There are also systems that evaluate credibility by utilizing facial expressions and voice data from interviews, but these systems cannot take real-time emotional information into account, making it difficult to determine an applicant's true aptitude.

[0361] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for collecting direct and indirect information about applicants, means for saving the collected information in a database, means for inputting the saved information into a generative AI and analyzing it, means for collecting the user's emotional state in real time, means for analyzing the collected emotional information, means for the generative AI to recommend the most appropriate department or division based on the analysis results and the emotional information, and means for displaying the recommendation results to the user. This makes it possible to comprehensively evaluate the skills and aptitude of applicants and assign them to the most appropriate department or division taking into account their real-time emotional information.

[0362] "Direct information from the applicant" refers to information provided by the applicant themselves, such as in a resume or application form.

[0363] "Indirect information about applicants" refers to information that applicants have made public but not directly provided, such as social media posts or blog articles.

[0364] A "database" is a system for systematically storing and managing collected information.

[0365] "Generative AI" is artificial intelligence that analyzes collected information and evaluates applicants' skills and aptitude.

[0366] "Analysis" is the process of evaluating the applicant's skills and suitability based on the collected information.

[0367] The "user's emotional state" is the emotional state recognized from the user's facial expressions and voice when using the system.

[0368] "Real-time collection means" refers to devices such as cameras and microphones for instantly collecting the user's emotional state.

[0369] "Emotional information" is collected data about a user's emotional state.

[0370] "Recommendation" means that the generative AI suggests the most suitable department or division for an applicant based on the analysis results and emotional information.

[0371] The "display means" refers to a device such as a display or monitor for visually presenting the recommendation results to the user.

[0372] This invention relates to a recruitment support system that uses generative AI. This system collects and analyzes direct and indirect information about applicants to evaluate their skills and aptitude and recommend the most suitable department or division. Furthermore, by collecting the user's emotional state in real time and incorporating that information into the analysis, more accurate recommendations can be made.

[0373] The server first collects direct information, such as resumes and application forms submitted by applicants. This information can be in PDF or Word format. The server also obtains indirect information, such as the applicant's social media accounts and blog URLs. This information is then stored in a database.

[0374] The server then inputs the stored information into a generative AI for analysis. The generative AI uses Python's NLTK and spaCy libraries to analyze text data. For example, it can analyze information such as "3 years of Python programming experience" on a resume and evaluate the applicant's skills.

[0375] The device collects facial expressions and voices in real time as the user uses the system. This is done using a camera and microphone. The collected emotion information is sent to the server and analyzed by the emotion engine. The emotion engine uses OpenCV and the Google® Cloud Speech-to-Text API to recognize the user's emotional state (joy, anger, sadness, surprise, etc.).

[0376] The server inputs applicant information and emotional information into the generative AI, which then recommends the most suitable department or division. The generative AI then comprehensively evaluates this information and generates a recommendation that takes into account the user's skills and emotional state. The recommendation result is sent to the device and displayed to the user.

[0377] For example, if an applicant's resume states that they have "three years of Python programming experience," generative AI will analyze this information and evaluate the applicant as having programming skills. Similarly, if information gathered from social media posts indicating an applicant's interest in data science is collected, the AI ​​will recommend a data science-related department.

[0378] Example prompt sentence:

[0379] "The applicant's resume states that he has three years of Python programming experience. His social media posts also indicate an interest in data science. Can you recommend a department that would be a good fit for him?"

[0380] In this way, a system is realized in which the server, terminals, and users work together to support optimal recruitment of personnel.

[0381] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0382] Step 1: Gather applicant information

[0383] The server receives resumes and application forms submitted by applicants. These can be in formats such as PDF or Word files. The server also collects URLs for applicants' social media accounts and blogs. The input is the files uploaded by applicants and the URLs entered, and the output is the collected applicant information.

[0384] Step 2: Save your applicant information

[0385] The server stores the collected applicant information in a database. Specifically, information such as the applicant's name, contact information, educational background, and work history is stored in the corresponding fields of the database. The input is the applicant information collected in step 1, and the output is the information stored in the database.

[0386] Step 3: Analyze applicant information

[0387] The server inputs the stored applicant information into the generative AI for analysis. The generative AI uses Python's NLTK and spaCy libraries to analyze text data. For example, it analyzes information such as "3 years of Python programming experience" and evaluates whether the applicant has programming skills. The input is the applicant information stored in the database, and the output is the analysis results.

[0388] Step 4: Collecting emotional information

[0389] The device collects facial expressions and voices in real time when the user uses the system. This is done using a camera and microphone. The input is the user's facial expressions and voice, and the output is the collected emotional information.

[0390] Step 5: Emotional Analysis

[0391] The server analyzes the emotion information sent from the device. The emotion engine uses OpenCV and Google Cloud Speech-to-Text API to recognize the user's emotional state (e.g., joy, anger, sadness, surprise). The input is the collected emotion information, and the output is the analyzed emotional state.

[0392] Step 6: Nominate a department or division

[0393] The server inputs applicant information and emotional information into the generative AI, which then recommends the most suitable department or division. The generative AI then comprehensively evaluates this information and generates a recommendation result, taking into account the user's skills and emotional state. The input is the analyzed applicant information and emotional information, and the output is a recommendation result.

[0394] Step 7: Viewing Recommendations

[0395] The terminal displays the recommendation results sent from the server to the user. For example, the screen may display "The best department for you is the Data Science Department." The input is the recommendation results sent from the server, and the output is the recommendation results displayed to the user.

[0396] (Application example 1)

[0397] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] Conventional recruitment support systems not only collect and analyze direct and indirect information about applicants, but also assess their credibility based on facial expressions and voice data during interviews. However, these systems lacked the ability to recognize applicants' emotional states in real time and provide that information to generative AI. Furthermore, conventional systems were not adequately equipped to evaluate the aptitude of autonomous vehicle drivers or recommend optimal driving tasks and routes. This made it difficult to make the most of a driver's skills and aptitude.

[0399] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes: means for using a generative AI to make a judgment based on direct and indirect information about the applicant and suggesting a department or division where the applicant's skills can be maximized based on information about the applicant's facial expressions and credibility from voice data during the interview; means for recognizing the user's emotional state in real time using an emotion engine and providing that information to the generative AI; means for collecting and analyzing the resumes, driving history, social media posts, and blog articles of autonomous vehicle drivers using the generative AI; and means for the generative AI to evaluate the driver's skills and aptitude and recommend optimal driving tasks and routes. This makes it possible to maximize the skills and aptitude of the applicant and driver and recommend optimal departments, divisions, driving tasks, and routes.

[0400] "Generative AI" is an artificial intelligence system that collects and analyzes information on applicants and drivers, and then uses the results to recommend the most suitable department, division, driving task, and route.

[0401] "Direct information" refers to official documents and data submitted by applicants and drivers, such as resumes and driving records.

[0402] "Indirect information" refers to unofficial data such as posts and articles published by applicants and drivers on social media, blogs, etc.

[0403] The "emotion engine" is a system that recognizes the user's emotional state in real time and provides that information to generative AI.

[0404] "Facial expression and voice data during the interview" refers to data such as the applicant's facial expressions and tone of voice collected during the interview.

[0405] "Credibility" is an indicator that evaluates whether the information and statements provided by applicants and drivers are true.

[0406] A "driver" is a person who operates and monitors an autonomous vehicle.

[0407] "Driving history" is data showing the driver's past driving records and experiences.

[0408] A "driving task" is a specific driving task or operation that a driver performs in an autonomous vehicle.

[0409] A "route" is the path or route that an autonomous vehicle will travel.

[0410] As an embodiment of the present invention, the following system is constructed.

[0411] System Program

[0412] Hardware and software used

[0413] Hardware: On-board computers, cameras, microphones, and emotion recognition sensors for autonomous vehicles

[0414] Software: Generative AI models (e.g., GPT-4), emotion engines (e.g., Affectiva), data collection and analysis tools (e.g., ElasticSearch®)

[0415] Program processing

[0416] The server first digitally collects the driver's resume and driving history, then uses a web scraping tool to collect the driver's social media posts and blog posts, and stores this data in Elasticsearch for search and analysis.

[0417] The generative AI model (GPT-4) evaluates the driver's skills and aptitude based on the collected data. Furthermore, an emotion recognition sensor collects real-time emotional data from the driver, which is then analyzed by the emotion engine (Affectiva). The emotional data analyzed by the emotion engine is provided to the generative AI model, which then makes an evaluation that takes into account the driver's emotional state.

[0418] Ultimately, the generative AI model recommends optimal driving tasks and routes based on the driver's skills, aptitudes, and emotional state.

[0419] Specific examples

[0420] For example, if Driver A applies for a new driving task, the system operates as follows:

[0421] 1. The server collects Driver A's resume and driving history.

[0422] 2. The server collects Driver A's social media posts and blog articles.

[0423] 3. The server collects real-time emotional data of Driver A using an emotion recognition sensor.

[0424] 4. A generative AI model (GPT-4) analyzes this data and evaluates Driver A's skills and aptitude.

[0425] 5. The emotion engine (Affectiva) analyzes Driver A's emotional state and provides it to the generative AI model.

[0426] 6. The generative AI model recommends optimal driving tasks and routes for Driver A.

[0427] Prompt Sentence Examples

[0428] Evaluate Driver A's skills and aptitude based on his / her resume, driving history, social media posts, blog articles, and real-time sentiment data, and recommend the optimal driving tasks and routes.

[0429] In this way, it is possible to make the most of the skills and aptitudes of applicants and drivers and recommend the most suitable departments, divisions, driving tasks and routes.

[0430] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0431] Step 1:

[0432] The server collects the driver's submitted resume and driving history in digital format. The input is the driver's resume and driving history, which are stored in a database. The output is the digitized resume and driving history.

[0433] Step 2:

[0434] The server uses a web scraping tool to collect the driver's social media posts and blog articles. The input is the driver's social media account and blog URL, which are scraped and saved as text data. The output is the collected text data of the social media posts and blog articles.

[0435] Step 3:

[0436] The server collects real-time emotional data from the driver using an emotion recognition sensor. The input is the driver's facial expressions and voice data, which are analyzed by the emotion recognition sensor. The output is the analyzed emotional data.

[0437] Step 4:

[0438] The server stores the collected data in Elasticsearch for search and analysis. The inputs include digitized resumes, driving records, social media posts, blog posts, and sentiment data, which are indexed in Elasticsearch. The output is the indexed data.

[0439] Step 5:

[0440] The generative AI model (GPT-4) evaluates the driver's skills and aptitude based on collected data. It takes indexed data as input, analyzes it, and evaluates the driver's skills and aptitude. The output is the evaluation result.

[0441] Step 6:

[0442] The emotion engine (Affectiva) analyzes the driver's emotional state and provides it to a generative AI model. It takes real-time emotional data as input, analyzes it, and evaluates the emotional state. The output is the evaluated emotional state.

[0443] Step 7:

[0444] The generative AI model recommends optimal driving tasks and routes based on the driver's skills, aptitude, and emotional state. The inputs are the evaluation results of skills and aptitude, and the emotional state, and the optimal driving tasks and routes are generated based on these. The output is the recommended driving tasks and routes.

[0445] Example 2

[0446] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0447] Conventional interview systems have difficulty accurately assessing the credibility and seriousness of applicants, and are unable to recommend the most appropriate department or division based on the applicant's emotional state and skills. This has resulted in low accuracy in hiring decisions and makes it difficult to assign the right person to the right position.

[0448] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting facial expression data of applicants, means for collecting voice data of applicants, means for storing the collected facial expression data and voice data in a database, means for transmitting the stored data to a generative AI model, means for the generative AI model to analyze the data using facial recognition software and voice recognition software, means for the generative AI model to evaluate the applicant's credibility and seriousness based on the analysis results, means for reflecting the evaluation results in the hiring decision system, means for an emotion engine to collect user emotion information, means for transmitting the collected emotion information to the generative AI model, means for the generative AI model to recommend the most appropriate department or division based on the emotion information and the user's skills and aptitude, and means for presenting the recommendation results to the user. This makes it possible to accurately evaluate the applicant's credibility and seriousness and further recommend the most appropriate department or division based on the emotion information and skills.

[0449] An "applicant" is a person who applies for a job interview.

[0450] "Facial expression data" is digital data relating to the facial expressions of applicants, captured using a photographic device such as a webcam.

[0451] "Audio data" is digital data relating to the applicant's speech or voice, captured using a recording device such as a microphone.

[0452] The "database" is an information management system for storing collected facial expression data and voice data.

[0453] A "generative AI model" is an artificial intelligence model that analyzes collected data and generates evaluation and recommendation results.

[0454] "Facial recognition software" is software that analyzes facial expression data and recognizes specific expressions and emotions.

[0455] "Voice recognition software" is software that analyzes voice data and recognizes the content and emotions of what is being said.

[0456] "Credibility" is an evaluation metric that indicates how trustworthy an applicant's statements and actions are.

[0457] "Seriousness" is an evaluation indicator that shows how seriously the applicant is taking the interview.

[0458] The "employment decision system" is a system for making final employment decisions based on the applicant's evaluation results.

[0459] The "emotion engine" is a system for collecting and analyzing users' emotional information in real time.

[0460] "Emotion information" is data that indicates the user's emotional state, and includes emotions such as joy, sadness, and anger.

[0461] "Skills" is information that indicates the techniques and abilities possessed by a user.

[0462] "Aptitude" is an evaluation index that indicates how well a user is suited to a particular job or role.

[0463] A "department" is a division within a company or organization that is responsible for a specific task.

[0464] A "department" is a large organizational unit that oversees multiple departments within a company or organization.

[0465] "Recommendation results" are suggestions for the department or division that is best suited to the user, generated by the generative AI model based on emotional information and skills.

[0466] This invention is a system that collects facial expression and voice data from applicants and analyzes this data using a generative AI model to evaluate the applicant's credibility and seriousness, and then recommends the most suitable department or division based on emotional information and skills.

[0467] First, the device uses a webcam (e.g., a general webcam) and a microphone (e.g., a general microphone) to collect facial expression data and voice data from the applicant. The collected data is then sent to a server in real time.

[0468] The server stores the received facial expression and voice data in a database, where each applicant is assigned a unique ID and the data is organized.

[0469] The server then transmits the stored data to a generative AI model (e.g., a general generative AI model), which analyzes the facial expression data using facial recognition software (e.g., general facial recognition software) and the voice data using voice recognition software (e.g., general voice recognition software).

[0470] The generative AI model evaluates the applicant's credibility and seriousness based on this data. The evaluation results are quantified as credibility and seriousness scores and returned to the server. The server reflects the evaluation results received from the generative AI model in the hiring decision system. Recruiters make the final hiring decision based on these evaluation results.

[0471] Furthermore, when a user uses the system, an emotion engine (e.g., a general emotion engine) collects the user's facial expressions and voice in real time using a webcam and microphone. The emotion engine generates the user's emotion information based on the collected data. The emotion information includes emotional states such as joy, sadness, and anger.

[0472] The emotion engine sends the generated emotion information to a server. The server provides this information to a generative AI model. The generative AI model takes into account the emotion information and the user's skills and aptitudes to recommend the most suitable department or division. The recommended results are presented as specific department or division names.

[0473] Examples:

[0474] As users are interviewed, a webcam and microphone are used to capture their facial expressions and voice.

[0475] The server sends the collected data to a generative AI model to evaluate its credibility and seriousness.

[0476] The generative AI model returns the evaluation results to the server, which are used to make hiring decisions.

[0477] The emotion engine collects emotion information in real time while the user is using the system.

[0478] The emotion engine transmits the collected emotion information to the server.

[0479] The generative AI model recommends the most suitable department or division based on emotional information and skills.

[0480] The server presents the recommendation results to the user.

[0481] Example prompt sentence:

[0482] "Analyze applicants' facial expressions and audio data to assess their authenticity and seriousness."

[0483] "Recommend the best department or division based on the user's sentiment and skills."

[0484] The above is an embodiment of the present invention.

[0485] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0486] Step 1:

[0487] The terminal uses a webcam and a microphone to collect facial expression data and voice data of the applicant. Specifically, the webcam captures a picture of the applicant's face and the microphone records the applicant's voice. These data are transmitted to a server in real time. The input is video and voice data from the webcam and microphone, and the output is digital data transmitted to the server.

[0488] Step 2:

[0489] The server stores the received facial expression data and voice data in a database. Specifically, it assigns a unique ID to each applicant and organizes and stores the data. The input is digital data sent from the terminal, and the output is structured data stored in the database.

[0490] Step 3:

[0491] The server sends the stored data to the generative AI model. Specifically, it extracts the necessary data from the database and calls an API to send it to the generative AI model. The input is the digital data extracted from the database, and the output is the data sent to the generative AI model.

[0492] Step 4:

[0493] The generative AI model analyzes data using facial recognition software and voice recognition software. Specifically, facial expression data is analyzed using facial recognition software, and voice data is analyzed using voice recognition software. The input is the digital data sent from the server, and the output is a credibility score and seriousness score as the analysis results.

[0494] Step 5:

[0495] The generative AI model evaluates the applicant's credibility and seriousness based on the analysis results. Specifically, it quantifies the analysis results and evaluates them as a credibility score and seriousness score. The input is the analysis results from facial recognition and voice recognition software, and the output is a credibility score and seriousness score.

[0496] Step 6:

[0497] The server reflects the evaluation results received from the generative AI model in the hiring decision system. Specifically, it inputs the evaluation results into the hiring decision system and makes them available to hiring managers. The input is the credibility score and seriousness score from the generative AI model, and the output is the evaluation results reflected in the hiring decision system.

[0498] Step 7:

[0499] When a user uses the system, the emotion engine collects the user's facial expressions and voice in real time using a webcam and microphone. Specifically, the webcam captures a picture of the user's face and the microphone records the user's voice. The input is video and audio data from the webcam and microphone, and the output is digital data sent to the emotion engine.

[0500] Step 8:

[0501] The emotion engine generates the user's emotional information based on the collected data. Specifically, it analyzes facial expression data and voice data to identify the user's emotional state. The input is digital data from the webcam and microphone, and the output is emotional information data.

[0502] Step 9:

[0503] The emotion engine sends the generated emotion information to a server. Specifically, it saves the emotion information in a database and calls an API to provide it to the generative AI model. The input is data as emotion information, and the output is the data sent to the server.

[0504] Step 10:

[0505] The generative AI model takes into account emotional information and the user's skills and aptitudes to recommend the most suitable department or division. Specifically, it analyzes emotional information and skill data to identify the most suitable department or division. The input is emotional information and skill data, and the output is data representing the recommendation result.

[0506] Step 11:

[0507] The server presents the recommendation results received from the generative AI model to the user. Specifically, it provides an interface for displaying the recommendation results on the user's screen. The input is the recommendation results from the generative AI model, and the output is the recommendation results presented to the user.

[0508] (Application example 2)

[0509] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0510] With conventional interview systems, it was difficult to accurately assess the credibility and seriousness of applicants, making it difficult to assign them to the appropriate department or division. In addition, there was a lack of means to evaluate the emotional state and work efficiency of employees working in factories in real time, which created challenges in improving employee aptitude and work efficiency. To solve these problems, a system is needed that analyzes the facial expressions and voice data of applicants and employees in real time and makes appropriate decisions.

[0511] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0512] In this invention, the server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about the applicant's credibility from facial expressions and voice data during the interview, means for analyzing facial expressions and voice data of employees working in the factory in real time to evaluate the employee's emotional state and work efficiency, means for collecting data through a camera and microphone mounted on a robot and analyzing the data with generative AI, and means for using the analysis results to help improve the aptitude and work efficiency of employees. This makes it possible to evaluate the aptitude of applicants and improve the work efficiency of employees.

[0513] "Generative AI" refers to artificial intelligence that generates new information and judgments based on input data.

[0514] "Applicant" means a person applying for a particular job or role.

[0515] "Direct information" refers to explicit data provided by applicants, such as resumes and curriculum vitae.

[0516] "Indirect information" refers to data obtained from the applicant's behavior and reactions, such as facial expressions and voice data during the interview.

[0517] "Credibility" is an indicator of how truthful an applicant's statements and actions are.

[0518] A "department" is a group responsible for a specific task or function within a company or organization.

[0519] A "department" is a further subdivided business unit within a department.

[0520] "Factory workers" refers to workers engaged in manufacturing and production industries.

[0521] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[0522] "Audio data" refers to sound information collected through a voice input device such as a microphone.

[0523] An "emotional state" is the emotion an individual is feeling at a particular moment.

[0524] "Work efficiency" is an index that indicates how much work can be done effectively within a certain amount of time.

[0525] A "robot" is a mechanical device that operates automatically to perform specific tasks.

[0526] A "camera" is a device for taking images or videos.

[0527] A "microphone" is a device for collecting sound.

[0528] "Analysis" is the process of analyzing collected data and extracting meaningful information.

[0529] "Aptitude" refers to an individual's suitability and ability for a particular job or role.

[0530] In order to implement the present invention, the following hardware and software are used.

[0531] Hardware

[0532] Webcam

[0533] microphone

[0534] robot

[0535] software

[0536] OpenCV (image processing library)

[0537] sounddevice (audio processing library)

[0538] EmotionRecognizer (emotion recognition library)

[0539] AIModel (generative AI model)

[0540] System configuration

[0541] The server collects facial and voice data from applicants and employees via webcams and microphones. The collected data is analyzed in real time by EmotionRecognizer. The analysis results are provided to AIModel, which evaluates the applicant's credibility and seriousness, as well as the employee's emotional state and work efficiency.

[0542] Data processing and calculation

[0543] The server processes video data acquired from the webcam using OpenCV to analyze facial expressions. Audio data acquired from the microphone is processed using a sound device to extract audio characteristics. This data is converted into emotional states by the Emotion Recognizer and input into the AI ​​Model. The AI ​​Model combines this emotional data with the applicant's skill information and employee work history to recommend the most suitable department or division.

[0544] Specific examples

[0545] For example, if Employee A working in a factory looks tired while working, the system will recognize that expression and update the AI ​​model with a "feeling of fatigue." Based on that information, the AI ​​model will suggest that Employee A take a break or instruct them to change their work.

[0546] Prompt Sentence Examples

[0547] "Analyze employees' facial expressions and voice data in real time to assess their emotional state. Based on the assessment results, make suggestions to improve their work efficiency."

[0548] In this way, the server can provide a system that evaluates the suitability of applicants and improves the work efficiency of employees.

[0549] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0550] Step 1:

[0551] The server captures video data of applicants and employees in real time through a webcam. The input is the video data from the webcam, and the output is the captured image frames. Specifically, the server uses OpenCV to acquire the video from the camera and process each frame.

[0552] Step 2:

[0553] The server captures voice data of applicants and employees in real time through a microphone. The input is the voice data from the microphone, and the output is the captured voice samples. Specifically, the server uses a sound device to acquire voice from the microphone and collects voice samples at regular time intervals.

[0554] Step 3:

[0555] The server analyzes the captured video data using EmotionRecognizer to extract facial expression data. The input is the captured image frame, and the output is the analyzed facial expression data. Specifically, the server inputs the image frame acquired by OpenCV into EmotionRecognizer, analyzes facial feature points, and identifies the emotional state.

[0556] Step 4:

[0557] The server analyzes the captured audio data using EmotionRecognizer and extracts the emotional state from the audio data. The input is the captured audio sample, and the output is the analyzed audio emotion data. Specifically, the server inputs the audio sample acquired by the sound device into EmotionRecognizer, analyzes the audio features, and identifies the emotional state.

[0558] Step 5:

[0559] The server inputs the analyzed facial expression data and voice data into the AIModel to evaluate the applicant's credibility and seriousness, as well as the employee's emotional state and work efficiency. The input is the analyzed facial expression data and voice data, and the output is the evaluation results. Specifically, the server inputs the emotional data obtained from the EmotionRecognizer into the AIModel and performs an integrated evaluation.

[0560] Step 6:

[0561] Based on the evaluation results of the AI ​​model, the server recommends the most suitable department or division for the applicant and makes suggestions to employees to improve work efficiency. The input is the evaluation results of the AI ​​model, and the output is the recommendation results and proposals. Specifically, the server analyzes the evaluation results of the AI ​​model and generates a prompt sentence to recommend an appropriate department or division.

[0562] Step 7:

[0563] The user checks the recommendation results and proposals provided by the server and takes action as necessary. The input is the recommendation results and proposals from the server, and the output is the user's response. Specifically, the user makes hiring decisions for applicants and adjusts the work content of employees based on the information from the server.

[0564] Example 3

[0565] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0566] Conventional recruitment systems have difficulty accurately assessing applicants' technical skills and aptitude, and recommending the most suitable departments and projects. Furthermore, there was a lack of methods for assessing credibility using facial expressions and voice data during interviews, which led to a problem of reduced accuracy in hiring decisions. This made it difficult for companies to find suitable talent, and reduced the efficiency of the recruitment process.

[0567] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0568] In this invention, the server includes: means for collecting technical information about an applicant; means for preprocessing the collected information; means for inputting the preprocessed information into a generative AI model to evaluate the applicant's technical skills and aptitude; means for collecting facial expression and voice data about the applicant using a webcam and microphone during the interview; means for analyzing the collected facial expression and voice data in real time with an emotion engine and providing the information to the generative AI model; means for the generative AI model to recommend the most suitable department or project based on the applicant's technical skills and aptitude and credibility during the interview; and means for presenting the recommendation results to the user. This makes it possible to accurately evaluate the applicant's multifaceted information and recommend the most suitable department or project.

[0569] "Means for collecting applicant's technical information" refers to methods and tools for collecting data that demonstrates technical skills and experience, such as the applicant's programming portfolio, commit history, technical blog posts, and question and answer history.

[0570] "Means for preprocessing collected information" refers to methods and tools that perform processes such as tokenization, removal of stop words, conversion of voice data to text, and extraction of features from image data in order to convert collected technical information into a format that is easier to analyze.

[0571] "Means for inputting into a generative AI model to assess an applicant's technical skills and aptitude" refers to methods or tools for inputting pre-processed information into a generative AI model and using the model to assess an applicant's technical skills and aptitude.

[0572] "Methods for collecting facial expression and voice data of applicants using a webcam and microphone during an interview" means methods or tools for capturing facial expressions and voice data of applicants in real time using a webcam and microphone during an interview.

[0573] "Means for analyzing collected facial expression and voice data in real time using an emotion engine and providing that information to a generative AI model" refers to methods and tools for analyzing collected facial expression and voice data in real time using an emotion engine and providing the analysis results to a generative AI model.

[0574] "A means for a generative AI model to recommend the most suitable department or project based on an applicant's technical skills and aptitude, as well as their credibility during the interview" refers to a method or tool that allows a generative AI model to comprehensively evaluate an applicant's technical skills and aptitude, as well as their credibility during the interview, and then recommend the most suitable department or project based on the results.

[0575] "Means for presenting recommendation results to users" refers to methods and tools for presenting the recommendation results produced by the generative AI model to users in an easy-to-understand manner.

[0576] This invention is a system that evaluates the technical skills and aptitude of applicants and recommends the most suitable departments and projects. A specific embodiment of this system will be described below.

[0577] The server first collects the applicant's technical information. Specifically, it uses the GitHub API to obtain the applicant's commit history and repository information, a web scraping tool (e.g., Beautiful Soup) to collect technical blog posts, and the Stack Overflow API to obtain the applicant's question and answer history.

[0578] The server then preprocesses the collected information. Technical blog posts and Stack Overflow posts are tokenized and stopwords are removed using natural language processing tools (e.g., NLTK). Audio data is converted to text using speech recognition tools (e.g., Google Speech-to-Text API). Image data (facial expressions) are subjected to feature extraction using image processing tools (e.g., OpenCV).

[0579] The preprocessed data is then fed into a generative AI model (e.g., GPT-4), which evaluates the applicant's technical skills and aptitude based on a prompt. For example, the following prompt might be used:

[0580] "Assess applicants' technical skills and aptitude based on their GitHub commit history, technical blog posts, and Stack Overflow activity. Also, consider facial and voice data from the interview, as well as emotional information provided by your emotion engine, to assess applicants' authenticity and recommend the best departments and projects for them."

[0581] During the interview, the device uses a webcam and microphone to collect facial expressions and voice data from the candidate in real time, which is then analyzed in real time by an emotion engine (e.g., Affectiva SDK) and the information is fed into a generative AI model.

[0582] The generative AI model comprehensively evaluates the applicant's technical skills and aptitude, as well as their credibility during the interview, and based on the results, recommends the most suitable department and project. The server presents the recommendation results obtained from the generative AI model to the user. The user reviews the recommendation results provided by the server and makes the final hiring decision.

[0583] As a concrete example, consider the case of candidate A, who has a long history of commits on GitHub and regularly posts articles on a technical blog. He is also active on Stack Overflow. During the interview, facial and voice data is collected via a webcam and microphone, and an emotion engine recognizes the candidate's emotional state in real time. Based on this information, the generative AI model evaluates candidate A's skill set and aptitude, and recommends the most suitable department and project for the candidate.

[0584] In this way, a system is realized in which the server, terminals, and users work together to evaluate and recommend applicants. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0585] Step 1:

[0586] The server collects technical information about applicants. Specifically, it uses the GitHub API to obtain the applicant's commit history and repository information, and a web scraping tool (e.g., Beautiful Soup) to collect technical blog posts. It also uses the Stack Overflow API to obtain the applicant's question and answer history. By collecting this data, it obtains input data for understanding the applicant's technical skills and experience.

[0587] Step 2:

[0588] The server preprocesses the collected information. Technical blog articles and Stack Overflow posts are tokenized and stop words are removed using natural language processing tools (e.g., NLTK). Audio data is converted to text using speech recognition tools (e.g., Google Speech-to-Text API). Image data (facial expressions) is subjected to feature extraction using image processing tools (e.g., OpenCV). This results in output data in an easy-to-analyze format.

[0589] Step 3:

[0590] The server inputs the preprocessed data into a generative AI model (e.g., GPT-4) to evaluate the applicant's technical skills and aptitude. The generative AI model analyzes the applicant's skill set and aptitude based on a prompt. For example, the prompt might be, "Please evaluate the applicant's technical skills and aptitude based on their GitHub commit history, technical blog posts, and Stack Overflow activity history." This outputs an evaluation result regarding the applicant's technical skills and aptitude.

[0591] Step 4:

[0592] The device uses a webcam and microphone to collect facial expressions and voice data from applicants during interviews in real time. The collected data is analyzed in real time by an emotion engine (e.g., Affectiva SDK), and the information is provided to a generative AI model. This provides facial expressions, voice data, and emotional information from the interview as input.

[0593] Step 5:

[0594] The server uses a generative AI model to comprehensively evaluate the applicant's technical skills and aptitude, as well as their credibility during the interview, and recommends the most suitable department and project based on the results. The generative AI model suggests the most suitable department and project based on the prompt text. For example, it generates a result such as "Applicant A has strengths in front-end development and is ideal for Project X." This provides the recommendation result as output.

[0595] Step 6:

[0596] The server presents the recommendation results to the user. The user then checks the recommendation results provided by the server and makes the final hiring decision. For example, a message such as "Applicant A is ideal for Project X" is displayed on the dashboard. This provides the user with the information they need to make the best hiring decision.

[0597] (Application example 3)

[0598] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0599] Conventional recruitment systems and factory robot deployment systems were unable to fully utilize the past history and real-time data of applicants and robots, making it difficult to recommend optimal deployments and roles. Furthermore, credibility assessments during interviews and reliability assessments of robots while they are in operation were not adequately performed, making it difficult to achieve efficient personnel and robot deployments. This resulted in issues such as reduced productivity and efficiency for companies.

[0600] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0601] In this invention, the server includes: means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview; means for collecting the robot's past work history, real-time data from sensors, and information about the work environment and analyzing these with the generative AI; means for collecting facial expressions and voice data while the robot is operating and analyzing this data in real time with an emotion engine; and means for evaluating the robot's reliability and efficiency using these data and recommending optimal placement and roles. This makes it possible to recommend optimal placement and roles by utilizing the past history and real-time data of applicants and robots.

[0602] "Generative AI" is an artificial intelligence that analyzes the past history and real-time data of applicants and robots to recommend optimal placements and roles.

[0603] "Direct information" is information provided by the applicant themselves, such as a programming portfolio submitted by the applicant or a GitHub commit history.

[0604] "Indirect information" is information based on the applicant's activity on other platforms, such as technical blog posts or activity history on Stack Overflow.

[0605] "Facial Expressions" refers to facial expression data of applicants collected via webcam during interviews.

[0606] "Audio data" refers to voice data of applicants collected via a microphone during an interview.

[0607] "Credibility" refers to the applicant's degree of honesty and trustworthiness, assessed based on facial expressions and voice data.

[0608] A "department or division" is an organizational unit within a company that is responsible for a specific task or project.

[0609] "Robot's past work history" is a record of work that the robot has done in the past.

[0610] "Real-time data from sensors" refers to data about the current environment and conditions collected by sensors while the robot is operating.

[0611] "Work environment information" is data relating to the environment in which the robot works.

[0612] The "emotion engine" is a system that analyzes facial expressions and voice data in real time while the robot is moving, and generates emotional information.

[0613] "Reliability" is the degree to which a robot can perform a task accurately and efficiently.

[0614] "Efficiency" is the degree to which a robot can achieve maximum results with minimum resources.

[0615] "Optimal placement and role" refers to the position or role in which the robot or applicant can perform the work most effectively.

[0616] To implement the present invention, the following system configuration and program are required.

[0617] First, the server uses generative AI to collect direct and indirect information about the applicant. Direct information includes the applicant's programming portfolio and GitHub commit history, while indirect information includes technical blog posts and activity history on Stack Overflow. Based on this information, the generative AI analyzes the applicant's skills and aptitude.

[0618] The server then uses a webcam and microphone during the interview to collect facial expressions and voice data from the applicant. This data is analyzed in real time by an emotion engine to evaluate the applicant's credibility. Based on this evaluation, the server recommends the department or division where the applicant would be best placed.

[0619] In addition, the server collects the factory robot's past work history, real-time data from sensors, and information on the work environment. This data is analyzed by generative AI, and facial expressions and voice data are also collected while the robot is in operation. An emotion engine analyzes this data in real time and evaluates the robot's reliability and efficiency. Based on this evaluation, it recommends the optimal placement and role of the robot.

[0620] Specific hardware requirements include a webcam, microphone, and various sensors, while the software uses the generative AI model, EmotionEngine.

[0621] For example, when optimizing the placement of welding robots within a factory, the server analyzes historical data on past welding work, real-time temperature and humidity data, and abnormal sounds and smoothness of the robots' movements while they are in operation, and can then recommend the most efficient locations and roles for the welding robots.

[0622] An example of a prompt sentence might be:

[0623] "Analyze the welding robot's past work history data, real-time temperature and humidity data, abnormal sounds while the robot is operating, and smoothness of operation, and recommend the optimal placement and role."

[0624] In this way, the server can utilize past history and real-time data of applicants and robots to recommend optimal placements and roles.

[0625] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0626] Step 1:

[0627] The server collects direct information about applicants (programming portfolios, GitHub commit histories) and indirect information (technical blog posts, Stack Overflow activity histories). Using this information as input, the generative AI analyzes the applicant's skills and aptitude. The output is an evaluation of the applicant's skills.

[0628] Step 2:

[0629] The server uses a webcam and microphone to collect facial expressions and voice data from applicants during interviews. The emotion engine analyzes these data in real time and evaluates the applicant's credibility. The output is a credibility evaluation result.

[0630] Step 3:

[0631] The server integrates the results of Steps 1 and 2 and recommends the department or division where the applicant will be best placed. Using the skill assessment results and credibility assessment results as input, the generative AI calculates the optimal placement. The recommended department or division is obtained as output.

[0632] Step 4:

[0633] The server collects the factory robot's past work history, real-time data from sensors, and information on the work environment. Using this data as input, the generative AI analyzes the robot's behavior. The output is an evaluation of the robot's behavior.

[0634] Step 5:

[0635] The server collects facial expressions and voice data while the robot is operating. The emotion engine analyzes this data in real time and evaluates the robot's reliability and efficiency. The output is a reliability evaluation result.

[0636] Step 6:

[0637] The server integrates the results of steps 4 and 5 and recommends optimal placement and roles for the robots. Using the behavior evaluation results and reliability evaluation results as input, the generative AI calculates the optimal placement. The output is the recommended placement and role.

[0638] Through these steps, the server can utilize the past history and real-time data of applicants and robots to recommend optimal placements and roles.

[0639] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0640] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0641] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0642] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0643] [Second embodiment]

[0644] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0645] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0646] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0647] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0648] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0649] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0650] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0651] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0652] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0653] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0654] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0655] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0656] "Example 1"

[0657] One embodiment of the present invention is a recruitment support system that uses generative AI. This system collects direct information such as resumes and application forms submitted by applicants, as well as indirect information such as social media posts and blog articles, and analyzes this information using generative AI. The generative AI uses this information to evaluate the applicant's skills and aptitude, and recommends the most suitable department or division for the job.

[0658] "Example 2"

[0659] Another embodiment of the present invention includes a function to analyze facial expressions and voice data during interviews. During interviews, facial expressions and voice data of applicants are collected via a webcam and microphone. The generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in the hiring decision.

[0660] "Example 3"

[0661] As a specific example, when considering hiring an engineer, generative AI collects direct information such as the applicant's programming portfolio and GitHub commit history, as well as indirect information such as technical blog posts and activity history on Stack Overflow, and analyzes these. Furthermore, during interviews, the applicant's facial expressions and voice data are collected via webcam and microphone, which are then analyzed by generative AI. Based on this information, the system recommends departments and projects where the applicant is likely to perform best.

[0662] The processing flow of each embodiment will be described below.

[0663] "Example 1"

[0664] Step 1: Collect direct information such as resumes and application forms submitted by applicants, as well as indirect information such as social media posts and blog articles.

[0665] Step 2: Input the collected information into the generative AI.

[0666] Step 3: Generative AI uses this information to evaluate the applicant's skills and aptitude, and recommends the most suitable department or division.

[0667] "Example 2"

[0668] Step 1: Collect facial expressions and audio data from applicants during the interview via webcam and microphone.

[0669] Step 2: Input the collected facial expressions and voice data into the generative AI.

[0670] Step 3: Generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in hiring decisions.

[0671] "Example 3"

[0672] Step 1: Collect direct information such as the applicant's programming portfolio and GitHub commit history, as well as indirect information such as technical blog posts and Stack Overflow activity.

[0673] Step 2: Input the collected information into the generative AI.

[0674] Step 3: Generative AI analyzes this information and evaluates the applicant's skills and aptitude.

[0675] Step 4: During the interview, facial expressions and audio data from the applicant are collected via a webcam and microphone.

[0676] Step 5: Input the collected facial expressions and voice data into the generative AI.

[0677] Step 6: Generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in hiring decisions.

[0678] Step 7: Based on this information, we recommend the department and project where the applicant would be most likely to excel.

[0679] Example 1

[0680] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0681] Conventional recruitment systems have difficulty accurately evaluating applicants' skills and aptitudes and recommending the most suitable department or division. Furthermore, they are unable to take into account not only direct information about applicants but also indirect information, resulting in a problem of reduced evaluation accuracy. Furthermore, there was a lack of a way to effectively provide evaluation results to human resources personnel.

[0682] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for collecting direct and indirect information about the applicant, a means for storing the collected information in a database, a means for analyzing the stored information using a generative AI model, a means for recommending an optimal department or division based on the analysis results, and a means for providing the recommendation results through a management screen. This makes it possible to evaluate the skills and aptitude of the applicant with high accuracy and recommend an optimal department or division.

[0683] "Direct information from the applicant" refers to information provided by the applicant themselves, such as in a resume or application form.

[0684] "Indirect information about applicants" refers to information that applicants have made public on the Internet, such as social media posts or blog articles.

[0685] A "database" is a system for organizing and storing collected information; examples include MySQL and MongoDB.

[0686] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to analyze applicant information and evaluate skills and aptitude.

[0687] The "analysis results" are the results of the skills and aptitudes evaluated by the generative AI model based on the applicant's information.

[0688] "Department or division recommendation" refers to suggesting the most suitable job or role for an applicant based on the analysis results.

[0689] The "administration screen" is an interface for providing recommendation results to human resources personnel, and is built using a front-end framework.

[0690] The present invention relates to a recruitment support system that uses generative AI. A specific embodiment of this system will be described below.

[0691] The server first collects direct and indirect information about applicants. Direct information includes resumes and application forms, which applicants provide themselves. Indirect information includes social media posts and blog articles, which applicants make public online. This information is collected using web scraping tools (e.g., BeautifulSoup and Scrapy) and APIs (e.g., Twitter API and Facebook Graph API).

[0692] Next, the server stores the collected information in a database (e.g., MySQL or MongoDB). When storing the information, it assigns a unique ID to each applicant and organizes the information. For example, the resume information for applicant A is stored in a table called "applicant_A_resume."

[0693] The stored information is analyzed using a generative AI model (e.g., OpenAI's GPT-4). The server inputs a prompt like this into the generative AI model:

[0694] The applicant's resume states that he has "5 years of Python programming experience," and his social media posts state that he is "interested in data science." Use this information to evaluate the applicant's skills and aptitude, and recommend the department or division that would be the best fit.

[0695] The generative AI model evaluates the applicant's skills and suitability based on this prompt and returns the results.

[0696] The server recommends the most suitable department or division for each applicant based on the evaluation results obtained from the generative AI model. For example, if the evaluation result shows that "Applicant A is suitable for the Data Science Division," the server will recommend Applicant A to the Data Science Division.

[0697] Finally, the server provides the recommendation results to the human resources personnel via an administration screen. The administration screen is built using a front-end framework (e.g., React or Vue.js) and can be accessed by the user via a browser. The user can check the recommendation results for applicant A on the administration screen and make a decision on assigning the candidate to an appropriate department.

[0698] In this way, the server, terminals, and users work together to create a system that provides efficient support for recruiting personnel.

[0699] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0700] Step 1: Gather applicant information

[0701] The server receives resumes and application forms submitted by applicants. This direct information is often uploaded in PDF or Word format. The server uses an OCR (optical character recognition) tool (e.g., Tesseract) to convert these files into text data. It also uses web scraping tools (e.g., BeautifulSoup or Scrapy) or APIs (e.g., Twitter API or Facebook Graph API) to collect indirect information such as applicants' social media posts and blog articles. The input is the resume or social media URL, and the output is text data.

[0702] Step 2: Save your applicant information

[0703] The server saves the collected text data in a database (e.g., MySQL or MongoDB). When saving, a unique ID is assigned to each applicant and the information is organized. For example, the resume information for applicant A is saved in a table called "applicant_A_resume." The input is text data, and the output is the information saved in the database.

[0704] Step 3: Analyze applicant information

[0705] The server inputs the stored information into a generative AI model (e.g., OpenAI's GPT-4) for analysis. Specifically, it inputs the following prompt sentence into the generative AI model:

[0706] The applicant's resume states that he has "5 years of Python programming experience," and his social media posts state that he is "interested in data science." Use this information to evaluate the applicant's skills and aptitude, and recommend the department or division that would be the best fit.

[0707] The generative AI model evaluates the applicant's skills and aptitude based on this prompt and returns the results. The input is the prompt and the output is the evaluation result.

[0708] Step 4: Department / Division Recommendations

[0709] The server recommends the most suitable department or division for the applicant based on the evaluation results obtained from the generative AI model. For example, if the evaluation result shows that "Applicant A is suitable for the Data Science Division," the server will recommend Applicant A to the Data Science Division. The input is the evaluation result, and the output is the recommendation result.

[0710] Step 5: Providing recommendations

[0711] The server provides the recommendation results to the human resources personnel via an administration screen. The administration screen is built using a front-end framework (e.g., React or Vue.js) and can be accessed by the user via a browser. The user checks the recommendation results for applicant A on the administration screen and makes a decision on assigning the applicant to an appropriate department. The input is the recommendation results, and the output is the information displayed on the administration screen.

[0712] (Application example 1)

[0713] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0714] Conventional recruitment systems have typically made decisions based on direct information such as applicants' resumes and application forms, but this makes it difficult to fully evaluate the applicant's aptitude and skills. Furthermore, credibility is not assessed based on facial expressions or voice data during interviews, making it difficult to assign candidates to the appropriate department or division. Furthermore, there is a lack of concrete methods for assessing suitability as a driver of an autonomous vehicle.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0716] In this invention, the server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for collecting the applicant's resume, application form, social media posts, blog articles, etc. and for the generative AI to analyze this information to evaluate the applicant's skills and aptitude, and means for evaluating the applicant's aptitude as a driver of an autonomous vehicle and recommending the most suitable position. This makes it possible to comprehensively evaluate the aptitude and skills of the applicant and assign them to the most suitable department or division.

[0717] "Generative AI" is an artificial intelligence technology that collects and analyzes information about applicants and evaluates their aptitude and skills based on the results.

[0718] "Direct information" refers to information contained in official documents submitted by applicants, such as resumes and application forms.

[0719] "Indirect information" refers to information obtained from sources other than official documents, such as applicants' social media posts or blog articles.

[0720] "Facial expressions during an interview" refers to the facial expressions and expressions shown by applicants during an interview.

[0721] "Audio data" refers to data related to the voice of the applicant during the interview, such as the tone of voice and speaking style.

[0722] "Credibility" is a measure of how trustworthy the information provided by the applicant and the expressions given during the interview are.

[0723] "Skills" refers to the specialized knowledge, techniques, and abilities possessed by the applicant.

[0724] "Qualities" are characteristics that indicate how well an applicant is suited to a particular job or role.

[0725] A "department" is a division within a company or organization that is responsible for a specific task or role.

[0726] A "department" refers to a large business unit within a company or organization, and may include multiple departments.

[0727] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0728] "Driver" refers to the person who operates and monitors the autonomous vehicle.

[0729] A "position" refers to a specific role or job within a company or organization.

[0730] The system for implementing this invention is composed of three main elements: a server, a terminal, and a user. The specific roles and processes of each element are explained below.

[0731] Server Roles and Operations

[0732] The server uses the following hardware and software to collect, analyze, and evaluate data:

[0733] Hardware used:

[0734] Server (data collection and execution of AI models)

[0735] Software used:

[0736] Python (data processing and running AI models)

[0737] TensorFlow (implementation of generative AI models)

[0738] Flask (Web application framework)

[0739] BeautifulSoup (web scraping library)

[0740] Data processing and calculation:

[0741] 1. Data Collection:

[0742] The server uses BeautifulSoup to scrape applicants' social media posts and blog posts, and converts the applicants' submitted resumes and application forms from PDFs and images to text.

[0743] 2. Data Preprocessing:

[0744] The server cleans the collected text data, removing unnecessary information, and then uses natural language processing (NLP) techniques to tokenize the text data and extract features.

[0745] 3. Analysis using generative AI models:

[0746] The server uses TensorFlow to build a generative AI model and uses the applicant's text data as input to evaluate their skills and aptitude.

[0747] 4. Displaying the results:

[0748] The server uses Flask to build a web application and display the results.

[0749] Terminal roles and processing

[0750] The terminal provides an interface for the user to enter applicant information and check the results.

[0751] Hardware used:

[0752] Smartphone (entering applicant information and displaying results)

[0753] Software used:

[0754] A web browser (accessing the Flask application)

[0755] User Roles and Actions

[0756] The user enters applicant information and checks the evaluation results generated by the generative AI model.

[0757] Examples:

[0758] Applicant A's resume states that he has "5 years of experience driving self-driving vehicles."

[0759] Applicant A's social media posts state that he "frequently shares articles about the latest autonomous driving technology."

[0760] The generative AI model analyzes this information and assesses that Applicant A has high suitability as a driver of an autonomous vehicle.

[0761] Example prompts to input to a generative AI model:

[0762] The applicant's resume states that he has "5 years of experience driving autonomous vehicles." His social media posts state that he "frequently shares articles about the latest autonomous driving technology." Please rate this applicant's suitability as a driver.

[0763] In this way, a system can be constructed in which the server, terminals, and users work together to comprehensively evaluate the aptitude and skills of applicants and assign them to the most suitable department or division.

[0764] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0765] Step 1:

[0766] The server collects applicants' social media posts and blog articles. Specifically, it uses BeautifulSoup to perform web scraping to obtain text data from applicants' social media accounts and blogs. The input is the URL of the applicants' social media accounts and blogs, and the output is the collected text data.

[0767] Step 2:

[0768] The server converts the resumes and application forms submitted by applicants from PDF or image to text. Specifically, it uses OCR (optical character recognition) technology to extract text data from PDF or image files. The input is the PDF or image file, and the output is the converted text data.

[0769] Step 3:

[0770] The server preprocesses the collected text data. Specifically, it cleans the text data and removes unnecessary information. Then, it uses natural language processing (NLP) techniques to tokenize the text data and extract features. The input is the collected text data, and the output is the preprocessed text data.

[0771] Step 4:

[0772] The server uses TensorFlow to build a generative AI model and evaluates skills and aptitude using preprocessed text data as input. Specifically, the generative AI model inputs text data and outputs a score that evaluates the applicant's skills and aptitude. The input is preprocessed text data, and the output is an evaluation score for skills and aptitude.

[0773] Step 5:

[0774] The server uses Flask to build a web application and display the evaluation results. Specifically, based on the evaluation scores, it generates recommendations for the most suitable department or division for the applicant and displays them through the web application. The input is the evaluation scores for skills and aptitude, and the output is the recommendation results.

[0775] Step 6:

[0776] The terminal provides an interface for users to enter applicant information and check the results. Specifically, users use a smartphone's web browser to access the Flask application and enter applicant information. The input is applicant information, and the output is the recommendation results.

[0777] Step 7:

[0778] The user checks the evaluation results of the generative AI model. Specifically, the user checks the recommendation results on their smartphone screen and inputs additional information as needed. The input is the recommendation results, and the output is the user's feedback.

[0779] In this way, a system can be constructed in which the server, terminals, and users work together to comprehensively evaluate the aptitude and skills of applicants and assign them to the most suitable department or division.

[0780] Example 2

[0781] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0782] With conventional interview systems, it was difficult to objectively evaluate the credibility and seriousness of applicants, and hiring decisions often involved subjective factors. It was also difficult to properly recommend departments or divisions that would best utilize an applicant's skills.

[0783] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for activating the webcam and microphone and starting data collection at the same time as the start of the interview, means for dividing the collected video data into frames and extracting facial regions, means for removing noise from the collected audio data and extracting audio features, means for integrating facial expression data and audio data and inputting the data into a generative AI model, means for the generative AI model to evaluate the credibility and seriousness based on the facial expression data and audio data, and means for providing the evaluation results to the hiring manager. This makes it possible to objectively evaluate the credibility and seriousness of the applicant and make an appropriate hiring decision.

[0784] A "webcam" is a digital camera for capturing video.

[0785] A "microphone" is an acoustic device for capturing sound.

[0786] "Data collection" is the process of capturing video and audio data using a webcam and microphone.

[0787] "Video Data" is a collection of video frames captured by a webcam.

[0788] A "frame" is an individual still image that makes up video data.

[0789] The "face area" refers to the part of the video data where a face is present.

[0790] "Audio Data" is a recording of sound captured by a microphone.

[0791] "Noise reduction" is the process of removing unwanted background sounds and noise from audio data.

[0792] "Voice features" are characteristics such as pitch, tone, and speed contained in the voice data.

[0793] "Facial expression data" is information about facial expressions extracted from video data.

[0794] A "generative AI model" is an algorithm that uses artificial intelligence to analyze data and make specific assessments or predictions.

[0795] "Credibility" refers to the reliability of whether an applicant's statements and actions are true.

[0796] "Seriousness" refers to how seriously the applicant takes the interview.

[0797] The "evaluation results" are scores and judgments of credibility and seriousness calculated by the generative AI model.

[0798] A "recruiter" is a person who makes hiring decisions based on the results of an applicant's interview.

[0799] This invention is a system for analyzing facial expressions and voice data of an applicant during an interview to evaluate the applicant's credibility and seriousness. A specific embodiment of this system will be described below.

[0800] Hardware and software used

[0801] Hardware: Webcam, microphone

[0802] Software: Generative AI models (e.g., general-purpose natural language processing models), facial expression recognition software (e.g., OpenFace), and speech analysis software (e.g., Praat)

[0803] Data collection

[0804] The device activates the webcam and microphone and begins collecting data as soon as the interview begins. The webcam captures video data, and the microphone captures audio data.

[0805] Data Preprocessing

[0806] The device splits the video data it collects into frames and extracts facial regions using facial expression recognition software, while it uses voice analysis software to remove noise and extract voice characteristics from the audio data.

[0807] Analyzing the data

[0808] The server receives the facial expression and voice data sent from the device and integrates them. The integrated data is then input into a generative AI model, which evaluates the authenticity and seriousness of the user based on the facial expression and voice data.

[0809] Output of evaluation results

[0810] The server receives the evaluation results from the generative AI model and provides them to recruiters. The evaluation results are displayed in the form of a dashboard or report, clearly indicating scores for credibility and seriousness.

[0811] Specific examples

[0812] For example, when an applicant introduces themselves, the device collects data via a webcam and microphone. The collected data is sent to a server and analyzed by a generative AI model. The resulting evaluation result may be something like, "The applicant's facial expression indicates nervousness, but their voice tone is stable, and they are evaluated as being highly credible."

[0813] Prompt Sentence Examples

[0814] "Please rate the applicant's credibility and seriousness based on their facial and voice data. Enter the following data: [Facial Expression Data], [Voice Data]"

[0815] In this way, the system analyzes the applicant's facial expressions and voice data during the interview to evaluate their credibility and seriousness, and the results of this evaluation are reflected in the hiring decision.

[0816] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0817] Step 1:

[0818] The device activates the webcam and microphone and starts collecting data at the same time as the interview begins. The input is the start signal for the interview, and the output is the video and audio data captured in real time. Specifically, the device captures the video stream from the webcam and the audio stream from the microphone in real time.

[0819] Step 2:

[0820] The device divides the collected video data into frames and extracts facial regions. The input is the captured video data, and the output is facial region data extracted from each frame. Specifically, the device uses facial expression recognition software (e.g., OpenFace) to extract facial feature points from each frame.

[0821] Step 3:

[0822] The device removes noise from the collected voice data and extracts voice features. The input is the captured voice data, and the output is the voice data with noise removed and features extracted. Specifically, the device uses voice analysis software (e.g., Praat) to extract voice features such as pitch, tone, and speed.

[0823] Step 4:

[0824] The server receives the facial expression data and voice data sent from the device and integrates them. The input is the facial expression data and voice data sent from the device, and the output is the integrated data. Specifically, the server combines the facial expression data and voice data into a single data set.

[0825] Step 5:

[0826] The server inputs the integrated data into the generative AI model to evaluate its credibility and seriousness. The input is the integrated data, and the output is the evaluation result of its credibility and seriousness. Specifically, the server inputs the integrated data as a prompt sentence to the generative AI model (e.g., GPT-4) and performs an evaluation.

[0827] Step 6:

[0828] The server receives the evaluation results from the generative AI model and provides them to recruiters. The input is the evaluation results from the generative AI model, and the output is the evaluation results provided to recruiters. Specifically, the server displays the evaluation results in the form of a dashboard or report, clearly indicating scores for credibility and seriousness.

[0829] (Application example 2)

[0830] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0831] Conventional interview systems make it difficult to accurately assess applicants' credibility and seriousness, and rely heavily on the interviewer's subjective judgment. Furthermore, there was a lack of a way to analyze participants' facial expressions and voices in real time during interviews and meetings to assess their credibility and seriousness, making it difficult to make appropriate personnel assignments and decisions. This hinders companies' efficient use of human resources and the improvement of the quality of important meetings.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0833] In this invention, the server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for analyzing the facial expressions and voice of participants in real time during interviews and important meetings using a smartphone camera and microphone, and means for the generative AI to analyze the data, evaluate the credibility and seriousness of the participants, and display the results in real time. This enables objective evaluation of the credibility and seriousness of applicants and participants, enabling appropriate personnel placement and decision-making.

[0834] "Generative AI" is an artificial intelligence technology that generates new data and information based on input data.

[0835] "Applicant" refers to a person applying for a particular job or role.

[0836] "Direct information" refers to explicit data provided by applicants, such as resumes and curriculum vitae.

[0837] "Indirect information" refers to data obtained from the applicant's behavior and reactions, such as facial expressions and voice data during the interview.

[0838] "Credibility" is a measure of whether an applicant's statements and actions are true.

[0839] "Seriousness" is a measure of how seriously an applicant takes an interview or meeting.

[0840] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and use applications.

[0841] A "camera" is a device for taking images or videos.

[0842] A "microphone" is a device for recording sound.

[0843] "Real-time" refers to data collection and processing occurring immediately.

[0844] A "department" is a group responsible for a specific task or function within a company or organization.

[0845] A "department" is a small group responsible for a specific task or role within a company or organization.

[0846] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[0847] "Display" refers to the visual presentation of data or information.

[0848] A system for carrying out this invention has the following configuration: The server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for analyzing the facial expressions and voices of participants in real time during interviews and important meetings using a smartphone camera and microphone, and means for the generative AI to analyze the data, evaluate the credibility and seriousness of the participants, and display the results in real time.

[0849] Hardware and software used

[0850] Hardware: Smartphone camera, smartphone microphone

[0851] Software: OpenCV (video capture and processing), SoundDevice (audio capture), TensorFlow (AI model for facial expression analysis), Transformers (generative AI model for audio data)

[0852] Data processing and calculation

[0853] 1. Video data capture and processing:

[0854] Use your smartphone camera to capture real-time footage of participants during interviews and meetings.

[0855] The video data is processed using OpenCV to extract the facial area.

[0856] The extracted facial area is subjected to facial expression analysis using TensorFlow to evaluate credibility and seriousness.

[0857] 2. Audio data capture and processing:

[0858] Use your smartphone's microphone to capture participants' voices in real time during interviews and meetings.

[0859] Audio data is recorded using SoundDevice and analyzed using Transformers.

[0860] The analysis results are used to evaluate the participants' credibility and seriousness.

[0861] Specific examples

[0862] For example, to evaluate in real time whether an applicant's facial expression is nervous during an interview, the applicant's face is captured with a smartphone camera and the facial region is extracted using OpenCV. The extracted facial region is analyzed using TensorFlow to evaluate the level of nervousness. Similarly, to analyze the seriousness of participants' comments during a meeting from audio data, the audio is captured with a smartphone microphone and recorded using SoundDevice. The recorded audio data is analyzed using Transformers to evaluate the level of seriousness.

[0863] Prompt Sentence Examples

[0864] "Analyze the applicant's facial expression data and assess their level of nervousness."

[0865] "Please analyze the seriousness of statements made during meetings from audio data."

[0866] In this way, the system can objectively assess the credibility and seriousness of applicants and participants, and support appropriate staffing and decision-making.

[0867] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0868] Step 1:

[0869] The server uses the smartphone camera to capture video of participants in interviews or meetings in real time. The input is video data from the smartphone camera, and the output is the captured video data. Specifically, the camera starts up and continuously acquires video frames.

[0870] Step 2:

[0871] The server processes the captured video data using OpenCV and extracts the face region. The input is the video data obtained in step 1, and the output is the extracted face region data. Specifically, it applies a face detection algorithm to the video frame to identify the position of the face.

[0872] Step 3:

[0873] The server performs facial expression analysis on the extracted facial region using TensorFlow. The input is the facial region data obtained in step 2, and the output is the result of the facial expression analysis. Specifically, the facial region data is preprocessed and input into an AI model to analyze facial expression features.

[0874] Step 4:

[0875] The server uses the smartphone's microphone to capture the voice of participants in an interview or meeting in real time. The input is the voice data from the smartphone's microphone, and the output is the captured voice data. Specifically, the microphone is turned on and continuously records voice.

[0876] Step 5:

[0877] The server records the audio data captured using SoundDevice and analyzes it using Transformers. The input is the audio data obtained in step 4, and the output is the result of the audio analysis. Specifically, the audio data is preprocessed and input into the AI ​​model to analyze the audio features.

[0878] Step 6:

[0879] The server integrates the results of the facial expression analysis and voice analysis obtained in steps 3 and 5 to evaluate the participants' credibility and seriousness. The inputs are the results of the facial expression analysis and voice analysis, and the output is the evaluation results of credibility and seriousness. Specifically, the server inputs both analysis results into an integration algorithm to perform a comprehensive evaluation.

[0880] Step 7:

[0881] The server displays the evaluation results in real time and provides feedback to the interviewer and the conference organizer. The input is the evaluation results obtained in step 6, and the output is the displayed feedback information. Specific operations include updating the interface to visually display the evaluation results.

[0882] Example 3

[0883] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] In the traditional recruitment process for engineers, it was difficult to accurately evaluate applicants' technical skills and experience and assign them to the most appropriate departments and projects. Furthermore, there was a lack of methods to evaluate applicants' emotions and communication skills using facial expressions and voice data during interviews, which led to a problem of reduced accuracy in recruitment.

[0885] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for collecting information about the applicant; means for transmitting the collected information to the server; means for the server to analyze the applicant's technical skills and experience using a generative AI model; means for the terminal to collect facial expression and voice data about the applicant via a webcam and microphone during the interview; means for transmitting the collected interview data to the server; means for the server to analyze the interview data using the generative AI model and evaluate the applicant's emotions and communication skills; means for the server to recommend a department or project that is most suitable for the applicant based on the analysis results; and means for notifying the user of the recommendation results. This makes it possible to comprehensively evaluate the applicant's technical skills, experience, emotions, and communication skills and assign them to the most suitable department or project.

[0886] "Means for collecting applicant information" refers to devices or software for obtaining data such as the applicant's programming portfolio, online platform history, technical blog posts, and activity history on Q&A sites.

[0887] The "means for transmitting collected information to the server" refers to a communication device or software for transferring applicant information from the terminal to the server.

[0888] "Means for analyzing applicants' technical skills and experience using generative AI models" refers to the AI ​​model and its execution environment used to analyze collected applicant information and evaluate technical skills and experience.

[0889] "Means for the device to collect facial expressions and voice data of the applicant through a webcam and microphone during the interview" refers to hardware and software for capturing the facial expressions and voice of the applicant in real time during the interview.

[0890] The "means for transmitting collected interview data to the server" refers to a communication device or software for transferring facial expression and voice data collected during the interview to the server.

[0891] "Means for analyzing interview data using a generative AI model to evaluate applicants' emotions and communication skills" refers to an AI model and its execution environment used to analyze collected interview data and evaluate applicants' emotions and communication skills.

[0892] "The means by which the server recommends the department or project that is most suitable for the applicant based on the analysis results" refers to the algorithm and its execution environment for determining and recommending the most suitable department or project for the applicant based on the analysis results of the AI ​​model.

[0893] The "means for notifying the user of the recommendation results" refers to a communication device or software for transmitting the recommendation results from the server to the user.

[0894] This invention is a system that streamlines the engineer recruitment process and comprehensively evaluates applicants' technical skills, experience, emotions, and communication skills. Specific embodiments of this system are described below.

[0895] First, the user initiates the engineer recruitment process. The terminal provides an interface for collecting information about the applicant. The terminal uses web scraping technology and APIs to obtain data such as the applicant's programming portfolio, online platform history, technical blog posts, and activity history on Q&A sites.

[0896] The collected information is sent from the device to a server, which then uses a generative AI model (such as OpenAI's GPT-4) to analyze the applicant's technical skills and experience. Specifically, the system evaluates the quality of the code, the frequency of commits, and the content of technical blog posts.

[0897] Next, as the user conducts the interview, the device collects the candidate's facial expressions and voice data through a webcam and microphone. Specifically, data is acquired using a Logitech webcam and a Blue Yeti microphone. The collected interview data is then sent from the device to a server.

[0898] The server then inputs the collected interview data back into the generative AI model, which uses facial recognition and voice analysis technologies to evaluate the applicant's emotions and communication skills. For example, it analyzes whether the applicant spoke confidently during the interview.

[0899] Based on the results of these analyses, the server recommends the department and project that best suits the applicant. It then comprehensively assesses the results of the evaluation of technical and communication skills to determine the most suitable assignment. The server notifies the user of the recommendation results. The user can then check the recommendation results on the system's dashboard and make the final hiring decision.

[0900] As a concrete example, a user starts a hiring process for an engineer, collects the commit history from Applicant A's GitHub account, and also retrieves technical blog posts. The server inputs the collected data into GPT-4 to analyze Applicant A's technical skills. For example, the result may be "Applicant A is strong in JavaScript and React."

[0901] During the interview, the user interviews Applicant A, and the device uses the Logitech webcam and Blue Yeti microphone to collect Applicant A's facial expressions and voice. The server inputs the collected interview data into GPT-4 to analyze Applicant A's communication skills. For example, the result may be, "Applicant A speaks confidently during the interview."

[0902] Finally, based on these analysis results, the server recommends Applicant A to the front-end team. The user can check the recommendation results on the system dashboard and make the final hiring decision.

[0903] An example of a prompt sentence is, "Please analyze applicant A's GitHub commit history and technical blog articles to evaluate his / her technical skills. Also, please evaluate his / her communication skills based on facial expressions and voice data during the interview and recommend the most suitable department for him / her." The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0904] Step 1:

[0905] The user initiates the hiring process for an engineer. The device displays a form for entering applicant information. The user enters information such as the applicant's programming portfolio, history on online platforms, technical blog posts, and activity history on Q&A sites. The entered information is saved on the device.

[0906] Step 2:

[0907] The device collects the information entered by the applicant. Specifically, it obtains data through links or files provided by the applicant. For example, it downloads the commit history from a GitHub URL and obtains technical blog posts as text data. The collected information is stored on the device.

[0908] Step 3:

[0909] The device sends the collected information to the server. Specifically, it transfers the data to the server using an HTTP request. The server stores the received data in a database. The input is the applicant's information, and the output is the data stored on the server.

[0910] Step 4:

[0911] The server retrieves applicant information from a database and inputs it into a generative AI model. Specifically, a Python script is used to pass the data to the AI ​​model. The AI ​​model analyzes the applicant's technical skills and experience. For example, it evaluates the quality of the code, the frequency of commits, and the content of technical blog posts. The input is the applicant's information, and the output is the evaluation of their technical skills and experience.

[0912] Step 5:

[0913] When a user conducts an interview, the device collects facial expression and voice data from the applicant through a webcam and microphone. Specifically, data is acquired using a Logitech webcam and a Blue Yeti microphone. The collected data is stored on the device. The input is facial expression and voice data from the interview, and the output is the data stored on the device.

[0914] Step 6:

[0915] The device sends the collected interview data to the server. Specifically, it transfers the data to the server using an HTTP request. The server stores the received data in a database. The input is the interview data, and the output is the data stored on the server.

[0916] Step 7:

[0917] The server retrieves interview data from the database and inputs it into the generative AI model. Specifically, a Python script is used to pass the data to the AI ​​model. The AI ​​model analyzes the applicant's emotions and communication skills from their facial expressions and voice. For example, it uses facial recognition technology and voice analysis technology to make the evaluation. The input is the interview data, and the output is the evaluation results of emotions and communication skills.

[0918] Step 8:

[0919] The server recommends the department and project that best suits the applicant based on the evaluation results of technical and communication skills. Specifically, it uses an algorithm to comprehensively judge the evaluation results and determine the most suitable assignment. The input is the evaluation results of technical and communication skills, and the output is the recommendation results.

[0920] Step 9:

[0921] The server notifies the user of the recommendation results. Specifically, the results are communicated via email or the system dashboard. The user checks the recommendation results and makes the final adoption decision. The input is the recommendation results, and the output is the results notified to the user.

[0922] (Application example 3)

[0923] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0924] Conventional engineer recruitment systems have difficulty accurately evaluating applicants' skills and aptitudes and recommending the most suitable departments and work content. Furthermore, there was no system that could assess the aptitude of robot operators working in factories and recommend the most suitable departments and work content, making it difficult to assign the right person to the right job.

[0925] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: a means for using generative AI to make a judgment based on direct and indirect information about the applicant and suggesting a department or division where the applicant's skills can be maximized based on information about credibility from facial expressions and voice data during the interview; a means for diagnosing the aptitude of robot operators working in factories and recommending the most appropriate department and work content; a means for capturing facial expressions using a webcam and analyzing emotions with a generative AI model; and a means for capturing voice data using a microphone, converting it into text with a generative AI model, and analyzing emotions and content. This makes it possible to accurately evaluate the aptitude of applicants and robot operators and recommend the most appropriate department and work content.

[0926] "Generative AI" is an artificial intelligence technology that generates new information and results based on input data.

[0927] "Direct information" refers to specific data provided by the applicant themselves, such as a programming portfolio submitted by the applicant or a GitHub commit history.

[0928] "Indirect information" refers to supplementary data obtained from an applicant's activities and behavior, such as technical blog posts or activity history on Stack Overflow.

[0929] "Facial expressions" refers to the applicant's facial movements and changes in expression captured via webcam during the interview.

[0930] "Audio data" refers to audio information such as the applicant's statements and tone of voice collected through a microphone during an interview.

[0931] "Credibility" refers to the reliability of the information provided by the applicant and the facial and voice data during the interview.

[0932] A "department or division" is a group or section within a company or organization that is responsible for a specific task or role.

[0933] A "robot operator working in a factory" is a person who operates robots in a factory and performs tasks such as manufacturing and assembly.

[0934] "Aptitude" refers to the ability and adaptability of an applicant or robot operator for a particular task or role.

[0935] A "webcam" is a camera device that connects to a computer or other device to capture video.

[0936] A "microphone" is a device for capturing sound and converting it into a digital signal.

[0937] "Analyzing emotions" means assessing the emotional state of applicants based on collected facial expressions and voice data.

[0938] "Convert to text" is the process of converting audio data into written information.

[0939] "Analyzing emotions and content" means evaluating the emotions and content of statements based on audio data converted into text.

[0940] The system for implementing this invention consists of three main elements: a server, a terminal, and a user. The following explains how each element functions.

[0941] server

[0942] The server uses generative AI to analyze direct and indirect information about applicants. Specifically, it collects information such as programming portfolios submitted by applicants, GitHub commit histories, technical blog posts, and activity history on Stack Overflow, and analyzes this data using a generative AI model. Furthermore, during interviews, facial expressions and voice data collected via a webcam and microphone are analyzed to assess credibility. This allows the server to recommend departments and divisions that can best utilize the applicant's skills and aptitudes.

[0943] Terminal

[0944] The devices are computers or smart devices equipped with webcams and microphones. These devices are used to capture the applicant's facial expressions and voice data during the interview. The webcam uses the OpenCV library to capture facial expressions and analyze emotions using a generative AI model. The microphone uses the SpeechRecognition library to capture voice data and convert it into text. The text is then analyzed for sentiment and content using a generative AI model.

[0945] User

[0946] Users are recruiters and factory managers at companies using the system. Based on the analysis results generated by the server, users evaluate the suitability of applicants and robot operators and determine the most suitable departments and work tasks. This makes it possible to assign the right people to the right jobs.

[0947] Specific examples

[0948] For example, when a new robot operator for a factory is interviewed, facial expressions and voice are collected via the device's webcam and microphone, and a generative AI on the server analyzes these to assess suitability. If the applicant says, "I'm good at detailed work," the generative AI analyzes the emotion and content of the statement and recommends a department that requires detailed work.

[0949] Prompt Sentence Examples

[0950] "Collect facial expressions and voice data from applicants and use generative AI to analyze their emotions and content. Based on the analysis results, recommend the most suitable department and work content."

[0951] In this way, a system is realized in which the server, terminals, and users work together to accurately evaluate the suitability of applicants and robot operators and recommend the most suitable departments and work contents.

[0952] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0953] Step 1:

[0954] The server collects direct information about applicants (e.g., programming portfolios, GitHub commit histories) and indirect information (e.g., technical blog posts, Stack Overflow activity histories). This information is obtained from data submitted online by applicants and publicly available databases. The input is various data about applicants, and the output is a dataset of the collected information.

[0955] Step 2:

[0956] The server inputs the collected information into a generative AI model to analyze the applicant's skills and aptitude. The generative AI model uses natural language processing and machine learning algorithms to evaluate the applicant's technical capabilities and past performance. The input is a dataset of collected information, and the output is an evaluation of the applicant's skills and aptitude.

[0957] Step 3:

[0958] The device captures the applicant's facial expressions using a webcam during the interview. It uses the OpenCV library to detect faces and analyze facial expressions. The input is the video data from the webcam, and the output is the analyzed facial expression data.

[0959] Step 4:

[0960] The device captures the applicant's voice data using a microphone during the interview. It converts the voice data into text using the SpeechRecognition library. The input is the voice data from the microphone, and the output is the converted text data.

[0961] Step 5:

[0962] The server inputs the captured facial expression data and converted text data into a generative AI model to analyze emotions and content. The generative AI model uses an emotion analysis algorithm to evaluate the applicant's emotional state and the content of their comments. The input is facial expression data and text data, and the output is an evaluation result regarding emotions and content.

[0963] Step 6:

[0964] The server integrates the evaluation results of the applicant's skills, aptitude, and emotions and content during the interview, and recommends the most suitable department or division. Based on these evaluation results, the generative AI model identifies the department or project where the applicant will be most effective. The input is the various evaluation results, and the output is a list of recommended departments and divisions.

[0965] Step 7:

[0966] The user checks the recommendation results generated by the server and makes the final decision. Based on the recommendation results, the user assigns the applicant to the appropriate department or project. The input is the recommendation results, and the output is the final assignment decision.

[0967] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0968] "Example 1"

[0969] One embodiment of the present invention is a system that incorporates an emotion engine. This system uses an emotion engine that recognizes the user's emotions, and this emotion information is included in the generative AI's decision-making. Specifically, the emotion engine recognizes the user's emotional state (joy, anger, sadness, surprise, etc.) in real time when using the system and provides this information to the generative AI. The generative AI takes this emotion information into consideration and recommends a department or division that will maximize the user's skills.

[0970] "Example 2"

[0971] Another embodiment of the present invention is a system that includes a function in which an emotion engine provides a user's emotional information to a generative AI and, based on that information, recommends the most appropriate department or division. Specifically, the emotion engine recognizes the user's emotional state in real time when using the system and provides that information to the generative AI. The generative AI then takes into account this emotional information and the user's skills and aptitudes to recommend the most appropriate department or division.

[0972] "Example 3"

[0973] In a further embodiment of the present invention, a system includes a function in which an emotion engine provides a user's emotional information to a generative AI and evaluates the user's credibility during an interview based on that information. Specifically, the emotion engine recognizes the user's emotional state during an interview in real time and provides that information to the generative AI. The generative AI evaluates the user's credibility by taking into account this emotional information, as well as facial expressions and voice data during the interview.

[0974] The processing flow of each embodiment will be described below.

[0975] "Example 1"

[0976] Step 1: When a user uses the system, the emotion engine recognizes the user's emotional state in real time.

[0977] Step 2: The emotion engine provides the recognized emotion information to the generative AI.

[0978] Step 3: The generative AI takes into account the provided emotional information and recommends departments or divisions that would best utilize the user's skills.

[0979] "Example 2"

[0980] Step 1: When a user uses the system, the emotion engine recognizes the user's emotional state in real time.

[0981] Step 2: The emotion engine provides the recognized emotion information to the generative AI.

[0982] Step 3: The generative AI takes into account the provided emotional information and the user's skills and aptitudes to recommend the most suitable department or division.

[0983] "Example 3"

[0984] Step 1: When a user is interviewed, the emotion engine recognizes the user's emotional state in real time.

[0985] Step 2: The emotion engine provides the recognized emotion information to the generative AI.

[0986] Step 3: The generative AI evaluates the user's credibility, taking into account the provided emotional information and facial expressions and voice data from the interview.

[0987] Example 1

[0988] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0989] Conventional recruitment systems have typically made decisions based on direct information such as applicants' resumes and application forms. However, this information alone makes it difficult to fully evaluate an applicant's skills and aptitude, making it difficult to assign them to the appropriate department or division. There are also systems that evaluate credibility by utilizing facial expressions and voice data from interviews, but these systems cannot take real-time emotional information into account, making it difficult to determine an applicant's true aptitude.

[0990] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes means for collecting direct and indirect information about applicants, means for saving the collected information in a database, means for inputting the saved information into a generative AI and analyzing it, means for collecting the user's emotional state in real time, means for analyzing the collected emotional information, means for the generative AI to recommend the most appropriate department or division based on the analysis results and the emotional information, and means for displaying the recommendation results to the user. This makes it possible to comprehensively evaluate the skills and aptitude of applicants and assign them to the most appropriate department or division taking into account their real-time emotional information.

[0991] "Direct information from the applicant" refers to information provided by the applicant themselves, such as in a resume or application form.

[0992] "Indirect information about applicants" refers to information that applicants have made public but not directly provided, such as social media posts or blog articles.

[0993] A "database" is a system for systematically storing and managing collected information.

[0994] "Generative AI" is artificial intelligence that analyzes collected information and evaluates applicants' skills and aptitude.

[0995] "Analysis" is the process of evaluating the applicant's skills and suitability based on the collected information.

[0996] The "user's emotional state" is the emotional state recognized from the user's facial expressions and voice when using the system.

[0997] "Real-time collection means" refers to devices such as cameras and microphones for instantly collecting the user's emotional state.

[0998] "Emotional information" is collected data about a user's emotional state.

[0999] "Recommendation" means that the generative AI suggests the most suitable department or division for an applicant based on the analysis results and emotional information.

[1000] The "display means" refers to a device such as a display or monitor for visually presenting the recommendation results to the user.

[1001] This invention relates to a recruitment support system that uses generative AI. This system collects and analyzes direct and indirect information about applicants to evaluate their skills and aptitude and recommend the most suitable department or division. Furthermore, by collecting the user's emotional state in real time and incorporating that information into the analysis, more accurate recommendations can be made.

[1002] The server first collects direct information, such as resumes and application forms submitted by applicants. This information can be in PDF or Word format. The server also obtains indirect information, such as the applicant's social media accounts and blog URLs. This information is then stored in a database.

[1003] The server then inputs the stored information into a generative AI for analysis. The generative AI uses Python's NLTK and spaCy libraries to analyze text data. For example, it can analyze information such as "3 years of Python programming experience" on a resume and evaluate the applicant's skills.

[1004] The device collects facial expressions and voices in real time as the user uses the system. This is done using a camera and microphone. The collected emotional information is sent to a server and analyzed by an emotion engine. The emotion engine uses OpenCV and the Google Cloud Speech-to-Text API to recognize the user's emotional state (joy, anger, sadness, surprise, etc.).

[1005] The server inputs applicant information and emotional information into the generative AI, which then recommends the most suitable department or division. The generative AI then comprehensively evaluates this information and generates a recommendation that takes into account the user's skills and emotional state. The recommendation result is sent to the device and displayed to the user.

[1006] For example, if an applicant's resume states that they have "three years of Python programming experience," generative AI will analyze this information and evaluate the applicant as having programming skills. Similarly, if information gathered from social media posts indicating an applicant's interest in data science is collected, the AI ​​will recommend a data science-related department.

[1007] Example prompt sentence:

[1008] "The applicant's resume states that he has three years of Python programming experience. His social media posts also indicate an interest in data science. Can you recommend a department that would be a good fit for him?"

[1009] In this way, a system is realized in which the server, terminals, and users work together to support optimal recruitment of personnel.

[1010] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1011] Step 1: Gather applicant information

[1012] The server receives resumes and application forms submitted by applicants. These can be in formats such as PDF or Word files. The server also collects URLs for applicants' social media accounts and blogs. The input is the files uploaded by applicants and the URLs entered, and the output is the collected applicant information.

[1013] Step 2: Save your applicant information

[1014] The server stores the collected applicant information in a database. Specifically, information such as the applicant's name, contact information, educational background, and work history is stored in the corresponding fields of the database. The input is the applicant information collected in step 1, and the output is the information stored in the database.

[1015] Step 3: Analyze applicant information

[1016] The server inputs the stored applicant information into the generative AI for analysis. The generative AI uses Python's NLTK and spaCy libraries to analyze text data. For example, it analyzes information such as "3 years of Python programming experience" and evaluates whether the applicant has programming skills. The input is the applicant information stored in the database, and the output is the analysis results.

[1017] Step 4: Collecting emotional information

[1018] The device collects facial expressions and voices in real time when the user uses the system. This is done using a camera and microphone. The input is the user's facial expressions and voice, and the output is the collected emotional information.

[1019] Step 5: Emotional Analysis

[1020] The server analyzes the emotion information sent from the device. The emotion engine uses OpenCV and Google Cloud Speech-to-Text API to recognize the user's emotional state (e.g., joy, anger, sadness, surprise). The input is the collected emotion information, and the output is the analyzed emotional state.

[1021] Step 6: Nominate a department or division

[1022] The server inputs applicant information and emotional information into the generative AI, which then recommends the most suitable department or division. The generative AI then comprehensively evaluates this information and generates a recommendation result, taking into account the user's skills and emotional state. The input is the analyzed applicant information and emotional information, and the output is a recommendation result.

[1023] Step 7: Viewing Recommendations

[1024] The terminal displays the recommendation results sent from the server to the user. For example, the screen may display "The best department for you is the Data Science Department." The input is the recommendation results sent from the server, and the output is the recommendation results displayed to the user.

[1025] (Application example 1)

[1026] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1027] Conventional recruitment support systems not only collect and analyze direct and indirect information about applicants, but also assess their credibility based on facial expressions and voice data during interviews. However, these systems lacked the ability to recognize applicants' emotional states in real time and provide that information to generative AI. Furthermore, conventional systems were not adequately equipped to evaluate the aptitude of autonomous vehicle drivers or recommend optimal driving tasks and routes. This made it difficult to make the most of a driver's skills and aptitude.

[1028] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes: means for using a generative AI to make a judgment based on direct and indirect information about the applicant and suggesting a department or division where the applicant's skills can be maximized based on information about the applicant's facial expressions and credibility from voice data during the interview; means for recognizing the user's emotional state in real time using an emotion engine and providing that information to the generative AI; means for collecting and analyzing the resumes, driving history, social media posts, and blog articles of autonomous vehicle drivers using the generative AI; and means for the generative AI to evaluate the driver's skills and aptitude and recommend optimal driving tasks and routes. This makes it possible to maximize the skills and aptitude of the applicant and driver and recommend optimal departments, divisions, driving tasks, and routes.

[1029] "Generative AI" is an artificial intelligence system that collects and analyzes information on applicants and drivers, and then uses the results to recommend the most suitable department, division, driving task, and route.

[1030] "Direct information" refers to official documents and data submitted by applicants and drivers, such as resumes and driving records.

[1031] "Indirect information" refers to unofficial data such as posts and articles published by applicants and drivers on social media, blogs, etc.

[1032] The "emotion engine" is a system that recognizes the user's emotional state in real time and provides that information to generative AI.

[1033] "Facial expression and voice data during the interview" refers to data such as the applicant's facial expressions and tone of voice collected during the interview.

[1034] "Credibility" is an indicator that evaluates whether the information and statements provided by applicants and drivers are true.

[1035] A "driver" is a person who operates and monitors an autonomous vehicle.

[1036] "Driving history" is data showing the driver's past driving records and experiences.

[1037] A "driving task" is a specific driving task or operation that a driver performs in an autonomous vehicle.

[1038] A "route" is the path or route that an autonomous vehicle will travel.

[1039] As an embodiment of the present invention, the following system is constructed.

[1040] System Program

[1041] Hardware and software used

[1042] Hardware: On-board computers, cameras, microphones, and emotion recognition sensors for autonomous vehicles

[1043] Software: Generative AI models (e.g., GPT-4), emotion engines (e.g., Affectiva), data collection and analysis tools (e.g., Elasticsearch)

[1044] Program processing

[1045] The server first digitally collects the driver's resume and driving history, then uses a web scraping tool to collect the driver's social media posts and blog posts, and stores this data in Elasticsearch for search and analysis.

[1046] The generative AI model (GPT-4) evaluates the driver's skills and aptitude based on the collected data. Furthermore, an emotion recognition sensor collects real-time emotional data from the driver, which is then analyzed by the emotion engine (Affectiva). The emotional data analyzed by the emotion engine is provided to the generative AI model, which then makes an evaluation that takes into account the driver's emotional state.

[1047] Ultimately, the generative AI model recommends optimal driving tasks and routes based on the driver's skills, aptitudes, and emotional state.

[1048] Specific examples

[1049] For example, if Driver A applies for a new driving task, the system operates as follows:

[1050] 1. The server collects Driver A's resume and driving history.

[1051] 2. The server collects Driver A's social media posts and blog articles.

[1052] 3. The server collects real-time emotional data of Driver A using an emotion recognition sensor.

[1053] 4. A generative AI model (GPT-4) analyzes this data and evaluates Driver A's skills and aptitude.

[1054] 5. The emotion engine (Affectiva) analyzes Driver A's emotional state and provides it to the generative AI model.

[1055] 6. The generative AI model recommends optimal driving tasks and routes for Driver A.

[1056] Prompt Sentence Examples

[1057] Evaluate Driver A's skills and aptitude based on his / her resume, driving history, social media posts, blog articles, and real-time sentiment data, and recommend the optimal driving tasks and routes.

[1058] In this way, it is possible to make the most of the skills and aptitudes of applicants and drivers and recommend the most suitable departments, divisions, driving tasks and routes.

[1059] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1060] Step 1:

[1061] The server collects the driver's submitted resume and driving history in digital format. The input is the driver's resume and driving history, which are stored in a database. The output is the digitized resume and driving history.

[1062] Step 2:

[1063] The server uses a web scraping tool to collect the driver's social media posts and blog articles. The input is the driver's social media account and blog URL, which are scraped and saved as text data. The output is the collected text data of the social media posts and blog articles.

[1064] Step 3:

[1065] The server collects real-time emotional data from the driver using an emotion recognition sensor. The input is the driver's facial expressions and voice data, which are analyzed by the emotion recognition sensor. The output is the analyzed emotional data.

[1066] Step 4:

[1067] The server stores the collected data in Elasticsearch for search and analysis. The inputs include digitized resumes, driving records, social media posts, blog posts, and sentiment data, which are indexed in Elasticsearch. The output is the indexed data.

[1068] Step 5:

[1069] The generative AI model (GPT-4) evaluates the driver's skills and aptitude based on collected data. It takes indexed data as input, analyzes it, and evaluates the driver's skills and aptitude. The output is the evaluation result.

[1070] Step 6:

[1071] The emotion engine (Affectiva) analyzes the driver's emotional state and provides it to a generative AI model. It takes real-time emotional data as input, analyzes it, and evaluates the emotional state. The output is the evaluated emotional state.

[1072] Step 7:

[1073] The generative AI model recommends optimal driving tasks and routes based on the driver's skills, aptitude, and emotional state. The inputs are the evaluation results of skills and aptitude, and the emotional state, and the optimal driving tasks and routes are generated based on these. The output is the recommended driving tasks and routes.

[1074] Example 2

[1075] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1076] Conventional interview systems have difficulty accurately assessing the credibility and seriousness of applicants, and are unable to recommend the most appropriate department or division based on the applicant's emotional state and skills. This has resulted in low accuracy in hiring decisions and makes it difficult to assign the right person to the right position.

[1077] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting facial expression data of applicants, means for collecting voice data of applicants, means for storing the collected facial expression data and voice data in a database, means for transmitting the stored data to a generative AI model, means for the generative AI model to analyze the data using facial recognition software and voice recognition software, means for the generative AI model to evaluate the applicant's credibility and seriousness based on the analysis results, means for reflecting the evaluation results in the hiring decision system, means for an emotion engine to collect user emotion information, means for transmitting the collected emotion information to the generative AI model, means for the generative AI model to recommend the most appropriate department or division based on the emotion information and the user's skills and aptitude, and means for presenting the recommendation results to the user. This makes it possible to accurately evaluate the applicant's credibility and seriousness and further recommend the most appropriate department or division based on the emotion information and skills.

[1078] An "applicant" is a person who applies for a job interview.

[1079] "Facial expression data" is digital data relating to the facial expressions of applicants, captured using a photographic device such as a webcam.

[1080] "Audio data" is digital data relating to the applicant's speech or voice, captured using a recording device such as a microphone.

[1081] The "database" is an information management system for storing collected facial expression data and voice data.

[1082] A "generative AI model" is an artificial intelligence model that analyzes collected data and generates evaluation and recommendation results.

[1083] "Facial recognition software" is software that analyzes facial expression data and recognizes specific expressions and emotions.

[1084] "Voice recognition software" is software that analyzes voice data and recognizes the content and emotions of what is being said.

[1085] "Credibility" is an evaluation metric that indicates how trustworthy an applicant's statements and actions are.

[1086] "Seriousness" is an evaluation indicator that shows how seriously the applicant is taking the interview.

[1087] The "employment decision system" is a system for making final employment decisions based on the applicant's evaluation results.

[1088] The "emotion engine" is a system for collecting and analyzing users' emotional information in real time.

[1089] "Emotion information" is data that indicates the user's emotional state, and includes emotions such as joy, sadness, and anger.

[1090] "Skills" is information that indicates the techniques and abilities possessed by a user.

[1091] "Aptitude" is an evaluation index that indicates how well a user is suited to a particular job or role.

[1092] A "department" is a division within a company or organization that is responsible for a specific task.

[1093] A "department" is a large organizational unit that oversees multiple departments within a company or organization.

[1094] "Recommendation results" are suggestions for the department or division that is best suited to the user, generated by the generative AI model based on emotional information and skills.

[1095] This invention is a system that collects facial expression and voice data from applicants and analyzes this data using a generative AI model to evaluate the applicant's credibility and seriousness, and then recommends the most suitable department or division based on emotional information and skills.

[1096] First, the device uses a webcam (e.g., a general webcam) and a microphone (e.g., a general microphone) to collect facial expression data and voice data from the applicant. The collected data is then sent to a server in real time.

[1097] The server stores the received facial expression and voice data in a database, where each applicant is assigned a unique ID and the data is organized.

[1098] The server then transmits the stored data to a generative AI model (e.g., a general generative AI model), which analyzes the facial expression data using facial recognition software (e.g., general facial recognition software) and the voice data using voice recognition software (e.g., general voice recognition software).

[1099] The generative AI model evaluates the applicant's credibility and seriousness based on this data. The evaluation results are quantified as credibility and seriousness scores and returned to the server. The server reflects the evaluation results received from the generative AI model in the hiring decision system. Recruiters make the final hiring decision based on these evaluation results.

[1100] Furthermore, when a user uses the system, an emotion engine (e.g., a general emotion engine) collects the user's facial expressions and voice in real time using a webcam and microphone. The emotion engine generates the user's emotion information based on the collected data. The emotion information includes emotional states such as joy, sadness, and anger.

[1101] The emotion engine sends the generated emotion information to a server. The server provides this information to a generative AI model. The generative AI model takes into account the emotion information and the user's skills and aptitudes to recommend the most suitable department or division. The recommended results are presented as specific department or division names.

[1102] Examples:

[1103] As users are interviewed, a webcam and microphone are used to capture their facial expressions and voice.

[1104] The server sends the collected data to a generative AI model to evaluate its credibility and seriousness.

[1105] The generative AI model returns the evaluation results to the server, which are used to make hiring decisions.

[1106] The emotion engine collects emotion information in real time while the user is using the system.

[1107] The emotion engine transmits the collected emotion information to the server.

[1108] The generative AI model recommends the most suitable department or division based on emotional information and skills.

[1109] The server presents the recommendation results to the user.

[1110] Example prompt sentence:

[1111] "Analyze applicants' facial expressions and audio data to assess their authenticity and seriousness."

[1112] "Recommend the best department or division based on the user's sentiment and skills."

[1113] The above is an embodiment of the present invention.

[1114] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1115] Step 1:

[1116] The terminal uses a webcam and a microphone to collect facial expression data and voice data of the applicant. Specifically, the webcam captures a picture of the applicant's face and the microphone records the applicant's voice. These data are transmitted to a server in real time. The input is video and voice data from the webcam and microphone, and the output is digital data transmitted to the server.

[1117] Step 2:

[1118] The server stores the received facial expression data and voice data in a database. Specifically, it assigns a unique ID to each applicant and organizes and stores the data. The input is digital data sent from the terminal, and the output is structured data stored in the database.

[1119] Step 3:

[1120] The server sends the stored data to the generative AI model. Specifically, it extracts the necessary data from the database and calls an API to send it to the generative AI model. The input is the digital data extracted from the database, and the output is the data sent to the generative AI model.

[1121] Step 4:

[1122] The generative AI model analyzes data using facial recognition software and voice recognition software. Specifically, facial expression data is analyzed using facial recognition software, and voice data is analyzed using voice recognition software. The input is the digital data sent from the server, and the output is a credibility score and seriousness score as the analysis results.

[1123] Step 5:

[1124] The generative AI model evaluates the applicant's credibility and seriousness based on the analysis results. Specifically, it quantifies the analysis results and evaluates them as a credibility score and seriousness score. The input is the analysis results from facial recognition and voice recognition software, and the output is a credibility score and seriousness score.

[1125] Step 6:

[1126] The server reflects the evaluation results received from the generative AI model in the hiring decision system. Specifically, it inputs the evaluation results into the hiring decision system and makes them available to hiring managers. The input is the credibility score and seriousness score from the generative AI model, and the output is the evaluation results reflected in the hiring decision system.

[1127] Step 7:

[1128] When a user uses the system, the emotion engine collects the user's facial expressions and voice in real time using a webcam and microphone. Specifically, the webcam captures a picture of the user's face and the microphone records the user's voice. The input is video and audio data from the webcam and microphone, and the output is digital data sent to the emotion engine.

[1129] Step 8:

[1130] The emotion engine generates the user's emotional information based on the collected data. Specifically, it analyzes facial expression data and voice data to identify the user's emotional state. The input is digital data from the webcam and microphone, and the output is emotional information data.

[1131] Step 9:

[1132] The emotion engine sends the generated emotion information to a server. Specifically, it saves the emotion information in a database and calls an API to provide it to the generative AI model. The input is data as emotion information, and the output is the data sent to the server.

[1133] Step 10:

[1134] The generative AI model takes into account emotional information and the user's skills and aptitudes to recommend the most suitable department or division. Specifically, it analyzes emotional information and skill data to identify the most suitable department or division. The input is emotional information and skill data, and the output is data representing the recommendation result.

[1135] Step 11:

[1136] The server presents the recommendation results received from the generative AI model to the user. Specifically, it provides an interface for displaying the recommendation results on the user's screen. The input is the recommendation results from the generative AI model, and the output is the recommendation results presented to the user.

[1137] (Application example 2)

[1138] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1139] With conventional interview systems, it was difficult to accurately assess the credibility and seriousness of applicants, making it difficult to assign them to the appropriate department or division. In addition, there was a lack of means to evaluate the emotional state and work efficiency of employees working in factories in real time, which created challenges in improving employee aptitude and work efficiency. To solve these problems, a system is needed that analyzes the facial expressions and voice data of applicants and employees in real time and makes appropriate decisions.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1141] In this invention, the server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about the applicant's credibility from facial expressions and voice data during the interview, means for analyzing facial expressions and voice data of employees working in the factory in real time to evaluate the employee's emotional state and work efficiency, means for collecting data through a camera and microphone mounted on a robot and analyzing the data with generative AI, and means for using the analysis results to help improve the aptitude and work efficiency of employees. This makes it possible to evaluate the aptitude of applicants and improve the work efficiency of employees.

[1142] "Generative AI" refers to artificial intelligence that generates new information and judgments based on input data.

[1143] "Applicant" means a person applying for a particular job or role.

[1144] "Direct information" refers to explicit data provided by applicants, such as resumes and curriculum vitae.

[1145] "Indirect information" refers to data obtained from the applicant's behavior and reactions, such as facial expressions and voice data during the interview.

[1146] "Credibility" is an indicator of how truthful an applicant's statements and actions are.

[1147] A "department" is a group responsible for a specific task or function within a company or organization.

[1148] A "department" is a further subdivided business unit within a department.

[1149] "Factory workers" refers to workers engaged in manufacturing and production industries.

[1150] "Facial expressions" refer to emotions and reactions shown by facial muscle movements.

[1151] "Audio data" refers to sound information collected through a voice input device such as a microphone.

[1152] An "emotional state" is the emotion an individual is feeling at a particular moment.

[1153] "Work efficiency" is an index that indicates how much work can be done effectively within a certain amount of time.

[1154] A "robot" is a mechanical device that operates automatically to perform specific tasks.

[1155] A "camera" is a device for taking images or videos.

[1156] A "microphone" is a device for collecting sound.

[1157] "Analysis" is the process of analyzing collected data and extracting meaningful information.

[1158] "Aptitude" refers to an individual's suitability and ability for a particular job or role.

[1159] In order to implement the present invention, the following hardware and software are used.

[1160] Hardware

[1161] Webcam

[1162] microphone

[1163] robot

[1164] software

[1165] OpenCV (image processing library)

[1166] sounddevice (audio processing library)

[1167] EmotionRecognizer (emotion recognition library)

[1168] AIModel (generative AI model)

[1169] System configuration

[1170] The server collects facial and voice data from applicants and employees via webcams and microphones. The collected data is analyzed in real time by EmotionRecognizer. The analysis results are provided to AIModel, which evaluates the applicant's credibility and seriousness, as well as the employee's emotional state and work efficiency.

[1171] Data processing and calculation

[1172] The server processes video data acquired from the webcam using OpenCV to analyze facial expressions. Audio data acquired from the microphone is processed using a sound device to extract audio characteristics. This data is converted into emotional states by the Emotion Recognizer and input into the AI ​​Model. The AI ​​Model combines this emotional data with the applicant's skill information and employee work history to recommend the most suitable department or division.

[1173] Specific examples

[1174] For example, if Employee A working in a factory looks tired while working, the system will recognize that expression and update the AI ​​model with a "feeling of fatigue." Based on that information, the AI ​​model will suggest that Employee A take a break or instruct them to change their work.

[1175] Prompt Sentence Examples

[1176] "Analyze employees' facial expressions and voice data in real time to assess their emotional state. Based on the assessment results, make suggestions to improve their work efficiency."

[1177] In this way, the server can provide a system that evaluates the suitability of applicants and improves the work efficiency of employees.

[1178] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1179] Step 1:

[1180] The server captures video data of applicants and employees in real time through a webcam. The input is the video data from the webcam, and the output is the captured image frames. Specifically, the server uses OpenCV to acquire the video from the camera and process each frame.

[1181] Step 2:

[1182] The server captures voice data of applicants and employees in real time through a microphone. The input is the voice data from the microphone, and the output is the captured voice samples. Specifically, the server uses a sound device to acquire voice from the microphone and collects voice samples at regular time intervals.

[1183] Step 3:

[1184] The server analyzes the captured video data using EmotionRecognizer to extract facial expression data. The input is the captured image frame, and the output is the analyzed facial expression data. Specifically, the server inputs the image frame acquired by OpenCV into EmotionRecognizer, analyzes facial feature points, and identifies the emotional state.

[1185] Step 4:

[1186] The server analyzes the captured audio data using EmotionRecognizer and extracts the emotional state from the audio data. The input is the captured audio sample, and the output is the analyzed audio emotion data. Specifically, the server inputs the audio sample acquired by the sound device into EmotionRecognizer, analyzes the audio features, and identifies the emotional state.

[1187] Step 5:

[1188] The server inputs the analyzed facial expression data and voice data into the AIModel to evaluate the applicant's credibility and seriousness, as well as the employee's emotional state and work efficiency. The input is the analyzed facial expression data and voice data, and the output is the evaluation results. Specifically, the server inputs the emotional data obtained from the EmotionRecognizer into the AIModel and performs an integrated evaluation.

[1189] Step 6:

[1190] Based on the evaluation results of the AI ​​model, the server recommends the most suitable department or division for the applicant and makes suggestions to employees to improve work efficiency. The input is the evaluation results of the AI ​​model, and the output is the recommendation results and proposals. Specifically, the server analyzes the evaluation results of the AI ​​model and generates a prompt sentence to recommend an appropriate department or division.

[1191] Step 7:

[1192] The user checks the recommendation results and proposals provided by the server and takes action as necessary. The input is the recommendation results and proposals from the server, and the output is the user's response. Specifically, the user makes hiring decisions for applicants and adjusts the work content of employees based on the information from the server.

[1193] Example 3

[1194] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1195] Conventional recruitment systems have difficulty accurately assessing applicants' technical skills and aptitude, and recommending the most suitable departments and projects. Furthermore, there was a lack of methods for assessing credibility using facial expressions and voice data during interviews, which led to a problem of reduced accuracy in hiring decisions. This made it difficult for companies to find suitable talent, and reduced the efficiency of the recruitment process.

[1196] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1197] In this invention, the server includes: means for collecting technical information about an applicant; means for preprocessing the collected information; means for inputting the preprocessed information into a generative AI model to evaluate the applicant's technical skills and aptitude; means for collecting facial expression and voice data about the applicant using a webcam and microphone during the interview; means for analyzing the collected facial expression and voice data in real time with an emotion engine and providing the information to the generative AI model; means for the generative AI model to recommend the most suitable department or project based on the applicant's technical skills and aptitude and credibility during the interview; and means for presenting the recommendation results to the user. This makes it possible to accurately evaluate the applicant's multifaceted information and recommend the most suitable department or project.

[1198] "Means for collecting applicant's technical information" refers to methods and tools for collecting data that demonstrates technical skills and experience, such as the applicant's programming portfolio, commit history, technical blog posts, and question and answer history.

[1199] "Means for preprocessing collected information" refers to methods and tools that perform processes such as tokenization, removal of stop words, conversion of voice data to text, and extraction of features from image data in order to convert collected technical information into a format that is easier to analyze.

[1200] "Means for inputting into a generative AI model to assess an applicant's technical skills and aptitude" refers to methods or tools for inputting pre-processed information into a generative AI model and using the model to assess an applicant's technical skills and aptitude.

[1201] "Methods for collecting facial expression and voice data of applicants using a webcam and microphone during an interview" means methods or tools for capturing facial expressions and voice data of applicants in real time using a webcam and microphone during an interview.

[1202] "Means for analyzing collected facial expression and voice data in real time using an emotion engine and providing that information to a generative AI model" refers to methods and tools for analyzing collected facial expression and voice data in real time using an emotion engine and providing the analysis results to a generative AI model.

[1203] "A means for a generative AI model to recommend the most suitable department or project based on an applicant's technical skills and aptitude, as well as their credibility during the interview" refers to a method or tool that allows a generative AI model to comprehensively evaluate an applicant's technical skills and aptitude, as well as their credibility during the interview, and then recommend the most suitable department or project based on the results.

[1204] "Means for presenting recommendation results to users" refers to methods and tools for presenting the recommendation results produced by the generative AI model to users in an easy-to-understand manner.

[1205] This invention is a system that evaluates the technical skills and aptitude of applicants and recommends the most suitable departments and projects. A specific embodiment of this system will be described below.

[1206] The server first collects the applicant's technical information. Specifically, it uses the GitHub API to obtain the applicant's commit history and repository information, a web scraping tool (e.g., Beautiful Soup) to collect technical blog posts, and the Stack Overflow API to obtain the applicant's question and answer history.

[1207] The server then preprocesses the collected information. Technical blog posts and Stack Overflow posts are tokenized and stopwords are removed using natural language processing tools (e.g., NLTK). Audio data is converted to text using speech recognition tools (e.g., Google Speech-to-Text API). Image data (facial expressions) are subjected to feature extraction using image processing tools (e.g., OpenCV).

[1208] The preprocessed data is then fed into a generative AI model (e.g., GPT-4), which evaluates the applicant's technical skills and aptitude based on a prompt. For example, the following prompt might be used:

[1209] "Assess applicants' technical skills and aptitude based on their GitHub commit history, technical blog posts, and Stack Overflow activity. Also, consider facial and voice data from the interview, as well as emotional information provided by your emotion engine, to assess applicants' authenticity and recommend the best departments and projects for them."

[1210] During the interview, the device uses a webcam and microphone to collect facial expressions and voice data from the candidate in real time, which is then analyzed in real time by an emotion engine (e.g., Affectiva SDK) and the information is fed into a generative AI model.

[1211] The generative AI model comprehensively evaluates the applicant's technical skills and aptitude, as well as their credibility during the interview, and based on the results, recommends the most suitable department and project. The server presents the recommendation results obtained from the generative AI model to the user. The user reviews the recommendation results provided by the server and makes the final hiring decision.

[1212] As a concrete example, consider the case of candidate A, who has a long history of commits on GitHub and regularly posts articles on a technical blog. He is also active on Stack Overflow. During the interview, facial and voice data is collected via a webcam and microphone, and an emotion engine recognizes the candidate's emotional state in real time. Based on this information, the generative AI model evaluates candidate A's skill set and aptitude, and recommends the most suitable department and project for the candidate.

[1213] In this way, a system is realized in which the server, terminals, and users work together to evaluate and recommend applicants. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1214] Step 1:

[1215] The server collects technical information about applicants. Specifically, it uses the GitHub API to obtain the applicant's commit history and repository information, and a web scraping tool (e.g., Beautiful Soup) to collect technical blog posts. It also uses the Stack Overflow API to obtain the applicant's question and answer history. By collecting this data, it obtains input data for understanding the applicant's technical skills and experience.

[1216] Step 2:

[1217] The server preprocesses the collected information. Technical blog articles and Stack Overflow posts are tokenized and stop words are removed using natural language processing tools (e.g., NLTK). Audio data is converted to text using speech recognition tools (e.g., Google Speech-to-Text API). Image data (facial expressions) is subjected to feature extraction using image processing tools (e.g., OpenCV). This results in output data in an easy-to-analyze format.

[1218] Step 3:

[1219] The server inputs the preprocessed data into a generative AI model (e.g., GPT-4) to evaluate the applicant's technical skills and aptitude. The generative AI model analyzes the applicant's skill set and aptitude based on a prompt. For example, the prompt might be, "Please evaluate the applicant's technical skills and aptitude based on their GitHub commit history, technical blog posts, and Stack Overflow activity history." This outputs an evaluation result regarding the applicant's technical skills and aptitude.

[1220] Step 4:

[1221] The device uses a webcam and microphone to collect facial expressions and voice data from applicants during interviews in real time. The collected data is analyzed in real time by an emotion engine (e.g., Affectiva SDK), and the information is provided to a generative AI model. This provides facial expressions, voice data, and emotional information from the interview as input.

[1222] Step 5:

[1223] The server uses a generative AI model to comprehensively evaluate the applicant's technical skills and aptitude, as well as their credibility during the interview, and recommends the most suitable department and project based on the results. The generative AI model suggests the most suitable department and project based on the prompt text. For example, it generates a result such as "Applicant A has strengths in front-end development and is ideal for Project X." This provides the recommendation result as output.

[1224] Step 6:

[1225] The server presents the recommendation results to the user. The user then checks the recommendation results provided by the server and makes the final hiring decision. For example, a message such as "Applicant A is ideal for Project X" is displayed on the dashboard. This provides the user with the information they need to make the best hiring decision.

[1226] (Application example 3)

[1227] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1228] Conventional recruitment systems and factory robot deployment systems were unable to fully utilize the past history and real-time data of applicants and robots, making it difficult to recommend optimal deployments and roles. Furthermore, credibility assessments during interviews and reliability assessments of robots while they are in operation were not adequately performed, making it difficult to achieve efficient personnel and robot deployments. This resulted in issues such as reduced productivity and efficiency for companies.

[1229] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1230] In this invention, the server includes: means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview; means for collecting the robot's past work history, real-time data from sensors, and information about the work environment and analyzing these with the generative AI; means for collecting facial expressions and voice data while the robot is operating and analyzing this data in real time with an emotion engine; and means for evaluating the robot's reliability and efficiency using these data and recommending optimal placement and roles. This makes it possible to recommend optimal placement and roles by utilizing the past history and real-time data of applicants and robots.

[1231] "Generative AI" is an artificial intelligence that analyzes the past history and real-time data of applicants and robots to recommend optimal placements and roles.

[1232] "Direct information" is information provided by the applicant themselves, such as a programming portfolio submitted by the applicant or a GitHub commit history.

[1233] "Indirect information" is information based on the applicant's activity on other platforms, such as technical blog posts or activity history on Stack Overflow.

[1234] "Facial Expressions" refers to facial expression data of applicants collected via webcam during interviews.

[1235] "Audio data" refers to voice data of applicants collected via a microphone during an interview.

[1236] "Credibility" refers to the applicant's degree of honesty and trustworthiness, assessed based on facial expressions and voice data.

[1237] A "department or division" is an organizational unit within a company that is responsible for a specific task or project.

[1238] "Robot's past work history" is a record of work that the robot has done in the past.

[1239] "Real-time data from sensors" refers to data about the current environment and conditions collected by sensors while the robot is operating.

[1240] "Work environment information" is data relating to the environment in which the robot works.

[1241] The "emotion engine" is a system that analyzes facial expressions and voice data in real time while the robot is moving, and generates emotional information.

[1242] "Reliability" is the degree to which a robot can perform a task accurately and efficiently.

[1243] "Efficiency" is the degree to which a robot can achieve maximum results with minimum resources.

[1244] "Optimal placement and role" refers to the position or role in which the robot or applicant can perform the work most effectively.

[1245] To implement the present invention, the following system configuration and program are required.

[1246] First, the server uses generative AI to collect direct and indirect information about the applicant. Direct information includes the applicant's programming portfolio and GitHub commit history, while indirect information includes technical blog posts and activity history on Stack Overflow. Based on this information, the generative AI analyzes the applicant's skills and aptitude.

[1247] The server then uses a webcam and microphone during the interview to collect facial expressions and voice data from the applicant. This data is analyzed in real time by an emotion engine to evaluate the applicant's credibility. Based on this evaluation, the server recommends the department or division where the applicant would be best placed.

[1248] In addition, the server collects the factory robot's past work history, real-time data from sensors, and information on the work environment. This data is analyzed by generative AI, and facial expressions and voice data are also collected while the robot is in operation. An emotion engine analyzes this data in real time and evaluates the robot's reliability and efficiency. Based on this evaluation, it recommends the optimal placement and role of the robot.

[1249] Specific hardware requirements include a webcam, microphone, and various sensors, while the software uses the generative AI model, EmotionEngine.

[1250] For example, when optimizing the placement of welding robots within a factory, the server analyzes historical data on past welding work, real-time temperature and humidity data, and abnormal sounds and smoothness of the robots' movements while they are in operation, and can then recommend the most efficient locations and roles for the welding robots.

[1251] An example of a prompt sentence might be:

[1252] "Analyze the welding robot's past work history data, real-time temperature and humidity data, abnormal sounds while the robot is operating, and smoothness of operation, and recommend the optimal placement and role."

[1253] In this way, the server can utilize past history and real-time data of applicants and robots to recommend optimal placements and roles.

[1254] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1255] Step 1:

[1256] The server collects direct information about applicants (programming portfolios, GitHub commit histories) and indirect information (technical blog posts, Stack Overflow activity histories). Using this information as input, the generative AI analyzes the applicant's skills and aptitude. The output is an evaluation of the applicant's skills.

[1257] Step 2:

[1258] The server uses a webcam and microphone to collect facial expressions and voice data from applicants during interviews. The emotion engine analyzes these data in real time and evaluates the applicant's credibility. The output is a credibility evaluation result.

[1259] Step 3:

[1260] The server integrates the results of Steps 1 and 2 and recommends the department or division where the applicant will be best placed. Using the skill assessment results and credibility assessment results as input, the generative AI calculates the optimal placement. The recommended department or division is obtained as output.

[1261] Step 4:

[1262] The server collects the factory robot's past work history, real-time data from sensors, and information on the work environment. Using this data as input, the generative AI analyzes the robot's behavior. The output is an evaluation of the robot's behavior.

[1263] Step 5:

[1264] The server collects facial expressions and voice data while the robot is operating. The emotion engine analyzes this data in real time and evaluates the robot's reliability and efficiency. The output is a reliability evaluation result.

[1265] Step 6:

[1266] The server integrates the results of steps 4 and 5 and recommends optimal placement and roles for the robots. Using the behavior evaluation results and reliability evaluation results as input, the generative AI calculates the optimal placement. The output is the recommended placement and role.

[1267] Through these steps, the server can utilize the past history and real-time data of applicants and robots to recommend optimal placements and roles.

[1268] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1269] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1270] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1271] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1272] [Third embodiment]

[1273] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1274] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1275] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1276] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1277] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1278] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1279] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1280] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1281] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1282] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1283] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1284] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1285] "Example 1"

[1286] One embodiment of the present invention is a recruitment support system that uses generative AI. This system collects direct information such as resumes and application forms submitted by applicants, as well as indirect information such as social media posts and blog articles, and analyzes this information using generative AI. The generative AI uses this information to evaluate the applicant's skills and aptitude, and recommends the most suitable department or division for the job.

[1287] "Example 2"

[1288] Another embodiment of the present invention includes a function to analyze facial expressions and voice data during interviews. During interviews, facial expressions and voice data of applicants are collected via a webcam and microphone. The generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in the hiring decision.

[1289] "Example 3"

[1290] As a specific example, when considering hiring an engineer, generative AI collects direct information such as the applicant's programming portfolio and GitHub commit history, as well as indirect information such as technical blog posts and activity history on Stack Overflow, and analyzes these. Furthermore, during interviews, the applicant's facial expressions and voice data are collected via webcam and microphone, which are then analyzed by generative AI. Based on this information, the system recommends departments and projects where the applicant is likely to perform best.

[1291] The processing flow of each embodiment will be described below.

[1292] "Example 1"

[1293] Step 1: Collect direct information such as resumes and application forms submitted by applicants, as well as indirect information such as social media posts and blog articles.

[1294] Step 2: Input the collected information into the generative AI.

[1295] Step 3: Generative AI uses this information to evaluate the applicant's skills and aptitude, and recommends the most suitable department or division.

[1296] "Example 2"

[1297] Step 1: Collect facial expressions and audio data from applicants during the interview via webcam and microphone.

[1298] Step 2: Input the collected facial expressions and voice data into the generative AI.

[1299] Step 3: Generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in hiring decisions.

[1300] "Example 3"

[1301] Step 1: Collect direct information such as the applicant's programming portfolio and GitHub commit history, as well as indirect information such as technical blog posts and Stack Overflow activity.

[1302] Step 2: Input the collected information into the generative AI.

[1303] Step 3: Generative AI analyzes this information and evaluates the applicant's skills and aptitude.

[1304] Step 4: During the interview, facial expressions and audio data from the applicant are collected via a webcam and microphone.

[1305] Step 5: Input the collected facial expressions and voice data into the generative AI.

[1306] Step 6: Generative AI uses this data to evaluate the applicant's credibility and seriousness, and reflects the results in hiring decisions.

[1307] Step 7: Based on this information, we recommend the department and project where the applicant would be most likely to excel.

[1308] Example 1

[1309] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1310] Conventional recruitment systems have difficulty accurately evaluating applicants' skills and aptitudes and recommending the most suitable department or division. Furthermore, they are unable to take into account not only direct information about applicants but also indirect information, resulting in a problem of reduced evaluation accuracy. Furthermore, there was a lack of a way to effectively provide evaluation results to human resources personnel.

[1311] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for collecting direct and indirect information about the applicant, a means for storing the collected information in a database, a means for analyzing the stored information using a generative AI model, a means for recommending an optimal department or division based on the analysis results, and a means for providing the recommendation results through a management screen. This makes it possible to evaluate the skills and aptitude of the applicant with high accuracy and recommend an optimal department or division.

[1312] "Direct information from the applicant" refers to information provided by the applicant themselves, such as in a resume or application form.

[1313] "Indirect information about applicants" refers to information that applicants have made public on the Internet, such as social media posts or blog articles.

[1314] A "database" is a system for organizing and storing collected information; examples include MySQL and MongoDB.

[1315] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to analyze applicant information and evaluate skills and aptitude.

[1316] The "analysis results" are the results of the skills and aptitudes evaluated by the generative AI model based on the applicant's information.

[1317] "Department or division recommendation" refers to suggesting the most suitable job or role for an applicant based on the analysis results.

[1318] The "administration screen" is an interface for providing recommendation results to human resources personnel, and is built using a front-end framework.

[1319] The present invention relates to a recruitment support system that uses generative AI. A specific embodiment of this system will be described below.

[1320] The server first collects direct and indirect information about applicants. Direct information includes resumes and application forms, which applicants provide themselves. Indirect information includes social media posts and blog articles, which applicants make public online. This information is collected using web scraping tools (e.g., BeautifulSoup and Scrapy) and APIs (e.g., Twitter API and Facebook Graph API).

[1321] Next, the server stores the collected information in a database (e.g., MySQL or MongoDB). When storing the information, it assigns a unique ID to each applicant and organizes the information. For example, the resume information for applicant A is stored in a table called "applicant_A_resume."

[1322] The stored information is analyzed using a generative AI model (e.g., OpenAI's GPT-4). The server inputs a prompt like this into the generative AI model:

[1323] The applicant's resume states that he has "5 years of Python programming experience," and his social media posts state that he is "interested in data science." Use this information to evaluate the applicant's skills and aptitude, and recommend the department or division that would be the best fit.

[1324] The generative AI model evaluates the applicant's skills and suitability based on this prompt and returns the results.

[1325] The server recommends the most suitable department or division for each applicant based on the evaluation results obtained from the generative AI model. For example, if the evaluation result shows that "Applicant A is suitable for the Data Science Division," the server will recommend Applicant A to the Data Science Division.

[1326] Finally, the server provides the recommendation results to the human resources personnel via an administration screen. The administration screen is built using a front-end framework (e.g., React or Vue.js) and can be accessed by the user via a browser. The user can check the recommendation results for applicant A on the administration screen and make a decision on assigning the candidate to an appropriate department.

[1327] In this way, the server, terminals, and users work together to create a system that provides efficient support for recruiting personnel.

[1328] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1329] Step 1: Gather applicant information

[1330] The server receives resumes and application forms submitted by applicants. This direct information is often uploaded in PDF or Word format. The server uses an OCR (optical character recognition) tool (e.g., Tesseract) to convert these files into text data. It also uses web scraping tools (e.g., BeautifulSoup or Scrapy) or APIs (e.g., Twitter API or Facebook Graph API) to collect indirect information such as applicants' social media posts and blog articles. The input is the resume or social media URL, and the output is text data.

[1331] Step 2: Save your applicant information

[1332] The server saves the collected text data in a database (e.g., MySQL or MongoDB). When saving, a unique ID is assigned to each applicant and the information is organized. For example, the resume information for applicant A is saved in a table called "applicant_A_resume." The input is text data, and the output is the information saved in the database.

[1333] Step 3: Analyze applicant information

[1334] The server inputs the stored information into a generative AI model (e.g., OpenAI's GPT-4) for analysis. Specifically, it inputs the following prompt sentence into the generative AI model:

[1335] The applicant's resume states that he has "5 years of Python programming experience," and his social media posts state that he is "interested in data science." Use this information to evaluate the applicant's skills and aptitude, and recommend the department or division that would be the best fit.

[1336] The generative AI model evaluates the applicant's skills and aptitude based on this prompt and returns the results. The input is the prompt and the output is the evaluation result.

[1337] Step 4: Department / Division Recommendations

[1338] The server recommends the most suitable department or division for the applicant based on the evaluation results obtained from the generative AI model. For example, if the evaluation result shows that "Applicant A is suitable for the Data Science Division," the server will recommend Applicant A to the Data Science Division. The input is the evaluation result, and the output is the recommendation result.

[1339] Step 5: Providing recommendations

[1340] The server provides the recommendation results to the human resources personnel via an administration screen. The administration screen is built using a front-end framework (e.g., React or Vue.js) and can be accessed by the user via a browser. The user checks the recommendation results for applicant A on the administration screen and makes a decision on assigning the applicant to an appropriate department. The input is the recommendation results, and the output is the information displayed on the administration screen.

[1341] (Application example 1)

[1342] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1343] Conventional recruitment systems have typically made decisions based on direct information such as applicants' resumes and application forms, but this makes it difficult to fully evaluate the applicant's aptitude and skills. Furthermore, credibility is not assessed based on facial expressions or voice data during interviews, making it difficult to assign candidates to the appropriate department or division. Furthermore, there is a lack of concrete methods for assessing suitability as a driver of an autonomous vehicle.

[1344] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1345] In this invention, the server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for collecting the applicant's resume, application form, social media posts, blog articles, etc. and for the generative AI to analyze this information to evaluate the applicant's skills and aptitude, and means for evaluating the applicant's aptitude as a driver of an autonomous vehicle and recommending the most suitable position. This makes it possible to comprehensively evaluate the aptitude and skills of the applicant and assign them to the most suitable department or division.

[1346] "Generative AI" is an artificial intelligence technology that collects and analyzes information about applicants and evaluates their aptitude and skills based on the results.

[1347] "Direct information" refers to information contained in official documents submitted by applicants, such as resumes and application forms.

[1348] "Indirect information" refers to information obtained from sources other than official documents, such as applicants' social media posts or blog articles.

[1349] "Facial expressions during an interview" refers to the facial expressions and expressions shown by applicants during an interview.

[1350] "Audio data" refers to data related to the voice of the applicant during the interview, such as the tone of voice and speaking style.

[1351] "Credibility" is a measure of how trustworthy the information provided by the applicant and the expressions given during the interview are.

[1352] "Skills" refers to the specialized knowledge, techniques, and abilities possessed by the applicant.

[1353] "Qualities" are characteristics that indicate how well an applicant is suited to a particular job or role.

[1354] A "department" is a division within a company or organization that is responsible for a specific task or role.

[1355] A "department" refers to a large business unit within a company or organization, and may include multiple departments.

[1356] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1357] "Driver" refers to the person who operates and monitors the autonomous vehicle.

[1358] A "position" refers to a specific role or job within a company or organization.

[1359] The system for implementing this invention is composed of three main elements: a server, a terminal, and a user. The specific roles and processes of each element are explained below.

[1360] Server Roles and Operations

[1361] The server uses the following hardware and software to collect, analyze, and evaluate data:

[1362] Hardware used:

[1363] Server (data collection and execution of AI models)

[1364] Software used:

[1365] Python (data processing and running AI models)

[1366] TensorFlow (implementation of generative AI models)

[1367] Flask (Web application framework)

[1368] BeautifulSoup (web scraping library)

[1369] Data processing and calculation:

[1370] 1. Data Collection:

[1371] The server uses BeautifulSoup to scrape applicants' social media posts and blog posts, and converts the applicants' submitted resumes and application forms from PDFs and images to text.

[1372] 2. Data Preprocessing:

[1373] The server cleans the collected text data, removing unnecessary information, and then uses natural language processing (NLP) techniques to tokenize the text data and extract features.

[1374] 3. Analysis using generative AI models:

[1375] The server uses TensorFlow to build a generative AI model and uses the applicant's text data as input to evaluate their skills and aptitude.

[1376] 4. Displaying the results:

[1377] The server uses Flask to build a web application and display the results.

[1378] Terminal roles and processing

[1379] The terminal provides an interface for the user to enter applicant information and check the results.

[1380] Hardware used:

[1381] Smartphone (entering applicant information and displaying results)

[1382] Software used:

[1383] A web browser (accessing the Flask application)

[1384] User Roles and Actions

[1385] The user enters applicant information and checks the evaluation results generated by the generative AI model.

[1386] Examples:

[1387] Applicant A's resume states that he has "5 years of experience driving self-driving vehicles."

[1388] Applicant A's social media posts state that he "frequently shares articles about the latest autonomous driving technology."

[1389] The generative AI model analyzes this information and assesses that Applicant A has high suitability as a driver of an autonomous vehicle.

[1390] Example prompts to input to a generative AI model:

[1391] The applicant's resume states that he has "5 years of experience driving autonomous vehicles." His social media posts state that he "frequently shares articles about the latest autonomous driving technology." Please rate this applicant's suitability as a driver.

[1392] In this way, a system can be constructed in which the server, terminals, and users work together to comprehensively evaluate the aptitude and skills of applicants and assign them to the most suitable department or division.

[1393] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1394] Step 1:

[1395] The server collects applicants' social media posts and blog articles. Specifically, it uses BeautifulSoup to perform web scraping to obtain text data from applicants' social media accounts and blogs. The input is the URL of the applicants' social media accounts and blogs, and the output is the collected text data.

[1396] Step 2:

[1397] The server converts the resumes and application forms submitted by applicants from PDF or image to text. Specifically, it uses OCR (optical character recognition) technology to extract text data from PDF or image files. The input is the PDF or image file, and the output is the converted text data.

[1398] Step 3:

[1399] The server preprocesses the collected text data. Specifically, it cleans the text data and removes unnecessary information. Then, it uses natural language processing (NLP) techniques to tokenize the text data and extract features. The input is the collected text data, and the output is the preprocessed text data.

[1400] Step 4:

[1401] The server uses TensorFlow to build a generative AI model and evaluates skills and aptitude using preprocessed text data as input. Specifically, the generative AI model inputs text data and outputs a score that evaluates the applicant's skills and aptitude. The input is preprocessed text data, and the output is an evaluation score for skills and aptitude.

[1402] Step 5:

[1403] The server uses Flask to build a web application and display the evaluation results. Specifically, based on the evaluation scores, it generates recommendations for the most suitable department or division for the applicant and displays them through the web application. The input is the evaluation scores for skills and aptitude, and the output is the recommendation results.

[1404] Step 6:

[1405] The terminal provides an interface for users to enter applicant information and check the results. Specifically, users use a smartphone's web browser to access the Flask application and enter applicant information. The input is applicant information, and the output is the recommendation results.

[1406] Step 7:

[1407] The user checks the evaluation results of the generative AI model. Specifically, the user checks the recommendation results on their smartphone screen and inputs additional information as needed. The input is the recommendation results, and the output is the user's feedback.

[1408] In this way, a system can be constructed in which the server, terminals, and users work together to comprehensively evaluate the aptitude and skills of applicants and assign them to the most suitable department or division.

[1409] Example 2

[1410] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1411] With conventional interview systems, it was difficult to objectively evaluate the credibility and seriousness of applicants, and hiring decisions often involved subjective factors. It was also difficult to properly recommend departments or divisions that would best utilize an applicant's skills.

[1412] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for activating the webcam and microphone and starting data collection at the same time as the start of the interview, means for dividing the collected video data into frames and extracting facial regions, means for removing noise from the collected audio data and extracting audio features, means for integrating facial expression data and audio data and inputting the data into a generative AI model, means for the generative AI model to evaluate the credibility and seriousness based on the facial expression data and audio data, and means for providing the evaluation results to the hiring manager. This makes it possible to objectively evaluate the credibility and seriousness of the applicant and make an appropriate hiring decision.

[1413] A "webcam" is a digital camera for capturing video.

[1414] A "microphone" is an acoustic device for capturing sound.

[1415] "Data collection" is the process of capturing video and audio data using a webcam and microphone.

[1416] "Video Data" is a collection of video frames captured by a webcam.

[1417] A "frame" is an individual still image that makes up video data.

[1418] The "face area" refers to the part of the video data where a face is present.

[1419] "Audio Data" is a recording of sound captured by a microphone.

[1420] "Noise reduction" is the process of removing unwanted background sounds and noise from audio data.

[1421] "Voice features" are characteristics such as pitch, tone, and speed contained in the voice data.

[1422] "Facial expression data" is information about facial expressions extracted from video data.

[1423] A "generative AI model" is an algorithm that uses artificial intelligence to analyze data and make specific assessments or predictions.

[1424] "Credibility" refers to the reliability of whether an applicant's statements and actions are true.

[1425] "Seriousness" refers to how seriously the applicant takes the interview.

[1426] The "evaluation results" are scores and judgments of credibility and seriousness calculated by the generative AI model.

[1427] A "recruiter" is a person who makes hiring decisions based on the results of an applicant's interview.

[1428] This invention is a system for analyzing facial expressions and voice data of an applicant during an interview to evaluate the applicant's credibility and seriousness. A specific embodiment of this system will be described below.

[1429] Hardware and software used

[1430] Hardware: Webcam, microphone

[1431] Software: Generative AI models (e.g., general-purpose natural language processing models), facial expression recognition software (e.g., OpenFace), and speech analysis software (e.g., Praat)

[1432] Data collection

[1433] The device activates the webcam and microphone and begins collecting data as soon as the interview begins. The webcam captures video data, and the microphone captures audio data.

[1434] Data Preprocessing

[1435] The device splits the video data it collects into frames and extracts facial regions using facial expression recognition software, while it uses voice analysis software to remove noise and extract voice characteristics from the audio data.

[1436] Analyzing the data

[1437] The server receives the facial expression and voice data sent from the device and integrates them. The integrated data is then input into a generative AI model, which evaluates the authenticity and seriousness of the user based on the facial expression and voice data.

[1438] Output of evaluation results

[1439] The server receives the evaluation results from the generative AI model and provides them to recruiters. The evaluation results are displayed in the form of a dashboard or report, clearly indicating scores for credibility and seriousness.

[1440] Specific examples

[1441] For example, when an applicant introduces themselves, the device collects data via a webcam and microphone. The collected data is sent to a server and analyzed by a generative AI model. The resulting evaluation result may be something like, "The applicant's facial expression indicates nervousness, but their voice tone is stable, and they are evaluated as being highly credible."

[1442] Prompt Sentence Examples

[1443] "Please rate the applicant's credibility and seriousness based on their facial and voice data. Enter the following data: [Facial Expression Data], [Voice Data]"

[1444] In this way, the system analyzes the applicant's facial expressions and voice data during the interview to evaluate their credibility and seriousness, and the results of this evaluation are reflected in the hiring decision.

[1445] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1446] Step 1:

[1447] The device activates the webcam and microphone and starts collecting data at the same time as the interview begins. The input is the start signal for the interview, and the output is the video and audio data captured in real time. Specifically, the device captures the video stream from the webcam and the audio stream from the microphone in real time.

[1448] Step 2:

[1449] The device divides the collected video data into frames and extracts facial regions. The input is the captured video data, and the output is facial region data extracted from each frame. Specifically, the device uses facial expression recognition software (e.g., OpenFace) to extract facial feature points from each frame.

[1450] Step 3:

[1451] The device removes noise from the collected voice data and extracts voice features. The input is the captured voice data, and the output is the voice data with noise removed and features extracted. Specifically, the device uses voice analysis software (e.g., Praat) to extract voice features such as pitch, tone, and speed.

[1452] Step 4:

[1453] The server receives the facial expression data and voice data sent from the device and integrates them. The input is the facial expression data and voice data sent from the device, and the output is the integrated data. Specifically, the server combines the facial expression data and voice data into a single data set.

[1454] Step 5:

[1455] The server inputs the integrated data into the generative AI model to evaluate its credibility and seriousness. The input is the integrated data, and the output is the evaluation result of its credibility and seriousness. Specifically, the server inputs the integrated data as a prompt sentence to the generative AI model (e.g., GPT-4) and performs an evaluation.

[1456] Step 6:

[1457] The server receives the evaluation results from the generative AI model and provides them to recruiters. The input is the evaluation results from the generative AI model, and the output is the evaluation results provided to recruiters. Specifically, the server displays the evaluation results in the form of a dashboard or report, clearly indicating scores for credibility and seriousness.

[1458] (Application example 2)

[1459] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1460] Conventional interview systems make it difficult to accurately assess applicants' credibility and seriousness, and rely heavily on the interviewer's subjective judgment. Furthermore, there was a lack of a way to analyze participants' facial expressions and voices in real time during interviews and meetings to assess their credibility and seriousness, making it difficult to make appropriate personnel assignments and decisions. This hinders companies' efficient use of human resources and the improvement of the quality of important meetings.

[1461] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1462] In this invention, the server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for analyzing the facial expressions and voice of participants in real time during interviews and important meetings using a smartphone camera and microphone, and means for the generative AI to analyze the data, evaluate the credibility and seriousness of the participants, and display the results in real time. This enables objective evaluation of the credibility and seriousness of applicants and participants, enabling appropriate personnel placement and decision-making.

[1463] "Generative AI" is an artificial intelligence technology that generates new data and information based on input data.

[1464] "Applicant" refers to a person applying for a particular job or role.

[1465] "Direct information" refers to explicit data provided by applicants, such as resumes and curriculum vitae.

[1466] "Indirect information" refers to data obtained from the applicant's behavior and reactions, such as facial expressions and voice data during the interview.

[1467] "Credibility" is a measure of whether an applicant's statements and actions are true.

[1468] "Seriousness" is a measure of how seriously an applicant takes an interview or meeting.

[1469] A "smartphone" is a type of mobile phone and a multi-function device that can connect to the Internet and use applications.

[1470] A "camera" is a device for taking images or videos.

[1471] A "microphone" is a device for recording sound.

[1472] "Real-time" refers to data collection and processing occurring immediately.

[1473] A "department" is a group responsible for a specific task or function within a company or organization.

[1474] A "department" is a small group responsible for a specific task or role within a company or organization.

[1475] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[1476] "Display" refers to the visual presentation of data or information.

[1477] A system for carrying out this invention has the following configuration: The server includes means for using generative AI to make judgments based on direct and indirect information about the applicant and suggesting departments or divisions where the applicant's skills can be utilized to the fullest based on information about credibility from facial expressions and voice data during the interview, means for analyzing the facial expressions and voices of participants in real time during interviews and important meetings using a smartphone camera and microphone, and means for the generative AI to analyze the data, evaluate the credibility and seriousness of the participants, and display the results in real time.

[1478] Hardware and software used

[1479] Hardware: Smartphone camera, smartphone microphone

[1480] Software: OpenCV (video capture and processing), SoundDevice (audio capture), TensorFlow (AI model for facial expression analysis), Transformers (generative AI model for audio data)

[1481] Data processing and calculation

[1482] 1. Video data capture and processing:

[1483] Use your smartphone camera to capture real-time footage of participants during interviews and meetings.

[1484] The video data is processed using OpenCV to extract the facial area.

[1485] The extracted facial area is subjected to facial expression analysis using TensorFlow to evaluate credibility and seriousness.

[1486] 2. Audio data capture and processing:

[1487] Use your smartphone's microphone to capture participants' voices in real time during interviews and meetings.

[1488] Audio data is recorded using SoundDevice and analyzed using Transformers.

[1489] The analysis results are used to evaluate the participants' credibility and seriousness.

[1490] Specific examples

[1491] For example, to evaluate in real time whether an applicant's facial expression is nervous during an interview, the applicant's face is captured with a smartphone camera and the facial region is extracted using OpenCV. The extracted facial region is analyzed using TensorFlow to evaluate the level of nervousness. Similarly, to analyze the seriousness of participants' comments during a meeting from audio data, the audio is captured with a smartphone microphone and recorded using SoundDevice. The recorded audio data is analyzed using Transformers to evaluate the level of seriousness.

[1492] Prompt Sentence Examples

[1493] "Analyze the applicant's facial expression data and assess their level of nervousness."

[1494] "Please analyze the seriousness of statements made during meetings from audio data."

[1495] In this way, the system can objectively assess the credibility and seriousness of applicants and participants, and support appropriate staffing and decision-making.

[1496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1497] Step 1:

[1498] The server uses the smartphone camera to capture video of participants in interviews or meetings in real time. The input is video data from the smartphone camera, and the output is the captured video data. Specifically, the camera starts up and continuously acquires video frames.

[1499] Step 2:

[1500] The server processes the captured video data using OpenCV and extracts the face region. The input is the video data obtained in step 1, and the output is the extracted face region data. Specifically, it applies a face detection algorithm to the video frame to identify the position of the face.

[1501] Step 3:

[1502] The server performs facial expression analysis on the extracted facial region using TensorFlow. The input is the facial region data obtained in step 2, and the output is the result of the facial expression analysis. Specifically, the facial region data is preprocessed and input into an AI model to analyze facial expression features.

[1503] Step 4:

[1504] The server uses the smartphone's microphone to capture the voice of participants in an interview or meeting in real time. The input is the voice data from the smartphone's microphone, and the output is the captured voice data. Specifically, the microphone is turned on and continuously records voice.

[1505] Step 5:

[1506] The server records the audio data captured using SoundDevice and analyzes it using Transformers. The input is the audio data obtained in step 4, and the output is the result of the audio analysis. Specifically, the audio data is preprocessed and input into the AI ​​model to analyze the audio features.

[1507] Step 6:

[1508] The server integrates the results of the facial expression analysis and voice analysis obtained in steps 3 and 5 to evaluate the participants' credibility and seriousness. The inputs are the results of the facial expression analysis and voice analysis, and the output is the evaluation results of credibility and seriousness. Specifically, the server inputs both analysis results into an integration algorithm to perform a comprehensive evaluation.

[1509] Step 7:

[1510] The server displays the evaluation results in real time and provides feedback to the interviewer and the conference organizer. The input is the evaluation results obtained in step 6, and the output is the displayed feedback information. Specific operations include updating the interface to visually display the evaluation results.

[1511] Example 3

[1512] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1513] In the traditional recruitment process for engineers, it was difficult to accurately evaluate applicants' technical skills and experience and assign them to the most appropriate departments and projects. Furthermore, there was a lack of methods to evaluate applicants' emotions and communication skills using facial expressions and voice data during interviews, which led to a problem of reduced accuracy in recruitment.

[1514] The identification process by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for collecting information about the applicant; means for transmitting the collected information to the server; means for the server to analyze the applicant's technical skills and experience using a generative AI model; means for the terminal to collect facial expression and voice data about the applicant via a webcam and microphone during the interview; means for transmitting the collected interview data to the server; means for the server to analyze the interview data using the generative AI model and evaluate the applicant's emotions and communication skills; means for the server to recommend a department or project that is most suitable for the applicant based on the analysis results; and means for notifying the user of the recommendation results. This makes it possible to comprehensively evaluate the applicant's technical skills, experience, emotions, and communication skills and assign them to the most suitable department or project.

[1515] "Means for collecting applicant information" refers to devices or software for obtaining data such as the applicant's programming portfolio, online platform history, technical blog posts, and activity history on Q&A sites.

[1516] The "means for transmitting collected information to the server" refers to a communication device or software for transferring applicant information from the terminal to the server.

[1517] "Means for analyzing applicants' technical skills and experience using generative AI models" refers to the AI ​​model and its execution environment used to analyze collected applicant information and evaluate technical skills and experience.

[1518] "Means for the device to collect facial expressions and voice data of the applicant through a webcam and microphone during the interview" refers to hardware and software for capturing the facial expressions and voice of the applicant in real time during the interview.

[1519] The "means for transmitting collected interview data to the server" refers to a communication device or software for transferring facial expression and voice data collected during the interview to the server.

[1520] "Means for analyzing interview data using a generative AI model to evaluate applicants' emotions and communication skills" refers to an AI model and its execution environment used to analyze collected interview data and evaluate applicants' emotions and communication s...

Claims

1. Means of collecting direct and indirect information about applicants; A means for storing the information collected by the collecting means in a database; A means for analyzing the information stored by the storing means using a generative AI model; A means for recommending a department or division that is most suitable for the applicant based on the analysis result by the analyzing means; a means for providing a recommendation result by the recommendation means through a management screen; A system including:

2. The system of claim 1 , wherein the generative AI model includes a function of analyzing applicant information and recommending the most suitable department or division based on the results.

3. The system of claim 1 , wherein the generative AI model includes a function for generating prompt sentences and performing analysis based on direct and indirect information about the applicant.

Images