Information processing device, information processing method, and information processing program

Abstract

By comprehensively analyzing linguistic and nonverbal information, we visualize the true state of mind. [Solution] A communication unit receives character data and facial expression data transmitted from an input device; a language processing unit analyzes the character data to extract linguistic information; a non-linguistic processing unit analyzes the facial expression data to extract non-linguistic information; and the system calculates the distance or area overlap between the location or shape of emotion estimated from the linguistic information and the location or shape of emotion estimated from the non-linguistic information, and determines whether the calculated distance or area overlap falls within a predetermined threshold range to evaluate the degree of agreement or mismatch between the linguistic information and the non-linguistic information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program for performing emotion analysis by combining language (character information) and non-language (video information such as expressions).

Background Art

[0002] Conventionally, personnel management in companies has been carried out based on objective indicators such as performance evaluation, skill evaluation, interview records, etc., and subjective evaluations such as questionnaires. However, with these methods, it has been difficult to accurately grasp the true feelings and the real mental state (mind state) of the employees who are the targets, and there have been limitations in taking appropriate management measures.

[0003] In recent years, with the development of emotion analysis technology using AI (artificial intelligence), technologies for estimating emotions from text data and for reading emotions from expressions have been put into practical use. For example, in the emotion analysis of documents, support vector machines (SVM), naive Bayes classifiers, random forests, etc. are used as supervised learning methods, and LSTM (Long Short-Term Memory) networks, BERT (Bidirectional Encoder Representations from Transformers), etc. are utilized as deep learning methods.

[0004] Also, in facial expression analysis technology, a face detection algorithm and a landmark detection technology are combined, and further expression recognition based on FACS (Facial Action Coding System) is performed. By using these technologies, it has become possible to estimate basic emotions (happiness, sadness, anger, fear, disgust, surprise).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] This method analyzes linguistic information (information extracted from text) and nonverbal information (information extracted from facial expressions) separately, making it difficult to comprehensively analyze both and then accurately estimate a person's true emotional state and intentions.

[0007] This invention was made in view of the above problems, and aims to accurately determine the true mental state of the subject. grasp The purpose is to do so. [Means for solving the problem]

[0008] To solve the above problems, the information processing apparatus of the present invention comprises a communication unit that receives character data and facial expression data input to an input device, a language processing unit that analyzes the character data and extracts language information, a non-language processing unit that analyzes the facial expression data and extracts non-language information, and an evaluation unit that evaluates the degree of agreement or disagreement between the language information and the non-language information. The language processing unit performs normalization processing on the character data, performs tokenization processing to divide it into word or morpheme units, performs syntactic analysis processing to identify the dependency structure and phrase structure between each smallest unit, and performs word semantic ambiguity resolution and anaphora analysis to identify words. The nonverbal processing unit generates a distributed representation of the expression, extracts expressions representing emotions and their intensity, detects face regions from the expression data, performs landmark detection to identify facial feature points, analyzes the expression based on the detected feature points, estimates emotion classification and emotion intensity based on a basic emotion model, and the evaluation unit calculates the distance or area overlap between the location or shape of the emotion estimated from the linguistic information and the location or shape of the emotion estimated from the nonverbal information, and evaluates the degree of agreement or mismatch between the linguistic information and the nonverbal information by determining whether the calculated distance or area overlap is within a predetermined threshold range.

[0009] Here, the evaluation unit calculates the distance or area overlap between the location or shape of emotion estimated from the linguistic information and the location or shape of emotion estimated from the non-linguistic information, and evaluates the degree of agreement or mismatch between the linguistic information and the non-linguistic information by determining whether the calculated distance or area overlap falls within a predetermined threshold range. Furthermore, if the visualization processing unit that constitutes the evaluation unit determines that the linguistic information and the non-linguistic information are consistent, it compares them with the standard input trends for each user, and if they are identical to the input trends, it generates a current situation assessment diagram based only on the linguistic information, and if they differ from the input trends, it generates a current situation assessment diagram based on both the linguistic and non-linguistic information.

[0010] An information processing method according to another aspect of the present invention is a computer comprising a communication unit that receives character data and facial expression data transmitted from an input device, and a processor that performs information processing, wherein the computer performs language processing to analyze the character data and extract language information, non-language processing to analyze the facial expression data and extract non-language information, and evaluation processing to evaluate the degree of agreement or disagreement between the language information and the non-language information, wherein the language processing performs normalization processing on the character data, tokenization processing to divide it into word or morpheme units, syntactic analysis processing to identify the dependency structure and phrase structure between each smallest unit, and semantic ambiguity resolution and The system generates distributed representations of words through coreference analysis and extracts expressions representing emotions and their intensities. In the non-verbal processing, it detects face regions from the facial expression data, performs landmark detection to identify facial feature points, analyzes facial expressions based on the detected feature points, and estimates emotion classification and emotion intensity based on a basic emotion model. In the evaluation process, it calculates the distance or area overlap between the location or shape of emotion estimated from the linguistic information and the location or shape of emotion estimated from the non-verbal information, and evaluates the degree of agreement or mismatch between the linguistic and non-verbal information by determining whether the calculated distance or area overlap falls within a predetermined threshold range.

[0011] An information processing program according to another aspect of the present invention is a computer comprising a communication unit that receives character data and facial expression data transmitted from an input device, and a processor that performs information processing, which causes the computer to perform language processing to analyze the character data and extract language information, non-language processing to analyze the facial expression data and extract non-language information, and evaluation processing to evaluate the degree of agreement or disagreement between the language information and the non-language information, wherein the language processing performs normalization processing on the character data, tokenization processing to divide it into word or morpheme units, syntactic analysis processing to identify the dependency structure and phrase structure between each smallest unit, and semantic ambiguity resolution. The system generates distributed representations of words through cosmological analysis and extracts expressions representing emotions and their intensities. In the nonverbal processing, it detects face regions from the facial expression data, performs landmark detection to identify facial feature points, analyzes facial expressions based on the detected feature points, and estimates emotion classification and emotion intensity based on a basic emotion model. In the evaluation process, it calculates the distance or area overlap between the location or shape of emotion estimated from the linguistic information and the location or shape of emotion estimated from the nonverbal information, and evaluates the degree of agreement or mismatch between the linguistic and nonverbal information by determining whether the calculated distance or area overlap falls within a predetermined threshold range. [Effects of the Invention]

[0012] According to the present invention, by comprehensively analyzing linguistic information extracted from text data entered by the subject and nonverbal information extracted from facial expression data indicating the subject's facial expressions, it is possible to grasp the subject's true feelings and underlying emotional state that are not expressed in words. In particular, quantifying the degree of agreement or mismatch between verbal and nonverbal information, and generating an appropriate situational assessment diagram based on the evaluation results, makes it possible to grasp the psychological state more accurately. This makes it possible to provide individualized HR management measures (e.g., interviews, training, and environmental improvements) tailored to the mindset of each individual. Furthermore, by detecting the consistency or inconsistency between verbal and nonverbal information, it is possible to understand the gap between an individual's true feelings and what they outwardly express, which can be used to improve the workplace environment and enhance the health of the organization.

[0013] Furthermore, the present invention has the effect of accurately capturing subtle changes in psychological state that could not be detected by conventional single-modality analysis, by quantitatively evaluating the degree of agreement or mismatch between verbal and nonverbal information. In particular, it can detect latent problems early, such as cases where employees use superficially positive verbal expressions but their facial expressions reveal dissatisfaction or fatigue, thereby preventing the risk of employee turnover. Furthermore, comparative analysis against standard input tendencies established for each individual allows for evaluation that takes individual differences into account, enabling appropriate evaluation tailored to each subject. In addition, by accumulating and analyzing situation assessment diagrams consisting of multiple emotional items over time, it becomes possible to continuously monitor the overall health of the organization and quantitatively evaluate the impact of organizational changes and environmental shifts. This not only provides an objective basis for management decisions, but also contributes to improving organizational culture and enhancing psychological safety. In addition, the automated analysis and personalized policy suggestions powered by AI technology are expected to improve the efficiency of HR personnel, allowing them to focus on developing more strategic HR policies. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 is a diagram showing the overall system configuration including an information processing device according to one embodiment of the present invention. [Figure 2] Figure 2 is a diagram illustrating the concept of the emotion model. [Figure 3] Figure 3 shows an example of a current situation assessment diagram (radar chart). [Figure 4] Figure 4 is a diagram showing the configuration of a computer. [Figure 5] Figure 5 shows an example of a screen where the user enters information. [Figure 6] Figure 6 shows an example of a calendar screen for managing the schedules of the subjects. [Modes for carrying out the invention]

[0015] An embodiment of the present invention will be described with reference to the drawings. Note that in the following embodiments, the technologies of each embodiment can be applied to each other including modified examples. Also, the following embodiments do not limit the content of the present invention, and modifications can be made without departing from the scope of the claims.

[0016] [Overall configuration of the system including the information processing device] FIG. 1 is a diagram showing the overall configuration of an information processing system according to an embodiment of the present invention. This information processing system includes an information processing device 1000 composed of a Web server or the like, an operation terminal 2000 (also referred to as an "input device") operated by an employee (also referred to as a "user", "target person", etc.), and a database 3000. The number of these devices is only shown for convenience of explanation, and it is of course possible to have a configuration in which a plurality of each device is provided. Note that this information processing system may exist alone or may be a part of another system. That is, it may be realized as a part of the functions in another system.

[0017] This information processing device is composed of a computer having a configuration as shown in FIG. 4, and includes a language processing unit 100, a non-language processing unit 200, an evaluation unit 300, and a communication unit 400 Human Resources Management Policy Development Department 500 and is configured. The communication unit 400 is connected to the input device via a network, has a function of receiving various data from the input device, and transmitting the data to each processing unit in the information processing device. This input device is composed of a computer having a configuration as shown in FIG. 4 and is an apparatus operable by a user.

[0018] This input device is capable of connecting to or mounting an imaging device (camera), and is capable of inputting text data, audio data, and video data. Furthermore, when inputting these types of data, the imaging device captures the subject's face, and the subject's face data (facial expression data) is input along with the various types of data mentioned above. This data is stored in memory and transmitted to an information processing device via a network.

[0019] Figure 5 shows a screen displaying a calendar related to work tasks, as displayed on the input device. The display example shown is as of today (the 11th). By manipulating the icons on the screen, the corresponding screen will be displayed. For example, if the target person (employee) manipulates the square icon or the star icon, an information input screen like the one shown in Figure 6 will be displayed, allowing the target person to input information. When the target person (employee) inputs information while the icon is in the square icon state, it changes to a star icon. Furthermore, by manipulating the black circle icon or the double circle icon, an information input screen similar to that shown in Figure 6 will appear, allowing users to enter information into the answer fields in Figure 6. However, these black circle and double circle icons are only displayed when operated by a person with managerial privileges who is the administrator of the target person (employee). When the double circle icon is displayed, if a person with managerial privileges enters information, it will change to a black circle icon.

[0020] Figure 6 shows an example of an information input screen for entering business-related information, which is displayed on this input device. The screen shown in Figure 6 is the same as the calendar screen shown in Figure 5. Transition from This screen is displayed for each individual, and displays an individual ID to identify the individual. Furthermore, the screen in Figure 6 allows for the input of both objective and subjective information. Objective information includes specific work details (number of items, amount, tasks, etc.), while subjective information includes self-evaluation (what they felt today, what they noticed, results), future actions (planned work from today onward), and matters to discuss / report. Furthermore, each input field is provided where the administrator (manager) can enter a response.

[0021] [Process Flow] The processing in this information processing device includes (1) a process of collecting and integrating information (business data) received by the communication unit, (2) a process of analyzing the information (analysis process), (3) an evaluation process of evaluating the analyzed information, and (4) a process of generating business-related advice based on the analysis results (advice generation process).

[0022] (1) In the data collection and integration process, the business data collection process collects data (text data, facial expression data) related to the business performed by the subject, which includes completed task information (completed business information, incomplete business information), project progress rate, and time spent. Of this data, data including completed task information (completed business information, incomplete business information), project progress rate, and time spent may be obtained from a task management system (sales progress management system) that manages business operations or a development project management system that manages the progress of development.

[0023] In addition, the additional data collection process may include collecting subjective evaluation data such as employee job satisfaction, sense of growth, motivation level, loyalty satisfaction, and communication satisfaction. This subjective evaluation data can be obtained through direct input by users, or extracted through sentiment analysis from periodic surveys, interview records, or daily reports. In other words, you may collect the components of the status assessment diagram (radar chart), which will be described later.

[0024] Next, in the data integration and cleansing process, the collected business data and subjective evaluation data are integrated, and cleansing processes such as data normalization, handling of missing values, and removal of outliers are performed. If there are any problems with the quality of the data at this stage, the process returns to the data collection stage to collect additional data or make corrections.

[0025] Next, in the process of extracting indicators from management and business policies, key indicators and keywords related to performance evaluation are extracted from the company's management and business policies. These include management target KPIs, priority project information, and expected behavioral indicators. The necessary data will be collected through the processes described above.

[0026] (2) The analysis process includes language processing, which performs morphological analysis, syntactic analysis, semantic analysis, and sentiment analysis on text data such as daily reports; nonverbal processing, which performs face detection, landmark detection, and facial expression analysis on facial expression data; policy-work relevance analysis, which analyzes the degree of agreement between management policies and work content; work pattern analysis, which identifies efficient / inefficient work patterns; and indicator-work performance correlation analysis, which analyzes the relationship between subjective evaluations and objective results.

[0027] (3) The evaluation process evaluates the degree of agreement (degree of agreement) and the degree of disagreement (degree of disagreement) between linguistic information and non-linguistic information.

[0028] (4) In the advice generation process, the challenges and gaps of each individual are identified, and their strengths and weaknesses are analyzed to generate personalized advice. Team support guidance for managers is also created.

[0029] [Details of the language processing unit] The language processing unit 100 consists of a preprocessing unit 110, a syntactic analysis unit 120, a semantic analysis unit 130, and a natural language understanding unit 140.

[0030] The preprocessing unit 110 performs normalization processing on the string (hereinafter also referred to as "character data") among the information input to the input device, and then performs tokenization processing to divide it into word or morpheme units. Furthermore, it performs morphological analysis processing to divide the divided words into morphemes and assign part-of-speech information, thereby dividing them into the smallest linguistic units. This extracts basic linguistic information (text, character data) from the character data.

[0031] In the normalization process, the unification of uppercase and lowercase letters, the processing of special characters, the normalization of whitespace, and the standardization of notations such as URLs, email addresses, dates, and times are performed. For example, date information described in different formats such as "2023 / 4 / 1" and "April 1, 2023" is converted into a unified format such as "2023-04-01". Also, for emotive symbols such as emojis and pictographs, they are extracted as information useful for sentiment classification and converted into standardized codes. This ensures consistency in subsequent processing.

[0032] The tokenization process is a process of splitting the normalized text into the smallest language units such as words and morphemes. For "non-segmented languages" such as Japanese, techniques such as splitting based on the longest match method based on a dictionary and delimiting at the change points of character types are used. In the morphological analysis process, the character string is split into morphemes and part-of-speech information is assigned.

[0033] The syntactic analysis unit 120 receives the character string and the processing result processed by the preprocessing unit 110, and performs processing to identify the dependency structure and sentence structure between each minimum unit (between morphemes), and performs "part-of-speech tagging processing" and "syntactic analysis processing". In this part-of-speech tagging process, part-of-speech information such as nouns, verbs, and adjectives is assigned to each token. Also, in the syntactic analysis process, the syntactic structure of the sentence is analyzed and a syntax tree is constructed. In the dependency structure analysis process, the dependency relationships (such as modification and modified relationships) between words in the sentence are identified. This enables the extraction of higher-level language information from the character data.

[0034] The semantic analysis unit 130 receives the output information from the syntactic analysis unit 120, performs word sense disambiguation (WSD) and anaphora analysis, and obtains a distributed representation using word embedding technology. In this word sense disambiguation process, an appropriate meaning corresponding to the context of the polysemous word is identified. Also, in the anaphora analysis process, it is identified which element in the context a pronoun or an omitted subject refers to. Furthermore, in the word embedding process, a distributed representation that represents a word as a point in a vector space is generated. Through these processes, semantic language information is extracted from the character data.

[0035] The natural language understanding unit 140 receives the output of the semantic analysis unit 130 and performs language understanding processing, including intent recognition processing and sentiment analysis processing. In this intent recognition processing, the intent is extracted from the user's utterance. In this sentiment analysis processing, based on this information, the emotional polarity (positive, negative, neutral), more detailed emotional classifications (joy, sadness, anger, etc.) and emotional levels contained in the text are identified, and the emotional items and emotional levels of the emotion model described later are identified.

[0036] [Details of the non-linguistic processing section] The nonverbal processing unit 200 comprises a facial expression analysis unit 210, a voice analysis unit 220, a posture / motion analysis unit 230, a time-series modeling unit 240, and an emotional state estimation unit 250.

[0037] The facial expression analysis unit 210 detects faces from input images or videos and performs facial expression analysis. In the face detection process, it identifies face regions from the input image and identifies who the detected face belongs to. It performs a landmark detection process to detect facial landmarks and an expression analysis process to analyze facial expressions based on those landmarks, recognizing what kind of expression the user (viewer) of that face is making. This allows for the extraction of basic nonverbal information from the expression data.

[0038] The landmark detection process identifies the locations of facial feature points (eyes, nose, mouth, etc.). This process uses methods such as the 68-point method to identify feature points. The facial expression analysis process analyzes facial expressions based on the detected facial landmarks. This process uses information stored as multiple types of facial expressions based on the movement of facial muscles to identify expressions. This allows nonverbal information related to facial expressions to be extracted from the facial expression data.

[0039] The face detection process identifies the face region from the input image and identifies who the detected face belongs to. After face detection, the face is scale-normalized and pose-corrected to convert it into a standard frontal face, thereby improving the accuracy of subsequent facial expression analysis.

[0040] Prosodic features such as pitch (fundamental frequency F0), jitter (subtle fluctuations in pitch), and shimmer (subtle fluctuations in amplitude) are extracted. Pitch is an important feature that reflects the speaker's emotional state, while jitter and shimmer serve as indicators of changes in voice quality and emotional state. This allows for the extraction of nonverbal information related to speech from the audio data.

[0041] The posture and motion analysis unit 230 performs processing to analyze a person's posture and movements from input images or videos. In the skeletal detection process, it identifies the joint positions of the person. This allows non-verbal information related to posture and movements to be extracted from the video data.

[0042] The time-series modeling unit 240 models the temporal change patterns of the extracted acoustic features and posture / motion features. This extracts the change patterns of non-verbal information over time.

[0043] The emotional state estimation unit 250 estimates the user's emotional state based on integrated multimodal data. This involves classification based on basic emotional models (such as Paul Ekman's six basic emotions) as shown in Figure 2, and representation based on emotional dimension models (VAD: Valence Arousal Dominance). Specifically, it identifies the basic emotion (at least one of several emotional items) from these emotional models, and then identifies an emotional level for each identified emotional item. Figure 2 shows how the "low" and "high" levels gradually change. Furthermore, it detects complex emotions and tracks and stores changes in emotions over time. This allows for the estimation of an overall emotional state from multiple pieces of nonverbal information.

[0044] [Evaluation process of linguistic and non-linguistic information (consistency analysis process)] The evaluation unit 300 analyzes the consistency between the linguistic information extracted from the input character data and the non-linguistic information extracted from the facial expression data. This analysis process evaluates the degree of agreement and disagreement between the linguistic and non-linguistic information.

[0045] Through the above language and non-language processing, the emotion items and emotion levels of the basic emotion model have been set (calculated) for both the text data and the facial expression data. For convenience, the position of the emotion intensity (emotion level) in the language information will be referred to as "Position A," and the position of the emotion intensity (emotion level) in the non-language information will be referred to as "Position B."

[0046] Next, the evaluation unit 300 is configured to include a visualization processing unit 310. The evaluation unit 300 calculates a distance Z (the distance between position A and position B) to measure the degree of discrepancy between position A and position B. If distance Z is smaller than a certain value (a predetermined threshold), it indicates a high degree of agreement. Conversely, if distance Z is larger than a certain value (a predetermined threshold), it indicates a low degree of agreement.

[0047] In addition, the evaluation unit 300 may, instead of using distance Z, generate a shape by connecting the emotional intensity (level) of each emotion classification with lines, and calculate the area formed by this shape to determine the degree of overlap (overlap rate) of this area. The shape that shows the emotional intensity of verbal information is called "emotional shape A," and the shape that shows the emotional intensity of nonverbal information is called "emotional shape B." In other words, a high overlap rate between emotional shape A and emotional shape B indicates a high degree of agreement between verbal and nonverbal information, while a low overlap rate between emotional shape A and emotional shape B indicates a low degree of agreement between verbal and nonverbal information.

[0048] Based on this distance Z or area overlap, it is determined whether the written content matches the emotion. More specifically, the written content matches the emotion if any of the following conditions are met: location A and location B belong to the same emotion category; distance Z is within a certain range (threshold range); or area overlap exceeds a certain area. For example, a match is determined if both verbal and nonverbal information are mapped to the "joy" position in the basic emotion model, and the distance between them is within a predetermined threshold (e.g., 20% or less in the normalized space) (i.e., within the threshold range), or if the area overlap between emotion shape A and emotion shape B is 50% or more (i.e., within the threshold range). This criterion can also be statistically optimized from the average trends of the user group or the individual's past data.

[0049] On the other hand, when the distance Z is not within a certain range, it is determined that the written content does not match the emotion and that the written content is different from the emotion. In addition, when the degree of overlap of areas is less than or equal to a certain area, it is determined that the written content does not match the emotion and that the written content is different from the emotion.

[0050] If the results of the above calculation (distance Z or area overlap) indicate that the written content (text data) and emotion (expression data) match, then the input characteristics (input tendencies) of each user are estimated for each emotion classification (emotion item) of the basic emotion model. In other words, when it is determined that the written content and the emotion match, the input trends of "verbal information" and "nonverbal information" are identified and stored for each user. Specifically, based on the calculation results, when it is determined that the verbal information and nonverbal information belong to the same emotion category, the differences between "verbal information" and "nonverbal information" for each emotion category are identified and stored as the standard input trends for each user. The difference in this case is the difference between the emotional levels of the emotional items in the emotional models for both verbal and nonverbal information.

[0051] Next, the visualization processing unit 310 uses the results of this calculation (distance Z or area overlap) to create a current situation assessment diagram (radar chart) consisting of pre-specified items that can visualize the mindset based on the input information (such as job satisfaction, sense of growth, motivation level, loyalty satisfaction, and satisfaction with peer communication). An example of this current situation assessment diagram (radar chart) is shown in Figure 3. Figure 3 shows five items: "Job satisfaction, sense of growth, motivation level, loyalty satisfaction, and satisfaction with peer communication." Each item is rated on a scale of 1 to 5. Level 1 represents the lowest value for the item, and Level 5 represents the highest value.

[0052] More specifically, the system determines whether the distance Z is above a certain value or whether the area overlap is above a certain value. If it is below a certain value (meaning the "linguistic information" and "non-linguistic information" match) and the difference is not above a certain value (meaning it is the same as the standard input tendencies (input characteristics) for each individual), then, in order to speed up processing, a current situation assessment diagram (radar chart) consisting of the above items is created based only on the linguistic information extracted from the input character data. Here, "not exceeding a certain value" means, for example, that the difference from the standard input trend falls within a range of one standard deviation from the mean of the difference between verbal and nonverbal information in the past 30 inputs. This makes it possible to streamline processing for normal input patterns and analyze only abnormal patterns in detail.

[0053] If the distance Z is greater than or equal to a certain value, or if the degree of overlap of the areas is less than a certain value (i.e., the "linguistic information" and "non-linguistic information" do not match), the levels of the items in the current situation assessment map (radar chart) are provisionally determined using the linguistic and non-linguistic information respectively, and the current situation assessment map is created by adjusting the results of these provisional determinations.

[0054] As a preliminary identification process, first, the level (item level) of each emotion item is preliminaryly identified (Identification 1) from location A or emotion shape A and linguistic information extracted from the character data from which location A or emotion shape A was estimated. Next, the level (item level) of each emotion item is preliminaryly identified (Identification 2) from location B or emotion shape B and non-linguistic information extracted from the facial expression data from which location B or emotion shape B was estimated.

[0055] Once each emotional item in the current situation assessment diagram (radar chart) has been tentatively identified, if there is a discrepancy between verbal and nonverbal information, the pattern of that discrepancy is then identified. One mismatch pattern is when verbal information is positive and nonverbal information is negative (e.g., indifference or fatigue), and another is when verbal information is negative and nonverbal information is positive (e.g., lively or energetic). Depending on these mismatch patterns, an update process is performed to update the tentatively identified item level. In this context, "favorable" refers to "linguistic expressions that include positive feelings or intentions," "linguistic information that shows positive evaluation or reaction," "linguistic content that expresses attitudes such as support, agreement, cooperation, and empathy," and "communication with constructive, beneficial, and facilitative intentions." Furthermore, "malicious" refers to "linguistic expressions that include negative emotions or intentions," "linguistic information that shows negative evaluations or reactions," "linguistic content that expresses attitudes such as criticism, rejection, attack, or hostility," and "communication with destructive, harmful, or disruptive intentions."

[0056] Specifically, in the case of Pattern 1 above, the update process updates the level of each emotion item to be closer to (Specific 2) between (Specific 1) and (Specific 2), depending on the content of the facial expression. In other words, the item level is updated prioritizing the emotion level of nonverbal information. Furthermore, in the case of Pattern 2 described above, the level of each emotion item is updated to be closer to (Specific 1) between (Specific 1) and (Specific 2), depending on the content of the facial expression. In other words, the emotion level of the linguistic information is prioritized when updating the item level.

[0057] Therefore, in the update process, a decision is made based on the results of processing nonverbal information to determine whether to prioritize the emotional level of verbal information or the emotional level of nonverbal information.

[0058] Thus, the present invention makes it possible to evaluate whether text input and facial expressions (emotional expression) match or not, and to evaluate the degree of match (degree of agreement) when they match, and the degree of mismatch (degree of mismatch) when they do not match. Furthermore, this evaluation allows for analysis of how information about complex states changes, enabling a more accurate understanding of psychological states and the generation of appropriate human resource management measures.

[0059] [Human Resource Management Policy Generation Process (Generation Process in the Policy Generation Department)] Human Resources Management Policy Development Department 5 00 describes the process for deriving the policy suitability score, growth indicator, and advice content classification as objective variables, with the current situation assessment diagram (radar chart) included as an explanatory variable, based on the evaluation results of the evaluation process performed by the evaluation unit 300.

[0060] The policy fit score is calculated using the following formula. Policy fit score = w1×K + w2×P + w3×U + w4×B + w5×E Here, K is the "keyword matching score (frequency and appropriateness of policy keywords in business texts)", P is the "priority project contribution (rate of time spent on high-priority projects)", U is the "policy understanding (appropriate use of policy-related terminology and contextual understanding)", B is the "behavioral fit (degree of agreement with expected behavioral patterns)", E is the "policy empathy (calculated using the formula below)", and w1 to w5 are the "weighting coefficients (adjustable according to the organization's situation)". This score calculation combines text mining techniques and sentiment analysis to evaluate, for example, whether policy keywords are used not just for their appearance, but also in a positive context. Priority project contribution is calculated using project-specific work time data extracted from the task management system.

[0061] The degree of alignment with the company's policies (E) is calculated from the "level of job satisfaction (M)" and the "level of loyalty satisfaction (L)" using the following formula. E = α×M + β×L + γ×(M×L) Here, α, β, and γ are coefficients, and they also represent the synergistic effect of "job satisfaction" and "loyalty satisfaction."

[0062] The growth indicator is calculated using the following formula. Growth index = v1×S + v2×E + v3×G + v4×F + v5×R + v6×A + v7×C Here, S represents "skill improvement (combination of objective evaluation and self-assessment)", E represents "efficiency rate (rate of change in workload per unit of time)", G represents "goal achievement rate (average degree of achievement against set goals)", F represents "perceived growth index (weighted average of the most recent value and rate of change of "perceived growth")", R represents "role expansion (degree of expansion of work scope and level of responsibility)", A represents "knowledge application (degree of practical application of learned content)", C represents "challenge behavior index (frequency and success rate of tackling new tasks)", and v1~v7 represent "weighting coefficients".

[0063] In calculating the skill improvement level (S), the following adjustments are made, taking into account the "perceived growth" and "motivation level". S = S0 × (δ×G + ε×M) Here, S0 represents the "degree of improvement in basic skills (objective evaluation)," G represents the "degree of perceived growth," M represents the "motivation level," and δ and ε represent the "adjustment coefficients."

[0064] The advice is calculated by considering the "degree of gap," the "predicted improvement effect," and the "degree of individual suitability."

[0065] This system generates personalized advice based on the policy suitability score and growth indicators calculated above. This advice outlines human resource management measures, and this generation process generates human resource management measures based on the generated current situation assessment diagram and work content data. In other words, the policy generation unit has the function of deriving policy suitability scores, growth indicators, and advice content, identifying challenges and gaps for each individual (target person), and proposing personalized human resource management measures. Specifically, it first detects gaps for each individual (target person), then uses the gap detection results to identify strengths and weaknesses, generates individual advice, and finally generates team advice (guidance) for the managers who manage those individuals.

[0066] The individual gap detection process identifies gaps between company policies and individual work content. These include insufficient understanding of policies, skill gaps, and imbalances in time allocation. Gap detection involves machine learning analysis combining work content analysis using text mining and scores for each item in a current situation assessment diagram. For example, a decision tree algorithm is used to classify and evaluate important gaps from text features and scores from the current situation assessment diagram. In particular, gaps in areas with low "perceived growth" are given higher weight to identify areas that should be prioritized for improvement.

[0067] The strengths / weaknesses identification process identifies each employee's strengths and areas for improvement based on their work patterns and data. Specifically, the process is carried out in the following steps. Step 1) Extract highly efficient business categories identified from business pattern analysis as potential strengths. Step 2) Identify work areas with a high "level of job satisfaction" and "level of motivation" as potential strengths. Step 3) Identify the common elements of Step 1) and Step 2) as key strengths. Step 4) Identify the common areas between low-efficiency task categories and tasks with low job satisfaction as the main weaknesses. Step 5) Identify communication strengths / weaknesses based on the scores for each subcategory of "communication satisfaction" (supervisors, colleagues, other departments, etc.). Step 6) Analyze the relevance with the "Degree of growth perception" for each of the strengths and weaknesses, and evaluate the magnitude of the growth potential.

[0068] In the personalized advice generation process, optimized advice for each employee is generated based on the detected gaps and identified strengths / weaknesses. This process is executed in the following steps. Step 1) Prioritize advice items based on the size, priority, and difficulty of elimination of the gaps. Step 2) Classify the advice content into categories such as "Action recommendation", "Skill development", "Time allocation", "Communication improvement", etc. Step 3) In each category, consider an approach to improving weaknesses by leveraging strengths. Step 4) Optimize the balance between challenging goals and achievable goals from the balance of the "Motivation level" and the "Degree of growth perception". Step 5) Adjust the strength of the association with corporate value according to the level of "Loyalty satisfaction". Step 6) Deterministically learn the most effective advice format and content from past advice and its effectiveness history. Step 7) Generate specific and actionable advice sentences using natural language generation technology.

[0069] In the manager-oriented team support guidance generation process, guidance for team support is generated for managers based on the trends and gaps of the entire team. Specific processing steps are shown below. Step 1) Visualize the data distribution among team members and identify the trends and outliers of the entire team. Step 2) Evaluate the team communication status from the average value and variance of the "Communication satisfaction" within the team. Step 3) Analyze the skill set distribution among members and identify the strengths / weaknesses of the entire team. Step 4) Evaluate the within-team differences regarding policy understanding from the comparison of the policy compliance score for each member. Step 5) Generate proposals for optimizing role allocation within the team. Step 6) Calculate team health indicators from the time-series changes in "motivation level" and "job satisfaction level." Step 7) Based on these analysis results, generate guidance that includes specific action plans to promote the growth of the entire team.

[0070] The advice generated by each of the above processes will be constantly updated and improved through the continued use of the system. The advice effectiveness measurement process measures the extent to which the generated advice contributed to behavioral change and performance improvement among employees. By continuously optimizing the advice generation logic based on these measurement results, it becomes possible to provide more effective personalized advice.

[0071] As an example of implementing the present invention, a camera is installed in an information processing system or a camera (imaging device) mounted on a computer is used to record the facial expressions of employees when they input their daily reports.

[0072] The input interface for this daily work report includes not only fields for recording objective information such as the content of work, but also fields for self-assessment of subjective information such as job satisfaction, sense of growth, motivation level, loyalty satisfaction, and communication satisfaction, using multiple scales.

[0073] This information processing system analyzes text input and facial expression data to evaluate the consistency between linguistic and nonverbal information. As a result of this evaluation, a certain percentage of inconsistencies are detected between linguistic information (text) and nonverbal information (facial expressions). For example, there are cases where someone writes "work is progressing smoothly" but their facial expression shows signs of fatigue or stress, or where someone writes "there are many challenges" but their facial expression suggests a willingness to take on challenges.

[0074] By analyzing these patterns of inconsistency, it becomes possible to grasp the true state of mind of employees (users) that cannot be captured by text alone, enabling the implementation of more appropriate human resource management measures. For example, if an employee writes positively but shows signs of fatigue on their face, it becomes possible to implement HR management measures such as redistributing tasks or providing support resources. On the other hand, if they write positively, teeth For employees who, despite writing negatively, show enthusiasm through their expressions, it becomes possible to implement personnel management measures such as acknowledging their daily achievements and creating an environment that allows them to strive for even greater heights.

[0075] Thus, by comprehensively analyzing verbal and nonverbal information, this invention makes it possible to visualize employees' true feelings and psychological states, which could not be captured by conventional methods, and to achieve more effective human resource management.

[0076] As shown in Figure 4, the computer comprises a processor 901, a storage device 902, an interface 903, and a display device 904.

[0077] The processor 901 controls processing in the computer by executing information processing programs stored in the storage device 902. For example, each functional unit and processing unit of the information processing device is realized by the processor 901 executing information processing programs stored in the storage device 902. Information processing programs are executed by the computer.

[0078] The storage device 902 temporarily stores the source code of the program executed by the processor 901 and data required when the program is executed, and is a volatile or non-volatile storage medium such as RAM, ROM, hard disk drive (HDD), or flash memory. This storage device 902 also serves as a database for storing the operating system (OS), various programs for implementing the above configurations, configuration information, tables, and so on.

[0079] The processor 901 reads and executes these programs, configuration information, table information, etc., to realize various functions.

[0080] The I / F903 is an input / output interface (also called a communication unit) for receiving user input from keyboards, mice, touch panels, various sensors, wearable devices, etc. It is an interface for inputting and outputting data from external devices to a computer, and a communication interface for communicating data between computers or with external devices via a communication network, either wired or wirelessly.

[0081] The display device 904 is a device that displays various types of information. Specific examples include liquid crystal displays, organic EL displays, and wearable devices. It may also be a touch panel or touch input device that combines an input / output interface with the display device 904.

[0082] The learning described above involves inputting data patterns into a computer based on training data, which consists of information paired with correct judgments. The computer then uses these patterns to discover new features and relationships. Another example is the computer's ability to identify commonalities and characteristic data within the input data.

[0083] In addition, learning can be exemplified by methods such as deep learning, in which a computer constructs a neural network to analyze, extract, and output local or characteristic data.

[0084] Of course, this learning process includes preprocessing, which consists of processes such as supplementing missing data, removing unnecessary data, and standardizing data formats, as well as postprocessing, which stores the learned results so that they can be used for other learning.

[0085] As used in the embodiments and claims described above, the terms “part,” “means,” “apparatus,” and “system” do not merely refer to physical means, but also include cases where the functions of these are realized by software or software services.

[0086] Furthermore, the functions of a single "part," "means," "apparatus," or "system" may not only be realized by a single physical means, software, software module, or apparatus, but may also be realized by multiple physical means, software, software modules, apparatus, or combinations thereof.

[0087] The terms used in the embodiments and claims described above should be interpreted as non-limiting terms. For example, the term "includes" should be interpreted as "not limited to those described as including." The term "contains" should be interpreted as "not limited to those described as containing." The term "equips" should be interpreted as "not limited to those described as equipped." The term "possesses" should be interpreted as "not limited to those described as possessing." The term "complements" should be interpreted as "not limited to those described as possessing."

[0088] Furthermore, in the estimation, generation, prediction, and classification processes of the various types of information described above, AI (artificial intelligence) deep learning and machine learning are employed, and the target variables and explanatory variables used at that time are as follows. The dependent variable is the data that is estimated or generated, also known as the "ground truth data," and is the data output through these processes. Therefore, this dependent variable is described using expressions such as "what is estimated or generated" or "what is estimated or generated." Furthermore, explanatory variables are variables used for estimation or generation, and are the data necessary for estimation and generation processes; they are also called "features." Therefore, these explanatory variables are described using expressions such as "what is used as the basis for estimation or generation" or "what is the basis for estimation or generation." [Industrial applicability]

[0089] This invention can be widely used as a human resource management system in companies. In particular, it is useful as a tool to solve the difficulty in understanding the psychological state of employees that arises from the decrease in face-to-face communication amidst the increase in remote work and hybrid work.

[0090] Furthermore, this technology can be applied not only to human resource management but also to various fields where understanding human emotions is crucial, such as customer service, healthcare, and education. For example, in online education systems, it can be used to evaluate learners' comprehension and concentration levels, enabling automatic adjustment of learning content difficulty and the suggestion of individually optimized learning plans. In the medical field, it can be used to assess patients' psychological states, aiding in mental illness screening and monitoring treatment effectiveness. In the customer service field, it can detect true customer satisfaction and dissatisfaction, leading to more appropriate service improvements. [Explanation of Symbols]

[0091] 100 Language Processing Unit 110 Pre-processing 120 Syntax Parsing Unit 130 Semantic analysis section 140 Natural Language Understanding Department 200 Nonverbal Processing Unit 210 Facial Expression Analysis Unit 220 Voice Analysis Unit 230 Posture / motion analysis department 240 Time Series Modeling Section 250 Emotional State Estimation Unit 300 Evaluation Department 400 Communications Department 5 00 Human Resources Management Policy Development Department

Claims

1. A communication unit that receives character data and facial expression data input to an input device, A language processing unit that analyzes the aforementioned character data and extracts language information, A nonverbal processing unit that analyzes the aforementioned facial expression data to extract nonverbal information, An evaluation unit that evaluates the degree of agreement or disagreement between the linguistic information and the non-linguistic information. It is equipped with, The language processing unit described above is The aforementioned character data is subjected to normalization processing and tokenization processing to divide it into word or morpheme units. The syntactic analysis process identifies the dependency structure and phrase structure between each smallest unit. By resolving semantic ambiguity and performing coreference resolution, distributed representations of words are generated. We extract the emotion classification that represents the emotion and the emotion intensity within that emotion classification. The aforementioned non-verbal processing unit is From the aforementioned facial expression data, a facial region is detected, and landmark detection is performed to identify the characteristic points of the face. Based on the detected feature points, facial expressions are analyzed, and emotion classification and emotion intensity are estimated based on a basic emotion model. The evaluation unit, The distance or area overlap between the location or shape of emotion estimated from the linguistic information and the location or shape of emotion estimated from the non-linguistic information is calculated. The degree of agreement or disagreement between linguistic and non-linguistic information is evaluated by determining whether the calculated distance or area overlap falls within a predetermined threshold range. Information processing device.

2. The system further comprises a visualization processing unit that generates a current situation assessment diagram consisting of multiple emotional items using the evaluation results evaluated by the evaluation unit, The visualization processing unit is If the linguistic and non-linguistic information is determined to match, it will be compared to the user's standard input patterns. If the input trend is the same, a current situation assessment diagram will be generated based solely on linguistic information. The information processing device according to claim 1, which generates a current situation assessment diagram based on both linguistic and non-linguistic information when the input trend differs from the said input trend.

3. The visualization processing unit is If a discrepancy is found between linguistic and non-linguistic information, the discrepancy pattern is identified. In cases where verbal information is positive but nonverbal information is negative, adjust the level of each emotion item according to the content of the nonverbal information. The information processing device according to claim 2, characterized in that, in cases where the linguistic information is malicious and the nonverbal information is favorable, the level of each emotion item is adjusted according to the content of the linguistic information.

4. A communication unit that receives character data and facial expression data transmitted from an input device, A processor that performs information processing, A computer equipped with, Language processing, which analyzes the aforementioned character data to extract linguistic information, Nonverbal processing, which analyzes the aforementioned facial expression data to extract nonverbal information, An evaluation process that evaluates the degree of agreement or mismatch between the linguistic information and the non-linguistic information. Execute, In the aforementioned language processing, The aforementioned character data is subjected to normalization processing and tokenization processing to divide it into word or morpheme units. The syntactic analysis process identifies the dependency structure and phrase structure between each smallest unit. By resolving semantic ambiguity and performing coreference resolution, distributed representations of words are generated. We extract the emotion classification that represents the emotion and the emotion intensity within that emotion classification. In the aforementioned nonverbal processing, From the aforementioned facial expression data, a facial region is detected, and landmark detection is performed to identify the characteristic points of the face. Based on the detected feature points, facial expressions are analyzed, and emotion classification and emotion intensity are estimated based on a basic emotion model. In the aforementioned evaluation process, The distance or area overlap between the location or shape of emotion estimated from the linguistic information and the location or shape of emotion estimated from the non-linguistic information is calculated. The degree of agreement or disagreement between linguistic and non-linguistic information is evaluated by determining whether the calculated distance or area overlap falls within a predetermined threshold range. Information processing methods.

5. A communication unit that receives character data and facial expression data transmitted from an input device, A processor that performs information processing and A computer equipped with, Language processing that analyzes the aforementioned character data to extract linguistic information, Nonverbal processing, which analyzes the aforementioned facial expression data to extract nonverbal information, An evaluation process that evaluates the degree of agreement or mismatch between the linguistic information and the non-linguistic information. Make it run, In the aforementioned language processing, The aforementioned character data is subjected to normalization processing and tokenization processing to divide it into word or morpheme units. The syntactic analysis process identifies the dependency structure and phrase structure between each smallest unit. By resolving semantic ambiguity and performing coreference resolution, distributed representations of words are generated. We extract the emotion classification that represents the emotion and the emotion intensity within that emotion classification. In the aforementioned nonverbal processing, From the aforementioned facial expression data, a facial region is detected, and landmark detection is performed to identify the characteristic points of the face. Based on the detected feature points, facial expressions are analyzed, and emotion classification and emotion intensity are estimated based on a basic emotion model. In the aforementioned evaluation process, The distance or area overlap between the location or shape of emotion estimated from the linguistic information and the location or shape of emotion estimated from the non-linguistic information is calculated. The degree of agreement or disagreement between linguistic and non-linguistic information is evaluated by determining whether the calculated distance or area overlap falls within a predetermined threshold range. Information processing program.

Images