system

Abstract

We provide the system. [Solution] Means for monitoring information obtained from users, Means for transmitting the aforementioned information to an analysis device, The aforementioned analytical device analyzes the information and provides means for evaluating the possibility of harassment, A means for issuing a notification to the user based on the evaluation results, Means for including suggestions for improvement in the aforementioned notification, A means to monitor family conversations in real time and guide the dialogue in a better direction, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, harassment has become a serious social problem in the workplace and online environments. Generally, harassers often lack the awareness that their actions are harassment against others. Therefore, harassment may occur unintentionally in a communication setting, which may have an adverse impact on human relationships and the workplace environment. Conventional countermeasures mainly focus on post - event responses, and there is a problem that it is difficult to prevent harassment in advance. Thus, there is a need for a system that can detect harassment in real time and point it out immediately.

Means for Solving the Problems

[0005] This invention monitors communication data in real time and transmits it to a data analysis device. The data analysis device uses natural language processing and speech recognition technology to analyze the received data and score expressions suspected of being harassment. Depending on the scoring results, the system notifies the user and provides suggestions for improvement to encourage user awareness. Furthermore, by periodically analyzing the communication data and generating reports showing trends, the system enables users to make long-term improvements. In this way, it provides a system that aims to prevent harassment.

[0006] A "user" refers to an individual or legal entity that uses a communication tool to send and receive messages.

[0007] "Communication data" refers to information including text messages and voice data exchanged between users.

[0008] A "data analysis device" refers to a device that analyzes collected communication data to detect harassment.

[0009] "Natural language processing" refers to the technology that enables machines to understand and process human language.

[0010] "Speech recognition technology" refers to the technology that takes speech data as input and converts it into text data.

[0011] "Scoring" refers to a method of quantifying the likelihood that the content of analyzed data constitutes harassment.

[0012] "Notification" refers to a message that informs a user of the possibility of harassment based on the results of data analysis.

[0013] "Improvement proposals" refer to specific actions or methods provided to users to mitigate the possibility of harassment.

[0014] A "report" refers to a documented report that analyzes regularly collected data and shows user communication trends and improvement progress. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system that analyzes communication data exchanged by users in real time and detects the possibility of harassment. This system can be implemented by integrating it with appropriate communication tools and following the process outlined below.

[0037] First, the terminal acquires text messages and voice data sent and received by the user in real time. This data is managed by a data acquisition module and sent to the server as needed. Data transmission is carried out using secure communication protocols and with consideration for data protection.

[0038] The server analyzes received text messages using natural language processing techniques to understand their meaning and context. Similarly, audio data is converted to text using speech recognition technology and analyzed in the same manner. The analysis results identify harassment-related keywords and expressions, and a score is assigned based on these. This score indicates an evaluation for each category—sexual harassment, power harassment, and moral harassment—and helps detect unintentional harassment by the user.

[0039] If the score exceeds a certain threshold, the server immediately issues a notification to the device. The device informs the user of the potential for harassment and suggests ways to improve. This helps the user become aware of the impact of their words and encourages them to improve their communication.

[0040] Furthermore, the server aggregates data at regular intervals and analyzes users' communication trends. Based on the analysis results, it generates a report and sends it to the terminal. This report includes information such as the frequency of harassment occurrences and areas for improvement, helping users to make long-term improvements.

[0041] As a concrete example, consider a case where a user says, "You failed again? Come on, give me a break," during an online meeting. This statement is captured by the device and analyzed as text on the server. In this case, a harassment score is calculated based on the wording and context, and if it exceeds the threshold, a warning is immediately displayed on the device. This notification also includes suggestions for improvement, such as, "Try to switch to positive feedback and encourage the other person." Based on this information, the user can review their statements and be motivated to improve their communication with others.

[0042] Through such a system, it is possible to prevent harassment throughout the entire organization and promote safe and healthy communication.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The device acquires text messages and voice data sent by the user using communication tools in real time.

[0046] Step 2:

[0047] The terminal performs initial processing on the acquired data and sends it to the server using a secure communication protocol. The voice data is sent in its original form and then analyzed on the server.

[0048] Step 3:

[0049] The server registers the received data in a parsing queue and begins processing it sequentially. Text messages are immediately parsed by the natural language processing module.

[0050] Step 4:

[0051] The server processes the audio data through a speech recognition engine to convert it into text. Then, it is analyzed using a natural language processing module, similar to how text messages are processed.

[0052] Step 5:

[0053] The server uses a machine learning model to score the likelihood of harassment based on the data analysis results. In this process, it analyzes specific keywords and contexts to numerically evaluate the likelihood of harassment.

[0054] Step 6:

[0055] If the score exceeds a predetermined threshold, the server generates an alert regarding that specific message.

[0056] Step 7:

[0057] The server sends the generated alert to the terminal. The terminal displays a notification to the user, providing information about the potential harassment and suggestions for improvement.

[0058] Step 8:

[0059] The server analyzes the accumulated data at regular intervals and generates a report summarizing communication trends.

[0060] Step 9:

[0061] The server sends the generated report to the terminal, and the user can review its contents to understand areas for long-term improvement.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] In today's communication environment, harassment can occur unconsciously, negatively impacting individuals and organizations. However, technology to monitor this in real time and propose effective solutions still does not exist. Addressing this problem is urgently needed.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for monitoring communication information acquired from a user, means for transmitting the communication information to a data processing device, and means for the data processing device to analyze the communication information and quantify the possibility of harassment. This makes it possible to detect harassment in real time and make appropriate improvement suggestions to the user, thereby improving the communication environment of individuals and organizations.

[0067] A "user" is an individual or group that uses the system to send and receive communication information.

[0068] "Communication information" refers to all information exchanged between users, including text messages and voice data.

[0069] "Monitoring methods" refer to processes and devices that collect communication information obtained from users in real time and check for any abnormalities.

[0070] A "data processing device" refers to hardware or software used to analyze collected communication information.

[0071] "Means of analysis" refers to the process of evaluating the content of communication information using natural language processing technology and speech recognition technology to detect specific patterns and keywords.

[0072] "Methods of quantification" refers to the process of evaluating the possibility of harassment based on the analysis results according to certain criteria and expressing it as a numerical value.

[0073] A "warning" refers to a message that notifies a user of potential harassment and encourages them to be aware of it.

[0074] "Improvement suggestions" refer to information that guides users on specific measures or changes in wording that they should take immediately to avoid harassment.

[0075] "Data in different formats" refers to information in various formats, such as text data and audio data.

[0076] "Detailed analysis" refers to a process that goes beyond simple keyword detection and includes comprehensive data evaluation, encompassing contextual and sentiment analysis.

[0077] This invention is a system that analyzes communication information exchanged between users in real time and evaluates the possibility of harassment. The system is implemented in the following way.

[0078] First, the terminal acquires text messages and voice data sent and received by the user in real time. This process utilizes the terminal's built-in microphone and keyboard input, and is managed by a data acquisition module. The terminal then sends the acquired communication information to the server using a secure communication protocol (e.g., TLS). This ensures the security of the data.

[0079] The server processes the received communication information. In particular, it analyzes text data using natural language processing techniques. This is done using open-source natural language processing libraries (e.g., SpaCy, NLTK). Audio data is also converted to text using speech recognition technology (e.g., Google® Speech-to-Text API) and analyzed in the same way.

[0080] Based on the analysis results, the server identifies specified harassment-related keywords and expressions and assigns numerical values ​​to them. This numerical value is scored based on a specific algorithm, and an evaluation is performed for each harassment category. If the score exceeds a set threshold, the server immediately notifies the terminal.

[0081] The device displays a warning to the user that their comments may be harassing. This warning also includes specific suggestions on how the user can improve their behavior. For example, it might advise, "Try to switch to positive feedback and encourage the other person."

[0082] Furthermore, the server aggregates data at regular intervals and analyzes user communication trends. Based on this analysis, a detailed report is generated for the user and sent to their terminal. This report contains information that can help improve communication.

[0083] As a concrete example, consider the message "Did you fail again? Enough is enough." spoken by a user during an online meeting. This message is captured on the device and transcribed and analyzed on the server. If problematic keywords are found, the potential for harassment is scored, and a warning is displayed if necessary.

[0084] An example of a prompt for a generative AI model is: "Analyze the following text in real time and score its likelihood of harassment: 'Did you fail again? Come on, give it a rest.'" Using this prompt, the AI ​​model analyzes the specified text and returns an appropriate score.

[0085] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0086] Step 1:

[0087] The terminal acquires text messages and voice data sent and received by the user in real time. The input is messages and call information from the user, and the output is the acquired raw communication information. The terminal manages this data using a data acquisition module and securely stores the data using encryption technology.

[0088] Step 2:

[0089] The terminal prepares to send the acquired communication information to the server. The input is the communication information collected in step 1, and the output is the data that has been formatted and encrypted for transmission. The terminal sends the data to the server using a security protocol such as Secure Sockets Layer (SSL).

[0090] Step 3:

[0091] The server receives data sent from the terminal. The input is encrypted communication information, and the output is decrypted data. The server verifies the integrity and validity of the received data and securely stores it in the database.

[0092] Step 4:

[0093] The server analyzes text messages using natural language processing techniques. The input is decoded text data, and the output is contextual understanding and keyword extraction information resulting from the analysis. The server uses open-source natural language processing libraries to perform this analysis and records identified keywords and context in a specific format.

[0094] Step 5:

[0095] The server uses speech recognition technology to convert audio data into text and then performs analysis. The input is decoded audio information, and the output is the text data and its analysis results. The server uses a speech recognition API to convert audio to text and then performs analysis.

[0096] Step 6:

[0097] The server quantifies the likelihood of harassment based on the analysis results. The input is the analysis results of text data, and the output is the harassment score for each category. The server uses a scoring algorithm to calculate this score and stores it in a database.

[0098] Step 7:

[0099] The server sends a notification to the device if the score exceeds a threshold. The input is the harassment score, and the output is a warning message sent to the user. The server includes improvement suggestions in the notification and relays them to the device.

[0100] Step 8:

[0101] The terminal displays received notifications to the user. Input is a warning message from the server, and output is an alert displayed on the user interface. The terminal allows the user to review the notifications and provides a feedback system as needed.

[0102] Step 9:

[0103] The server analyzes data collected at regular intervals and generates reports showing user communication trends. The input is aggregated communication data, and the output is a detailed report. The server uses a database management system to evaluate the data and periodically sends reports to the terminals.

[0104] (Application Example 1)

[0105] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0106] In modern society, communication within families often breaks down, contributing to increased stress within households. In particular, the prevalence of unconscious or intentional verbal harassment within families negatively impacts the home environment. Effective measures are needed to prevent this problem and maintain healthy communication within families.

[0107] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0108] In this invention, the server includes means for monitoring information obtained from the user, means for transmitting the information to an analysis device, and means for analyzing the information and evaluating the possibility of harassment. This makes it possible to improve the quality of communication within the family, reduce stress among family members, and promote better dialogue.

[0109] "Information obtained from users" refers to communication-related information such as text data and audio data generated or transmitted by users.

[0110] "Monitoring methods" refer to functions that continuously check information obtained from users and collect necessary data.

[0111] "Means for transmitting to an analytical device" refers to the function of appropriately transmitting data to a data analysis device in order to process the acquired information.

[0112] "Means for analyzing and evaluating the possibility of harassment" refers to a process for analyzing received information and evaluating whether a particular action constitutes harassment.

[0113] "Means of issuing notifications" refers to a function that provides users with warnings and suggestions for improvement based on evaluation results.

[0114] "Means for including suggestions for improvement" refers to a function that provides users with specific suggestions to encourage better communication.

[0115] "A means of monitoring conversations in real time and guiding dialogue in a better direction" refers to a function that instantly analyzes conversations between users and provides feedback to maintain smooth communication.

[0116] To implement this invention, it is necessary to implement a dedicated system as a program and run it on appropriate hardware. A specific example of this is an information analysis system used in a home environment.

[0117] The server uses the Google Cloud Speech-to-Text API to convert speech data received from users into text data in real time. This process makes the speech data easily parseable. The converted text data is sent to an analysis device, where text analysis is performed using the natural language processing libraries NLTK or spaCy.

[0118] Based on these analysis results, the device provides feedback to the user. This feedback includes notifications if potential harassment is detected and specific suggestions for improving communication. Furthermore, the device monitors family conversations in real time and provides feedback to facilitate better communication.

[0119] For example, if a family is having a conversation in the living room and an argument breaks out between the children, the device will analyze the conversation and immediately provide advice such as, "Let's try to respect each other's opinions and have a more constructive discussion."

[0120] An example of a prompt to input into the generative AI model would be, "Analyze aggressive or negative remarks heard in the home and consider how to address them each time." This would allow the system to provide appropriate feedback to support healthy communication within the home.

[0121] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0122] Step 1:

[0123] The server acquires voice data from the user. This voice data is collected in real time from conversations within the home. The input is received as voice data through the microphone, and the output becomes an audio file for analysis.

[0124] Step 2:

[0125] The server uses the Google Cloud Speech-to-Text API to convert the acquired audio data into text data. Speech recognition technology is applied to the audio file as input. The output is data in a parseable text format.

[0126] Step 3:

[0127] The server analyzes the text data using a natural language processing library (NLTK or spaCy) to identify expressions that may constitute harassment. This step involves analyzing the text data as input for harassment-related keywords and context. The output is identification information for the relevant sections in the text.

[0128] Step 4:

[0129] The server evaluates the analysis results and generates alerts as needed. If the identified harassment exceeds a certain threshold, it sends a notification to the terminal. The input is the analysis results and evaluation criteria, and the output is a warning notification.

[0130] Step 5:

[0131] The terminal presents notifications to the user based on the generated alerts and provides necessary improvement suggestions. In this step, the alerts and suggestions sent from the server are used as input, and the user is notified visually or audibly. The output is a warning message and improvement suggestions for the user.

[0132] Step 6:

[0133] The user attempts to improve the conversation based on the suggestions presented. This step involves the user, upon receiving a notification from their device, deciding how to modify their behavior to improve the quality of communication. Specifically, this requires choosing more assertive or gentle language.

[0134] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0135] This invention is a system that not only analyzes communication data to detect the possibility of harassment, but also analyzes the user's emotions simultaneously, enabling more adaptive improvement suggestions. The basic configuration for implementing this system is as follows.

[0136] First, the terminal acquires text messages and voice data exchanged by users using communication tools in real time. This acquired data is promptly sent to the server. Upon receiving this data, the server performs analysis using natural language processing and speech recognition technologies.

[0137] This system further incorporates an emotion engine into the data analysis device to identify the user's emotional state from text messages and voice data. Emotion identification is based on the context of the user's text messages, as well as their tone and speed of voice.

[0138] After the server scores the harassment behavior, it further considers this emotional data. Based on the scoring results and emotional state, it immediately sends a notification to the user. The device receives this notification and displays it to the user. The notification includes improvement suggestions based on the emotional state and the harassment behavior score. These suggestions are tailored to the user's current emotions and provide more effective feedback.

[0139] For example, if a user sends the message, "You never understand anything," the device retrieves this message as data and sends it to the server. The server uses natural language processing to evaluate the aggression of the message and scores its potential for harassment. Simultaneously, an emotion engine recognizes the user's feelings of frustration and irritation. As a result, the server sends an emotion-sensitive notification to the device, such as, "This statement may be harassment. Please calm down and let's work together to find a concrete solution." Upon receiving this notification, the user can reflect on their own emotions and be motivated to choose a better way of communicating.

[0140] Thus, this invention prevents harassment by analyzing user emotions and encouraging more appropriate behavior. As a result, it is possible to promote safe and constructive communication throughout the organization.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] The device acquires text messages and voice data sent by the user using communication tools in real time.

[0144] Step 2:

[0145] The terminal sends the acquired data to the server using a secure protocol. Voice data is sent to the server as is, but it is not converted to text.

[0146] Step 3:

[0147] The server adds the received data to a parsing queue, and the text messages are analyzed using a natural language processing module. Similarly, audio data is converted to text using a speech recognition engine and then analyzed.

[0148] Step 4:

[0149] The server inputs text data and analyzed audio data into the emotion engine to identify the user's emotional state. This emotion analysis is based on the tone of the text and keywords that suggest emotion.

[0150] Step 5:

[0151] The server integrates emotional information and text analysis results to score the likelihood of harassment. In addition to the likelihood of harassment, the emotional state is also considered in the scoring.

[0152] Step 6:

[0153] The server creates an alert to notify the user if the score exceeds a predetermined threshold.

[0154] Step 7:

[0155] The server sends a notification to the device that includes improvement suggestions tailored to the user's emotional state.

[0156] Step 8:

[0157] The terminal displays notifications received from the server to the user, offering suggestions for improvement that are sensitive to the user's feelings, along with information about the possibility of harassment.

[0158] Step 9:

[0159] The server analyzes the accumulated data and generates reports, including individual user communication trends, at regular intervals, sending them to the user's device. Users can then use these reports to implement long-term behavioral changes.

[0160] (Example 2)

[0161] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0162] In today's information and communication environment, while communication between users is increasing, the risk of unintentional harassment is also rising. Such behavior can lead to a deterioration of interpersonal relationships and cause serious problems for organizations and individuals. However, users themselves are often unaware of it, so there is a need for technology that can efficiently and automatically detect harassment and provide appropriate feedback.

[0163] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0164] In this invention, the server includes means for monitoring information acquired from the user, means for transmitting the information to a processing device, and means for the processing device to analyze the information and score the likelihood of the behavior. This makes it possible to analyze behavior in real time, quickly detect potential harassment behavior, and provide the user with appropriate improvement suggestions.

[0165] A "user" refers to an individual or group that uses the system to communicate.

[0166] "Information" refers to digital data, including text messages and audio data, that users generate or receive.

[0167] "Means of monitoring" refers to functions for acquiring information in real time and continuously observing its content and characteristics.

[0168] A "processing unit" refers to a computer or a part thereof that analyzes received information and performs necessary calculations and decisions.

[0169] "Means of analysis" refers to technical means used to break down information using natural language processing and speech recognition technologies and derive conclusions that align with a specific purpose.

[0170] "Means for scoring the likelihood of an action" refers to a function that quantifies or evaluates the degree of a specific action or impact based on the analysis results.

[0171] "Notifications" refer to messages generated based on analysis and scoring results that convey information to the user.

[0172] "Adaptive improvement suggestions" refer to suggestions that include specific and effective advice on behavioral improvements tailored to the user's current situation and emotional state.

[0173] This invention is a system for detecting potential harassment by acquiring information exchanged by users through communication tools in real time. The system includes a terminal used by the user, a server for analyzing the data, and a function to notify the user of improvement suggestions based on the analysis results.

[0174] First, the device continuously acquires text messages and voice data sent and received by the user. Specific target tools include instant messaging software and collaboration platforms. This acquired information is then transmitted to a server via a secure protocol.

[0175] Next, the server receives this information and performs analysis using natural language processing and speech recognition technologies. The technologies used here include generative AI models; for example, open-source libraries and commercial APIs are used for natural language processing, and APIs from speech technology providers are used for speech recognition. Based on this, the server evaluates the aggressiveness and negative tone of the message and scores the likelihood of the action based on the results.

[0176] Furthermore, the server uses an emotion engine to identify the user's emotional state from the information. This involves analyzing text and voice features to quantify the user's emotions. For example, if the user is feeling frustrated or stressed, this will be reflected in the analysis results and influence the final feedback.

[0177] The server generates notifications for the user based on the scoring results and emotional state. The notifications include potential harassment behavior and specific suggestions for improvement. The suggestions are tailored to the user's emotional state, aiming to provide more effective feedback. For example, if the message "You never understand anything" is analyzed, a notification such as "This statement may be considered harassment. Let's calm down and work together to find a concrete solution" will be sent.

[0178] An example of a prompt might be: "If you receive the following message, how should you interpret it and determine your emotional state? Also, generate suggestions for behavioral improvement. Message: 'You never understand anything.'"

[0179] In this way, the system can provide constructive feedback to users and prevent harassment from occurring.

[0180] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0181] Step 1:

[0182] The terminal acquires text messages and voice data sent and received by the user using communication tools in real time. The input is information sent by the user, and the output is the acquired digital data. This data is temporarily stored on the terminal and then prepared to be sent to the server. The terminal checks the data format, encodes it as needed, and converts it into a format suitable for transmission.

[0183] Step 2:

[0184] The terminal sends acquired text messages and voice data to the server. Input is digital data temporarily stored within the terminal, and output is data sent to the server. The terminal ensures the confidentiality and integrity of the data by sending it using a secure protocol (e.g., HTTPS). Error checking is also performed during the transmission process to confirm that the data was sent correctly.

[0185] Step 3:

[0186] The server receives data sent from the terminal and prepares it for analysis. The input consists of text and audio data sent from the terminal, and the output is the data ready for analysis. The server verifies the data format and checks for any invalid data. After confirming the data is clean, it is passed to the analysis module.

[0187] Step 4:

[0188] The server analyzes text data using natural language processing techniques. The input is text data ready for analysis, and the output is the result of extracting contextual and emotional features. The server utilizes a generative AI model to analyze linguistic features and score aggression and negative emotions. This includes contextual analysis, keyword extraction, and emotional tone analysis.

[0189] Step 5:

[0190] The server uses speech recognition technology to convert audio data into text, and then performs natural language processing. The input is audio data ready for analysis, and the output is the text analysis result with emotions and context extracted. Speech recognition accurately transcribes speech into text, followed by emotion analysis. This process utilizes the tone and speed of the user's voice to identify emotions.

[0191] Step 6:

[0192] The server scores the likelihood of an action and determines the user's emotional state based on the analysis of text and audio. The input is the analyzed data, and the output is feedback data that includes improvement suggestions to notify the user. The server applies a scoring algorithm to identify the likelihood of an incident while generating improvement suggestions based on the detected emotional state.

[0193] Step 7:

[0194] The server sends the generated feedback data to the terminal and notifies the user. The input is the feedback data, and the output is the notification message displayed on the terminal. The server sends the feedback, ready for transmission, to the terminal and formats the feedback message appropriately so that it can be displayed as a notification on the user's screen. This allows the user to immediately reflect on their actions and make improvements.

[0195] (Application Example 2)

[0196] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0197] In modern society, misunderstandings and harassment due to communication are major problems within organizations and families. In particular, misunderstandings in family conversations can cause significant stress and discord. This can hinder constructive dialogue between parents and children, and within families. Therefore, it is necessary to mitigate these problems and provide a safe and healthy communication environment.

[0198] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0199] In this invention, the server includes means for monitoring communication data acquired from users, means for transmitting the communication data to a data analysis device, and means for the data analysis device to analyze the communication data and score the likelihood of harassment. This enables the analysis of conversations between family members in real time and, if necessary, provides improvement suggestions that take emotions into consideration, thereby reducing stress and discord within the family and realizing a better communication environment.

[0200] "Means for monitoring communication data obtained from users" refers to devices or methods for continuously detecting and recording text messages and voice data sent or received by users.

[0201] "Means of transmitting to a data analysis device" refers to the technology for immediately or periodically transferring monitored communication data to a data analysis device, which is usually done via a network.

[0202] "Methods for analyzing communication data and scoring the likelihood of harassment" refer to devices and methods that use natural language processing technology and speech recognition technology based on acquired data to quantify and evaluate the likelihood of harassing behavior.

[0203] "Means for issuing harassment notifications to users based on the scoring results and emotional state" refers to devices or methods for providing warnings and suggestions for improvement in real time, taking into account the user's emotional state based on the scoring results obtained through analysis.

[0204] "A means installed in a home information processing device to analyze conversations between family members and provide appropriate improvement suggestions" refers to a device or method for continuously monitoring voice and text conversations between family members in a home environment, performing sentiment analysis, and automatically communicating improvement suggestions to promote calm dialogue.

[0205] The invention will now be described in terms of its embodiments. This system monitors user communication data in real time and provides analysis and notifications as needed. It basically includes a home information processing device and a data analysis device, and promotes good communication within the family.

[0206] First, the device uses sensors such as microphones and cameras to acquire the user's text messages and voice data. This data is sent to the server in real time. Upon receiving the communication, the server uses the natural language processing library NLTK and the Google Speech Recognition API to analyze the data. In particular, the voice data is converted into text data by Google's API, and harassment scoring and sentiment analysis are performed on that text data.

[0207] The data analysis device evaluates the likelihood of harassment based on the scoring results and the user's emotional state, and issues notifications to the user as needed. These notifications are emotionally sensitive and include guidance on smooth communication within the family. For example, it might suggest to parents regarding how to deal with their children, "It would be good to have time for your child to relax and talk." Based on these improvement suggestions, the server reduces stress and discomfort within the family and provides a sophisticated and healthy communication environment.

[0208] As a concrete example, when the server analyzes conversations between parents and children, it recommends that "in situations where parents are being harsh, they should speak in a calm tone to create a safe environment for the child." An example of a prompt to generate this improvement suggestion would be, "Explain how to automatically detect negative emotions and harassment occurring within the home and provide appropriate advice." By implementing this invention, the quality of communication in the home environment is improved.

[0209] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0210] Step 1:

[0211] The device uses a microphone and camera to capture user text messages and voice data in real time. The captured data is sent to a server via the network. The input is voice and text data, and the output is the transmission of this data to the server. As the data is transmitted, it is encrypted and transmitted in a secure state.

[0212] Step 2:

[0213] The server converts the received audio data into text data using the Google Speech Recognition API. This conversion process analyzes the input audio data and converts it into text as output. Specifically, it recognizes the frequency components of the speech and outputs the text that best matches them.

[0214] Step 3:

[0215] The server performs natural language processing using NLTK on the text data. Here, the input is the text data obtained in step 2, and the output is numerical data indicating the sentiment score and degree of aggression of the words spoken by the user. Specifically, it analyzes the context of the text and the word choices, and identifies positive or negative emotions using the sentiment engine.

[0216] Step 4:

[0217] The server generates appropriate notifications for the user based on the scoring results and sentiment analysis results. The inputs used are the scores and sentiment state data generated in step 3, and the output is a notification that includes sentiment-sensitive alerts and improvement suggestions. Specifically, it generates positive feedback and advice to help the user take appropriate action.

[0218] Step 5:

[0219] The device communicates notifications from the server to the user via display or audio. Input is emotion-based notifications from the server, and output is a visual or auditory display of the notification on the user's device. Specifically, the user's attention is drawn by displaying a message on the screen or playing an audio message through the speaker.

[0220] Step 6:

[0221] Based on the notifications and advice provided, users reflect on their own emotions and actions. Here, the input is the notification information received via display or audio, and the output is the user's decision to change their behavior or make improvements. Specifically, it serves as a catalyst for users to consider suggestions and engage in better communication.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] This invention is a system that analyzes communication data exchanged by users in real time and detects the possibility of harassment. This system can be implemented by integrating it with appropriate communication tools and following the process outlined below.

[0239] First, the terminal acquires text messages and voice data sent and received by the user in real time. This data is managed by a data acquisition module and sent to the server as needed. Data transmission is carried out using secure communication protocols and with consideration for data protection.

[0240] The server analyzes received text messages using natural language processing techniques to understand their meaning and context. Similarly, audio data is converted to text using speech recognition technology and analyzed in the same manner. The analysis results identify harassment-related keywords and expressions, and a score is assigned based on these. This score indicates an evaluation for each category—sexual harassment, power harassment, and moral harassment—and helps detect unintentional harassment by the user.

[0241] If the score exceeds a certain threshold, the server immediately issues a notification to the device. The device informs the user of the potential for harassment and suggests ways to improve. This helps the user become aware of the impact of their words and encourages them to improve their communication.

[0242] Furthermore, the server aggregates data at regular intervals and analyzes users' communication trends. Based on the analysis results, it generates a report and sends it to the terminal. This report includes information such as the frequency of harassment occurrences and areas for improvement, helping users to make long-term improvements.

[0243] As a concrete example, consider a case where a user says, "You failed again? Come on, give me a break," during an online meeting. This statement is captured by the device and analyzed as text on the server. In this case, a harassment score is calculated based on the wording and context, and if it exceeds the threshold, a warning is immediately displayed on the device. This notification also includes suggestions for improvement, such as, "Try to switch to positive feedback and encourage the other person." Based on this information, the user can review their statements and be motivated to improve their communication with others.

[0244] Through such a system, it is possible to prevent harassment throughout the entire organization and promote safe and healthy communication.

[0245] The following describes the processing flow.

[0246] Step 1:

[0247] The device acquires text messages and voice data sent by the user using communication tools in real time.

[0248] Step 2:

[0249] The terminal performs initial processing on the acquired data and sends it to the server using a secure communication protocol. The voice data is sent in its original form and then analyzed on the server.

[0250] Step 3:

[0251] The server registers the received data in a parsing queue and begins processing it sequentially. Text messages are immediately parsed by the natural language processing module.

[0252] Step 4:

[0253] The server processes the audio data through a speech recognition engine to convert it into text. Then, it is analyzed using a natural language processing module, similar to how text messages are processed.

[0254] Step 5:

[0255] The server uses a machine learning model to score the likelihood of harassment based on the data analysis results. In this process, it analyzes specific keywords and contexts to numerically evaluate the likelihood of harassment.

[0256] Step 6:

[0257] If the score exceeds a predetermined threshold, the server generates an alert regarding that specific message.

[0258] Step 7:

[0259] The server sends the generated alert to the terminal. The terminal displays a notification to the user, providing information about the potential harassment and suggestions for improvement.

[0260] Step 8:

[0261] The server analyzes the accumulated data at regular intervals and generates a report summarizing communication trends.

[0262] Step 9:

[0263] The server sends the generated report to the terminal, and the user can review its contents to understand areas for long-term improvement.

[0264] (Example 1)

[0265] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0266] In today's communication environment, harassment can occur unconsciously, negatively impacting individuals and organizations. However, technology to monitor this in real time and propose effective solutions still does not exist. Addressing this problem is urgently needed.

[0267] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0268] In this invention, the server includes means for monitoring communication information acquired from a user, means for transmitting the communication information to a data processing device, and means for the data processing device to analyze the communication information and quantify the possibility of harassment. This makes it possible to detect harassment in real time and make appropriate improvement suggestions to the user, thereby improving the communication environment of individuals and organizations.

[0269] A "user" is an individual or group that uses the system to send and receive communication information.

[0270] "Communication information" refers to all information exchanged between users, including text messages and voice data.

[0271] "Monitoring methods" refer to processes and devices that collect communication information obtained from users in real time and check for any abnormalities.

[0272] A "data processing device" refers to hardware or software used to analyze collected communication information.

[0273] "Means of analysis" refers to the process of evaluating the content of communication information using natural language processing technology and speech recognition technology to detect specific patterns and keywords.

[0274] "Methods of quantification" refers to the process of evaluating the possibility of harassment based on the analysis results according to certain criteria and expressing it as a numerical value.

[0275] A "warning" refers to a message that notifies a user of potential harassment and encourages them to be aware of it.

[0276] "Improvement suggestions" refer to information that guides users on specific measures or changes in wording that they should take immediately to avoid harassment.

[0277] "Data in different formats" refers to information in various formats, such as text data and audio data.

[0278] "Detailed analysis" refers to a process that goes beyond simple keyword detection and includes comprehensive data evaluation, encompassing contextual and sentiment analysis.

[0279] This invention is a system that analyzes communication information exchanged between users in real time and evaluates the possibility of harassment. The system is implemented in the following way.

[0280] First, the terminal obtains in real-time the text messages and voice data that the user sends and receives. In this process, the built-in microphone and keyboard input of the terminal are used, and the data acquisition module manages this. The terminal sends the acquired communication information to the server using a secure communication protocol (e.g., TLS). This ensures the security of the data.

[0281] The server processes the received communication information. In particular, it analyzes the text data using natural language processing technology. For this, open-source natural language processing libraries (e.g., SpaCy, NLTK) are used. Also, the voice data is converted into text using voice recognition technology (e.g., Google Speech-to-Text API) and analyzed in the same way.

[0282] From the analysis results, the server identifies the specified harassment-related keywords and expressions and performs quantification based on this. This quantification is scored based on a specific algorithm, and an evaluation is made for each harassment category. If the score exceeds the set threshold, the server immediately notifies the terminal.

[0283] The terminal displays a warning to the user that there may be harassment. This warning also includes specific improvement suggestions on how the user should improve their speech. For example, advice such as "Switch to positive feedback and try to encourage the other person" is presented.

[0284] Furthermore, the server aggregates the data at regular intervals and analyzes the communication trends of the user. Based on this analysis, a detailed report for the user is generated and sent to the terminal. This report contains information useful for improving communication.

[0285] As a specific example, consider the message "Did you fail again? I wish you would be more serious" that a user spoke during an online meeting. This utterance is captured by the terminal and undergoes text conversion and analysis on the server. If a problematic keyword is found in this scenario, the likelihood of power harassment is scored, and a warning is displayed if necessary.

[0286] As an example of a prompt sentence for the generative AI model, "Please analyze the following text in real time and score the likelihood of harassment: 'Did you fail again? I wish you would be more serious.'" can be cited. Using this prompt, the AI model analyzes the specified text and returns an appropriate score.

[0287] The flow of the specific process in Example 1 will be described using FIG. 11.

[0288] Step 1:

[0289] The terminal acquires in real time the text messages and voice data sent and received by the user. The input is the message or call information from the user, and the output is the raw communication information obtained. The terminal manages this using a data acquisition module and securely stores the data using encryption technology.

[0290] Step 2:

[0291] The terminal prepares to send the acquired communication information to the server. The input is the communication information collected in Step 1, and the output is the data formatted and encrypted for transmission. The terminal sends the data to the server using a security protocol such as Secure Sockets Layer (SSL).

[0292] Step 3:

[0293] The server receives data sent from the terminal. The input is encrypted communication information, and the output is decrypted data. The server verifies the integrity and validity of the received data and securely stores it in the database.

[0294] Step 4:

[0295] The server analyzes text messages using natural language processing techniques. The input is decoded text data, and the output is contextual understanding and keyword extraction information resulting from the analysis. The server uses open-source natural language processing libraries to perform this analysis and records identified keywords and context in a specific format.

[0296] Step 5:

[0297] The server uses speech recognition technology to convert audio data into text and then performs analysis. The input is decoded audio information, and the output is the text data and its analysis results. The server uses a speech recognition API to convert audio to text and then performs analysis.

[0298] Step 6:

[0299] The server quantifies the likelihood of harassment based on the analysis results. The input is the analysis results of text data, and the output is the harassment score for each category. The server uses a scoring algorithm to calculate this score and stores it in a database.

[0300] Step 7:

[0301] The server sends a notification to the device if the score exceeds a threshold. The input is the harassment score, and the output is a warning message sent to the user. The server includes improvement suggestions in the notification and relays them to the device.

[0302] Step 8:

[0303] The terminal displays the received notifications to the user. The input is a warning message from the server, and the output is an alert displayed on the user interface. The terminal enables the user to view the notifications and provides a feedback system as needed.

[0304] Step 9:

[0305] The server analyzes the data collected at regular intervals and generates a report indicating the user's communication trends. The input is the aggregated communication data, and the output is a detailed report. The server uses a database management system to evaluate the data and periodically sends the report to the terminal.

[0306] (Application Example 1)

[0307] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0308] In modern society, communication within the family often does not proceed smoothly, which sometimes leads to an increase in stress within the family. In particular, the spread of unconscious or intentional annoying behavior through words within the family has an adverse impact on the family environment. An effective means to prevent this problem and maintain healthy communication within the family is required.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0310] In this invention, the server includes means for monitoring information acquired from the user, means for transmitting the information to an analysis device, and means for analyzing the information and evaluating the possibility of annoying behavior. Thereby, it becomes possible to improve the quality of communication within the family, reduce stress among family members, and promote better conversations.

[0311] "Information obtained from users" refers to communication-related information such as text data and audio data generated or transmitted by users.

[0312] "Monitoring methods" refer to functions that continuously check information obtained from users and collect necessary data.

[0313] "Means for transmitting to an analytical device" refers to the function of appropriately transmitting data to a data analysis device in order to process the acquired information.

[0314] "Means for analyzing and evaluating the possibility of harassment" refers to a process for analyzing received information and evaluating whether a particular action constitutes harassment.

[0315] "Means of issuing notifications" refers to a function that provides users with warnings and suggestions for improvement based on evaluation results.

[0316] "Means for including suggestions for improvement" refers to a function that provides users with specific suggestions to encourage better communication.

[0317] "A means of monitoring conversations in real time and guiding dialogue in a better direction" refers to a function that instantly analyzes conversations between users and provides feedback to maintain smooth communication.

[0318] To implement this invention, it is necessary to implement a dedicated system as a program and run it on appropriate hardware. A specific example of this is an information analysis system used in a home environment.

[0319] The server uses the Google Cloud Speech-to-Text API to convert speech data received from users into text data in real time. This process makes the speech data easily parseable. The converted text data is sent to an analysis device, where text analysis is performed using the natural language processing libraries NLTK or spaCy.

[0320] Based on these analysis results, the device provides feedback to the user. This feedback includes notifications if potential harassment is detected and specific suggestions for improving communication. Furthermore, the device monitors family conversations in real time and provides feedback to facilitate better communication.

[0321] For example, if a family is having a conversation in the living room and an argument breaks out between the children, the device will analyze the conversation and immediately provide advice such as, "Let's try to respect each other's opinions and have a more constructive discussion."

[0322] An example of a prompt to input into the generative AI model would be, "Analyze aggressive or negative remarks heard in the home and consider how to address them each time." This would allow the system to provide appropriate feedback to support healthy communication within the home.

[0323] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0324] Step 1:

[0325] The server acquires voice data from the user. This voice data is collected in real time from conversations within the home. The input is received as voice data through the microphone, and the output becomes an audio file for analysis.

[0326] Step 2:

[0327] The server uses the Google Cloud Speech-to-Text API to convert the acquired audio data into text data. Speech recognition technology is applied to the audio file as input. The output is data in a parseable text format.

[0328] Step 3:

[0329] The server analyzes the text data using a natural language processing library (NLTK or spaCy) to identify expressions that may constitute harassment. This step involves analyzing the text data as input for harassment-related keywords and context. The output is identification information for the relevant sections in the text.

[0330] Step 4:

[0331] The server evaluates the analysis results and generates alerts as needed. If the identified harassment exceeds a certain threshold, it sends a notification to the terminal. The input is the analysis results and evaluation criteria, and the output is a warning notification.

[0332] Step 5:

[0333] The terminal presents notifications to the user based on the generated alerts and provides necessary improvement suggestions. In this step, the alerts and suggestions sent from the server are used as input, and the user is notified visually or audibly. The output is a warning message and improvement suggestions for the user.

[0334] Step 6:

[0335] The user attempts to improve the conversation based on the suggestions presented. This step involves the user, upon receiving a notification from their device, deciding how to modify their behavior to improve the quality of communication. Specifically, this requires choosing more assertive or gentle language.

[0336] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0337] This invention is a system that not only analyzes communication data to detect the possibility of harassment, but also analyzes the user's emotions simultaneously, enabling more adaptive improvement suggestions. The basic configuration for implementing this system is as follows.

[0338] First, the terminal acquires text messages and voice data exchanged by users using communication tools in real time. This acquired data is promptly sent to the server. Upon receiving this data, the server performs analysis using natural language processing and speech recognition technologies.

[0339] This system further incorporates an emotion engine into the data analysis device to identify the user's emotional state from text messages and voice data. Emotion identification is based on the context of the user's text messages, as well as their tone and speed of voice.

[0340] After the server scores the harassment behavior, it further considers this emotional data. Based on the scoring results and emotional state, it immediately sends a notification to the user. The device receives this notification and displays it to the user. The notification includes improvement suggestions based on the emotional state and the harassment behavior score. These suggestions are tailored to the user's current emotions and provide more effective feedback.

[0341] For example, if a user sends the message, "You never understand anything," the device retrieves this message as data and sends it to the server. The server uses natural language processing to evaluate the aggression of the message and scores its potential for harassment. Simultaneously, an emotion engine recognizes the user's feelings of frustration and irritation. As a result, the server sends an emotion-sensitive notification to the device, such as, "This statement may be harassment. Please calm down and let's work together to find a concrete solution." Upon receiving this notification, the user can reflect on their own emotions and be motivated to choose a better way of communicating.

[0342] Thus, this invention prevents harassment by analyzing user emotions and encouraging more appropriate behavior. As a result, it is possible to promote safe and constructive communication throughout the organization.

[0343] The following describes the processing flow.

[0344] Step 1:

[0345] The device acquires text messages and voice data sent by the user using communication tools in real time.

[0346] Step 2:

[0347] The terminal sends the acquired data to the server using a secure protocol. Voice data is sent to the server as is, but it is not converted to text.

[0348] Step 3:

[0349] The server adds the received data to a parsing queue, and the text messages are analyzed using a natural language processing module. Similarly, audio data is converted to text using a speech recognition engine and then analyzed.

[0350] Step 4:

[0351] The server inputs text data and analyzed audio data into the emotion engine to identify the user's emotional state. This emotion analysis is based on the tone of the text and keywords that suggest emotion.

[0352] Step 5:

[0353] The server integrates emotional information and text analysis results to score the likelihood of harassment. In addition to the likelihood of harassment, the emotional state is also considered in the scoring.

[0354] Step 6:

[0355] The server creates an alert to notify the user if the score exceeds a predetermined threshold.

[0356] Step 7:

[0357] The server sends a notification to the device that includes improvement suggestions tailored to the user's emotional state.

[0358] Step 8:

[0359] The terminal displays notifications received from the server to the user, offering suggestions for improvement that are sensitive to the user's feelings, along with information about the possibility of harassment.

[0360] Step 9:

[0361] The server analyzes the accumulated data and generates reports, including individual user communication trends, at regular intervals, sending them to the user's device. Users can then use these reports to implement long-term behavioral changes.

[0362] (Example 2)

[0363] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0364] In today's information and communication environment, while communication between users is increasing, the risk of unintentional harassment is also rising. Such behavior can lead to a deterioration of interpersonal relationships and cause serious problems for organizations and individuals. However, users themselves are often unaware of it, so there is a need for technology that can efficiently and automatically detect harassment and provide appropriate feedback.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0366] In this invention, the server includes means for monitoring information acquired from the user, means for transmitting the information to a processing device, and means for the processing device to analyze the information and score the likelihood of the behavior. This makes it possible to analyze behavior in real time, quickly detect potential harassment behavior, and provide the user with appropriate improvement suggestions.

[0367] A "user" refers to an individual or group that uses the system to communicate.

[0368] "Information" refers to digital data, including text messages and audio data, that users generate or receive.

[0369] "Means of monitoring" refers to functions for acquiring information in real time and continuously observing its content and characteristics.

[0370] A "processing unit" refers to a computer or a part thereof that analyzes received information and performs necessary calculations and decisions.

[0371] "Means of analysis" refers to technical means used to break down information using natural language processing and speech recognition technologies and derive conclusions that align with a specific purpose.

[0372] "Means for scoring the likelihood of an action" refers to a function that quantifies or evaluates the degree of a specific action or impact based on the analysis results.

[0373] "Notifications" refer to messages generated based on analysis and scoring results that convey information to the user.

[0374] "Adaptive improvement suggestions" refer to suggestions that include specific and effective advice on behavioral improvements tailored to the user's current situation and emotional state.

[0375] This invention is a system for detecting potential harassment by acquiring information exchanged by users through communication tools in real time. The system includes a terminal used by the user, a server for analyzing the data, and a function to notify the user of improvement suggestions based on the analysis results.

[0376] First, the device continuously acquires text messages and voice data sent and received by the user. Specific target tools include instant messaging software and collaboration platforms. This acquired information is then transmitted to a server via a secure protocol.

[0377] Next, the server receives this information and performs analysis using natural language processing and speech recognition technologies. The technologies used here include generative AI models; for example, open-source libraries and commercial APIs are used for natural language processing, and APIs from speech technology providers are used for speech recognition. Based on this, the server evaluates the aggressiveness and negative tone of the message and scores the likelihood of the action based on the results.

[0378] Furthermore, the server uses an emotion engine to identify the user's emotional state from the information. This involves analyzing text and voice features to quantify the user's emotions. For example, if the user is feeling frustrated or stressed, this will be reflected in the analysis results and influence the final feedback.

[0379] The server generates notifications for the user based on the scoring results and emotional state. The notifications include potential harassment behavior and specific suggestions for improvement. The suggestions are tailored to the user's emotional state, aiming to provide more effective feedback. For example, if the message "You never understand anything" is analyzed, a notification such as "This statement may be considered harassment. Let's calm down and work together to find a concrete solution" will be sent.

[0380] An example of a prompt might be: "If you receive the following message, how should you interpret it and determine your emotional state? Also, generate suggestions for behavioral improvement. Message: 'You never understand anything.'"

[0381] In this way, the system can provide constructive feedback to users and prevent harassment from occurring.

[0382] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0383] Step 1:

[0384] The terminal acquires text messages and voice data sent and received by the user using communication tools in real time. The input is information sent by the user, and the output is the acquired digital data. This data is temporarily stored on the terminal and then prepared to be sent to the server. The terminal checks the data format, encodes it as needed, and converts it into a format suitable for transmission.

[0385] Step 2:

[0386] The terminal sends acquired text messages and voice data to the server. Input is digital data temporarily stored within the terminal, and output is data sent to the server. The terminal ensures the confidentiality and integrity of the data by sending it using a secure protocol (e.g., HTTPS). Error checking is also performed during the transmission process to confirm that the data was sent correctly.

[0387] Step 3:

[0388] The server receives data sent from the terminal and prepares it for analysis. The input consists of text and audio data sent from the terminal, and the output is the data ready for analysis. The server verifies the data format and checks for any invalid data. After confirming the data is clean, it is passed to the analysis module.

[0389] Step 4:

[0390] The server analyzes text data using natural language processing techniques. The input is text data ready for analysis, and the output is the result of extracting contextual and emotional features. The server utilizes a generative AI model to analyze linguistic features and score aggression and negative emotions. This includes contextual analysis, keyword extraction, and emotional tone analysis.

[0391] Step 5:

[0392] The server uses speech recognition technology to convert audio data into text, and then performs natural language processing. The input is audio data ready for analysis, and the output is the text analysis result with emotions and context extracted. Speech recognition accurately transcribes speech into text, followed by emotion analysis. This process utilizes the tone and speed of the user's voice to identify emotions.

[0393] Step 6:

[0394] The server scores the likelihood of an action and determines the user's emotional state based on the analysis of text and audio. The input is the analyzed data, and the output is feedback data that includes improvement suggestions to notify the user. The server applies a scoring algorithm to identify the likelihood of an incident while generating improvement suggestions based on the detected emotional state.

[0395] Step 7:

[0396] The server sends the generated feedback data to the terminal and notifies the user. The input is the feedback data, and the output is the notification message displayed on the terminal. The server sends the feedback, ready for transmission, to the terminal and formats the feedback message appropriately so that it can be displayed as a notification on the user's screen. This allows the user to immediately reflect on their actions and make improvements.

[0397] (Application Example 2)

[0398] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0399] In modern society, misunderstandings and harassment due to communication are major problems within organizations and families. In particular, misunderstandings in family conversations can cause significant stress and discord. This can hinder constructive dialogue between parents and children, and within families. Therefore, it is necessary to mitigate these problems and provide a safe and healthy communication environment.

[0400] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0401] In this invention, the server includes means for monitoring communication data acquired from users, means for transmitting the communication data to a data analysis device, and means for the data analysis device to analyze the communication data and score the likelihood of harassment. This enables the analysis of conversations between family members in real time and, if necessary, provides improvement suggestions that take emotions into consideration, thereby reducing stress and discord within the family and realizing a better communication environment.

[0402] "Means for monitoring communication data obtained from users" refers to devices or methods for continuously detecting and recording text messages and voice data sent or received by users.

[0403] "Means of transmitting to a data analysis device" refers to the technology for immediately or periodically transferring monitored communication data to a data analysis device, which is usually done via a network.

[0404] "Methods for analyzing communication data and scoring the likelihood of harassment" refer to devices and methods that use natural language processing technology and speech recognition technology based on acquired data to quantify and evaluate the likelihood of harassing behavior.

[0405] "Means for issuing harassment notifications to users based on the scoring results and emotional state" refers to devices or methods for providing warnings and suggestions for improvement in real time, taking into account the user's emotional state based on the scoring results obtained through analysis.

[0406] "A means installed in a home information processing device to analyze conversations between family members and provide appropriate improvement suggestions" refers to a device or method for continuously monitoring voice and text conversations between family members in a home environment, performing sentiment analysis, and automatically communicating improvement suggestions to promote calm dialogue.

[0407] The invention will now be described in terms of its embodiments. This system monitors user communication data in real time and provides analysis and notifications as needed. It basically includes a home information processing device and a data analysis device, and promotes good communication within the family.

[0408] First, the device uses sensors such as microphones and cameras to acquire the user's text messages and voice data. This data is sent to the server in real time. Upon receiving the communication, the server uses the natural language processing library NLTK and the Google Speech Recognition API to analyze the data. In particular, the voice data is converted into text data by Google's API, and harassment scoring and sentiment analysis are performed on that text data.

[0409] The data analysis device evaluates the likelihood of harassment based on the scoring results and the user's emotional state, and issues notifications to the user as needed. These notifications are emotionally sensitive and include guidance on smooth communication within the family. For example, it might suggest to parents regarding how to deal with their children, "It would be good to have time for your child to relax and talk." Based on these improvement suggestions, the server reduces stress and discomfort within the family and provides a sophisticated and healthy communication environment.

[0410] As a concrete example, when the server analyzes conversations between parents and children, it recommends that "in situations where parents are being harsh, they should speak in a calm tone to create a safe environment for the child." An example of a prompt to generate this improvement suggestion would be, "Explain how to automatically detect negative emotions and harassment occurring within the home and provide appropriate advice." By implementing this invention, the quality of communication in the home environment is improved.

[0411] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0412] Step 1:

[0413] The device uses a microphone and camera to capture user text messages and voice data in real time. The captured data is sent to a server via the network. The input is voice and text data, and the output is the transmission of this data to the server. As the data is transmitted, it is encrypted and transmitted in a secure state.

[0414] Step 2:

[0415] The server converts the received audio data into text data using the Google Speech Recognition API. This conversion process analyzes the input audio data and converts it into text as output. Specifically, it recognizes the frequency components of the speech and outputs the text that best matches them.

[0416] Step 3:

[0417] The server performs natural language processing using NLTK on the text data. Here, the input is the text data obtained in step 2, and the output is numerical data indicating the sentiment score and degree of aggression of the words spoken by the user. Specifically, it analyzes the context of the text and the word choices, and identifies positive or negative emotions using the sentiment engine.

[0418] Step 4:

[0419] The server generates appropriate notifications for the user based on the scoring results and sentiment analysis results. The inputs used are the scores and sentiment state data generated in step 3, and the output is a notification that includes sentiment-sensitive alerts and improvement suggestions. Specifically, it generates positive feedback and advice to help the user take appropriate action.

[0420] Step 5:

[0421] The device communicates notifications from the server to the user via display or audio. Input is emotion-based notifications from the server, and output is a visual or auditory display of the notification on the user's device. Specifically, the user's attention is drawn by displaying a message on the screen or playing an audio message through the speaker.

[0422] Step 6:

[0423] Based on the notifications and advice provided, users reflect on their own emotions and actions. Here, the input is the notification information received via display or audio, and the output is the user's decision to change their behavior or make improvements. Specifically, it serves as a catalyst for users to consider suggestions and engage in better communication.

[0424] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0425] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0426] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0427] [Third Embodiment]

[0428] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0429] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0430] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0431] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0432] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0433] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0434] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0435] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0436] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0437] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0438] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0439] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0440] This invention is a system that analyzes communication data exchanged by users in real time and detects the possibility of harassment. This system can be implemented by integrating it with appropriate communication tools and following the process outlined below.

[0441] First, the terminal acquires text messages and voice data sent and received by the user in real time. This data is managed by a data acquisition module and sent to the server as needed. Data transmission is carried out using secure communication protocols and with consideration for data protection.

[0442] The server analyzes received text messages using natural language processing techniques to understand their meaning and context. Similarly, audio data is converted to text using speech recognition technology and analyzed in the same manner. The analysis results identify harassment-related keywords and expressions, and a score is assigned based on these. This score indicates an evaluation for each category—sexual harassment, power harassment, and moral harassment—and helps detect unintentional harassment by the user.

[0443] If the score exceeds a certain threshold, the server immediately issues a notification to the device. The device informs the user of the potential for harassment and suggests ways to improve. This helps the user become aware of the impact of their words and encourages them to improve their communication.

[0444] Furthermore, the server aggregates data at regular intervals and analyzes users' communication trends. Based on the analysis results, it generates a report and sends it to the terminal. This report includes information such as the frequency of harassment occurrences and areas for improvement, helping users to make long-term improvements.

[0445] As a concrete example, consider a case where a user says, "You failed again? Come on, give me a break," during an online meeting. This statement is captured by the device and analyzed as text on the server. In this case, a harassment score is calculated based on the wording and context, and if it exceeds the threshold, a warning is immediately displayed on the device. This notification also includes suggestions for improvement, such as, "Try to switch to positive feedback and encourage the other person." Based on this information, the user can review their statements and be motivated to improve their communication with others.

[0446] Through such a system, it is possible to prevent harassment throughout the entire organization and promote safe and healthy communication.

[0447] The following describes the processing flow.

[0448] Step 1:

[0449] The device acquires text messages and voice data sent by the user using communication tools in real time.

[0450] Step 2:

[0451] The terminal performs initial processing on the acquired data and sends it to the server using a secure communication protocol. The voice data is sent in its original form and then analyzed on the server.

[0452] Step 3:

[0453] The server registers the received data in a parsing queue and begins processing it sequentially. Text messages are immediately parsed by the natural language processing module.

[0454] Step 4:

[0455] The server processes the audio data through a speech recognition engine to convert it into text. Then, it is analyzed using a natural language processing module, similar to how text messages are processed.

[0456] Step 5:

[0457] The server uses a machine learning model to score the likelihood of harassment based on the data analysis results. In this process, it analyzes specific keywords and contexts to numerically evaluate the likelihood of harassment.

[0458] Step 6:

[0459] If the score exceeds a predetermined threshold, the server generates an alert regarding that specific message.

[0460] Step 7:

[0461] The server sends the generated alert to the terminal. The terminal displays a notification to the user, providing information about the potential harassment and suggestions for improvement.

[0462] Step 8:

[0463] The server analyzes the accumulated data at regular intervals and generates a report summarizing communication trends.

[0464] Step 9:

[0465] The server sends the generated report to the terminal, and the user can review its contents to understand areas for long-term improvement.

[0466] (Example 1)

[0467] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0468] In today's communication environment, harassment can occur unconsciously, negatively impacting individuals and organizations. However, technology to monitor this in real time and propose effective solutions still does not exist. Addressing this problem is urgently needed.

[0469] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0470] In this invention, the server includes means for monitoring communication information acquired from a user, means for transmitting the communication information to a data processing device, and means for the data processing device to analyze the communication information and quantify the possibility of harassment. This makes it possible to detect harassment in real time and make appropriate improvement suggestions to the user, thereby improving the communication environment of individuals and organizations.

[0471] A "user" is an individual or group that uses the system to send and receive communication information.

[0472] "Communication information" refers to all information exchanged between users, including text messages and voice data.

[0473] "Monitoring methods" refer to processes and devices that collect communication information obtained from users in real time and check for any abnormalities.

[0474] A "data processing device" refers to hardware or software used to analyze collected communication information.

[0475] "Means of analysis" refers to the process of evaluating the content of communication information using natural language processing technology and speech recognition technology to detect specific patterns and keywords.

[0476] "Methods of quantification" refers to the process of evaluating the possibility of harassment based on the analysis results according to certain criteria and expressing it as a numerical value.

[0477] A "warning" refers to a message that notifies a user of potential harassment and encourages them to be aware of it.

[0478] "Improvement suggestions" refer to information that guides users on specific measures or changes in wording that they should take immediately to avoid harassment.

[0479] "Data in different formats" refers to information in various formats, such as text data and audio data.

[0480] "Detailed analysis" refers to a process that goes beyond simple keyword detection and includes comprehensive data evaluation, encompassing contextual and sentiment analysis.

[0481] This invention is a system that analyzes communication information exchanged between users in real time and evaluates the possibility of harassment. The system is implemented in the following way.

[0482] First, the terminal acquires text messages and voice data sent and received by the user in real time. This process utilizes the terminal's built-in microphone and keyboard input, and is managed by a data acquisition module. The terminal then sends the acquired communication information to the server using a secure communication protocol (e.g., TLS). This ensures the security of the data.

[0483] The server processes the received communication information. In particular, it analyzes text data using natural language processing techniques. This is done using open-source natural language processing libraries (e.g., SpaCy, NLTK). Audio data is converted to text using speech recognition technology (e.g., Google Speech-to-Text API) and analyzed in the same way.

[0484] Based on the analysis results, the server identifies specified harassment-related keywords and expressions and assigns numerical values ​​to them. This numerical value is scored based on a specific algorithm, and an evaluation is performed for each harassment category. If the score exceeds a set threshold, the server immediately notifies the terminal.

[0485] The device displays a warning to the user that their comments may be harassing. This warning also includes specific suggestions on how the user can improve their behavior. For example, it might advise, "Try to switch to positive feedback and encourage the other person."

[0486] Furthermore, the server aggregates data at regular intervals and analyzes user communication trends. Based on this analysis, a detailed report is generated for the user and sent to their terminal. This report contains information that can help improve communication.

[0487] As a concrete example, consider the message "Did you fail again? Enough is enough." spoken by a user during an online meeting. This message is captured on the device and transcribed and analyzed on the server. If problematic keywords are found, the potential for harassment is scored, and a warning is displayed if necessary.

[0488] An example of a prompt for a generative AI model is: "Analyze the following text in real time and score its likelihood of harassment: 'Did you fail again? Come on, give it a rest.'" Using this prompt, the AI ​​model analyzes the specified text and returns an appropriate score.

[0489] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0490] Step 1:

[0491] The terminal acquires text messages and voice data sent and received by the user in real time. The input is messages and call information from the user, and the output is the acquired raw communication information. The terminal manages this data using a data acquisition module and securely stores the data using encryption technology.

[0492] Step 2:

[0493] The terminal prepares to send the acquired communication information to the server. The input is the communication information collected in step 1, and the output is the data that has been formatted and encrypted for transmission. The terminal sends the data to the server using a security protocol such as Secure Sockets Layer (SSL).

[0494] Step 3:

[0495] The server receives data sent from the terminal. The input is encrypted communication information, and the output is decrypted data. The server verifies the integrity and validity of the received data and securely stores it in the database.

[0496] Step 4:

[0497] The server analyzes text messages using natural language processing techniques. The input is decoded text data, and the output is contextual understanding and keyword extraction information resulting from the analysis. The server uses open-source natural language processing libraries to perform this analysis and records identified keywords and context in a specific format.

[0498] Step 5:

[0499] The server uses speech recognition technology to convert audio data into text and then performs analysis. The input is decoded audio information, and the output is the text data and its analysis results. The server uses a speech recognition API to convert audio to text and then performs analysis.

[0500] Step 6:

[0501] The server quantifies the likelihood of harassment based on the analysis results. The input is the analysis results of text data, and the output is the harassment score for each category. The server uses a scoring algorithm to calculate this score and stores it in a database.

[0502] Step 7:

[0503] The server sends a notification to the device if the score exceeds a threshold. The input is the harassment score, and the output is a warning message sent to the user. The server includes improvement suggestions in the notification and relays them to the device.

[0504] Step 8:

[0505] The terminal displays received notifications to the user. Input is a warning message from the server, and output is an alert displayed on the user interface. The terminal allows the user to review the notifications and provides a feedback system as needed.

[0506] Step 9:

[0507] The server analyzes data collected at regular intervals and generates reports showing user communication trends. The input is aggregated communication data, and the output is a detailed report. The server uses a database management system to evaluate the data and periodically sends reports to the terminals.

[0508] (Application Example 1)

[0509] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0510] In modern society, communication within families often breaks down, contributing to increased stress within households. In particular, the prevalence of unconscious or intentional verbal harassment within families negatively impacts the home environment. Effective measures are needed to prevent this problem and maintain healthy communication within families.

[0511] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0512] In this invention, the server includes means for monitoring information obtained from the user, means for transmitting the information to an analysis device, and means for analyzing the information and evaluating the possibility of harassment. This makes it possible to improve the quality of communication within the family, reduce stress among family members, and promote better dialogue.

[0513] "Information obtained from users" refers to communication-related information such as text data and audio data generated or transmitted by users.

[0514] "Monitoring methods" refer to functions that continuously check information obtained from users and collect necessary data.

[0515] "Means for transmitting to an analytical device" refers to the function of appropriately transmitting data to a data analysis device in order to process the acquired information.

[0516] "Means for analyzing and evaluating the possibility of harassment" refers to a process for analyzing received information and evaluating whether a particular action constitutes harassment.

[0517] "Means of issuing notifications" refers to a function that provides users with warnings and suggestions for improvement based on evaluation results.

[0518] "Means for including suggestions for improvement" refers to a function that provides users with specific suggestions to encourage better communication.

[0519] "A means of monitoring conversations in real time and guiding dialogue in a better direction" refers to a function that instantly analyzes conversations between users and provides feedback to maintain smooth communication.

[0520] To implement this invention, it is necessary to implement a dedicated system as a program and run it on appropriate hardware. A specific example of this is an information analysis system used in a home environment.

[0521] The server uses the Google Cloud Speech-to-Text API to convert speech data received from users into text data in real time. This process makes the speech data easily parseable. The converted text data is sent to an analysis device, where text analysis is performed using the natural language processing libraries NLTK or spaCy.

[0522] Based on these analysis results, the device provides feedback to the user. This feedback includes notifications if potential harassment is detected and specific suggestions for improving communication. Furthermore, the device monitors family conversations in real time and provides feedback to facilitate better communication.

[0523] For example, if a family is having a conversation in the living room and an argument breaks out between the children, the device will analyze the conversation and immediately provide advice such as, "Let's try to respect each other's opinions and have a more constructive discussion."

[0524] An example of a prompt to input into the generative AI model would be, "Analyze aggressive or negative remarks heard in the home and consider how to address them each time." This would allow the system to provide appropriate feedback to support healthy communication within the home.

[0525] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0526] Step 1:

[0527] The server acquires voice data from the user. This voice data is collected in real time from conversations within the home. The input is received as voice data through the microphone, and the output becomes an audio file for analysis.

[0528] Step 2:

[0529] The server uses the Google Cloud Speech-to-Text API to convert the acquired audio data into text data. Speech recognition technology is applied to the audio file as input. The output is data in a parseable text format.

[0530] Step 3:

[0531] The server analyzes the text data using a natural language processing library (NLTK or spaCy) to identify expressions that may constitute harassment. This step involves analyzing the text data as input for harassment-related keywords and context. The output is identification information for the relevant sections in the text.

[0532] Step 4:

[0533] The server evaluates the analysis results and generates alerts as needed. If the identified harassment exceeds a certain threshold, it sends a notification to the terminal. The input is the analysis results and evaluation criteria, and the output is a warning notification.

[0534] Step 5:

[0535] The terminal presents notifications to the user based on the generated alerts and provides necessary improvement suggestions. In this step, the alerts and suggestions sent from the server are used as input, and the user is notified visually or audibly. The output is a warning message and improvement suggestions for the user.

[0536] Step 6:

[0537] The user attempts to improve the conversation based on the suggestions presented. This step involves the user, upon receiving a notification from their device, deciding how to modify their behavior to improve the quality of communication. Specifically, this requires choosing more assertive or gentle language.

[0538] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0539] This invention is a system that not only analyzes communication data to detect the possibility of harassment, but also analyzes the user's emotions simultaneously, enabling more adaptive improvement suggestions. The basic configuration for implementing this system is as follows.

[0540] First, the terminal acquires text messages and voice data exchanged by users using communication tools in real time. This acquired data is promptly sent to the server. Upon receiving this data, the server performs analysis using natural language processing and speech recognition technologies.

[0541] This system further incorporates an emotion engine into the data analysis device to identify the user's emotional state from text messages and voice data. Emotion identification is based on the context of the user's text messages, as well as their tone and speed of voice.

[0542] After the server scores the harassment behavior, it further considers this emotional data. Based on the scoring results and emotional state, it immediately sends a notification to the user. The device receives this notification and displays it to the user. The notification includes improvement suggestions based on the emotional state and the harassment behavior score. These suggestions are tailored to the user's current emotions and provide more effective feedback.

[0543] For example, if a user sends the message, "You never understand anything," the device retrieves this message as data and sends it to the server. The server uses natural language processing to evaluate the aggression of the message and scores its potential for harassment. Simultaneously, an emotion engine recognizes the user's feelings of frustration and irritation. As a result, the server sends an emotion-sensitive notification to the device, such as, "This statement may be harassment. Please calm down and let's work together to find a concrete solution." Upon receiving this notification, the user can reflect on their own emotions and be motivated to choose a better way of communicating.

[0544] Thus, this invention prevents harassment by analyzing user emotions and encouraging more appropriate behavior. As a result, it is possible to promote safe and constructive communication throughout the organization.

[0545] The following describes the processing flow.

[0546] Step 1:

[0547] The device acquires text messages and voice data sent by the user using communication tools in real time.

[0548] Step 2:

[0549] The terminal sends the acquired data to the server using a secure protocol. Voice data is sent to the server as is, but it is not converted to text.

[0550] Step 3:

[0551] The server adds the received data to a parsing queue, and the text messages are analyzed using a natural language processing module. Similarly, audio data is converted to text using a speech recognition engine and then analyzed.

[0552] Step 4:

[0553] The server inputs text data and analyzed audio data into the emotion engine to identify the user's emotional state. This emotion analysis is based on the tone of the text and keywords that suggest emotion.

[0554] Step 5:

[0555] The server integrates emotional information and text analysis results to score the likelihood of harassment. In addition to the likelihood of harassment, the emotional state is also considered in the scoring.

[0556] Step 6:

[0557] The server creates an alert to notify the user if the score exceeds a predetermined threshold.

[0558] Step 7:

[0559] The server sends a notification to the device that includes improvement suggestions tailored to the user's emotional state.

[0560] Step 8:

[0561] The terminal displays notifications received from the server to the user, offering suggestions for improvement that are sensitive to the user's feelings, along with information about the possibility of harassment.

[0562] Step 9:

[0563] The server analyzes the accumulated data and generates reports, including individual user communication trends, at regular intervals, sending them to the user's device. Users can then use these reports to implement long-term behavioral changes.

[0564] (Example 2)

[0565] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0566] In today's information and communication environment, while communication between users is increasing, the risk of unintentional harassment is also rising. Such behavior can lead to a deterioration of interpersonal relationships and cause serious problems for organizations and individuals. However, users themselves are often unaware of it, so there is a need for technology that can efficiently and automatically detect harassment and provide appropriate feedback.

[0567] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0568] In this invention, the server includes means for monitoring information acquired from the user, means for transmitting the information to a processing device, and means for the processing device to analyze the information and score the likelihood of the behavior. This makes it possible to analyze behavior in real time, quickly detect potential harassment behavior, and provide the user with appropriate improvement suggestions.

[0569] A "user" refers to an individual or group that uses the system to communicate.

[0570] "Information" refers to digital data, including text messages and audio data, that users generate or receive.

[0571] "Means of monitoring" refers to functions for acquiring information in real time and continuously observing its content and characteristics.

[0572] A "processing unit" refers to a computer or a part thereof that analyzes received information and performs necessary calculations and decisions.

[0573] "Means of analysis" refers to technical means used to break down information using natural language processing and speech recognition technologies and derive conclusions that align with a specific purpose.

[0574] "Means for scoring the likelihood of an action" refers to a function that quantifies or evaluates the degree of a specific action or impact based on the analysis results.

[0575] "Notifications" refer to messages generated based on analysis and scoring results that convey information to the user.

[0576] "Adaptive improvement suggestions" refer to suggestions that include specific and effective advice on behavioral improvements tailored to the user's current situation and emotional state.

[0577] This invention is a system for detecting potential harassment by acquiring information exchanged by users through communication tools in real time. The system includes a terminal used by the user, a server for analyzing the data, and a function to notify the user of improvement suggestions based on the analysis results.

[0578] First, the device continuously acquires text messages and voice data sent and received by the user. Specific target tools include instant messaging software and collaboration platforms. This acquired information is then transmitted to a server via a secure protocol.

[0579] Next, the server receives this information and performs analysis using natural language processing and speech recognition technologies. The technologies used here include generative AI models; for example, open-source libraries and commercial APIs are used for natural language processing, and APIs from speech technology providers are used for speech recognition. Based on this, the server evaluates the aggressiveness and negative tone of the message and scores the likelihood of the action based on the results.

[0580] Furthermore, the server uses an emotion engine to identify the user's emotional state from the information. This involves analyzing text and voice features to quantify the user's emotions. For example, if the user is feeling frustrated or stressed, this will be reflected in the analysis results and influence the final feedback.

[0581] The server generates notifications for the user based on the scoring results and emotional state. The notifications include potential harassment behavior and specific suggestions for improvement. The suggestions are tailored to the user's emotional state, aiming to provide more effective feedback. For example, if the message "You never understand anything" is analyzed, a notification such as "This statement may be considered harassment. Let's calm down and work together to find a concrete solution" will be sent.

[0582] An example of a prompt might be: "If you receive the following message, how should you interpret it and determine your emotional state? Also, generate suggestions for behavioral improvement. Message: 'You never understand anything.'"

[0583] In this way, the system can provide constructive feedback to users and prevent harassment from occurring.

[0584] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0585] Step 1:

[0586] The terminal acquires text messages and voice data sent and received by the user using communication tools in real time. The input is information sent by the user, and the output is the acquired digital data. This data is temporarily stored on the terminal and then prepared to be sent to the server. The terminal checks the data format, encodes it as needed, and converts it into a format suitable for transmission.

[0587] Step 2:

[0588] The terminal sends acquired text messages and voice data to the server. Input is digital data temporarily stored within the terminal, and output is data sent to the server. The terminal ensures the confidentiality and integrity of the data by sending it using a secure protocol (e.g., HTTPS). Error checking is also performed during the transmission process to confirm that the data was sent correctly.

[0589] Step 3:

[0590] The server receives data sent from the terminal and prepares it for analysis. The input consists of text and audio data sent from the terminal, and the output is the data ready for analysis. The server verifies the data format and checks for any invalid data. After confirming the data is clean, it is passed to the analysis module.

[0591] Step 4:

[0592] The server analyzes text data using natural language processing techniques. The input is text data ready for analysis, and the output is the result of extracting contextual and emotional features. The server utilizes a generative AI model to analyze linguistic features and score aggression and negative emotions. This includes contextual analysis, keyword extraction, and emotional tone analysis.

[0593] Step 5:

[0594] The server uses speech recognition technology to convert audio data into text, and then performs natural language processing. The input is audio data ready for analysis, and the output is the text analysis result with emotions and context extracted. Speech recognition accurately transcribes speech into text, followed by emotion analysis. This process utilizes the tone and speed of the user's voice to identify emotions.

[0595] Step 6:

[0596] The server scores the likelihood of an action and determines the user's emotional state based on the analysis of text and audio. The input is the analyzed data, and the output is feedback data that includes improvement suggestions to notify the user. The server applies a scoring algorithm to identify the likelihood of an incident while generating improvement suggestions based on the detected emotional state.

[0597] Step 7:

[0598] The server sends the generated feedback data to the terminal and notifies the user. The input is the feedback data, and the output is the notification message displayed on the terminal. The server sends the feedback, ready for transmission, to the terminal and formats the feedback message appropriately so that it can be displayed as a notification on the user's screen. This allows the user to immediately reflect on their actions and make improvements.

[0599] (Application Example 2)

[0600] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0601] In modern society, misunderstandings and harassment due to communication are major problems within organizations and families. In particular, misunderstandings in family conversations can cause significant stress and discord. This can hinder constructive dialogue between parents and children, and within families. Therefore, it is necessary to mitigate these problems and provide a safe and healthy communication environment.

[0602] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0603] In this invention, the server includes means for monitoring communication data acquired from users, means for transmitting the communication data to a data analysis device, and means for the data analysis device to analyze the communication data and score the likelihood of harassment. This enables the analysis of conversations between family members in real time and, if necessary, provides improvement suggestions that take emotions into consideration, thereby reducing stress and discord within the family and realizing a better communication environment.

[0604] "Means for monitoring communication data obtained from users" refers to devices or methods for continuously detecting and recording text messages and voice data sent or received by users.

[0605] "Means of transmitting to a data analysis device" refers to the technology for immediately or periodically transferring monitored communication data to a data analysis device, which is usually done via a network.

[0606] "Methods for analyzing communication data and scoring the likelihood of harassment" refer to devices and methods that use natural language processing technology and speech recognition technology based on acquired data to quantify and evaluate the likelihood of harassing behavior.

[0607] "Means for issuing harassment notifications to users based on the scoring results and emotional state" refers to devices or methods for providing warnings and suggestions for improvement in real time, taking into account the user's emotional state based on the scoring results obtained through analysis.

[0608] "A means installed in a home information processing device to analyze conversations between family members and provide appropriate improvement suggestions" refers to a device or method for continuously monitoring voice and text conversations between family members in a home environment, performing sentiment analysis, and automatically communicating improvement suggestions to promote calm dialogue.

[0609] The invention will now be described in terms of its embodiments. This system monitors user communication data in real time and provides analysis and notifications as needed. It basically includes a home information processing device and a data analysis device, and promotes good communication within the family.

[0610] First, the device uses sensors such as microphones and cameras to acquire the user's text messages and voice data. This data is sent to the server in real time. Upon receiving the communication, the server uses the natural language processing library NLTK and the Google Speech Recognition API to analyze the data. In particular, the voice data is converted into text data by Google's API, and harassment scoring and sentiment analysis are performed on that text data.

[0611] The data analysis device evaluates the likelihood of harassment based on the scoring results and the user's emotional state, and issues notifications to the user as needed. These notifications are emotionally sensitive and include guidance on smooth communication within the family. For example, it might suggest to parents regarding how to deal with their children, "It would be good to have time for your child to relax and talk." Based on these improvement suggestions, the server reduces stress and discomfort within the family and provides a sophisticated and healthy communication environment.

[0612] As a concrete example, when the server analyzes conversations between parents and children, it recommends that "in situations where parents are being harsh, they should speak in a calm tone to create a safe environment for the child." An example of a prompt to generate this improvement suggestion would be, "Explain how to automatically detect negative emotions and harassment occurring within the home and provide appropriate advice." By implementing this invention, the quality of communication in the home environment is improved.

[0613] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0614] Step 1:

[0615] The device uses a microphone and camera to capture user text messages and voice data in real time. The captured data is sent to a server via the network. The input is voice and text data, and the output is the transmission of this data to the server. As the data is transmitted, it is encrypted and transmitted in a secure state.

[0616] Step 2:

[0617] The server converts the received audio data into text data using the Google Speech Recognition API. This conversion process analyzes the input audio data and converts it into text as output. Specifically, it recognizes the frequency components of the speech and outputs the text that best matches them.

[0618] Step 3:

[0619] The server performs natural language processing using NLTK on the text data. Here, the input is the text data obtained in step 2, and the output is numerical data indicating the sentiment score and degree of aggression of the words spoken by the user. Specifically, it analyzes the context of the text and the word choices, and identifies positive or negative emotions using the sentiment engine.

[0620] Step 4:

[0621] The server generates appropriate notifications for the user based on the scoring results and sentiment analysis results. The inputs used are the scores and sentiment state data generated in step 3, and the output is a notification that includes sentiment-sensitive alerts and improvement suggestions. Specifically, it generates positive feedback and advice to help the user take appropriate action.

[0622] Step 5:

[0623] The device communicates notifications from the server to the user via display or audio. Input is emotion-based notifications from the server, and output is a visual or auditory display of the notification on the user's device. Specifically, the user's attention is drawn by displaying a message on the screen or playing an audio message through the speaker.

[0624] Step 6:

[0625] Based on the notifications and advice provided, users reflect on their own emotions and actions. Here, the input is the notification information received via display or audio, and the output is the user's decision to change their behavior or make improvements. Specifically, it serves as a catalyst for users to consider suggestions and engage in better communication.

[0626] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0627] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0628] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0629] [Fourth Embodiment]

[0630] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0631] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0632] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0633] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0634] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0635] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0636] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0637] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0638] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0639] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0640] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0641] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0642] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0643] This invention is a system that analyzes communication data exchanged by users in real time and detects the possibility of harassment. This system can be implemented by integrating it with appropriate communication tools and following the process outlined below.

[0644] First, the terminal acquires text messages and voice data sent and received by the user in real time. This data is managed by a data acquisition module and sent to the server as needed. Data transmission is carried out using secure communication protocols and with consideration for data protection.

[0645] The server analyzes received text messages using natural language processing techniques to understand their meaning and context. Similarly, audio data is converted to text using speech recognition technology and analyzed in the same manner. The analysis results identify harassment-related keywords and expressions, and a score is assigned based on these. This score indicates an evaluation for each category—sexual harassment, power harassment, and moral harassment—and helps detect unintentional harassment by the user.

[0646] If the score exceeds a certain threshold, the server immediately issues a notification to the device. The device informs the user of the potential for harassment and suggests ways to improve. This helps the user become aware of the impact of their words and encourages them to improve their communication.

[0647] Furthermore, the server aggregates data at regular intervals and analyzes users' communication trends. Based on the analysis results, it generates a report and sends it to the terminal. This report includes information such as the frequency of harassment occurrences and areas for improvement, helping users to make long-term improvements.

[0648] As a concrete example, consider a case where a user says, "You failed again? Come on, give me a break," during an online meeting. This statement is captured by the device and analyzed as text on the server. In this case, a harassment score is calculated based on the wording and context, and if it exceeds the threshold, a warning is immediately displayed on the device. This notification also includes suggestions for improvement, such as, "Try to switch to positive feedback and encourage the other person." Based on this information, the user can review their statements and be motivated to improve their communication with others.

[0649] Through such a system, it is possible to prevent harassment throughout the entire organization and promote safe and healthy communication.

[0650] The following describes the processing flow.

[0651] Step 1:

[0652] The device acquires text messages and voice data sent by the user using communication tools in real time.

[0653] Step 2:

[0654] The terminal performs initial processing on the acquired data and sends it to the server using a secure communication protocol. The voice data is sent in its original form and then analyzed on the server.

[0655] Step 3:

[0656] The server registers the received data in a parsing queue and begins processing it sequentially. Text messages are immediately parsed by the natural language processing module.

[0657] Step 4:

[0658] The server processes the audio data through a speech recognition engine to convert it into text. Then, it is analyzed using a natural language processing module, similar to how text messages are processed.

[0659] Step 5:

[0660] The server uses a machine learning model to score the likelihood of harassment based on the data analysis results. In this process, it analyzes specific keywords and contexts to numerically evaluate the likelihood of harassment.

[0661] Step 6:

[0662] If the score exceeds a predetermined threshold, the server generates an alert regarding that specific message.

[0663] Step 7:

[0664] The server sends the generated alert to the terminal. The terminal displays a notification to the user, providing information about the potential harassment and suggestions for improvement.

[0665] Step 8:

[0666] The server analyzes the accumulated data at regular intervals and generates a report summarizing communication trends.

[0667] Step 9:

[0668] The server sends the generated report to the terminal, and the user can review its contents to understand areas for long-term improvement.

[0669] (Example 1)

[0670] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0671] In today's communication environment, harassment can occur unconsciously, negatively impacting individuals and organizations. However, technology to monitor this in real time and propose effective solutions still does not exist. Addressing this problem is urgently needed.

[0672] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0673] In this invention, the server includes means for monitoring communication information acquired from a user, means for transmitting the communication information to a data processing device, and means for the data processing device to analyze the communication information and quantify the possibility of harassment. This makes it possible to detect harassment in real time and make appropriate improvement suggestions to the user, thereby improving the communication environment of individuals and organizations.

[0674] A "user" is an individual or group that uses the system to send and receive communication information.

[0675] "Communication information" refers to all information exchanged between users, including text messages and voice data.

[0676] "Monitoring methods" refer to processes and devices that collect communication information obtained from users in real time and check for any abnormalities.

[0677] A "data processing device" refers to hardware or software used to analyze collected communication information.

[0678] "Means of analysis" refers to the process of evaluating the content of communication information using natural language processing technology and speech recognition technology to detect specific patterns and keywords.

[0679] "Methods of quantification" refers to the process of evaluating the possibility of harassment based on the analysis results according to certain criteria and expressing it as a numerical value.

[0680] A "warning" refers to a message that notifies a user of potential harassment and encourages them to be aware of it.

[0681] "Improvement suggestions" refer to information that guides users on specific measures or changes in wording that they should take immediately to avoid harassment.

[0682] "Data in different formats" refers to information in various formats, such as text data and audio data.

[0683] "Detailed analysis" refers to a process that goes beyond simple keyword detection and includes comprehensive data evaluation, encompassing contextual and sentiment analysis.

[0684] This invention is a system that analyzes communication information exchanged between users in real time and evaluates the possibility of harassment. The system is implemented in the following way.

[0685] First, the terminal acquires text messages and voice data sent and received by the user in real time. This process utilizes the terminal's built-in microphone and keyboard input, and is managed by a data acquisition module. The terminal then sends the acquired communication information to the server using a secure communication protocol (e.g., TLS). This ensures the security of the data.

[0686] The server processes the received communication information. In particular, it analyzes text data using natural language processing techniques. This is done using open-source natural language processing libraries (e.g., SpaCy, NLTK). Audio data is converted to text using speech recognition technology (e.g., Google Speech-to-Text API) and analyzed in the same way.

[0687] Based on the analysis results, the server identifies specified harassment-related keywords and expressions and assigns numerical values ​​to them. This numerical value is scored based on a specific algorithm, and an evaluation is performed for each harassment category. If the score exceeds a set threshold, the server immediately notifies the terminal.

[0688] The device displays a warning to the user that their comments may be harassing. This warning also includes specific suggestions on how the user can improve their behavior. For example, it might advise, "Try to switch to positive feedback and encourage the other person."

[0689] Furthermore, the server aggregates data at regular intervals and analyzes user communication trends. Based on this analysis, a detailed report is generated for the user and sent to their terminal. This report contains information that can help improve communication.

[0690] As a concrete example, consider the message "Did you fail again? Enough is enough." spoken by a user during an online meeting. This message is captured on the device and transcribed and analyzed on the server. If problematic keywords are found, the potential for harassment is scored, and a warning is displayed if necessary.

[0691] An example of a prompt for a generative AI model is: "Analyze the following text in real time and score its likelihood of harassment: 'Did you fail again? Come on, give it a rest.'" Using this prompt, the AI ​​model analyzes the specified text and returns an appropriate score.

[0692] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0693] Step 1:

[0694] The terminal acquires text messages and voice data sent and received by the user in real time. The input is messages and call information from the user, and the output is the acquired raw communication information. The terminal manages this data using a data acquisition module and securely stores the data using encryption technology.

[0695] Step 2:

[0696] The terminal prepares to send the acquired communication information to the server. The input is the communication information collected in step 1, and the output is the data that has been formatted and encrypted for transmission. The terminal sends the data to the server using a security protocol such as Secure Sockets Layer (SSL).

[0697] Step 3:

[0698] The server receives data sent from the terminal. The input is encrypted communication information, and the output is decrypted data. The server verifies the integrity and validity of the received data and securely stores it in the database.

[0699] Step 4:

[0700] The server analyzes text messages using natural language processing techniques. The input is decoded text data, and the output is contextual understanding and keyword extraction information resulting from the analysis. The server uses open-source natural language processing libraries to perform this analysis and records identified keywords and context in a specific format.

[0701] Step 5:

[0702] The server uses speech recognition technology to convert audio data into text and then performs analysis. The input is decoded audio information, and the output is the text data and its analysis results. The server uses a speech recognition API to convert audio to text and then performs analysis.

[0703] Step 6:

[0704] The server quantifies the likelihood of harassment based on the analysis results. The input is the analysis results of text data, and the output is the harassment score for each category. The server uses a scoring algorithm to calculate this score and stores it in a database.

[0705] Step 7:

[0706] The server sends a notification to the device if the score exceeds a threshold. The input is the harassment score, and the output is a warning message sent to the user. The server includes improvement suggestions in the notification and relays them to the device.

[0707] Step 8:

[0708] The terminal displays received notifications to the user. Input is a warning message from the server, and output is an alert displayed on the user interface. The terminal allows the user to review the notifications and provides a feedback system as needed.

[0709] Step 9:

[0710] The server analyzes data collected at regular intervals and generates reports showing user communication trends. The input is aggregated communication data, and the output is a detailed report. The server uses a database management system to evaluate the data and periodically sends reports to the terminals.

[0711] (Application Example 1)

[0712] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0713] In modern society, communication within families often breaks down, contributing to increased stress within households. In particular, the prevalence of unconscious or intentional verbal harassment within families negatively impacts the home environment. Effective measures are needed to prevent this problem and maintain healthy communication within families.

[0714] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0715] In this invention, the server includes means for monitoring information obtained from the user, means for transmitting the information to an analysis device, and means for analyzing the information and evaluating the possibility of harassment. This makes it possible to improve the quality of communication within the family, reduce stress among family members, and promote better dialogue.

[0716] "Information obtained from users" refers to communication-related information such as text data and audio data generated or transmitted by users.

[0717] "Monitoring methods" refer to functions that continuously check information obtained from users and collect necessary data.

[0718] "Means for transmitting to an analytical device" refers to the function of appropriately transmitting data to a data analysis device in order to process the acquired information.

[0719] "Means for analyzing and evaluating the possibility of harassment" refers to a process for analyzing received information and evaluating whether a particular action constitutes harassment.

[0720] "Means of issuing notifications" refers to a function that provides users with warnings and suggestions for improvement based on evaluation results.

[0721] "Means for including suggestions for improvement" refers to a function that provides users with specific suggestions to encourage better communication.

[0722] "A means of monitoring conversations in real time and guiding dialogue in a better direction" refers to a function that instantly analyzes conversations between users and provides feedback to maintain smooth communication.

[0723] To implement this invention, it is necessary to implement a dedicated system as a program and run it on appropriate hardware. A specific example of this is an information analysis system used in a home environment.

[0724] The server uses the Google Cloud Speech-to-Text API to convert speech data received from users into text data in real time. This process makes the speech data easily parseable. The converted text data is sent to an analysis device, where text analysis is performed using the natural language processing libraries NLTK or spaCy.

[0725] Based on these analysis results, the device provides feedback to the user. This feedback includes notifications if potential harassment is detected and specific suggestions for improving communication. Furthermore, the device monitors family conversations in real time and provides feedback to facilitate better communication.

[0726] For example, if a family is having a conversation in the living room and an argument breaks out between the children, the device will analyze the conversation and immediately provide advice such as, "Let's try to respect each other's opinions and have a more constructive discussion."

[0727] An example of a prompt to input into the generative AI model would be, "Analyze aggressive or negative remarks heard in the home and consider how to address them each time." This would allow the system to provide appropriate feedback to support healthy communication within the home.

[0728] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0729] Step 1:

[0730] The server acquires voice data from the user. This voice data is collected in real time from conversations within the home. The input is received as voice data through the microphone, and the output becomes an audio file for analysis.

[0731] Step 2:

[0732] The server uses the Google Cloud Speech-to-Text API to convert the acquired audio data into text data. Speech recognition technology is applied to the audio file as input. The output is data in a parseable text format.

[0733] Step 3:

[0734] The server analyzes the text data using a natural language processing library (NLTK or spaCy) to identify expressions that may constitute harassment. This step involves analyzing the text data as input for harassment-related keywords and context. The output is identification information for the relevant sections in the text.

[0735] Step 4:

[0736] The server evaluates the analysis results and generates alerts as needed. If the identified harassment exceeds a certain threshold, it sends a notification to the terminal. The input is the analysis results and evaluation criteria, and the output is a warning notification.

[0737] Step 5:

[0738] The terminal presents notifications to the user based on the generated alerts and provides necessary improvement suggestions. In this step, the alerts and suggestions sent from the server are used as input, and the user is notified visually or audibly. The output is a warning message and improvement suggestions for the user.

[0739] Step 6:

[0740] The user attempts to improve the conversation based on the suggestions presented. This step involves the user, upon receiving a notification from their device, deciding how to modify their behavior to improve the quality of communication. Specifically, this requires choosing more assertive or gentle language.

[0741] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0742] This invention is a system that not only analyzes communication data to detect the possibility of harassment, but also analyzes the user's emotions simultaneously, enabling more adaptive improvement suggestions. The basic configuration for implementing this system is as follows.

[0743] First, the terminal acquires text messages and voice data exchanged by users using communication tools in real time. This acquired data is promptly sent to the server. Upon receiving this data, the server performs analysis using natural language processing and speech recognition technologies.

[0744] This system further incorporates an emotion engine into the data analysis device to identify the user's emotional state from text messages and voice data. Emotion identification is based on the context of the user's text messages, as well as their tone and speed of voice.

[0745] After the server scores the harassment behavior, it further considers this emotional data. Based on the scoring results and emotional state, it immediately sends a notification to the user. The device receives this notification and displays it to the user. The notification includes improvement suggestions based on the emotional state and the harassment behavior score. These suggestions are tailored to the user's current emotions and provide more effective feedback.

[0746] For example, if a user sends the message, "You never understand anything," the device retrieves this message as data and sends it to the server. The server uses natural language processing to evaluate the aggression of the message and scores its potential for harassment. Simultaneously, an emotion engine recognizes the user's feelings of frustration and irritation. As a result, the server sends an emotion-sensitive notification to the device, such as, "This statement may be harassment. Please calm down and let's work together to find a concrete solution." Upon receiving this notification, the user can reflect on their own emotions and be motivated to choose a better way of communicating.

[0747] Thus, this invention prevents harassment by analyzing user emotions and encouraging more appropriate behavior. As a result, it is possible to promote safe and constructive communication throughout the organization.

[0748] The following describes the processing flow.

[0749] Step 1:

[0750] The device acquires text messages and voice data sent by the user using communication tools in real time.

[0751] Step 2:

[0752] The terminal sends the acquired data to the server using a secure protocol. Voice data is sent to the server as is, but it is not converted to text.

[0753] Step 3:

[0754] The server adds the received data to a parsing queue, and the text messages are analyzed using a natural language processing module. Similarly, audio data is converted to text using a speech recognition engine and then analyzed.

[0755] Step 4:

[0756] The server inputs text data and analyzed audio data into the emotion engine to identify the user's emotional state. This emotion analysis is based on the tone of the text and keywords that suggest emotion.

[0757] Step 5:

[0758] The server integrates emotional information and text analysis results to score the likelihood of harassment. In addition to the likelihood of harassment, the emotional state is also considered in the scoring.

[0759] Step 6:

[0760] The server creates an alert to notify the user if the score exceeds a predetermined threshold.

[0761] Step 7:

[0762] The server sends a notification to the device that includes improvement suggestions tailored to the user's emotional state.

[0763] Step 8:

[0764] The terminal displays notifications received from the server to the user, offering suggestions for improvement that are sensitive to the user's feelings, along with information about the possibility of harassment.

[0765] Step 9:

[0766] The server analyzes the accumulated data and generates reports, including individual user communication trends, at regular intervals, sending them to the user's device. Users can then use these reports to implement long-term behavioral changes.

[0767] (Example 2)

[0768] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0769] In today's information and communication environment, while communication between users is increasing, the risk of unintentional harassment is also rising. Such behavior can lead to a deterioration of interpersonal relationships and cause serious problems for organizations and individuals. However, users themselves are often unaware of it, so there is a need for technology that can efficiently and automatically detect harassment and provide appropriate feedback.

[0770] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0771] In this invention, the server includes means for monitoring information acquired from the user, means for transmitting the information to a processing device, and means for the processing device to analyze the information and score the likelihood of the behavior. This makes it possible to analyze behavior in real time, quickly detect potential harassment behavior, and provide the user with appropriate improvement suggestions.

[0772] A "user" refers to an individual or group that uses the system to communicate.

[0773] "Information" refers to digital data, including text messages and audio data, that users generate or receive.

[0774] "Means of monitoring" refers to functions for acquiring information in real time and continuously observing its content and characteristics.

[0775] A "processing unit" refers to a computer or a part thereof that analyzes received information and performs necessary calculations and decisions.

[0776] "Means of analysis" refers to technical means used to break down information using natural language processing and speech recognition technologies and derive conclusions that align with a specific purpose.

[0777] "Means for scoring the likelihood of an action" refers to a function that quantifies or evaluates the degree of a specific action or impact based on the analysis results.

[0778] "Notifications" refer to messages generated based on analysis and scoring results that convey information to the user.

[0779] "Adaptive improvement suggestions" refer to suggestions that include specific and effective advice on behavioral improvements tailored to the user's current situation and emotional state.

[0780] This invention is a system for detecting potential harassment by acquiring information exchanged by users through communication tools in real time. The system includes a terminal used by the user, a server for analyzing the data, and a function to notify the user of improvement suggestions based on the analysis results.

[0781] First, the device continuously acquires text messages and voice data sent and received by the user. Specific target tools include instant messaging software and collaboration platforms. This acquired information is then transmitted to a server via a secure protocol.

[0782] Next, the server receives this information and performs analysis using natural language processing and speech recognition technologies. The technologies used here include generative AI models; for example, open-source libraries and commercial APIs are used for natural language processing, and APIs from speech technology providers are used for speech recognition. Based on this, the server evaluates the aggressiveness and negative tone of the message and scores the likelihood of the action based on the results.

[0783] Furthermore, the server uses an emotion engine to identify the user's emotional state from the information. This involves analyzing text and voice features to quantify the user's emotions. For example, if the user is feeling frustrated or stressed, this will be reflected in the analysis results and influence the final feedback.

[0784] The server generates notifications for the user based on the scoring results and emotional state. The notifications include potential harassment behavior and specific suggestions for improvement. The suggestions are tailored to the user's emotional state, aiming to provide more effective feedback. For example, if the message "You never understand anything" is analyzed, a notification such as "This statement may be considered harassment. Let's calm down and work together to find a concrete solution" will be sent.

[0785] An example of a prompt might be: "If you receive the following message, how should you interpret it and determine your emotional state? Also, generate suggestions for behavioral improvement. Message: 'You never understand anything.'"

[0786] In this way, the system can provide constructive feedback to users and prevent harassment from occurring.

[0787] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0788] Step 1:

[0789] The terminal acquires text messages and voice data sent and received by the user using communication tools in real time. The input is information sent by the user, and the output is the acquired digital data. This data is temporarily stored on the terminal and then prepared to be sent to the server. The terminal checks the data format, encodes it as needed, and converts it into a format suitable for transmission.

[0790] Step 2:

[0791] The terminal sends acquired text messages and voice data to the server. Input is digital data temporarily stored within the terminal, and output is data sent to the server. The terminal ensures the confidentiality and integrity of the data by sending it using a secure protocol (e.g., HTTPS). Error checking is also performed during the transmission process to confirm that the data was sent correctly.

[0792] Step 3:

[0793] The server receives data sent from the terminal and prepares it for analysis. The input consists of text and audio data sent from the terminal, and the output is the data ready for analysis. The server verifies the data format and checks for any invalid data. After confirming the data is clean, it is passed to the analysis module.

[0794] Step 4:

[0795] The server analyzes text data using natural language processing techniques. The input is text data ready for analysis, and the output is the result of extracting contextual and emotional features. The server utilizes a generative AI model to analyze linguistic features and score aggression and negative emotions. This includes contextual analysis, keyword extraction, and emotional tone analysis.

[0796] Step 5:

[0797] The server uses speech recognition technology to convert audio data into text, and then performs natural language processing. The input is audio data ready for analysis, and the output is the text analysis result with emotions and context extracted. Speech recognition accurately transcribes speech into text, followed by emotion analysis. This process utilizes the tone and speed of the user's voice to identify emotions.

[0798] Step 6:

[0799] The server scores the likelihood of an action and determines the user's emotional state based on the analysis of text and audio. The input is the analyzed data, and the output is feedback data that includes improvement suggestions to notify the user. The server applies a scoring algorithm to identify the likelihood of an incident while generating improvement suggestions based on the detected emotional state.

[0800] Step 7:

[0801] The server sends the generated feedback data to the terminal and notifies the user. The input is the feedback data, and the output is the notification message displayed on the terminal. The server sends the feedback, ready for transmission, to the terminal and formats the feedback message appropriately so that it can be displayed as a notification on the user's screen. This allows the user to immediately reflect on their actions and make improvements.

[0802] (Application Example 2)

[0803] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0804] In modern society, misunderstandings and harassment due to communication are major problems within organizations and families. In particular, misunderstandings in family conversations can cause significant stress and discord. This can hinder constructive dialogue between parents and children, and within families. Therefore, it is necessary to mitigate these problems and provide a safe and healthy communication environment.

[0805] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0806] In this invention, the server includes means for monitoring communication data acquired from users, means for transmitting the communication data to a data analysis device, and means for the data analysis device to analyze the communication data and score the likelihood of harassment. This enables the analysis of conversations between family members in real time and, if necessary, provides improvement suggestions that take emotions into consideration, thereby reducing stress and discord within the family and realizing a better communication environment.

[0807] "Means for monitoring communication data obtained from users" refers to devices or methods for continuously detecting and recording text messages and voice data sent or received by users.

[0808] "Means of transmitting to a data analysis device" refers to the technology for immediately or periodically transferring monitored communication data to a data analysis device, which is usually done via a network.

[0809] "Methods for analyzing communication data and scoring the likelihood of harassment" refer to devices and methods that use natural language processing technology and speech recognition technology based on acquired data to quantify and evaluate the likelihood of harassing behavior.

[0810] "Means for issuing harassment notifications to users based on the scoring results and emotional state" refers to devices or methods for providing warnings and suggestions for improvement in real time, taking into account the user's emotional state based on the scoring results obtained through analysis.

[0811] "A means installed in a home information processing device to analyze conversations between family members and provide appropriate improvement suggestions" refers to a device or method for continuously monitoring voice and text conversations between family members in a home environment, performing sentiment analysis, and automatically communicating improvement suggestions to promote calm dialogue.

[0812] The invention will now be described in terms of its embodiments. This system monitors user communication data in real time and provides analysis and notifications as needed. It basically includes a home information processing device and a data analysis device, and promotes good communication within the family.

[0813] First, the device uses sensors such as microphones and cameras to acquire the user's text messages and voice data. This data is sent to the server in real time. Upon receiving the communication, the server uses the natural language processing library NLTK and the Google Speech Recognition API to analyze the data. In particular, the voice data is converted into text data by Google's API, and harassment scoring and sentiment analysis are performed on that text data.

[0814] The data analysis device evaluates the likelihood of harassment based on the scoring results and the user's emotional state, and issues notifications to the user as needed. These notifications are emotionally sensitive and include guidance on smooth communication within the family. For example, it might suggest to parents regarding how to deal with their children, "It would be good to have time for your child to relax and talk." Based on these improvement suggestions, the server reduces stress and discomfort within the family and provides a sophisticated and healthy communication environment.

[0815] As a concrete example, when the server analyzes conversations between parents and children, it recommends that "in situations where parents are being harsh, they should speak in a calm tone to create a safe environment for the child." An example of a prompt to generate this improvement suggestion would be, "Explain how to automatically detect negative emotions and harassment occurring within the home and provide appropriate advice." By implementing this invention, the quality of communication in the home environment is improved.

[0816] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0817] Step 1:

[0818] The device uses a microphone and camera to capture user text messages and voice data in real time. The captured data is sent to a server via the network. The input is voice and text data, and the output is the transmission of this data to the server. As the data is transmitted, it is encrypted and transmitted in a secure state.

[0819] Step 2:

[0820] The server converts the received audio data into text data using the Google Speech Recognition API. This conversion process analyzes the input audio data and converts it into text as output. Specifically, it recognizes the frequency components of the speech and outputs the text that best matches them.

[0821] Step 3:

[0822] The server performs natural language processing using NLTK on the text data. Here, the input is the text data obtained in step 2, and the output is numerical data indicating the sentiment score and degree of aggression of the words spoken by the user. Specifically, it analyzes the context of the text and the word choices, and identifies positive or negative emotions using the sentiment engine.

[0823] Step 4:

[0824] The server generates appropriate notifications for the user based on the scoring results and sentiment analysis results. The inputs used are the scores and sentiment state data generated in step 3, and the output is a notification that includes sentiment-sensitive alerts and improvement suggestions. Specifically, it generates positive feedback and advice to help the user take appropriate action.

[0825] Step 5:

[0826] The device communicates notifications from the server to the user via display or audio. Input is emotion-based notifications from the server, and output is a visual or auditory display of the notification on the user's device. Specifically, the user's attention is drawn by displaying a message on the screen or playing an audio message through the speaker.

[0827] Step 6:

[0828] Based on the notifications and advice provided, users reflect on their own emotions and actions. Here, the input is the notification information received via display or audio, and the output is the user's decision to change their behavior or make improvements. Specifically, it serves as a catalyst for users to consider suggestions and engage in better communication.

[0829] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0830] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0831] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0832] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0833] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0834] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0835] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0836] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0837] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0838] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0839] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0840] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0841] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0842] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0843] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0844] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0845] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0846] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0847] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0848] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0849] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0850] The following is further disclosed regarding the embodiments described above.

[0851] (Claim 1)

[0852] A means of monitoring communication data obtained from users,

[0853] Means for transmitting the aforementioned communication data to a data analysis device,

[0854] The data analysis device analyzes the communication data and scores the likelihood of harassment,

[0855] A means for issuing a harassment notification to the user based on the aforementioned scoring results,

[0856] Means for including improvement suggestions in the aforementioned notification,

[0857] ...

[0858] A system that includes this.

[0859] (Claim 2)

[0860] The system according to claim 1, characterized in that the data analysis device analyzes text messages and voice data using natural language processing and speech recognition technology.

[0861] (Claim 3)

[0862] The system according to claim 1, characterized in that the data analysis device periodically analyzes collected data and generates a report showing the user's communication trends.

[0863] "Example 1"

[0864] (Claim 1)

[0865] A means of monitoring communication information obtained from users,

[0866] means for transmitting the aforementioned communication information to a data processing device,

[0867] The data processing device includes means for analyzing the communication information and quantifying the possibility of harassment,

[0868] A means for issuing a warning to the user based on the aforementioned numerical results,

[0869] Means of including improvement suggestions in the aforementioned warning,

[0870] The data processing device includes means for processing data of different formats and performing detailed analysis,

[0871] A system that includes this.

[0872] (Claim 2)

[0873] The system according to claim 1, characterized in that the data processing device analyzes written messages and voice information using natural language processing technology and speech recognition technology.

[0874] (Claim 3)

[0875] The system according to claim 1, characterized in that the data processing device periodically evaluates the collected data and generates a report showing the user's communication trends.

[0876] "Application Example 1"

[0877] (Claim 1)

[0878] Means for monitoring information obtained from users,

[0879] Means for transmitting the aforementioned information to an analysis device,

[0880] The aforementioned analytical device analyzes the information and provides means for evaluating the possibility of harassment,

[0881] A means for issuing a notification to the user based on the evaluation results,

[0882] Means for including suggestions for improvement in the aforementioned notification,

[0883] A means to monitor family conversations in real time and guide the dialogue in a better direction,

[0884] A system that includes this.

[0885] (Claim 2)

[0886] The system according to claim 1, characterized in that the analysis device analyzes language data and speech using natural language processing and speech recognition technology.

[0887] (Claim 3)

[0888] The system according to claim 1, characterized in that the analysis device periodically analyzes collected information and generates a report showing the user's conversational tendencies.

[0889] "Example 2 of combining an emotion engine"

[0890] (Claim 1)

[0891] Means for monitoring information obtained from users,

[0892] means for transmitting the aforementioned information to a processing device,

[0893] The processing device includes means for analyzing the information and scoring the likelihood of an action,

[0894] The processing device includes means for identifying an emotional state from the information,

[0895] A means for issuing a notification to the user based on the aforementioned scoring results and emotional state,

[0896] Means for including adaptive improvement suggestions in the aforementioned notification,

[0897] ...

[0898] A system that includes this.

[0899] (Claim 2)

[0900] The system according to claim 1, characterized in that the processing device analyzes text and speech information using natural language processing and speech recognition technology.

[0901] (Claim 3)

[0902] The system according to claim 1, characterized in that the processing device analyzes periodically collected data and generates records indicating user behavioral trends.

[0903] "Application example 2 when combining with an emotional engine"

[0904] (Claim 1)

[0905] A means of monitoring communication data obtained from users,

[0906] Means for transmitting the aforementioned communication data to a data analysis device,

[0907] The data analysis device analyzes the communication data and scores the likelihood of harassment,

[0908] A means for issuing a harassment notification to the user based on the aforementioned scoring results and emotional state,

[0909] The aforementioned notification includes means of including improvement suggestions that correspond to the emotional state,

[0910] A means installed in a home information processing device that analyzes conversations between family members and provides appropriate improvement suggestions,

[0911] ...

[0912] A system that includes this.

[0913] (Claim 2)

[0914] The system according to claim 1, characterized in that the data analysis device analyzes text messages and voice data using natural language processing and speech recognition technology to recognize emotions.

[0915] (Claim 3)

[0916] The system according to claim 1, characterized in that the data analysis device periodically analyzes collected data and generates a report showing the user's communication tendencies and emotional changes. [Explanation of symbols]

[0917] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for monitoring information obtained from users, Means for transmitting the aforementioned information to an analysis device, The aforementioned analytical device analyzes the information and provides means for evaluating the possibility of harassment, A means for issuing a notification to the user based on the evaluation results, Means for including suggestions for improvement in the aforementioned notification, A means to monitor family conversations in real time and guide the dialogue in a better direction, A system that includes this.

2. The system according to claim 1, characterized in that the analysis device analyzes language data and speech using natural language processing and speech recognition technology.

3. The system according to claim 1, characterized in that the analysis device periodically analyzes collected information and generates a report showing the user's conversational tendencies.

Images