system

Abstract

We provide the system. [Solution] Means for collecting data on harassment behavior, A means of identifying harassment behavior by analyzing the aforementioned collected data, A means of generating improvement suggestions based on identified harassment behaviors, A means of providing the aforementioned improvement suggestions to the user, A system that includes this.

Description

Technical Field

[0004] , , , ,

[0005] , , , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, harassment behavior in the workplace has become a social problem, and there is a need to prevent this and construct a healthy workplace environment that respects diversity. However, with conventional methods, it is difficult to accurately detect harassment behavior and present specific improvement measures according to individual situations. In addition, due to the lack of a highly reliable approach based on objective data, effective harassment prevention measures have not been established.

Means for Solving the Problems

[0005] This invention provides a system that collects, analyzes, and identifies various data related to harassment in order to prevent harassment behavior, and provides users with specific, individualized improvement suggestions. Specifically, it includes means for collecting information including text data and video data, and analyzes this information using natural language processing technology and image recognition technology. By doing so, it identifies harassment behavior, generates accurate and effective improvement suggestions, and provides them to users, thereby improving the workplace environment.

[0006] "Harassment behavior" refers to actions such as inappropriate words or attitudes towards other individuals in the workplace that cause mental or physical distress to the victim.

[0007] "Data" refers to any form of information, including text and images, that is collected for the purpose of analyzing harassment behavior.

[0008] "Data collection" is the process of gathering necessary information from diverse sources and is an activity aimed at accumulating data.

[0009] "Analysis" is the process of interpretation and evaluation carried out using collected data to identify patterns and trends present within it.

[0010] "Identification" is the activity of identifying and clearly distinguishing specific characteristic behaviors or events based on analysis.

[0011] An "improvement suggestion" is the provision of specific guidelines and advice to correct identified problematic behaviors and improve the workplace environment.

[0012] A "user" is the ultimate beneficiary who receives suggestions and information from the system, and is primarily an individual who is asked to improve their harassment behavior.

[0013] A "system" is a technological framework in which multiple means interact with each other to provide an integrated set of processes for achieving a specific objective. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention relates to a system for appropriately detecting harassment behavior in the workplace and proposing effective corrective measures. This system consists of a server, terminals, and users.

[0036] Processing performed by the server

[0037] The server collects various data from the company's digital infrastructure. Specifically, it obtains information from emails, chat messages, and surveillance camera footage. In addition, it can also collect publicly available data from social media. The server then analyzes the collected data. This analysis uses natural language processing technology to scrutinize message content and identify aggressive behavior and negative emotions. It also uses image recognition technology to detect nonverbal harassment behavior from video footage. Based on the analysis results, the server automatically generates improvement suggestions. This includes utilizing a database of past cases and insights based on behavioral science.

[0038] Processing performed by the terminal

[0039] The terminal is a device used by the user and is responsible for receiving improvement suggestions sent from the server. The terminal provides an interface to display suggestions to the user and prompt action. Suggestions can be presented in various formats (e.g., text-based reports and visual content) to allow users to easily access the information they need.

[0040] User-performed processes

[0041] Based on improvement suggestions received via the device, users review their own behavior and take specific actions to improve it. If necessary, they can utilize additional resources provided by the server or device (e.g., training modules, workshop information) to correct harassing behavior.

[0042] As a concrete example, if communication between employee A and a subordinate is deemed problematic in a workplace, the server analyzes A's email history and video footage from meetings to detect overbearing attitudes and aggressive language. Next, based on this information, the server generates improvement suggestions for A, recommending modules that introduce communication skills improvement and appropriate teaching methods. These suggestions are provided to A via their device, allowing them to use them to improve their behavior.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The server collects data potentially related to harassment behavior from sources such as the company's email system, chat tools, surveillance cameras, and public social media data. It is programmed to comply with access control and privacy policies during this process.

[0046] Step 2:

[0047] The server preprocesses the collected data. Specifically, text data is tokenized and unwanted noise is removed. For video data, frames are extracted and converted into a format suitable for analysis.

[0048] Step 3:

[0049] The server analyzes pre-processed data. Using natural language processing techniques, it performs sentiment analysis on text to identify negative or aggressive language. Simultaneously, it uses image recognition techniques to detect nonverbal problematic behaviors from video.

[0050] Step 4:

[0051] The server generates improvement suggestions based on the analysis results. Through analysis of similar cases based on past database data, it forms appropriate improvement measures and guidelines for behavioral change. These suggestions include detailed advice for specific behavioral improvements and skill enhancements.

[0052] Step 5:

[0053] The terminal receives improvement suggestions sent from the server and presents them to the user through the user interface. Visuals and audio guides are used to display the information in a way that is easy for the user to understand.

[0054] Step 6:

[0055] Users refer to suggestions from their devices and take the suggested actions. If necessary, they can access training videos and guidelines included in the suggestions to improve their behavior.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] Harassment in the workplace negatively impacts organizational health and productivity. However, systems for early identification of harassment and implementation of appropriate corrective measures are not adequately in place. Therefore, effective means for the early detection and resolution of harassment are needed.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for collecting harassment-related information from a communication medium, means for analyzing the collected information to identify harassment behavior, and means for generating corrective measures for the identified harassment behavior. This makes it possible to quickly identify harassment behavior and present effective corrective measures.

[0061] A "communication medium" is an electronic or optical means used to send and receive digital information.

[0062] "Harassment-related information" refers to data concerning aggressive or inappropriate behavior towards individuals or organizations.

[0063] "Analysis" is the process of analyzing collected data to extract and evaluate specific information.

[0064] "Harassment" refers to any behavior that causes psychological or physical discomfort to an individual or group.

[0065] A "solution" is a set of actionable steps proposed to resolve an identified problem.

[0066] A "terminal device" is a device used by a user to receive or display digital information.

[0067] A "user" is an individual or organization that receives information and services from a system and utilizes them.

[0068] A "natural language processing unit" is a computer system designed to analyze and understand human language.

[0069] An "image analysis device" is a computer system designed to extract and recognize information from still images or videos.

[0070] This invention is a system for efficiently detecting and addressing harassment in the workplace. The system consists of a server, terminal devices, and users, and in order to implement the invention, it is necessary to specifically understand the role and technology of each component.

[0071] Server roles and configuration

[0072] The server functions as the central processing unit of this system. The server collects information from workplace emails, chat systems, monitoring systems, and publicly available social networking services (SNS) through communication media. For analyzing the collected information, it uses the Python NLTK library as a natural language processing unit and the OpenCV library as an image analysis unit. Furthermore, the server automatically generates appropriate improvement measures based on the collected and analyzed information using a generative AI model. These improvement measures are based on a database of past cases and insights from behavioral science.

[0073] Role and function of terminal devices

[0074] The terminal device functions as a device for exchanging information between the user and the server. It receives improvement suggestions sent from the server and presents them to the user in an easy-to-understand manner. This presentation is provided in various report formats. The terminal device has the ability to display pop-up notifications and detailed information to help the user easily accept the improvement suggestions.

[0075] User roles and responses

[0076] Users are required to review their own behavior based on the improvement measures received via the terminal device and take specific actions to improve it. If necessary, the terminal will provide additional resources such as training modules and workshop information, which users can utilize to improve their harassment behavior.

[0077] For example, if a user is identified as having communication problems at work, the server analyzes the user's email history and meeting videos. This detects aggressive attitudes and negative expressions, and based on this information, improvement measures are generated. The user receives these improvement measures through a terminal device and can take training modules to improve their communication skills.

[0078] An example of a prompt to input into the generating AI model would be: "Develop a system that analyzes email and chat history within the workplace, identifies aggressive behavior, generates improvement measures based on that information, and distributes them to users."

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1: Data Collection

[0081] The server collects harassment-related information from workplace email servers, chat platforms, monitoring systems, and publicly available social media. Input data includes text messages, images, and videos. Specifically, the server periodically accesses each communication medium, retrieves new data, and securely stores it in a database. As output, the collected raw data is sent to the next analysis step.

[0082] Step 2: Data Analysis

[0083] The server applies a natural language processing unit and an image analysis unit to the collected data. The input consists of text and image data collected in the previous step. Here, the text data is analyzed using a natural language processing algorithm to identify aggressive language and negative expressions. Simultaneously, image analysis is utilized to detect nonverbal harassment behavior from video data. Specifically, the text is analyzed using the Python NLTK library, and image recognition is performed using the OpenCV library. The output generates the detected suspicious behavior and its details.

[0084] Step 3: Generating improvement measures

[0085] The server generates improvement measures based on the analysis results. The input is detailed information about the harassment behavior identified in the previous step. Based on behavioral science and similar past cases, a generative AI model creates specific and effective improvement measures. Specifically, the analysis results are input into the generative AI model, and the resulting improvement measures are output.

[0086] Step 4: Distributing the proposal

[0087] The terminal receives improvement suggestions generated from the server. The input is improvement suggestions customized for each user. The terminal device displays these suggestions clearly to the user and runs a notification function to encourage behavioral improvement. The output is the details of the improvement suggestions displayed to the user and the corresponding action steps. In terms of specific actions, the terminal presents suggestions in the form of pop-up alerts or reports.

[0088] Step 5: User Actions

[0089] Based on the improvement suggestions received through the device, users review their behavior and take concrete actions. Inputs include the content of the suggestions and additional resources (e.g., online training modules and workshop information). Users actively utilize these resources provided through the device to improve their harassment behavior. Specifically, users follow the suggestions, attend training, and practice new communication techniques. Expected outputs include improved behavior and enhanced communication skills.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Harassment in the workplace is a serious problem that harms employees' mental and physical health and reduces productivity. Traditional systems have struggled to quickly and accurately detect such behavior and provide effective countermeasures. In particular, there is a challenge in making informed decisions in situations requiring immediate on-site response.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes means for collecting digital information relating to harassment behavior, means for analyzing the collected digital information to identify harassment behavior, and means for generating behavioral improvement measures based on the identified harassment behavior. This enables real-time detection and rapid response to problematic behavior in the workplace, reduces employee stress, and improves productivity.

[0095] "Harassment behavior" refers to aggressive words or actions towards employees in the workplace, or any behavior that harms their mental or physical health.

[0096] "Digital information" refers to all data, including emails, chat messages, and video data, that are generated and collected within the workplace for communication and work performance.

[0097] "Data collection means" refers to a combination of hardware and software used to acquire relevant information from various digital infrastructures within the workplace.

[0098] "Data analysis means" refers to algorithms or methods for identifying harassment behaviors based on collected digital information, using natural language processing technology and image recognition technology.

[0099] "Behavioral improvement measures" refer to specific guidelines or procedures, including suggestions or policies, that are generated to correct identified harassment behaviors.

[0100] "Real-time analysis" refers to the process of instantly processing video data and providing rapid information and warnings about identified behaviors.

[0101] The system for implementing the present invention mainly consists of three elements: a server, a terminal, and a user. The server collects digital information by acquiring emails, chat messages, and surveillance camera footage from various digital infrastructures within the workplace, and also collects publicly available data from social networking services (SNS) as much as possible. To analyze this data in real time, the server uses image generation programs such as TENSORFLOW® and OpenCV to detect nonverbal harassment behavior. It also analyzes text data using natural language processing techniques through libraries such as spaCy and Transformers to identify aggressive words and negative emotions.

[0102] Based on the analysis, the server automatically generates behavioral improvement measures and sends them to the terminal. These measures utilize past case data and insights based on behavioral science, and include specific countermeasures and training modules. The terminals used by users can be various devices such as smart glasses or head-mounted displays, which receive information sent from the server and display warnings of problematic behaviors and improvement measures in the user's field of vision in real time. Based on this, users are expected to review their own behavior and take appropriate action.

[0103] As a concrete example, if an employee exhibits aggressive behavior, the server analyzes the data and issues a warning to highlight the problem. The administrator, acting as the user, receives the warning through smart glasses, recognizes the issue, and can quickly take corrective action. In this application, the prompt used for the generative AI model is: "Please provide prompts to detect harassment behavior in the workplace and, based on that, suggest specific improvement measures that the user (administrator) should take."

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The server collects emails, chat messages, surveillance camera footage, and new publicly available social media data from the workplace's digital infrastructure. The input is digital data from each device, which is then prepared for storage on the server. The output forms a digital information storage system for the entire workplace environment.

[0107] Step 2:

[0108] The server analyzes the collected digital information in real time. For text data, natural language processing techniques are used, employing libraries such as spaCy and Transformers to identify aggressive language and negative emotions. The input is extracted text data, and the output generates an index of the identified aggressive language and emotions. For video data, TensorFlow and OpenCV are used to detect nonverbal harassment behavior. Here, the input is video frames, and the output is tagged with the identified problematic behavior.

[0109] Step 3:

[0110] The server automatically generates behavioral improvement measures based on the harassment behaviors identified through analysis. It proposes specific countermeasures and training modules, utilizing past cases and insights from behavioral science. Inputs include indexes and tags from the analysis results, while output is text data of the generated improvement measures.

[0111] Step 4:

[0112] The device displays improvement information received from the server to the user in real time. Warnings and suggestions are displayed visually through smart glasses or similar devices. Input is text data sent from the server, and output is visual notifications and follow-up signals.

[0113] Step 5:

[0114] Users review their actions based on notifications from their devices. Specifically, they recognize problems based on visually presented information and implement suggested behavioral improvements. The action plan that users should take follows the text of the suggested improvements.

[0115] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0116] This invention provides a system that combines an emotion engine to effectively detect and improve harassment behavior in the workplace. This system consists of a server, terminals, users, and an emotion engine.

[0117] Processing performed by the server

[0118] The server collects data from various workplace data sources, such as emails, chat logs, surveillance camera feeds, and publicly available data from social media. It also collects emotional data from audio and video recordings. After preprocessing this data, the server uses natural language processing and image recognition techniques to identify harassing behaviors and problematic emotional expressions.

[0119] The function of the emotional engine

[0120] The emotion engine analyzes collected audio and video data to determine the user's emotional state. For audio data analysis, it estimates emotions from the tone and volume of speech. For video data analysis, it identifies emotions by analyzing facial expressions based on facial recognition technology. The emotional data determined by the emotion engine is used for further analysis on the server.

[0121] Server-based generation of improvement suggestions

[0122] The server automatically generates specific improvement suggestions based on data analysis of harassment behaviors and emotional states. These suggestions include guidance on how users should control their emotions and engage in constructive communication.

[0123] Terminal role

[0124] The terminal is a device that delivers improvement suggestions sent from the server to the user. The terminal displays the improvement suggestions to the user in an intuitively understandable format and proposes an action plan to encourage improvement in behavior.

[0125] User reactions and actions

[0126] Users will implement specific behavioral improvement measures based on the information provided via their devices. If necessary, they will utilize additional training resources and support to improve their emotional management skills and correct harassing behaviors.

[0127] For example, if person B makes an inappropriate remark during a meeting at a certain workplace, this system analyzes the nature of the remark and person B's emotional state. The server identifies the behavior as problematic and generates improvement suggestions that take into account the stress indicators detected by the emotion engine. These suggestions are provided to person B via a terminal, and based on them, person B can learn how to correct their behavior.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The server collects information from email, chat logs, surveillance camera footage, and publicly available data from social media via the corporate network. During this process, it adheres to security and privacy policies and performs necessary access authentication.

[0131] Step 2:

[0132] The server preprocesses the collected data. Text data is tokenized and converted into a format that can be grammatically analyzed, and video data is extracted frame by frame. For audio data, noise is removed and the characteristics of the audio waveform are extracted.

[0133] Step 3:

[0134] The emotion engine analyzes audio and video data to determine the user's emotions. For audio, it analyzes voice tone and pitch to assess the level of stress and anger in adverse situations. For video, it uses facial recognition technology to capture subtle changes in facial expressions and identify emotions.

[0135] Step 4:

[0136] The server integrates and analyzes pre-processed data and the results of the emotion engine. It uses natural language processing techniques to check text for offensive or negative words and image recognition techniques to detect nonverbal harassment behavior in videos.

[0137] Step 5:

[0138] Based on the analysis results, the server generates improvement suggestions tailored to each individual user. From the analysis, which includes emotional states, it creates specific advice on how users should improve their behavior and control their emotions.

[0139] Step 6:

[0140] The terminal receives improvement suggestions generated from the server and presents them to the user. The improvement suggestions are displayed in a visualized interface and provided in a way that is easy for the user to understand intuitively.

[0141] Step 7:

[0142] Users implement improvement suggestions received via their devices. They review their behavior and manage their emotions based on the suggestions. They also utilize additional support materials and applications as needed to achieve continuous improvement.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0145] Harassment is a serious problem in the workplace, and prompt detection and corrective measures are required. Traditional methods have faced challenges in effective countermeasures due to fragmented data collection and analysis, and an inability to fully grasp emotional states. Furthermore, there was a need for a system that could accurately grasp emotional changes, quickly generate concrete improvement proposals based on those findings, and provide them to users.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes means for collecting information relating to harassment behavior, means for analyzing the collected information to identify harassment behavior, and means for analyzing emotions from audio data and video data. This makes it possible to accurately detect harassment behavior in the workplace environment and quickly provide users with effective improvement suggestions for taking appropriate corrective measures.

[0148] "Harassment behavior" refers to actions that cause psychological or physical discomfort or harm to others in social settings such as the workplace.

[0149] "Information" refers to facts and data expressed in any form, including data collected from emails, chats, audio recordings, video recordings, and social networking services.

[0150] "Analyzing emotions" refers to the process of determining and identifying the emotional state expressed in collected audio and video data.

[0151] "Improvement suggestions" refer to content that provides guidelines and action plans for the individual to improve their behavior and emotions, as well as their communication, based on identified harassment behaviors and analyzed emotions.

[0152] "Users" refers to individuals or groups who receive improvement suggestions provided through this system with the aim of improving their own behavior and emotions.

[0153] This invention is a system for effectively detecting and improving harassment behavior in the workplace environment. The system consists of a server, terminals, users, and an emotion engine.

[0154] The server collects data from emails, chats, surveillance camera footage, and social networking services. Audio recordings are also included. The server preprocesses the collected data and analyzes it using natural language processing and image recognition technologies. The hardware used includes a high-performance database server, and the software utilizes natural language processing libraries and image recognition algorithms. Through this analysis, the server identifies harassment behaviors and users' emotional states.

[0155] The emotion engine analyzes audio and video data to determine the user's emotional state. It identifies emotions by analyzing speech tone and volume from audio data, and facial expressions from video data. This resulting emotional data is then further analyzed by the server.

[0156] Improvement suggestions are automatically generated based on the analysis results and sent to the terminal by the server. The terminal displays the improvement suggestions from the server to the user in an easy-to-understand format. This allows the user to obtain concrete actionable improvement measures. The terminal has the function of providing information in a format that is easy for the user to understand intuitively.

[0157] For example, if the server detects inappropriate remarks in a particular meeting, it will generate improvement suggestions, taking into account the emotional state behind those remarks. These suggestions will be displayed on the terminal, showing the user how to correct their behavior.

[0158] By using a generative AI model, the server can improve the accuracy of its information analysis and respond flexibly to user needs. An example of a prompt message would be, "Please describe a system that automatically generates suggestions for improving harassment behavior in the workplace. Please explain in detail, including specific data processing methods using an emotion engine and natural language processing technology." This input optimizes the system's operation.

[0159] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0160] Step 1:

[0161] The server collects information from various data sources in the work environment. Inputs include email, chat history, surveillance camera footage, and data from social networks. The output is the result of this raw data collection, which is stored in a database. This process integrates the data into an appropriate format and prepares it for subsequent processing.

[0162] Step 2:

[0163] The server preprocesses the collected data. The input includes various data collected in Step 1. Data processing, such as data cleaning, noise reduction, and data format standardization, is performed to obtain clean, analyzable data as output. This process prepares the information necessary to improve the accuracy of the analysis.

[0164] Step 3:

[0165] The server performs emotion analysis using an emotion engine. The input includes clean audio and video data obtained in step 2. For the audio data, the tone and volume of speech are analyzed, and for the video data, facial recognition is used to analyze facial expressions and identify emotions. The output is data of the identified emotional state, which is used in the next analysis step.

[0166] Step 4:

[0167] The server uses natural language processing and image recognition technologies to perform analysis to identify harassment behaviors. The input includes text data obtained in step 2 and sentiment data obtained in step 3. The processing analyzes the text content and related sentiment indicators to determine whether a behavior constitutes harassment. The output is data representing the evaluation results for each behavior.

[0168] Step 5:

[0169] The server automatically generates improvement suggestions based on identified behavioral assessments and emotional data. The input includes the analysis results from step 4. This step generates specific suggestions on how to manage emotions and engage in constructive communication. This provides guidance for necessary behavioral improvements.

[0170] Step 6:

[0171] The terminal delivers improvement suggestions sent from the server to the user. The input includes improvement suggestions automatically generated in step 5. The terminal displays these suggestions visually and clearly to the user, making them intuitively understandable. The output is the information presented to the user.

[0172] Step 7:

[0173] Users improve their behavior based on improvement suggestions provided through their devices. They implement specific behavioral improvements and utilize additional resources and training as needed. This aims to improve emotional management and correct harassing behaviors. The output is the user's improved behavior and its effects.

[0174] (Application Example 2)

[0175] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0176] When harassment occurs in the workplace, it can significantly impair employees' mental health and work efficiency. Traditional methods have struggled to quickly recognize and correct harassment, making immediate action to prevent the problem from escalating difficult. Therefore, there is a need for a system that can analyze emotional states in real time, detect inappropriate behavior early, and provide appropriate improvement suggestions.

[0177] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0178] In this invention, the server includes means for collecting information on harassment behavior, means for analyzing the collected information to identify harassment behavior, means for generating improvement suggestions based on the identified harassment behavior, and means for detecting inappropriate behavior in real time, including an emotion engine that analyzes emotions from the user's voice and facial expressions. This makes it possible to quickly and effectively detect harassment behavior in the workplace and provide specific improvement measures.

[0179] "Harassment behavior" refers to inappropriate or offensive actions or remarks in the workplace that harm the mental health of employees or the work environment.

[0180] "Means of information gathering" refers to processes that include various technologies and equipment for collecting relevant data from data sources within the workplace.

[0181] "Means of analyzing information" refers to techniques that use collected data to perform analyses in order to identify harassment behaviors.

[0182] "Means for generating improvement suggestions" refers to algorithms or software used to propose appropriate countermeasures or behavioral modifications for identified harassment behaviors.

[0183] An "emotion engine" is a technology that analyzes an individual's emotional state from data such as voice and facial expressions, and detects changes in emotions in real time.

[0184] A "means for detecting inappropriate behavior" refers to a mechanism that identifies problematic or risky behaviors in the workplace in real time, based on the results of sentiment analysis.

[0185] The system for implementing the present invention mainly consists of a server, terminals, and users. The server has the function of collecting information from various data sources within the work environment, and this information includes text information, audio data, and image data. Specifically, it receives data from emails, chat records, video from surveillance systems, and audio recordings.

[0186] The collected information is analyzed on the server using natural language processing and visual pattern recognition technologies. First, textual information is analyzed by natural language processing algorithms to evaluate the possibility of harassment based on specific keywords and context. Next, from image data, facial recognition technology and an emotion engine are used to identify emotional states from facial expressions. For audio data, speech analysis technology is used to infer emotions from factors such as tone and volume of voice. This allows for sensitive detection of emotional changes in real time.

[0187] The terminal serves to communicate improvement suggestions generated from the server to the user. Through the terminal, the user receives specific suggestions for behavioral improvement and uses them to improve the situation. For example, if a user shows strong frustration during a meeting, the terminal will immediately display suggestions for calming down. These include specific actions such as leaving the room or taking deep breaths.

[0188] An example of a prompt using a generative AI model is, "Please check what emotional state was detected from this data." Another prompt used is, "Please generate specific behavioral improvement measures based on the emotion analysis results." These prompts enable the generation of more flexible and appropriate improvement suggestions.

[0189] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0190] Step 1:

[0191] The server collects information from various data sources within the work environment. It receives emails, chat logs, video and audio recordings from monitoring systems as input. This data is temporarily stored in storage as raw data.

[0192] Step 2:

[0193] The server preprocesses the collected raw data. Unnecessary tags and special characters are removed from the input text information, and audio data is converted to text. Image data is also converted to a standard format, and noise is removed to produce clean analytical data as output.

[0194] Step 3:

[0195] The server applies natural language processing techniques to the pre-processed text information. This involves extracting keywords and performing sentiment analysis from the input text. This identifies potentially harassing phrases, and the relevant sections are output as analysis results.

[0196] Step 4:

[0197] The server applies face recognition technology and an emotion engine to pre-processed image data. It recognizes individual faces from the input image data and identifies emotional states through facial expression analysis. The obtained emotional state information is output as the analysis result.

[0198] Step 5:

[0199] The server applies speech analysis technology to the audio data. It analyzes the tone and volume of the input audio and estimates the emotional state. The estimated emotional information is output as the analysis result.

[0200] Step 6:

[0201] The server integrates the analysis results to identify harassment behaviors and generate necessary improvement suggestions. Here, a generative AI model is used to create prompt messages based on the input analysis results and output behavioral improvement measures.

[0202] Step 7:

[0203] The terminal receives improvement suggestions sent from the server and displays them to the user in an intuitively understandable format. Based on the suggestions received as input, it outputs visualized action plans and presents specific improvement actions.

[0204] Step 8:

[0205] Users implement improvement suggestions displayed on their devices. Following these suggestions, they might, for example, learn communication techniques to avoid misunderstandings and adjust their behavior according to the situation. This manifests as actual changes in behavior in the output.

[0206] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0207] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0208] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0209] [Second Embodiment]

[0210] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0211] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0212] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0213] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0214] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0215] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0216] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0217] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0218] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0219] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0220] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0221] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0222] This invention relates to a system for appropriately detecting harassment behavior in the workplace and proposing effective corrective measures. This system consists of a server, terminals, and users.

[0223] Processing performed by the server

[0224] The server collects various data from the company's digital infrastructure. Specifically, it obtains information from emails, chat messages, and surveillance camera footage. In addition, it can also collect publicly available data from social media. The server then analyzes the collected data. This analysis uses natural language processing technology to scrutinize message content and identify aggressive behavior and negative emotions. It also uses image recognition technology to detect nonverbal harassment behavior from video footage. Based on the analysis results, the server automatically generates improvement suggestions. This includes utilizing a database of past cases and insights based on behavioral science.

[0225] Processing performed by the terminal

[0226] The terminal is a device used by the user and is responsible for receiving improvement suggestions sent from the server. The terminal provides an interface to display suggestions to the user and prompt action. Suggestions can be presented in various formats (e.g., text-based reports and visual content) to allow users to easily access the information they need.

[0227] User-performed processes

[0228] Based on improvement suggestions received via the device, users review their own behavior and take specific actions to improve it. If necessary, they can utilize additional resources provided by the server or device (e.g., training modules, workshop information) to correct harassing behavior.

[0229] As a concrete example, if communication between employee A and a subordinate is deemed problematic in a workplace, the server analyzes A's email history and video footage from meetings to detect overbearing attitudes and aggressive language. Next, based on this information, the server generates improvement suggestions for A, recommending modules that introduce communication skills improvement and appropriate teaching methods. These suggestions are provided to A via their device, allowing them to use them to improve their behavior.

[0230] The following describes the processing flow.

[0231] Step 1:

[0232] The server collects data potentially related to harassment behavior from sources such as the company's email system, chat tools, surveillance cameras, and public social media data. It is programmed to comply with access control and privacy policies during this process.

[0233] Step 2:

[0234] The server preprocesses the collected data. Specifically, text data is tokenized and unwanted noise is removed. For video data, frames are extracted and converted into a format suitable for analysis.

[0235] Step 3:

[0236] The server analyzes pre-processed data. Using natural language processing techniques, it performs sentiment analysis on text to identify negative or aggressive language. Simultaneously, it uses image recognition techniques to detect nonverbal problematic behaviors from video.

[0237] Step 4:

[0238] The server generates improvement suggestions based on the analysis results. Through analysis of similar cases based on past database data, it forms appropriate improvement measures and guidelines for behavioral change. These suggestions include detailed advice for specific behavioral improvements and skill enhancements.

[0239] Step 5:

[0240] The terminal receives improvement suggestions sent from the server and presents them to the user through the user interface. Visuals and audio guides are used to display the information in a way that is easy for the user to understand.

[0241] Step 6:

[0242] Users refer to suggestions from their devices and take the suggested actions. If necessary, they can access training videos and guidelines included in the suggestions to improve their behavior.

[0243] (Example 1)

[0244] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0245] Harassment in the workplace negatively impacts organizational health and productivity. However, systems for early identification of harassment and implementation of appropriate corrective measures are not adequately in place. Therefore, effective means for the early detection and resolution of harassment are needed.

[0246] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0247] In this invention, the server includes means for collecting harassment-related information from a communication medium, means for analyzing the collected information to identify harassment behavior, and means for generating corrective measures for the identified harassment behavior. This makes it possible to quickly identify harassment behavior and present effective corrective measures.

[0248] A "communication medium" is an electronic or optical means used to send and receive digital information.

[0249] "Harassment-related information" refers to data concerning aggressive or inappropriate behavior towards individuals or organizations.

[0250] "Analysis" is the process of analyzing collected data to extract and evaluate specific information.

[0251] "Harassment" refers to any behavior that causes psychological or physical discomfort to an individual or group.

[0252] A "solution" is a set of actionable steps proposed to resolve an identified problem.

[0253] A "terminal device" is a device used by a user to receive or display digital information.

[0254] A "user" is an individual or organization that receives information and services from a system and utilizes them.

[0255] A "natural language processing unit" is a computer system designed to analyze and understand human language.

[0256] An "image analysis device" is a computer system designed to extract and recognize information from still images or videos.

[0257] This invention is a system for efficiently detecting and addressing harassment in the workplace. The system consists of a server, terminal devices, and users, and in order to implement the invention, it is necessary to specifically understand the role and technology of each component.

[0258] Server roles and configuration

[0259] The server functions as the central processing unit of this system. The server collects information from workplace emails, chat systems, monitoring systems, and publicly available social networking services (SNS) through communication media. For analyzing the collected information, it uses the Python NLTK library as a natural language processing unit and the OpenCV library as an image analysis unit. Furthermore, the server automatically generates appropriate improvement measures based on the collected and analyzed information using a generative AI model. These improvement measures are based on a database of past cases and insights from behavioral science.

[0260] Role and function of terminal devices

[0261] The terminal device functions as a device for exchanging information between the user and the server. It receives improvement suggestions sent from the server and presents them to the user in an easy-to-understand manner. This presentation is provided in various report formats. The terminal device has the ability to display pop-up notifications and detailed information to help the user easily accept the improvement suggestions.

[0262] User roles and responses

[0263] Users are required to review their own behavior based on the improvement measures received via the terminal device and take specific actions to improve it. If necessary, the terminal will provide additional resources such as training modules and workshop information, which users can utilize to improve their harassment behavior.

[0264] For example, if a user is identified as having communication problems at work, the server analyzes the user's email history and meeting videos. This detects aggressive attitudes and negative expressions, and based on this information, improvement measures are generated. The user receives these improvement measures through a terminal device and can take training modules to improve their communication skills.

[0265] An example of a prompt to input into the generating AI model would be: "Develop a system that analyzes email and chat history within the workplace, identifies aggressive behavior, generates improvement measures based on that information, and distributes them to users."

[0266] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0267] Step 1: Data Collection

[0268] The server collects harassment-related information from workplace email servers, chat platforms, monitoring systems, and publicly available social media. Input data includes text messages, images, and videos. Specifically, the server periodically accesses each communication medium, retrieves new data, and securely stores it in a database. As output, the collected raw data is sent to the next analysis step.

[0269] Step 2: Data Analysis

[0270] The server applies a natural language processing unit and an image analysis unit to the collected data. The input consists of text and image data collected in the previous step. Here, the text data is analyzed using a natural language processing algorithm to identify aggressive language and negative expressions. Simultaneously, image analysis is utilized to detect nonverbal harassment behavior from video data. Specifically, the text is analyzed using the Python NLTK library, and image recognition is performed using the OpenCV library. The output generates the detected suspicious behavior and its details.

[0271] Step 3: Generating improvement measures

[0272] The server generates improvement measures based on the analysis results. The input is detailed information about the harassment behavior identified in the previous step. Based on behavioral science and similar past cases, a generative AI model creates specific and effective improvement measures. Specifically, the analysis results are input into the generative AI model, and the resulting improvement measures are output.

[0273] Step 4: Distributing the proposal

[0274] The terminal receives improvement suggestions generated from the server. The input is improvement suggestions customized for each user. The terminal device displays these suggestions clearly to the user and runs a notification function to encourage behavioral improvement. The output is the details of the improvement suggestions displayed to the user and the corresponding action steps. In terms of specific actions, the terminal presents suggestions in the form of pop-up alerts or reports.

[0275] Step 5: User Actions

[0276] Based on the improvement suggestions received through the device, users review their behavior and take concrete actions. Inputs include the content of the suggestions and additional resources (e.g., online training modules and workshop information). Users actively utilize these resources provided through the device to improve their harassment behavior. Specifically, users follow the suggestions, attend training, and practice new communication techniques. Expected outputs include improved behavior and enhanced communication skills.

[0277] (Application Example 1)

[0278] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0279] Harassment behavior in the workplace environment is a serious problem that harms the mental and physical health of employees and reduces productivity. In conventional systems, it has been difficult to detect such behavior quickly and accurately and to provide more effective improvement measures. In particular, there is a problem that it is difficult to make a decision with appropriate information in a situation where immediate response at the site is required.

[0280] The specific processing by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is realized by the following respective means.

[0281] In this invention, the server includes means for collecting digital information related to harassment behavior, means for analyzing the collected digital information to identify harassment behavior, and means for generating action improvement measures based on the identified harassment behavior. As a result, it becomes possible to detect real-time problem behavior in the workplace and respond quickly, reduce the stress of employees, and improve productivity.

[0282] "Harassment behavior" refers to aggressive words, actions, or attitudes towards employees in the workplace environment, or actions that damage mental and physical health thereby.

[0283] "Digital information" refers to all data including information such as e-mails, chat messages, and video data generated and collected in communication and business operations within the workplace.

[0284] "Data collection means" refers to a combination of hardware and software for acquiring relevant information from various digital infrastructures within the workplace.

[0285] "Data analysis means" refers to an algorithm or method for identifying harassment behavior using natural language processing technology and image recognition technology based on the collected digital information.

[0286] The "action improvement measures" refer to specific guidelines and procedures that include proposals or policies generated to correct the identified harassment behaviors.

[0287] "Real-time analysis" refers to the process of immediately processing video data and providing rapid information and warnings about the identified behaviors.

[0288] The system for implementing the present invention mainly consists of three elements: a server, a terminal, and a user. The server acquires emails, chat messages, and videos from surveillance cameras from various digital infrastructures within the workplace in order to collect digital information, and also collects as much public data from SNS as possible. In order to analyze these data in real time, the server uses image generation programs such as TensorFlow and OpenCV to detect non-verbal harassment behaviors. Also, through libraries such as spaCy and Transformers, it analyzes text data using natural language processing technology to identify aggressive words and negative emotions.

[0289] As a result of the analysis, the server automatically generates action improvement measures and sends them to the terminal. These improvement measures utilize past case data and findings based on behavioral science, and include specific coping methods and training modules. The terminal used by the user can be various devices such as smart glasses and head-mounted displays. It receives the information sent from the server and displays warnings about problem behaviors and improvement measures to the user in real time. The user is required to review their own behavior based on this and take appropriate actions.

[0290] As a specific example, when an employee takes a coercive attitude, the server analyzes the data and issues a warning informing of the problem. The manager, who is the user, can receive the warning through smart glasses, recognize the problem, and quickly take countermeasures. In this application, the prompt sentence "Please teach me the prompt for detecting harassment behaviors in the workplace and presenting specific improvement proposals that the user (manager) should take based on them" is used for the generative AI model.

[0291] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0292] Step 1:

[0293] The server collects emails, chat messages, surveillance camera footage, and new publicly available social media data from the workplace's digital infrastructure. The input is digital data from each device, which is then prepared for storage on the server. The output forms a digital information storage system for the entire workplace environment.

[0294] Step 2:

[0295] The server analyzes the collected digital information in real time. For text data, natural language processing techniques are used, employing libraries such as spaCy and Transformers to identify aggressive language and negative emotions. The input is extracted text data, and the output generates an index of the identified aggressive language and emotions. For video data, TensorFlow and OpenCV are used to detect nonverbal harassment behavior. Here, the input is video frames, and the output is tagged with the identified problematic behavior.

[0296] Step 3:

[0297] The server automatically generates behavioral improvement measures based on the harassment behaviors identified through analysis. It proposes specific countermeasures and training modules, utilizing past cases and insights from behavioral science. Inputs include indexes and tags from the analysis results, while output is text data of the generated improvement measures.

[0298] Step 4:

[0299] The device displays improvement information received from the server to the user in real time. Warnings and suggestions are displayed visually through smart glasses or similar devices. Input is text data sent from the server, and output is visual notifications and follow-up signals.

[0300] Step 5:

[0301] Users review their actions based on notifications from their devices. Specifically, they recognize problems based on visually presented information and implement suggested behavioral improvements. The action plan that users should take follows the text of the suggested improvements.

[0302] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0303] This invention provides a system that combines an emotion engine to effectively detect and improve harassment behavior in the workplace. This system consists of a server, terminals, users, and an emotion engine.

[0304] Processing performed by the server

[0305] The server collects data from various workplace data sources, such as emails, chat logs, surveillance camera feeds, and publicly available data from social media. It also collects emotional data from audio and video recordings. After preprocessing this data, the server uses natural language processing and image recognition techniques to identify harassing behaviors and problematic emotional expressions.

[0306] The function of the emotional engine

[0307] The emotion engine analyzes the collected voice data and video data to determine the user's emotional state. In the analysis of voice data, emotions are estimated from the tone and volume of speech. In the analysis of video data, facial recognition technology is used to analyze expressions and identify emotions. The emotion data determined by the emotion engine is used for further analysis on the server.

[0308] Improvement proposal generation by the server

[0309] The server automatically generates specific improvement proposals based on the results of data analysis based on harassment behavior and emotional states. This proposal includes guidelines on how the user should control their emotions and engage in constructive communication.

[0310] The role of the terminal

[0311] The terminal is a device that delivers the improvement proposals sent from the server to the user. The terminal displays the improvement proposals in a form that the user can intuitively understand and proposes an action plan to promote behavior improvement.

[0312] The user's reaction and actions

[0313] The user implements specific behavior improvement measures based on the information provided via the terminal. If necessary, additional training resources and support are utilized to improve emotional management capabilities and correct harassment behavior.

[0314] For example, in a certain workplace, if Mr. B makes inappropriate remarks during a meeting, this system analyzes the nature of the remarks and Mr. B's emotional state. The server generates an improvement proposal that takes into account the stress indicators detected by the emotion engine along with the identification of the problem behavior. This proposal is provided to Mr. B through the terminal, and Mr. B can learn how to correct his own behavior based on this.

[0315] The processing flow will be described below.

[0316] Step 1:

[0317] The server collects information from email, chat logs, surveillance camera footage, and publicly available data from social media via the corporate network. During this process, it adheres to security and privacy policies and performs necessary access authentication.

[0318] Step 2:

[0319] The server preprocesses the collected data. Text data is tokenized and converted into a format that can be grammatically analyzed, and video data is extracted frame by frame. For audio data, noise is removed and the characteristics of the audio waveform are extracted.

[0320] Step 3:

[0321] The emotion engine analyzes audio and video data to determine the user's emotions. For audio, it analyzes voice tone and pitch to assess the level of stress and anger in adverse situations. For video, it uses facial recognition technology to capture subtle changes in facial expressions and identify emotions.

[0322] Step 4:

[0323] The server integrates and analyzes pre-processed data and the results of the emotion engine. It uses natural language processing techniques to check text for offensive or negative words and image recognition techniques to detect nonverbal harassment behavior in videos.

[0324] Step 5:

[0325] Based on the analysis results, the server generates improvement suggestions tailored to each individual user. From the analysis, which includes emotional states, it creates specific advice on how users should improve their behavior and control their emotions.

[0326] Step 6:

[0327] The terminal receives improvement suggestions generated from the server and presents them to the user. The improvement suggestions are displayed in a visualized interface and provided in a way that is easy for the user to understand intuitively.

[0328] Step 7:

[0329] Users implement improvement suggestions received via their devices. They review their behavior and manage their emotions based on the suggestions. They also utilize additional support materials and applications as needed to achieve continuous improvement.

[0330] (Example 2)

[0331] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0332] Harassment is a serious problem in the workplace, and prompt detection and corrective measures are required. Traditional methods have faced challenges in effective countermeasures due to fragmented data collection and analysis, and an inability to fully grasp emotional states. Furthermore, there was a need for a system that could accurately grasp emotional changes, quickly generate concrete improvement proposals based on those findings, and provide them to users.

[0333] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0334] In this invention, the server includes means for collecting information relating to harassment behavior, means for analyzing the collected information to identify harassment behavior, and means for analyzing emotions from audio data and video data. This makes it possible to accurately detect harassment behavior in the workplace environment and quickly provide users with effective improvement suggestions for taking appropriate corrective measures.

[0335] "Harassment behavior" refers to actions that cause psychological or physical discomfort or harm to others in social settings such as the workplace.

[0336] "Information" refers to facts and data expressed in any form, including data collected from emails, chats, audio recordings, video recordings, and social networking services.

[0337] "Analyzing emotions" refers to the process of determining and identifying the emotional state expressed in collected audio and video data.

[0338] "Improvement suggestions" refer to content that provides guidelines and action plans for the individual to improve their behavior and emotions, as well as their communication, based on identified harassment behaviors and analyzed emotions.

[0339] "Users" refers to individuals or groups who receive improvement suggestions provided through this system with the aim of improving their own behavior and emotions.

[0340] This invention is a system for effectively detecting and improving harassment behavior in the workplace environment. The system consists of a server, terminals, users, and an emotion engine.

[0341] The server collects data from emails, chats, surveillance camera footage, and social networking services. Audio recordings are also included. The server preprocesses the collected data and analyzes it using natural language processing and image recognition technologies. The hardware used includes a high-performance database server, and the software utilizes natural language processing libraries and image recognition algorithms. Through this analysis, the server identifies harassment behaviors and users' emotional states.

[0342] The emotion engine analyzes audio and video data to determine the user's emotional state. It identifies emotions by analyzing speech tone and volume from audio data, and facial expressions from video data. This resulting emotional data is then further analyzed by the server.

[0343] Improvement suggestions are automatically generated based on the analysis results and sent to the terminal by the server. The terminal displays the improvement suggestions from the server to the user in an easy-to-understand format. This allows the user to obtain concrete actionable improvement measures. The terminal has the function of providing information in a format that is easy for the user to understand intuitively.

[0344] For example, if the server detects inappropriate remarks in a particular meeting, it will generate improvement suggestions, taking into account the emotional state behind those remarks. These suggestions will be displayed on the terminal, showing the user how to correct their behavior.

[0345] By using a generative AI model, the server can improve the accuracy of its information analysis and respond flexibly to user needs. An example of a prompt message would be, "Please describe a system that automatically generates suggestions for improving harassment behavior in the workplace. Please explain in detail, including specific data processing methods using an emotion engine and natural language processing technology." This input optimizes the system's operation.

[0346] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0347] Step 1:

[0348] The server collects information from various data sources in the work environment. Inputs include email, chat history, surveillance camera footage, and data from social networks. The output is the result of this raw data collection, which is stored in a database. This process integrates the data into an appropriate format and prepares it for subsequent processing.

[0349] Step 2:

[0350] The server preprocesses the collected data. The input includes various data collected in Step 1. Data processing, such as data cleaning, noise reduction, and data format standardization, is performed to obtain clean, analyzable data as output. This process prepares the information necessary to improve the accuracy of the analysis.

[0351] Step 3:

[0352] The server performs emotion analysis using an emotion engine. The input includes clean audio and video data obtained in step 2. For the audio data, the tone and volume of speech are analyzed, and for the video data, facial recognition is used to analyze facial expressions and identify emotions. The output is data of the identified emotional state, which is used in the next analysis step.

[0353] Step 4:

[0354] The server uses natural language processing and image recognition technologies to perform analysis to identify harassment behaviors. The input includes text data obtained in step 2 and sentiment data obtained in step 3. The processing analyzes the text content and related sentiment indicators to determine whether a behavior constitutes harassment. The output is data representing the evaluation results for each behavior.

[0355] Step 5:

[0356] The server automatically generates improvement suggestions based on identified behavioral assessments and emotional data. The input includes the analysis results from step 4. This step generates specific suggestions on how to manage emotions and engage in constructive communication. This provides guidance for necessary behavioral improvements.

[0357] Step 6:

[0358] The terminal delivers improvement suggestions sent from the server to the user. The input includes improvement suggestions automatically generated in step 5. The terminal displays these suggestions visually and clearly to the user, making them intuitively understandable. The output is the information presented to the user.

[0359] Step 7:

[0360] Users improve their behavior based on improvement suggestions provided through their devices. They implement specific behavioral improvements and utilize additional resources and training as needed. This aims to improve emotional management and correct harassing behaviors. The output is the user's improved behavior and its effects.

[0361] (Application Example 2)

[0362] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0363] When harassment occurs in the workplace, it can significantly impair employees' mental health and work efficiency. Traditional methods have struggled to quickly recognize and correct harassment, making immediate action to prevent the problem from escalating difficult. Therefore, there is a need for a system that can analyze emotional states in real time, detect inappropriate behavior early, and provide appropriate improvement suggestions.

[0364] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0365] In this invention, the server includes means for collecting information on harassment behavior, means for analyzing the collected information to identify harassment behavior, means for generating improvement suggestions based on the identified harassment behavior, and means for detecting inappropriate behavior in real time, including an emotion engine that analyzes emotions from the user's voice and facial expressions. This makes it possible to quickly and effectively detect harassment behavior in the workplace and provide specific improvement measures.

[0366] "Harassment behavior" refers to inappropriate or offensive actions or remarks in the workplace that harm the mental health of employees or the work environment.

[0367] "Means of information gathering" refers to processes that include various technologies and equipment for collecting relevant data from data sources within the workplace.

[0368] "Means of analyzing information" refers to techniques that use collected data to perform analyses in order to identify harassment behaviors.

[0369] "Means for generating improvement suggestions" refers to algorithms or software used to propose appropriate countermeasures or behavioral modifications for identified harassment behaviors.

[0370] An "emotion engine" is a technology that analyzes an individual's emotional state from data such as voice and facial expressions, and detects changes in emotions in real time.

[0371] A "means for detecting inappropriate behavior" refers to a mechanism that identifies problematic or risky behaviors in the workplace in real time, based on the results of sentiment analysis.

[0372] The system for implementing the present invention mainly consists of a server, terminals, and users. The server has the function of collecting information from various data sources within the work environment, and this information includes text information, audio data, and image data. Specifically, it receives data from emails, chat records, video from surveillance systems, and audio recordings.

[0373] The collected information is analyzed on the server using natural language processing and visual pattern recognition technologies. First, textual information is analyzed by natural language processing algorithms to evaluate the possibility of harassment based on specific keywords and context. Next, from image data, facial recognition technology and an emotion engine are used to identify emotional states from facial expressions. For audio data, speech analysis technology is used to infer emotions from factors such as tone and volume of voice. This allows for sensitive detection of emotional changes in real time.

[0374] The terminal serves to communicate improvement suggestions generated from the server to the user. Through the terminal, the user receives specific suggestions for behavioral improvement and uses them to improve the situation. For example, if a user shows strong frustration during a meeting, the terminal will immediately display suggestions for calming down. These include specific actions such as leaving the room or taking deep breaths.

[0375] An example of a prompt using a generative AI model is, "Please check what emotional state was detected from this data." Another prompt used is, "Please generate specific behavioral improvement measures based on the emotion analysis results." These prompts enable the generation of more flexible and appropriate improvement suggestions.

[0376] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0377] Step 1:

[0378] The server collects information from various data sources within the work environment. It receives emails, chat logs, video and audio recordings from monitoring systems as input. This data is temporarily stored in storage as raw data.

[0379] Step 2:

[0380] The server preprocesses the collected raw data. Unnecessary tags and special characters are removed from the input text information, and audio data is converted to text. Image data is also converted to a standard format, and noise is removed to produce clean analytical data as output.

[0381] Step 3:

[0382] The server applies natural language processing techniques to the pre-processed text information. This involves extracting keywords and performing sentiment analysis from the input text. This identifies potentially harassing phrases, and the relevant sections are output as analysis results.

[0383] Step 4:

[0384] The server applies face recognition technology and an emotion engine to pre-processed image data. It recognizes individual faces from the input image data and identifies emotional states through facial expression analysis. The obtained emotional state information is output as the analysis result.

[0385] Step 5:

[0386] The server applies speech analysis technology to the audio data. It analyzes the tone and volume of the input audio and estimates the emotional state. The estimated emotional information is output as the analysis result.

[0387] Step 6:

[0388] The server integrates the analysis results to identify harassment behaviors and generate necessary improvement suggestions. Here, a generative AI model is used to create prompt messages based on the input analysis results and output behavioral improvement measures.

[0389] Step 7:

[0390] The terminal receives improvement suggestions sent from the server and displays them to the user in an intuitively understandable format. Based on the suggestions received as input, it outputs visualized action plans and presents specific improvement actions.

[0391] Step 8:

[0392] Users implement improvement suggestions displayed on their devices. Following these suggestions, they might, for example, learn communication techniques to avoid misunderstandings and adjust their behavior according to the situation. This manifests as actual changes in behavior in the output.

[0393] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0394] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0395] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0396] [Third Embodiment]

[0397] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0398] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0399] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0400] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0401] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0403] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0404] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0405] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0406] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0407] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0408] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0409] This invention relates to a system for appropriately detecting harassment behavior in the workplace and proposing effective corrective measures. This system consists of a server, terminals, and users.

[0410] Processing performed by the server

[0411] The server collects various data from the company's digital infrastructure. Specifically, it obtains information from emails, chat messages, and surveillance camera footage. In addition, it can also collect publicly available data from social media. The server then analyzes the collected data. This analysis uses natural language processing technology to scrutinize message content and identify aggressive behavior and negative emotions. It also uses image recognition technology to detect nonverbal harassment behavior from video footage. Based on the analysis results, the server automatically generates improvement suggestions. This includes utilizing a database of past cases and insights based on behavioral science.

[0412] Processing performed by the terminal

[0413] The terminal is a device used by the user and is responsible for receiving improvement suggestions sent from the server. The terminal provides an interface to display suggestions to the user and prompt action. Suggestions can be presented in various formats (e.g., text-based reports and visual content) to allow users to easily access the information they need.

[0414] User-performed processes

[0415] Based on improvement suggestions received via the device, users review their own behavior and take specific actions to improve it. If necessary, they can utilize additional resources provided by the server or device (e.g., training modules, workshop information) to correct harassing behavior.

[0416] As a concrete example, if communication between employee A and a subordinate is deemed problematic in a workplace, the server analyzes A's email history and video footage from meetings to detect overbearing attitudes and aggressive language. Next, based on this information, the server generates improvement suggestions for A, recommending modules that introduce communication skills improvement and appropriate teaching methods. These suggestions are provided to A via their device, allowing them to use them to improve their behavior.

[0417] The following describes the processing flow.

[0418] Step 1:

[0419] The server collects data potentially related to harassment behavior from sources such as the company's email system, chat tools, surveillance cameras, and public social media data. It is programmed to comply with access control and privacy policies during this process.

[0420] Step 2:

[0421] The server preprocesses the collected data. Specifically, text data is tokenized and unwanted noise is removed. For video data, frames are extracted and converted into a format suitable for analysis.

[0422] Step 3:

[0423] The server analyzes pre-processed data. Using natural language processing techniques, it performs sentiment analysis on text to identify negative or aggressive language. Simultaneously, it uses image recognition techniques to detect nonverbal problematic behaviors from video.

[0424] Step 4:

[0425] The server generates improvement suggestions based on the analysis results. Through analysis of similar cases based on past database data, it forms appropriate improvement measures and guidelines for behavioral change. These suggestions include detailed advice for specific behavioral improvements and skill enhancements.

[0426] Step 5:

[0427] The terminal receives improvement suggestions sent from the server and presents them to the user through the user interface. Visuals and audio guides are used to display the information in a way that is easy for the user to understand.

[0428] Step 6:

[0429] Users refer to suggestions from their devices and take the suggested actions. If necessary, they can access training videos and guidelines included in the suggestions to improve their behavior.

[0430] (Example 1)

[0431] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0432] Harassment in the workplace negatively impacts organizational health and productivity. However, systems for early identification of harassment and implementation of appropriate corrective measures are not adequately in place. Therefore, effective means for the early detection and resolution of harassment are needed.

[0433] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0434] In this invention, the server includes means for collecting harassment-related information from a communication medium, means for analyzing the collected information to identify harassment behavior, and means for generating corrective measures for the identified harassment behavior. This makes it possible to quickly identify harassment behavior and present effective corrective measures.

[0435] A "communication medium" is an electronic or optical means used to send and receive digital information.

[0436] "Harassment-related information" refers to data concerning aggressive or inappropriate behavior towards individuals or organizations.

[0437] "Analysis" is the process of analyzing collected data to extract and evaluate specific information.

[0438] "Harassment" refers to any behavior that causes psychological or physical discomfort to an individual or group.

[0439] A "solution" is a set of actionable steps proposed to resolve an identified problem.

[0440] A "terminal device" is a device used by a user to receive or display digital information.

[0441] A "user" is an individual or organization that receives information and services from a system and utilizes them.

[0442] A "natural language processing unit" is a computer system designed to analyze and understand human language.

[0443] An "image analysis device" is a computer system designed to extract and recognize information from still images or videos.

[0444] This invention is a system for efficiently detecting and addressing harassment in the workplace. The system consists of a server, terminal devices, and users, and in order to implement the invention, it is necessary to specifically understand the role and technology of each component.

[0445] Server roles and configuration

[0446] The server functions as the central processing unit of this system. The server collects information from workplace emails, chat systems, monitoring systems, and publicly available social networking services (SNS) through communication media. For analyzing the collected information, it uses the Python NLTK library as a natural language processing unit and the OpenCV library as an image analysis unit. Furthermore, the server automatically generates appropriate improvement measures based on the collected and analyzed information using a generative AI model. These improvement measures are based on a database of past cases and insights from behavioral science.

[0447] Role and function of terminal devices

[0448] The terminal device functions as a device for exchanging information between the user and the server. It receives improvement suggestions sent from the server and presents them to the user in an easy-to-understand manner. This presentation is provided in various report formats. The terminal device has the ability to display pop-up notifications and detailed information to help the user easily accept the improvement suggestions.

[0449] User roles and responses

[0450] Users are required to review their own behavior based on the improvement measures received via the terminal device and take specific actions to improve it. If necessary, the terminal will provide additional resources such as training modules and workshop information, which users can utilize to improve their harassment behavior.

[0451] For example, if a user is identified as having communication problems at work, the server analyzes the user's email history and meeting videos. This detects aggressive attitudes and negative expressions, and based on this information, improvement measures are generated. The user receives these improvement measures through a terminal device and can take training modules to improve their communication skills.

[0452] An example of a prompt to input into the generating AI model would be: "Develop a system that analyzes email and chat history within the workplace, identifies aggressive behavior, generates improvement measures based on that information, and distributes them to users."

[0453] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0454] Step 1: Data Collection

[0455] The server collects harassment-related information from workplace email servers, chat platforms, monitoring systems, and publicly available social media. Input data includes text messages, images, and videos. Specifically, the server periodically accesses each communication medium, retrieves new data, and securely stores it in a database. As output, the collected raw data is sent to the next analysis step.

[0456] Step 2: Data Analysis

[0457] The server applies a natural language processing unit and an image analysis unit to the collected data. The input consists of text and image data collected in the previous step. Here, the text data is analyzed using a natural language processing algorithm to identify aggressive language and negative expressions. Simultaneously, image analysis is utilized to detect nonverbal harassment behavior from video data. Specifically, the text is analyzed using the Python NLTK library, and image recognition is performed using the OpenCV library. The output generates the detected suspicious behavior and its details.

[0458] Step 3: Generating improvement measures

[0459] The server generates improvement measures based on the analysis results. The input is detailed information about the harassment behavior identified in the previous step. Based on behavioral science and similar past cases, a generative AI model creates specific and effective improvement measures. Specifically, the analysis results are input into the generative AI model, and the resulting improvement measures are output.

[0460] Step 4: Distributing the proposal

[0461] The terminal receives improvement suggestions generated from the server. The input is improvement suggestions customized for each user. The terminal device displays these suggestions clearly to the user and runs a notification function to encourage behavioral improvement. The output is the details of the improvement suggestions displayed to the user and the corresponding action steps. In terms of specific actions, the terminal presents suggestions in the form of pop-up alerts or reports.

[0462] Step 5: User Actions

[0463] Based on the improvement suggestions received through the device, users review their behavior and take concrete actions. Inputs include the content of the suggestions and additional resources (e.g., online training modules and workshop information). Users actively utilize these resources provided through the device to improve their harassment behavior. Specifically, users follow the suggestions, attend training, and practice new communication techniques. Expected outputs include improved behavior and enhanced communication skills.

[0464] (Application Example 1)

[0465] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0466] Harassment in the workplace is a serious problem that harms employees' mental and physical health and reduces productivity. Traditional systems have struggled to quickly and accurately detect such behavior and provide effective countermeasures. In particular, there is a challenge in making informed decisions in situations requiring immediate on-site response.

[0467] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0468] In this invention, the server includes means for collecting digital information relating to harassment behavior, means for analyzing the collected digital information to identify harassment behavior, and means for generating behavioral improvement measures based on the identified harassment behavior. This enables real-time detection and rapid response to problematic behavior in the workplace, reduces employee stress, and improves productivity.

[0469] "Harassment behavior" refers to aggressive words or actions towards employees in the workplace, or any behavior that harms their mental or physical health.

[0470] "Digital information" refers to all data, including emails, chat messages, and video data, that are generated and collected within the workplace for communication and work performance.

[0471] "Data collection means" refers to a combination of hardware and software used to acquire relevant information from various digital infrastructures within the workplace.

[0472] "Data analysis means" refers to algorithms or methods for identifying harassment behaviors based on collected digital information, using natural language processing technology and image recognition technology.

[0473] "Behavioral improvement measures" refer to specific guidelines or procedures, including suggestions or policies, that are generated to correct identified harassment behaviors.

[0474] "Real-time analysis" refers to the process of instantly processing video data and providing rapid information and warnings about identified behaviors.

[0475] The system for implementing this invention mainly consists of three elements: a server, a terminal, and a user. The server collects digital information by acquiring emails, chat messages, and surveillance camera footage from various digital infrastructures within the workplace, and also collects publicly available data from social networking services (SNS) as much as possible. To analyze this data in real time, the server uses image generation programs such as TensorFlow and OpenCV to detect nonverbal harassment behavior. It also analyzes text data using natural language processing techniques through libraries such as spaCy and Transformers to identify aggressive words and negative emotions.

[0476] Based on the analysis, the server automatically generates behavioral improvement measures and sends them to the terminal. These measures utilize past case data and insights based on behavioral science, and include specific countermeasures and training modules. The terminals used by users can be various devices such as smart glasses or head-mounted displays, which receive information sent from the server and display warnings of problematic behaviors and improvement measures in the user's field of vision in real time. Based on this, users are expected to review their own behavior and take appropriate action.

[0477] As a concrete example, if an employee exhibits aggressive behavior, the server analyzes the data and issues a warning to highlight the problem. The administrator, acting as the user, receives the warning through smart glasses, recognizes the issue, and can quickly take corrective action. In this application, the prompt used for the generative AI model is: "Please provide prompts to detect harassment behavior in the workplace and, based on that, suggest specific improvement measures that the user (administrator) should take."

[0478] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0479] Step 1:

[0480] The server collects emails, chat messages, surveillance camera footage, and new publicly available social media data from the workplace's digital infrastructure. The input is digital data from each device, which is then prepared for storage on the server. The output forms a digital information storage system for the entire workplace environment.

[0481] Step 2:

[0482] The server analyzes the collected digital information in real time. For text data, natural language processing techniques are used, employing libraries such as spaCy and Transformers to identify aggressive language and negative emotions. The input is extracted text data, and the output generates an index of the identified aggressive language and emotions. For video data, TensorFlow and OpenCV are used to detect nonverbal harassment behavior. Here, the input is video frames, and the output is tagged with the identified problematic behavior.

[0483] Step 3:

[0484] The server automatically generates behavioral improvement measures based on the harassment behaviors identified through analysis. It proposes specific countermeasures and training modules, utilizing past cases and insights from behavioral science. Inputs include indexes and tags from the analysis results, while output is text data of the generated improvement measures.

[0485] Step 4:

[0486] The device displays improvement information received from the server to the user in real time. Warnings and suggestions are displayed visually through smart glasses or similar devices. Input is text data sent from the server, and output is visual notifications and follow-up signals.

[0487] Step 5:

[0488] Users review their actions based on notifications from their devices. Specifically, they recognize problems based on visually presented information and implement suggested behavioral improvements. The action plan that users should take follows the text of the suggested improvements.

[0489] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0490] This invention provides a system that combines an emotion engine to effectively detect and improve harassment behavior in the workplace. This system consists of a server, terminals, users, and an emotion engine.

[0491] Processing performed by the server

[0492] The server collects data from various workplace data sources, such as emails, chat logs, surveillance camera feeds, and publicly available data from social media. It also collects emotional data from audio and video recordings. After preprocessing this data, the server uses natural language processing and image recognition techniques to identify harassing behaviors and problematic emotional expressions.

[0493] The function of the emotional engine

[0494] The emotion engine analyzes collected audio and video data to determine the user's emotional state. For audio data analysis, it estimates emotions from the tone and volume of speech. For video data analysis, it identifies emotions by analyzing facial expressions based on facial recognition technology. The emotional data determined by the emotion engine is used for further analysis on the server.

[0495] Server-based generation of improvement suggestions

[0496] The server automatically generates specific improvement suggestions based on data analysis of harassment behaviors and emotional states. These suggestions include guidance on how users should control their emotions and engage in constructive communication.

[0497] Terminal role

[0498] The terminal is a device that delivers improvement suggestions sent from the server to the user. The terminal displays the improvement suggestions to the user in an intuitively understandable format and proposes an action plan to encourage improvement in behavior.

[0499] User reactions and actions

[0500] Users will implement specific behavioral improvement measures based on the information provided via their devices. If necessary, they will utilize additional training resources and support to improve their emotional management skills and correct harassing behaviors.

[0501] For example, if person B makes an inappropriate remark during a meeting at a certain workplace, this system analyzes the nature of the remark and person B's emotional state. The server identifies the behavior as problematic and generates improvement suggestions that take into account the stress indicators detected by the emotion engine. These suggestions are provided to person B via a terminal, and based on them, person B can learn how to correct their behavior.

[0502] The following describes the processing flow.

[0503] Step 1:

[0504] The server collects information from email, chat logs, surveillance camera footage, and publicly available data from social media via the corporate network. During this process, it adheres to security and privacy policies and performs necessary access authentication.

[0505] Step 2:

[0506] The server preprocesses the collected data. Text data is tokenized and converted into a format that can be grammatically analyzed, and video data is extracted frame by frame. For audio data, noise is removed and the characteristics of the audio waveform are extracted.

[0507] Step 3:

[0508] The emotion engine analyzes audio and video data to determine the user's emotions. For audio, it analyzes voice tone and pitch to assess the level of stress and anger in adverse situations. For video, it uses facial recognition technology to capture subtle changes in facial expressions and identify emotions.

[0509] Step 4:

[0510] The server integrates and analyzes pre-processed data and the results of the emotion engine. It uses natural language processing techniques to check text for offensive or negative words and image recognition techniques to detect nonverbal harassment behavior in videos.

[0511] Step 5:

[0512] Based on the analysis results, the server generates improvement suggestions tailored to each individual user. From the analysis, which includes emotional states, it creates specific advice on how users should improve their behavior and control their emotions.

[0513] Step 6:

[0514] The terminal receives improvement suggestions generated from the server and presents them to the user. The improvement suggestions are displayed in a visualized interface and provided in a way that is easy for the user to understand intuitively.

[0515] Step 7:

[0516] Users implement improvement suggestions received via their devices. They review their behavior and manage their emotions based on the suggestions. They also utilize additional support materials and applications as needed to achieve continuous improvement.

[0517] (Example 2)

[0518] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0519] Harassment is a serious problem in the workplace, and prompt detection and corrective measures are required. Traditional methods have faced challenges in effective countermeasures due to fragmented data collection and analysis, and an inability to fully grasp emotional states. Furthermore, there was a need for a system that could accurately grasp emotional changes, quickly generate concrete improvement proposals based on those findings, and provide them to users.

[0520] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0521] In this invention, the server includes means for collecting information relating to harassment behavior, means for analyzing the collected information to identify harassment behavior, and means for analyzing emotions from audio data and video data. This makes it possible to accurately detect harassment behavior in the workplace environment and quickly provide users with effective improvement suggestions for taking appropriate corrective measures.

[0522] "Harassment behavior" refers to actions that cause psychological or physical discomfort or harm to others in social settings such as the workplace.

[0523] "Information" refers to facts and data expressed in any form, including data collected from emails, chats, audio recordings, video recordings, and social networking services.

[0524] "Analyzing emotions" refers to the process of determining and identifying the emotional state expressed in collected audio and video data.

[0525] "Improvement suggestions" refer to content that provides guidelines and action plans for the individual to improve their behavior and emotions, as well as their communication, based on identified harassment behaviors and analyzed emotions.

[0526] "Users" refers to individuals or groups who receive improvement suggestions provided through this system with the aim of improving their own behavior and emotions.

[0527] This invention is a system for effectively detecting and improving harassment behavior in the workplace environment. The system consists of a server, terminals, users, and an emotion engine.

[0528] The server collects data from emails, chats, surveillance camera footage, and social networking services. Audio recordings are also included. The server preprocesses the collected data and analyzes it using natural language processing and image recognition technologies. The hardware used includes a high-performance database server, and the software utilizes natural language processing libraries and image recognition algorithms. Through this analysis, the server identifies harassment behaviors and users' emotional states.

[0529] The emotion engine analyzes audio and video data to determine the user's emotional state. It identifies emotions by analyzing speech tone and volume from audio data, and facial expressions from video data. This resulting emotional data is then further analyzed by the server.

[0530] Improvement suggestions are automatically generated based on the analysis results and sent to the terminal by the server. The terminal displays the improvement suggestions from the server to the user in an easy-to-understand format. This allows the user to obtain concrete actionable improvement measures. The terminal has the function of providing information in a format that is easy for the user to understand intuitively.

[0531] For example, if the server detects inappropriate remarks in a particular meeting, it will generate improvement suggestions, taking into account the emotional state behind those remarks. These suggestions will be displayed on the terminal, showing the user how to correct their behavior.

[0532] By using a generative AI model, the server can improve the accuracy of its information analysis and respond flexibly to user needs. An example of a prompt message would be, "Please describe a system that automatically generates suggestions for improving harassment behavior in the workplace. Please explain in detail, including specific data processing methods using an emotion engine and natural language processing technology." This input optimizes the system's operation.

[0533] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0534] Step 1:

[0535] The server collects information from various data sources in the work environment. Inputs include email, chat history, surveillance camera footage, and data from social networks. The output is the result of this raw data collection, which is stored in a database. This process integrates the data into an appropriate format and prepares it for subsequent processing.

[0536] Step 2:

[0537] The server preprocesses the collected data. The input includes various data collected in Step 1. Data processing, such as data cleaning, noise reduction, and data format standardization, is performed to obtain clean, analyzable data as output. This process prepares the information necessary to improve the accuracy of the analysis.

[0538] Step 3:

[0539] The server performs emotion analysis using an emotion engine. The input includes clean audio and video data obtained in step 2. For the audio data, the tone and volume of speech are analyzed, and for the video data, facial recognition is used to analyze facial expressions and identify emotions. The output is data of the identified emotional state, which is used in the next analysis step.

[0540] Step 4:

[0541] The server uses natural language processing and image recognition technologies to perform analysis to identify harassment behaviors. The input includes text data obtained in step 2 and sentiment data obtained in step 3. The processing analyzes the text content and related sentiment indicators to determine whether a behavior constitutes harassment. The output is data representing the evaluation results for each behavior.

[0542] Step 5:

[0543] The server automatically generates improvement suggestions based on identified behavioral assessments and emotional data. The input includes the analysis results from step 4. This step generates specific suggestions on how to manage emotions and engage in constructive communication. This provides guidance for necessary behavioral improvements.

[0544] Step 6:

[0545] The terminal delivers improvement suggestions sent from the server to the user. The input includes improvement suggestions automatically generated in step 5. The terminal displays these suggestions visually and clearly to the user, making them intuitively understandable. The output is the information presented to the user.

[0546] Step 7:

[0547] Users improve their behavior based on improvement suggestions provided through their devices. They implement specific behavioral improvements and utilize additional resources and training as needed. This aims to improve emotional management and correct harassing behaviors. The output is the user's improved behavior and its effects.

[0548] (Application Example 2)

[0549] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0550] When harassment occurs in the workplace, it can significantly impair employees' mental health and work efficiency. Traditional methods have struggled to quickly recognize and correct harassment, making immediate action to prevent the problem from escalating difficult. Therefore, there is a need for a system that can analyze emotional states in real time, detect inappropriate behavior early, and provide appropriate improvement suggestions.

[0551] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0552] In this invention, the server includes means for collecting information on harassment behavior, means for analyzing the collected information to identify harassment behavior, means for generating improvement suggestions based on the identified harassment behavior, and means for detecting inappropriate behavior in real time, including an emotion engine that analyzes emotions from the user's voice and facial expressions. This makes it possible to quickly and effectively detect harassment behavior in the workplace and provide specific improvement measures.

[0553] "Harassment behavior" refers to inappropriate or offensive actions or remarks in the workplace that harm the mental health of employees or the work environment.

[0554] "Means of information gathering" refers to processes that include various technologies and equipment for collecting relevant data from data sources within the workplace.

[0555] "Means of analyzing information" refers to techniques that use collected data to perform analyses in order to identify harassment behaviors.

[0556] "Means for generating improvement suggestions" refers to algorithms or software used to propose appropriate countermeasures or behavioral modifications for identified harassment behaviors.

[0557] An "emotion engine" is a technology that analyzes an individual's emotional state from data such as voice and facial expressions, and detects changes in emotions in real time.

[0558] A "means for detecting inappropriate behavior" refers to a mechanism that identifies problematic or risky behaviors in the workplace in real time, based on the results of sentiment analysis.

[0559] The system for implementing the present invention mainly consists of a server, terminals, and users. The server has the function of collecting information from various data sources within the work environment, and this information includes text information, audio data, and image data. Specifically, it receives data from emails, chat records, video from surveillance systems, and audio recordings.

[0560] The collected information is analyzed on the server using natural language processing and visual pattern recognition technologies. First, textual information is analyzed by natural language processing algorithms to evaluate the possibility of harassment based on specific keywords and context. Next, from image data, facial recognition technology and an emotion engine are used to identify emotional states from facial expressions. For audio data, speech analysis technology is used to infer emotions from factors such as tone and volume of voice. This allows for sensitive detection of emotional changes in real time.

[0561] The terminal serves to communicate improvement suggestions generated from the server to the user. Through the terminal, the user receives specific suggestions for behavioral improvement and uses them to improve the situation. For example, if a user shows strong frustration during a meeting, the terminal will immediately display suggestions for calming down. These include specific actions such as leaving the room or taking deep breaths.

[0562] An example of a prompt using a generative AI model is, "Please check what emotional state was detected from this data." Another prompt used is, "Please generate specific behavioral improvement measures based on the emotion analysis results." These prompts enable the generation of more flexible and appropriate improvement suggestions.

[0563] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0564] Step 1:

[0565] The server collects information from various data sources within the work environment. It receives emails, chat logs, video and audio recordings from monitoring systems as input. This data is temporarily stored in storage as raw data.

[0566] Step 2:

[0567] The server preprocesses the collected raw data. Unnecessary tags and special characters are removed from the input text information, and audio data is converted to text. Image data is also converted to a standard format, and noise is removed to produce clean analytical data as output.

[0568] Step 3:

[0569] The server applies natural language processing techniques to the pre-processed text information. This involves extracting keywords and performing sentiment analysis from the input text. This identifies potentially harassing phrases, and the relevant sections are output as analysis results.

[0570] Step 4:

[0571] The server applies face recognition technology and an emotion engine to pre-processed image data. It recognizes individual faces from the input image data and identifies emotional states through facial expression analysis. The obtained emotional state information is output as the analysis result.

[0572] Step 5:

[0573] The server applies speech analysis technology to the audio data. It analyzes the tone and volume of the input audio and estimates the emotional state. The estimated emotional information is output as the analysis result.

[0574] Step 6:

[0575] The server integrates the analysis results to identify harassment behaviors and generate necessary improvement suggestions. Here, a generative AI model is used to create prompt messages based on the input analysis results and output behavioral improvement measures.

[0576] Step 7:

[0577] The terminal receives improvement suggestions sent from the server and displays them to the user in an intuitively understandable format. Based on the suggestions received as input, it outputs visualized action plans and presents specific improvement actions.

[0578] Step 8:

[0579] Users implement improvement suggestions displayed on their devices. Following these suggestions, they might, for example, learn communication techniques to avoid misunderstandings and adjust their behavior according to the situation. This manifests as actual changes in behavior in the output.

[0580] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0581] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0582] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0583] [Fourth Embodiment]

[0584] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0585] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0586] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0587] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0588] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0589] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0590] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0591] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0592] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0593] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0594] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0595] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0596] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0597] This invention relates to a system for appropriately detecting harassment behavior in the workplace and proposing effective corrective measures. This system consists of a server, terminals, and users.

[0598] Processing performed by the server

[0599] The server collects various data from the company's digital infrastructure. Specifically, it obtains information from emails, chat messages, and surveillance camera footage. In addition, it can also collect publicly available data from social media. The server then analyzes the collected data. This analysis uses natural language processing technology to scrutinize message content and identify aggressive behavior and negative emotions. It also uses image recognition technology to detect nonverbal harassment behavior from video footage. Based on the analysis results, the server automatically generates improvement suggestions. This includes utilizing a database of past cases and insights based on behavioral science.

[0600] Processing performed by the terminal

[0601] The terminal is a device used by the user and is responsible for receiving improvement suggestions sent from the server. The terminal provides an interface to display suggestions to the user and prompt action. Suggestions can be presented in various formats (e.g., text-based reports and visual content) to allow users to easily access the information they need.

[0602] User-performed processes

[0603] Based on improvement suggestions received via the device, users review their own behavior and take specific actions to improve it. If necessary, they can utilize additional resources provided by the server or device (e.g., training modules, workshop information) to correct harassing behavior.

[0604] As a concrete example, if communication between employee A and a subordinate is deemed problematic in a workplace, the server analyzes A's email history and video footage from meetings to detect overbearing attitudes and aggressive language. Next, based on this information, the server generates improvement suggestions for A, recommending modules that introduce communication skills improvement and appropriate teaching methods. These suggestions are provided to A via their device, allowing them to use them to improve their behavior.

[0605] The following describes the processing flow.

[0606] Step 1:

[0607] The server collects data potentially related to harassment behavior from sources such as the company's email system, chat tools, surveillance cameras, and public social media data. It is programmed to comply with access control and privacy policies during this process.

[0608] Step 2:

[0609] The server preprocesses the collected data. Specifically, text data is tokenized and unwanted noise is removed. For video data, frames are extracted and converted into a format suitable for analysis.

[0610] Step 3:

[0611] The server analyzes pre-processed data. Using natural language processing techniques, it performs sentiment analysis on text to identify negative or aggressive language. Simultaneously, it uses image recognition techniques to detect nonverbal problematic behaviors from video.

[0612] Step 4:

[0613] The server generates improvement suggestions based on the analysis results. Through analysis of similar cases based on past database data, it forms appropriate improvement measures and guidelines for behavioral change. These suggestions include detailed advice for specific behavioral improvements and skill enhancements.

[0614] Step 5:

[0615] The terminal receives improvement suggestions sent from the server and presents them to the user through the user interface. Visuals and audio guides are used to display the information in a way that is easy for the user to understand.

[0616] Step 6:

[0617] Users refer to suggestions from their devices and take the suggested actions. If necessary, they can access training videos and guidelines included in the suggestions to improve their behavior.

[0618] (Example 1)

[0619] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0620] Harassment in the workplace negatively impacts organizational health and productivity. However, systems for early identification of harassment and implementation of appropriate corrective measures are not adequately in place. Therefore, effective means for the early detection and resolution of harassment are needed.

[0621] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0622] In this invention, the server includes means for collecting harassment-related information from a communication medium, means for analyzing the collected information to identify harassment behavior, and means for generating corrective measures for the identified harassment behavior. This makes it possible to quickly identify harassment behavior and present effective corrective measures.

[0623] A "communication medium" is an electronic or optical means used to send and receive digital information.

[0624] "Harassment-related information" refers to data concerning aggressive or inappropriate behavior towards individuals or organizations.

[0625] "Analysis" is the process of analyzing collected data to extract and evaluate specific information.

[0626] "Harassment" refers to any behavior that causes psychological or physical discomfort to an individual or group.

[0627] A "solution" is a set of actionable steps proposed to resolve an identified problem.

[0628] A "terminal device" is a device used by a user to receive or display digital information.

[0629] A "user" is an individual or organization that receives information and services from a system and utilizes them.

[0630] A "natural language processing unit" is a computer system designed to analyze and understand human language.

[0631] An "image analysis device" is a computer system designed to extract and recognize information from still images or videos.

[0632] This invention is a system for efficiently detecting and addressing harassment in the workplace. The system consists of a server, terminal devices, and users, and in order to implement the invention, it is necessary to specifically understand the role and technology of each component.

[0633] Server roles and configuration

[0634] The server functions as the central processing unit of this system. The server collects information from workplace emails, chat systems, monitoring systems, and publicly available social networking services (SNS) through communication media. For analyzing the collected information, it uses the Python NLTK library as a natural language processing unit and the OpenCV library as an image analysis unit. Furthermore, the server automatically generates appropriate improvement measures based on the collected and analyzed information using a generative AI model. These improvement measures are based on a database of past cases and insights from behavioral science.

[0635] Role and function of terminal devices

[0636] The terminal device functions as a device for exchanging information between the user and the server. It receives improvement suggestions sent from the server and presents them to the user in an easy-to-understand manner. This presentation is provided in various report formats. The terminal device has the ability to display pop-up notifications and detailed information to help the user easily accept the improvement suggestions.

[0637] User roles and responses

[0638] Users are required to review their own behavior based on the improvement measures received via the terminal device and take specific actions to improve it. If necessary, the terminal will provide additional resources such as training modules and workshop information, which users can utilize to improve their harassment behavior.

[0639] For example, if a user is identified as having communication problems at work, the server analyzes the user's email history and meeting videos. This detects aggressive attitudes and negative expressions, and based on this information, improvement measures are generated. The user receives these improvement measures through a terminal device and can take training modules to improve their communication skills.

[0640] An example of a prompt to input into the generating AI model would be: "Develop a system that analyzes email and chat history within the workplace, identifies aggressive behavior, generates improvement measures based on that information, and distributes them to users."

[0641] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0642] Step 1: Data Collection

[0643] The server collects harassment-related information from workplace email servers, chat platforms, monitoring systems, and publicly available social media. Input data includes text messages, images, and videos. Specifically, the server periodically accesses each communication medium, retrieves new data, and securely stores it in a database. As output, the collected raw data is sent to the next analysis step.

[0644] Step 2: Data Analysis

[0645] The server applies a natural language processing unit and an image analysis unit to the collected data. The input consists of text and image data collected in the previous step. Here, the text data is analyzed using a natural language processing algorithm to identify aggressive language and negative expressions. Simultaneously, image analysis is utilized to detect nonverbal harassment behavior from video data. Specifically, the text is analyzed using the Python NLTK library, and image recognition is performed using the OpenCV library. The output generates the detected suspicious behavior and its details.

[0646] Step 3: Generating improvement measures

[0647] The server generates improvement measures based on the analysis results. The input is detailed information about the harassment behavior identified in the previous step. Based on behavioral science and similar past cases, a generative AI model creates specific and effective improvement measures. Specifically, the analysis results are input into the generative AI model, and the resulting improvement measures are output.

[0648] Step 4: Distributing the proposal

[0649] The terminal receives improvement suggestions generated from the server. The input is improvement suggestions customized for each user. The terminal device displays these suggestions clearly to the user and runs a notification function to encourage behavioral improvement. The output is the details of the improvement suggestions displayed to the user and the corresponding action steps. In terms of specific actions, the terminal presents suggestions in the form of pop-up alerts or reports.

[0650] Step 5: User Actions

[0651] Based on the improvement suggestions received through the device, users review their behavior and take concrete actions. Inputs include the content of the suggestions and additional resources (e.g., online training modules and workshop information). Users actively utilize these resources provided through the device to improve their harassment behavior. Specifically, users follow the suggestions, attend training, and practice new communication techniques. Expected outputs include improved behavior and enhanced communication skills.

[0652] (Application Example 1)

[0653] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0654] Harassment in the workplace is a serious problem that harms employees' mental and physical health and reduces productivity. Traditional systems have struggled to quickly and accurately detect such behavior and provide effective countermeasures. In particular, there is a challenge in making informed decisions in situations requiring immediate on-site response.

[0655] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0656] In this invention, the server includes means for collecting digital information relating to harassment behavior, means for analyzing the collected digital information to identify harassment behavior, and means for generating behavioral improvement measures based on the identified harassment behavior. This enables real-time detection and rapid response to problematic behavior in the workplace, reduces employee stress, and improves productivity.

[0657] "Harassment behavior" refers to aggressive words or actions towards employees in the workplace, or any behavior that harms their mental or physical health.

[0658] "Digital information" refers to all data, including emails, chat messages, and video data, that are generated and collected within the workplace for communication and work performance.

[0659] "Data collection means" refers to a combination of hardware and software used to acquire relevant information from various digital infrastructures within the workplace.

[0660] "Data analysis means" refers to algorithms or methods for identifying harassment behaviors based on collected digital information, using natural language processing technology and image recognition technology.

[0661] "Behavioral improvement measures" refer to specific guidelines or procedures, including suggestions or policies, that are generated to correct identified harassment behaviors.

[0662] "Real-time analysis" refers to the process of instantly processing video data and providing rapid information and warnings about identified behaviors.

[0663] The system for implementing this invention mainly consists of three elements: a server, a terminal, and a user. The server collects digital information by acquiring emails, chat messages, and surveillance camera footage from various digital infrastructures within the workplace, and also collects publicly available data from social networking services (SNS) as much as possible. To analyze this data in real time, the server uses image generation programs such as TensorFlow and OpenCV to detect nonverbal harassment behavior. It also analyzes text data using natural language processing techniques through libraries such as spaCy and Transformers to identify aggressive words and negative emotions.

[0664] Based on the analysis, the server automatically generates behavioral improvement measures and sends them to the terminal. These measures utilize past case data and insights based on behavioral science, and include specific countermeasures and training modules. The terminals used by users can be various devices such as smart glasses or head-mounted displays, which receive information sent from the server and display warnings of problematic behaviors and improvement measures in the user's field of vision in real time. Based on this, users are expected to review their own behavior and take appropriate action.

[0665] As a concrete example, if an employee exhibits aggressive behavior, the server analyzes the data and issues a warning to highlight the problem. The administrator, acting as the user, receives the warning through smart glasses, recognizes the issue, and can quickly take corrective action. In this application, the prompt used for the generative AI model is: "Please provide prompts to detect harassment behavior in the workplace and, based on that, suggest specific improvement measures that the user (administrator) should take."

[0666] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0667] Step 1:

[0668] The server collects emails, chat messages, surveillance camera footage, and new publicly available social media data from the workplace's digital infrastructure. The input is digital data from each device, which is then prepared for storage on the server. The output forms a digital information storage system for the entire workplace environment.

[0669] Step 2:

[0670] The server analyzes the collected digital information in real time. For text data, natural language processing techniques are used, employing libraries such as spaCy and Transformers to identify aggressive language and negative emotions. The input is extracted text data, and the output generates an index of the identified aggressive language and emotions. For video data, TensorFlow and OpenCV are used to detect nonverbal harassment behavior. Here, the input is video frames, and the output is tagged with the identified problematic behavior.

[0671] Step 3:

[0672] The server automatically generates behavioral improvement measures based on the harassment behaviors identified through analysis. It proposes specific countermeasures and training modules, utilizing past cases and insights from behavioral science. Inputs include indexes and tags from the analysis results, while output is text data of the generated improvement measures.

[0673] Step 4:

[0674] The device displays improvement information received from the server to the user in real time. Warnings and suggestions are displayed visually through smart glasses or similar devices. Input is text data sent from the server, and output is visual notifications and follow-up signals.

[0675] Step 5:

[0676] Users review their actions based on notifications from their devices. Specifically, they recognize problems based on visually presented information and implement suggested behavioral improvements. The action plan that users should take follows the text of the suggested improvements.

[0677] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0678] This invention provides a system that combines an emotion engine to effectively detect and improve harassment behavior in the workplace. This system consists of a server, terminals, users, and an emotion engine.

[0679] Processing performed by the server

[0680] The server collects data from various workplace data sources, such as emails, chat logs, surveillance camera feeds, and publicly available data from social media. It also collects emotional data from audio and video recordings. After preprocessing this data, the server uses natural language processing and image recognition techniques to identify harassing behaviors and problematic emotional expressions.

[0681] The function of the emotional engine

[0682] The emotion engine analyzes collected audio and video data to determine the user's emotional state. For audio data analysis, it estimates emotions from the tone and volume of speech. For video data analysis, it identifies emotions by analyzing facial expressions based on facial recognition technology. The emotional data determined by the emotion engine is used for further analysis on the server.

[0683] Server-based generation of improvement suggestions

[0684] The server automatically generates specific improvement suggestions based on data analysis of harassment behaviors and emotional states. These suggestions include guidance on how users should control their emotions and engage in constructive communication.

[0685] Terminal role

[0686] The terminal is a device that delivers improvement suggestions sent from the server to the user. The terminal displays the improvement suggestions to the user in an intuitively understandable format and proposes an action plan to encourage improvement in behavior.

[0687] User reactions and actions

[0688] Users will implement specific behavioral improvement measures based on the information provided via their devices. If necessary, they will utilize additional training resources and support to improve their emotional management skills and correct harassing behaviors.

[0689] For example, if person B makes an inappropriate remark during a meeting at a certain workplace, this system analyzes the nature of the remark and person B's emotional state. The server identifies the behavior as problematic and generates improvement suggestions that take into account the stress indicators detected by the emotion engine. These suggestions are provided to person B via a terminal, and based on them, person B can learn how to correct their behavior.

[0690] The following describes the processing flow.

[0691] Step 1:

[0692] The server collects information from email, chat logs, surveillance camera footage, and publicly available data from social media via the corporate network. During this process, it adheres to security and privacy policies and performs necessary access authentication.

[0693] Step 2:

[0694] The server preprocesses the collected data. Text data is tokenized and converted into a format that can be grammatically analyzed, and video data is extracted frame by frame. For audio data, noise is removed and the characteristics of the audio waveform are extracted.

[0695] Step 3:

[0696] The emotion engine analyzes audio and video data to determine the user's emotions. For audio, it analyzes voice tone and pitch to assess the level of stress and anger in adverse situations. For video, it uses facial recognition technology to capture subtle changes in facial expressions and identify emotions.

[0697] Step 4:

[0698] The server integrates and analyzes pre-processed data and the results of the emotion engine. It uses natural language processing techniques to check text for offensive or negative words and image recognition techniques to detect nonverbal harassment behavior in videos.

[0699] Step 5:

[0700] Based on the analysis results, the server generates improvement suggestions tailored to each individual user. From the analysis, which includes emotional states, it creates specific advice on how users should improve their behavior and control their emotions.

[0701] Step 6:

[0702] The terminal receives improvement suggestions generated from the server and presents them to the user. The improvement suggestions are displayed in a visualized interface and provided in a way that is easy for the user to understand intuitively.

[0703] Step 7:

[0704] Users implement improvement suggestions received via their devices. They review their behavior and manage their emotions based on the suggestions. They also utilize additional support materials and applications as needed to achieve continuous improvement.

[0705] (Example 2)

[0706] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0707] Harassment is a serious problem in the workplace, and prompt detection and corrective measures are required. Traditional methods have faced challenges in effective countermeasures due to fragmented data collection and analysis, and an inability to fully grasp emotional states. Furthermore, there was a need for a system that could accurately grasp emotional changes, quickly generate concrete improvement proposals based on those findings, and provide them to users.

[0708] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0709] In this invention, the server includes means for collecting information relating to harassment behavior, means for analyzing the collected information to identify harassment behavior, and means for analyzing emotions from audio data and video data. This makes it possible to accurately detect harassment behavior in the workplace environment and quickly provide users with effective improvement suggestions for taking appropriate corrective measures.

[0710] "Harassment behavior" refers to actions that cause psychological or physical discomfort or harm to others in social settings such as the workplace.

[0711] "Information" refers to facts and data expressed in any form, including data collected from emails, chats, audio recordings, video recordings, and social networking services.

[0712] "Analyzing emotions" refers to the process of determining and identifying the emotional state expressed in collected audio and video data.

[0713] "Improvement suggestions" refer to content that provides guidelines and action plans for the individual to improve their behavior and emotions, as well as their communication, based on identified harassment behaviors and analyzed emotions.

[0714] "Users" refers to individuals or groups who receive improvement suggestions provided through this system with the aim of improving their own behavior and emotions.

[0715] This invention is a system for effectively detecting and improving harassment behavior in the workplace environment. The system consists of a server, terminals, users, and an emotion engine.

[0716] The server collects data from emails, chats, surveillance camera footage, and social networking services. Audio recordings are also included. The server preprocesses the collected data and analyzes it using natural language processing and image recognition technologies. The hardware used includes a high-performance database server, and the software utilizes natural language processing libraries and image recognition algorithms. Through this analysis, the server identifies harassment behaviors and users' emotional states.

[0717] The emotion engine analyzes audio and video data to determine the user's emotional state. It identifies emotions by analyzing speech tone and volume from audio data, and facial expressions from video data. This resulting emotional data is then further analyzed by the server.

[0718] Improvement suggestions are automatically generated based on the analysis results and sent to the terminal by the server. The terminal displays the improvement suggestions from the server to the user in an easy-to-understand format. This allows the user to obtain concrete actionable improvement measures. The terminal has the function of providing information in a format that is easy for the user to understand intuitively.

[0719] For example, if the server detects inappropriate remarks in a particular meeting, it will generate improvement suggestions, taking into account the emotional state behind those remarks. These suggestions will be displayed on the terminal, showing the user how to correct their behavior.

[0720] By using a generative AI model, the server can improve the accuracy of its information analysis and respond flexibly to user needs. An example of a prompt message would be, "Please describe a system that automatically generates suggestions for improving harassment behavior in the workplace. Please explain in detail, including specific data processing methods using an emotion engine and natural language processing technology." This input optimizes the system's operation.

[0721] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0722] Step 1:

[0723] The server collects information from various data sources in the work environment. Inputs include email, chat history, surveillance camera footage, and data from social networks. The output is the result of this raw data collection, which is stored in a database. This process integrates the data into an appropriate format and prepares it for subsequent processing.

[0724] Step 2:

[0725] The server preprocesses the collected data. The input includes various data collected in Step 1. Data processing, such as data cleaning, noise reduction, and data format standardization, is performed to obtain clean, analyzable data as output. This process prepares the information necessary to improve the accuracy of the analysis.

[0726] Step 3:

[0727] The server performs emotion analysis using an emotion engine. The input includes clean audio and video data obtained in step 2. For the audio data, the tone and volume of speech are analyzed, and for the video data, facial recognition is used to analyze facial expressions and identify emotions. The output is data of the identified emotional state, which is used in the next analysis step.

[0728] Step 4:

[0729] The server uses natural language processing and image recognition technologies to perform analysis to identify harassment behaviors. The input includes text data obtained in step 2 and sentiment data obtained in step 3. The processing analyzes the text content and related sentiment indicators to determine whether a behavior constitutes harassment. The output is data representing the evaluation results for each behavior.

[0730] Step 5:

[0731] The server automatically generates improvement suggestions based on identified behavioral assessments and emotional data. The input includes the analysis results from step 4. This step generates specific suggestions on how to manage emotions and engage in constructive communication. This provides guidance for necessary behavioral improvements.

[0732] Step 6:

[0733] The terminal delivers improvement suggestions sent from the server to the user. The input includes improvement suggestions automatically generated in step 5. The terminal displays these suggestions visually and clearly to the user, making them intuitively understandable. The output is the information presented to the user.

[0734] Step 7:

[0735] Users improve their behavior based on improvement suggestions provided through their devices. They implement specific behavioral improvements and utilize additional resources and training as needed. This aims to improve emotional management and correct harassing behaviors. The output is the user's improved behavior and its effects.

[0736] (Application Example 2)

[0737] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0738] When harassment occurs in the workplace, it can significantly impair employees' mental health and work efficiency. Traditional methods have struggled to quickly recognize and correct harassment, making immediate action to prevent the problem from escalating difficult. Therefore, there is a need for a system that can analyze emotional states in real time, detect inappropriate behavior early, and provide appropriate improvement suggestions.

[0739] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0740] In this invention, the server includes means for collecting information on harassment behavior, means for analyzing the collected information to identify harassment behavior, means for generating improvement suggestions based on the identified harassment behavior, and means for detecting inappropriate behavior in real time, including an emotion engine that analyzes emotions from the user's voice and facial expressions. This makes it possible to quickly and effectively detect harassment behavior in the workplace and provide specific improvement measures.

[0741] "Harassment behavior" refers to inappropriate or offensive actions or remarks in the workplace that harm the mental health of employees or the work environment.

[0742] "Means of information gathering" refers to processes that include various technologies and equipment for collecting relevant data from data sources within the workplace.

[0743] "Means of analyzing information" refers to techniques that use collected data to perform analyses in order to identify harassment behaviors.

[0744] "Means for generating improvement suggestions" refers to algorithms or software used to propose appropriate countermeasures or behavioral modifications for identified harassment behaviors.

[0745] An "emotion engine" is a technology that analyzes an individual's emotional state from data such as voice and facial expressions, and detects changes in emotions in real time.

[0746] A "means for detecting inappropriate behavior" refers to a mechanism that identifies problematic or risky behaviors in the workplace in real time, based on the results of sentiment analysis.

[0747] The system for implementing the present invention mainly consists of a server, terminals, and users. The server has the function of collecting information from various data sources within the work environment, and this information includes text information, audio data, and image data. Specifically, it receives data from emails, chat records, video from surveillance systems, and audio recordings.

[0748] The collected information is analyzed on the server using natural language processing and visual pattern recognition technologies. First, textual information is analyzed by natural language processing algorithms to evaluate the possibility of harassment based on specific keywords and context. Next, from image data, facial recognition technology and an emotion engine are used to identify emotional states from facial expressions. For audio data, speech analysis technology is used to infer emotions from factors such as tone and volume of voice. This allows for sensitive detection of emotional changes in real time.

[0749] The terminal serves to communicate improvement suggestions generated from the server to the user. Through the terminal, the user receives specific suggestions for behavioral improvement and uses them to improve the situation. For example, if a user shows strong frustration during a meeting, the terminal will immediately display suggestions for calming down. These include specific actions such as leaving the room or taking deep breaths.

[0750] An example of a prompt using a generative AI model is, "Please check what emotional state was detected from this data." Another prompt used is, "Please generate specific behavioral improvement measures based on the emotion analysis results." These prompts enable the generation of more flexible and appropriate improvement suggestions.

[0751] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0752] Step 1:

[0753] The server collects information from various data sources within the work environment. It receives emails, chat logs, video and audio recordings from monitoring systems as input. This data is temporarily stored in storage as raw data.

[0754] Step 2:

[0755] The server preprocesses the collected raw data. Unnecessary tags and special characters are removed from the input text information, and audio data is converted to text. Image data is also converted to a standard format, and noise is removed to produce clean analytical data as output.

[0756] Step 3:

[0757] The server applies natural language processing techniques to the pre-processed text information. This involves extracting keywords and performing sentiment analysis from the input text. This identifies potentially harassing phrases, and the relevant sections are output as analysis results.

[0758] Step 4:

[0759] The server applies face recognition technology and an emotion engine to pre-processed image data. It recognizes individual faces from the input image data and identifies emotional states through facial expression analysis. The obtained emotional state information is output as the analysis result.

[0760] Step 5:

[0761] The server applies speech analysis technology to the audio data. It analyzes the tone and volume of the input audio and estimates the emotional state. The estimated emotional information is output as the analysis result.

[0762] Step 6:

[0763] The server integrates the analysis results to identify harassment behaviors and generate necessary improvement suggestions. Here, a generative AI model is used to create prompt messages based on the input analysis results and output behavioral improvement measures.

[0764] Step 7:

[0765] The terminal receives improvement suggestions sent from the server and displays them to the user in an intuitively understandable format. Based on the suggestions received as input, it outputs visualized action plans and presents specific improvement actions.

[0766] Step 8:

[0767] Users implement improvement suggestions displayed on their devices. Following these suggestions, they might, for example, learn communication techniques to avoid misunderstandings and adjust their behavior according to the situation. This manifests as actual changes in behavior in the output.

[0768] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0769] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0770] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0771] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0772] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0773] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0774] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0775] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0776] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0777] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0778] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0779] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0780] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0781] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0782] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0783] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0784] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0785] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0786] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0787] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0788] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0789] The following is further disclosed regarding the embodiments described above.

[0790] (Claim 1)

[0791] Means for collecting data on harassment behavior,

[0792] A means of identifying harassment behavior by analyzing the aforementioned collected data,

[0793] A means of generating improvement suggestions based on identified harassment behaviors,

[0794] A means of providing the aforementioned improvement suggestions to the user,

[0795] A system that includes this.

[0796] (Claim 2)

[0797] The system according to claim 1, characterized in that the data includes text data and video data.

[0798] (Claim 3)

[0799] The system according to claim 1, characterized in that the analysis uses natural language processing technology and image recognition technology.

[0800] "Example 1"

[0801] (Claim 1)

[0802] Means of collecting harassment-related information from communication media,

[0803] A means of analyzing the collected information and identifying harassment behavior,

[0804] A means of generating corrective measures for identified harassment behaviors,

[0805] A means for presenting the aforementioned improvement measures to the user via a terminal device,

[0806] A system that includes this.

[0807] (Claim 2)

[0808] The system according to claim 1, characterized in that it includes text information and image information.

[0809] (Claim 3)

[0810] The system according to claim 1, characterized by using a natural language processing device and an image analysis device.

[0811] "Application Example 1"

[0812] (Claim 1)

[0813] Means for collecting digital information on harassment behavior,

[0814] A means of identifying harassment behavior by analyzing the digital information collected as described above,

[0815] A means of generating behavioral improvement measures based on identified harassment behaviors,

[0816] Means for providing the aforementioned behavioral improvement measures to users,

[0817] A means of analyzing video data in real time and notifying identified actions,

[0818] A system that includes this.

[0819] (Claim 2)

[0820] The system according to claim 1, characterized in that the digital information includes text information and image information.

[0821] (Claim 3)

[0822] The system according to claim 1, characterized in that the analysis uses language processing technology and image analysis technology.

[0823] "Example 2 of combining an emotion engine"

[0824] (Claim 1)

[0825] Means for collecting information on harassment behavior,

[0826] A means of identifying harassment behavior by analyzing the information collected above,

[0827] A means of analyzing emotions from audio and video data,

[0828] A means for generating improvement suggestions based on identified harassment behaviors and analyzed emotions,

[0829] A means of providing the aforementioned improvement suggestions to users,

[0830] A system that includes this.

[0831] (Claim 2)

[0832] The system according to claim 1, characterized in that the information includes text information, image information, and audio information.

[0833] (Claim 3)

[0834] The system according to claim 1, characterized in that the analysis uses natural language processing technology, image recognition technology, and speech analysis technology.

[0835] "Application example 2 when combining with an emotional engine"

[0836] (Claim 1)

[0837] Means for collecting information on harassment behavior,

[0838] A means of identifying harassment behavior by analyzing the information collected above,

[0839] A means of generating improvement suggestions based on identified harassment behaviors,

[0840] A means of providing the aforementioned improvement suggestions to users,

[0841] It includes an emotion engine that analyzes emotions from the user's voice and facial expressions, and means for detecting inappropriate behavior in real time.

[0842] A system that includes this.

[0843] (Claim 2)

[0844] The system according to claim 1, characterized in that the information includes text information and image information.

[0845] (Claim 3)

[0846] The system according to claim 1, characterized in that the analysis uses natural language processing technology and visual pattern recognition technology. [Explanation of symbols]

[0847] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for collecting data on harassment behavior, A means of identifying harassment behavior by analyzing the aforementioned collected data, A means of generating improvement suggestions based on identified harassment behaviors, A means of providing the aforementioned improvement suggestions to the user, A system that includes this.

2. The system according to claim 1, characterized in that the data includes text data and video data.

3. The system according to claim 1, characterized in that the analysis uses natural language processing technology and image recognition technology.

Images