system

Abstract

Provide a system. A means for recording audio data of meetings and interviews; means for converting the recorded voice data into text data; A means for analyzing text data and detecting harassment behavior; means for notifying the user of the analysis results; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, harassment during internal meetings and interviews is a serious problem. While proper and accurate communication is essential to improving the quality of communication within a company, it is difficult to continuously monitor and appropriately evaluate it. Furthermore, when an employee experiences harassment, early detection and appropriate countermeasures are required. The purpose of this invention is to provide an effective means for solving these problems and improving internal communication. [Means for solving the problem]

[0005] The present invention is a system that includes a means for recording audio data of meetings and interviews, a means for converting the recorded audio data into text data, a means for analyzing the text data to detect harassment, and a means for notifying the user of the analysis results. The system also includes a means for transmitting manually entered text data along with the recorded audio data to a server, and a means for receiving the analysis results from the server. Furthermore, the system includes a means for extracting problematic terms and phrases from the text data using natural language processing technology, evaluating harassment behavior, and generating recommended countermeasures based on the evaluation results. This system improves the quality of internal communication and enables early detection and countermeasures of harassment.

[0006] A "meeting" refers to a meeting where multiple people gather to share business-related information and solve problems.

[0007] "Interview" refers to business communication conducted in the form of an individual dialogue, including, for example, interview evaluations and individual consultations between superiors and subordinates.

[0008] "Voice data" refers to data that has been recorded and stored in digital form.

[0009] "Text data" is data in a format in which voice data is converted into text.

[0010] "Harassment" refers to inappropriate verbal or physical behavior toward another person, and includes psychological or physical harassment and bullying that are particularly problematic in the workplace.

[0011] "Recording means" refers to devices and software for capturing and storing audio data.

[0012] "Conversion means" refers to speech recognition technology or software for converting voice data into text data.

[0013] "Analysis means" refers to natural language processing technology or software used to analyze text data and detect specific problematic behaviors or terms.

[0014] "Notification means" refers to devices or communication means for notifying the user of the analysis results.

[0015] "Server" means a central computer system for processing and storing data.

[0016] "Natural language processing technology" refers to the technology and algorithms that enable machines to understand and analyze human language.

[0017] "Countermeasures" refers to specific actions or procedures recommended when harassing behavior is detected. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention relates to a system for detecting harassment behavior during internal company meetings and interviews. An embodiment of the system will be described in detail below.

[0040] This system consists of a user-owned device, an associated server, and data communications between them. Users use an application to record audio from meetings and interviews in real time. The device converts the recorded audio data into text data and sends it to the server.

[0041] System Overview

[0042] 1. Recording audio data

[0043] User: At the start of a meeting or interview, launch the dedicated application and tap the "Start Recording" button.

[0044] Device: Uses the built-in microphone to begin collecting audio data, which is temporarily stored in local storage.

[0045] 2. Converting voice data to text

[0046] Terminal: Once the recording is complete, the collected voice data is converted into text data using speech recognition software, which analyzes the voice waveform and generates the most appropriate string of characters.

[0047] 3. Data transmission

[0048] Terminal: The converted text data, and manually entered minutes data if necessary, are sent to the server. Before being sent, the data is encrypted to ensure its security.

[0049] 4. Text Data Analysis

[0050] Server: The received text data is passed through a natural language processing (NLP) engine, which extracts specific keywords and phrases from the text and evaluates whether they constitute harassment.

[0051] Server: Based on the analysis results, if harassment is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0052] 5. Notification of analysis results

[0053] Server: Organizes the analysis results and generates them in report format.

[0054] Server: Sends the generated report to the terminal.

[0055] Terminal: Notifies the user of the received analysis results and displays them so that the user can easily check them.

[0056] Specific examples

[0057] Example 1: Meeting recording and harassment detection

[0058] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0059] 2. Terminal: Collects recorded data and converts it into text data using speech recognition software after the meeting is over.

[0060] 3. Terminal: Sends the converted text data to the server.

[0061] 4. Server: The text data is analyzed and phrases such as "If we fail to achieve our goal, it's your fault" are detected.

[0062] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0063] 6. Terminal: Receives the report and notifies the user.

[0064] 7. User: Review the report and take any necessary follow-up actions.

[0065] Example 2: Minutes of individual consultations and harassment detection

[0066] 1. User: Enter the details of the individual consultation into the app as minutes.

[0067] 2. Terminal: Sends the minutes data to the server.

[0068] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0069] 4. Server: Determines that this constitutes harassment and generates a report with recommended actions.

[0070] 5. Terminal: Receives the report and notifies the user.

[0071] 6. User: Review the report and take action to resolve the issue with your manager if necessary.

[0072] This system can provide an effective means of improving the quality of business communication, early detection of harassment, and appropriate responses.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[0076] Step 2:

[0077] Device: Begins collecting audio data using the built-in microphone, which is saved to local storage in real time.

[0078] Step 3:

[0079] User: When the meeting or interview is over, tap the "Stop Recording" button in the application.

[0080] Step 4:

[0081] Terminal: The recorded voice data is input into voice recognition software, which converts the voice data into text data. The voice recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[0082] Step 5:

[0083] Terminal: The converted text data and, if necessary, manually entered minutes data are sent to the server. The data is encrypted before being sent to ensure its security.

[0084] Step 6:

[0085] Server: Receives data packets sent from the device and decompresses and decrypts the data.

[0086] Step 7:

[0087] Server: Inputs the received text data into a natural language processing (NLP) engine to extract specific keywords and phrases from the text, and evaluates whether the extracted keywords and phrases constitute harassment.

[0088] Step 8:

[0089] Server: Based on the output of the NLP engine, if harassment is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0090] Step 9:

[0091] Server: Sends the generated analysis result report to the terminal.

[0092] Step 10:

[0093] Terminal: Decodes the received analysis results and displays them in the user interface, formatting them so that the user can easily view them.

[0094] Step 11:

[0095] User: Review the analysis results and take any necessary follow-up actions, such as reporting to a manager or relevant department, or providing additional training.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] There is a need to prevent harassment during company meetings and interviews, and to quickly detect it if it does occur and take appropriate measures. However, there is currently a lack of systems that can detect harassment in real time and quickly propose appropriate measures. This has led to problems with delays in the early detection of harassment and effective responses.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for converting voice data into text data, means for encrypting and transmitting the text data to the server, means for analyzing the text data using natural language processing technology to detect harassment, means for generating recommended countermeasures based on the analysis results, and means for notifying the user of the analysis results and the countermeasures, thereby enabling early detection of harassment and prompt provision of appropriate countermeasures.

[0101] "Audio data" refers to digital data that records audio from meetings, interviews, etc.

[0102] "Text data" is character string data generated by analyzing voice data.

[0103] "Encryption" is a technology that converts data using a special algorithm to prevent it from being intercepted by third parties.

[0104] A "server" is a system that receives requests from clients via a network and processes and stores data.

[0105] A "user" is a person who operates the system and records audio data of meetings and interviews.

[0106] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0107] "Analysis" is the process of examining data in detail and understanding its meaning and structure.

[0108] "Harassment" is any act that causes discomfort and mental stress to an individual.

[0109] "Countermeasures" are specific methods or measures proposed to solve a problem.

[0110] "Notification" is the act of conveying information to a specific recipient.

[0111] A "generative AI model" is a model that uses artificial intelligence to automatically generate new information and suggestions from data.

[0112] A "prompt sentence" is a specific instruction sentence that is input to a generative AI model.

[0113] The present invention relates to a system for detecting harassment behavior during internal meetings and interviews, and an embodiment thereof is described in detail below. The system is composed of a terminal owned by a user, an associated server, and data communications between them.

[0114] First, the user installs a dedicated application on their device (such as a smartphone or PC). After launching the application and tapping the "Start Recording" button at the start of a meeting or interview, the device will use its built-in microphone to record audio data in real time. The recorded audio data is temporarily stored in the device's local storage.

[0115] Next, once the meeting or interview is over, the device converts the collected voice data into text data using speech recognition software (e.g., Google® Speech-to-Text API or Amazon Transcribe). This speech recognition software analyzes the voice waveform and generates the most appropriate string of characters. The generated text data is then encrypted (e.g., using TLS / SSL protocol) and sent to the server.

[0116] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., spaCy, Google Cloud Natural Language API, etc.). The NLP engine extracts specific keywords and phrases from the text and evaluates whether they constitute harassing behavior. In particular, it applies a set of rules based on past cases and laws to perform a detailed analysis of the text data.

[0117] Based on the analysis results, the server uses a generative AI model to generate recommended countermeasures, including specific measures (e.g., "early reporting of problems" and "provision of mental health care"). The generated analysis results and countermeasures are then organized into a report and sent to the device.

[0118] Finally, the device notifies the user of the received report and can display the report within the application, where the user can open the report, review the details, and take any necessary action.

[0119] For example, the following specific example is shown.

[0120] Example 1: Meeting recording and harassment detection

[0121] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0122] 2. Terminal: Collects recorded data and converts it into text data using voice recognition software after the meeting is over.

[0123] 3. Terminal: Sends the converted text data to the server.

[0124] 4. Server: The text data is analyzed and phrases such as "If we fail to achieve our goal, it's your fault" are detected.

[0125] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0126] 6. Terminal: Receives the report and notifies the user.

[0127] 7. User: Review the report and take any necessary follow-up actions.

[0128] Example 2: Minutes of individual consultations and harassment detection

[0129] 1. User: Enter the details of the individual consultation into the app as minutes.

[0130] 2. Terminal: Sends the minutes data to the server.

[0131] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0132] 4. Server: Determines that this constitutes harassment and generates a report with recommended actions.

[0133] 5. Terminal: Receives the report and notifies the user.

[0134] 6. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[0135] The following can be used as an example prompt:

[0136] Example prompt 1:

[0137] Please analyze the following conversation and rate whether it contains harassing behavior:

[0138] 'Boss: If you fail to achieve your goals, it's all your fault.'

[0139] Based on the analysis results, generate a report and propose appropriate countermeasures.

[0140] Example prompt 2:

[0141] Please analyze the following minutes and assess whether they contain any harassing behavior:

[0142] Subordinate: I think there is room for improvement in this proposal.

[0143] Boss: You have no room for improvement.'

[0144] Based on the analysis results, generate a report and propose appropriate countermeasures.

[0145] This system can improve the quality of business communication and provide an effective means of supporting the early detection of harassment and appropriate responses.

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1:

[0148] User: Launch the dedicated application on a device such as a smartphone or PC, and tap the "Start Recording" button at the start of the meeting or interview.

[0149] Input: Application launch and button tap.

[0150] Specific operation: When the application is launched and the "Start Recording" button is pressed, the built-in microphone starts working.

[0151] Output: Recording start trigger set by device.

[0152] Step 2:

[0153] Device: Uses the built-in microphone to collect voice data in real time and temporarily stores it in local storage.

[0154] Input: Audio during a meeting or interview.

[0155] What it does: The microphone captures ambient sounds and stores them as digital audio data in file formats such as WAV and MP3.

[0156] Output: Audio data file saved to local storage.

[0157] Step 3:

[0158] Device: After recording is complete, the collected voice data is sent to speech recognition software (such as Google Speech-to-Text API), which begins analyzing the voice waveform and converting it into text data.

[0159] Input: Saved audio data file.

[0160] What it does: Speech recognition software analyzes the audio data and converts the audio waveform into text.

[0161] Output: The generated text data.

[0162] Step 4:

[0163] Terminal: The converted text data and manually entered minutes data are encrypted and sent to the server.

[0164] Input: Text data and meeting minutes data.

[0165] What it does: Encodes data using the TLS / SSL protocol and sends it securely to the server.

[0166] Output: Encrypted data reaches the server.

[0167] Step 5:

[0168] Server: Decodes the received data and analyzes the text data using a natural language processing (NLP) engine (such as spaCy).

[0169] Input: Encrypted text data.

[0170] What it does: The decoded text data is analyzed by an NLP engine to extract specific keywords and phrases.

[0171] Output: A dataset of the analysis results.

[0172] Step 6:

[0173] Server: Based on the analysis results, a generative AI model is used to generate recommended countermeasures.

[0174] Input: NLP engine analysis result dataset.

[0175] How it works: The generative AI model takes the analysis results as input and generates appropriate countermeasures based on past data and case studies.

[0176] Output: A report with recommended actions.

[0177] Step 7:

[0178] Server: Sends the generated report to the terminal.

[0179] Input: A report with recommended actions.

[0180] Specific operation: Encodes the report data and transmits it securely to the terminal.

[0181] Output: The report arrives at the terminal.

[0182] Step 8:

[0183] Terminal: Notifies and displays received reports to the user.

[0184] Input: Report received from server.

[0185] Specific operation: The terminal decodes the report data and notifies the user through the user interface. The contents of the report are displayed on the screen.

[0186] Output: The report is available for the user to review.

[0187] Through the above processing steps, this system enables early detection of harassment behavior and provides appropriate countermeasures.

[0188] (Application example 1)

[0189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0190] There is a need for a system that can quickly detect harassment that occurs in communication between workers in factories and take appropriate measures. However, existing systems lack real-time capabilities and can be slow to notify managers, making it difficult to respond quickly and effectively.

[0191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0192] In this invention, the server includes means for encrypting voice data and transmitting it to the server, means for analyzing text data using a natural language processing engine and generating recommended countermeasures when harassment is detected, and means for receiving the report generated by the server and notifying the administrator in PDF format. This makes it possible to detect harassment in real time and quickly notify the administrator.

[0193] "Audio Data" means audio information recorded in analog or digital format.

[0194] "Text data" is character string information obtained by processing voice data.

[0195] "Encryption" is the process of transforming data using a specific algorithm to ensure its security.

[0196] A "natural language processing engine" is a computer program that analyzes text data and understands its meaning and context.

[0197] "Recommended actions" indicate actions or countermeasures to be taken when a problem is detected.

[0198] A "report" is a document summarizing the analysis results and countermeasures.

[0199] A "server" is a central computer system for storing and processing data.

[0200] "Real-time" refers to a time frame in which data processing and information provision occurs immediately without delay.

[0201] This invention relates to a system for quickly detecting harassment behavior in communication between workers in a factory and taking appropriate measures. This system is composed of a robot assistant that patrols the factory, an associated server, and data communication between them.

[0202] System Overview

[0203] Recording audio data

[0204] The robot uses a built-in microphone to record conversations between workers in the factory in real time, and the recordings are temporarily saved in local storage.

[0205] Converting audio data to text

[0206] Once the recording is complete, the collected audio data is converted to text using speech recognition software (e.g., Google Cloud Speech-to-Text API), which analyzes the audio waveform and generates the most appropriate string of characters.

[0207] Sending data

[0208] The converted text data is encrypted using a data encryption tool (e.g., AES encryption) and sent to a server via a reliable communication (e.g., HTTPS communication).

[0209] Text data analysis

[0210] The server runs the received text data through a natural language processing (NLP) engine (such as OpenAI's GPT-4). The NLP engine extracts specific keywords and phrases from the text and evaluates whether they constitute harassing behavior. Based on the evaluation results, if harassing behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0211] Notification of analysis results

[0212] The server compiles the analysis results and generates a report. This report is output in PDF format and sent to the administrator, who can then take appropriate measures promptly based on the report.

[0213] Specific examples of hardware and software used

[0214] Hardware: Robot for patrolling the factory, built-in microphone, local storage, internet communication module

[0215] software:

[0216] Speech recognition software (Google Cloud Speech-to-Text API)

[0217] Natural language processing engine (OpenAI's GPT-4)

[0218] Data encryption tool (AES encryption)

[0219] Data transmission module (HTTPS communication)

[0220] Report generation tools (e.g. PDF generation libraries)

[0221] Specific examples

[0222] Prompt Sentence Examples

[0223] markdown

[0224] Analyze the following conversation text to detect harassing behavior, and if so, generate details and appropriate countermeasures.

[0225] Conversation text:

[0226] Worker A: "You caused the line to stop, and it's all your fault."

[0227] Worker B: "Sorry, I'll be more careful next time."

[0228] Worker A: "It's not enough to just be careful. If you do it again, you're fired."

[0229] result:

[0230] 1. Type of conduct: Harassment

[0231] 2. Key phrases detected: "It's your fault", "You're fired"

[0232] 3. Recommended measures:

[0233] Recommend that Worker A take harassment prevention training

[0234] Follow-up meetings held to improve the work environment

[0235] In this way, the system that is the subject of this patent is extremely effective in improving human relationships and the working environment within a factory.

[0236] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0237] Step 1:

[0238] The user activates the robot assistant in an area of ​​the factory where communication between workers takes place. The robot uses a built-in microphone to record conversations in real time. The recorded data (input) is temporarily saved in local storage (output).

[0239] Step 2:

[0240] The robot's terminal converts the temporarily stored voice data into text data using voice recognition software (Google Cloud Speech-to-Text API). The voice data (input) is converted into text data (output) as character string information through analysis of the voice waveform. Specifically, the voice file is sent via the cloud to the voice recognition service, and the returned text data is received.

[0241] Step 3:

[0242] The terminal encrypts the converted text data using the AES encryption tool. The text data (input) is encrypted using a specified key to produce secure ciphertext data (output). During the encryption process, the text data is transformed through an algorithm to make it unintelligible to outsiders.

[0243] Step 4:

[0244] The terminal sends encrypted text data to the server via HTTPS communication. At this time, the encrypted text data (input) is sent to the server (output) using a secure communication protocol. Specifically, SSL / TLS is used to prevent data interception.

[0245] Step 5:

[0246] The server decrypts the ciphertext data it receives. The server decrypts the ciphertext data (input) with the appropriate key to obtain the original text data (output). Decryption is performed by reversing the encryption process.

[0247] Step 6:

[0248] The server analyzes the text data using a natural language processing engine (OpenAI's GPT-4). Specific keywords and phrases are extracted from the text data (input) and evaluated to determine whether they constitute harassment. The natural language processing engine generates the analysis results, and the output indicates whether or not the behavior constitutes harassment.

[0249] Step 7:

[0250] The server generates recommended countermeasures based on the analysis results. The analysis results (input) are converted into report-format data (output) that includes countermeasures. Specifically, the NLP engine proposes candidate countermeasures and compiles them into a report.

[0251] Step 8:

[0252] The server generates a report in PDF format and notifies the administrator. This report data (input) is output as a PDF file (output) using a PDF generation library and sent to the administrator's terminal.

[0253] Step 9:

[0254] The administrator checks the received report and takes appropriate measures. The administrator's device displays the received PDF file (input) and allows the contents to be checked (output). Specifically, the administrator uses a PDF viewer to view the report and select and execute the necessary actions.

[0255] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0256] The present invention relates to a system for effectively detecting harassment behavior in internal meetings and interviews, analyzing the user's emotions, and notifying the user. Specific embodiments of the system will be described in detail below.

[0257] This system consists of multiple components, including the user's device, a server, and an emotion engine. Users can use the application to record the contents of meetings and interviews and analyze the data in real time.

[0258] System Overview

[0259] 1. Recording audio data

[0260] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[0261] Device: Begins collecting audio data using the built-in microphone, which is temporarily stored in local storage.

[0262] 2. Converting voice data to text

[0263] Terminal: Once the recording is complete, the voice data is input into speech recognition software, which converts it into text data. The speech recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[0264] 3. Emotion Analysis

[0265] Device: During and after recording, the voice data is input into an emotion engine to analyze the user's emotional state (e.g., anger, anxiety, joy, etc.).

[0266] Emotion engine: Analyzes emotional expressions contained in voice data and evaluates the user's emotional state.

[0267] 4. Data transmission

[0268] Terminal: The converted text data and the emotion engine analysis results are sent to the server. The data is encrypted before being sent to ensure its security.

[0269] 5. Analysis of Text Data

[0270] Server: Inputs the received text data into a natural language processing (NLP) engine to extract specific keywords and phrases from the text, and evaluates whether the extracted keywords and phrases constitute harassment.

[0271] Server: Uses the analysis results of the emotion engine as additional data to complement the assessment results of harassment behavior.

[0272] 6. Generation and notification of analysis results

[0273] Server: Based on the output of the NLP engine and emotion engine, if harassment behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0274] Server: Sends the generated analysis result report to the terminal.

[0275] Terminal: Decodes the received analysis results and displays them on the user interface. Formats the information so that the user can easily check the results.

[0276] Specific examples

[0277] Example 1: Meeting recording, harassment detection, and emotion analysis

[0278] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0279] 2. Device: Collects recorded data and converts it into text using speech recognition software, while simultaneously analyzing the user's emotions using an emotion engine.

[0280] 3. Terminal: The converted text data and the emotion engine analysis results are sent to the server.

[0281] 4. Server: Analyzes the text data and detects phrases such as "If we fail to achieve our goal, it's your fault." Using sentiment analysis, the server adds the user's strong anxiety to the score.

[0282] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0283] 6. Terminal: Receives the report and notifies the user.

[0284] 7. User: Review the report and take any necessary follow-up actions.

[0285] Example 2: Individual consultation minutes, harassment detection, and sentiment analysis

[0286] 1. User: Enter the details of the individual consultation into the app as minutes.

[0287] 2. Terminal: The minutes data is sent to the server. Emotion analysis is also performed by the emotion engine in parallel.

[0288] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0289] 4. Server: Based on the results of sentiment analysis, the user indicates strong disappointment and adds it to the rating.

[0290] 5. Server: Determines that this constitutes harassment and generates a report with recommended action.

[0291] 6. Terminal: Receives the report and notifies the user.

[0292] 7. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[0293] This system will improve the quality of internal communication, enable early detection of harassment, and enable appropriate responses. Furthermore, by analyzing the user's emotional state, more accurate evaluations and countermeasures can be achieved.

[0294] The processing flow will be explained below.

[0295] Step 1:

[0296] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[0297] Step 2:

[0298] Device: Begins collecting audio data using the built-in microphone, which is saved to local storage in real time.

[0299] Step 3:

[0300] On the device, the voice data collected at the same time is input into the emotion engine, which analyzes the user's emotional state in real time by analyzing voice characteristics such as tone, speed, and volume.

[0301] Step 4:

[0302] User: When the meeting or interview is over, tap the "Stop Recording" button in the application.

[0303] Step 5:

[0304] Terminal: The recorded voice data is input into voice recognition software, which converts the voice data into text data. The voice recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[0305] Step 6:

[0306] Device: The converted text data and the emotion engine analysis results are sent to the server. The data is encrypted before being sent to ensure its security.

[0307] Step 7:

[0308] Server: Receives data packets sent from the device and decompresses and decrypts the data.

[0309] Step 8:

[0310] Server: The received text data is input into a natural language processing (NLP) engine to extract specific keywords and phrases from the text. The NLP engine then analyzes the text data and evaluates whether it constitutes harassment.

[0311] Step 9:

[0312] Server: Use the results of the emotion engine analysis as additional data to complement the assessment of harassment behavior, paying particular attention to when the user has negative emotions.

[0313] Step 10:

[0314] Server: Based on the output of the NLP engine and emotion engine, if harassment behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0315] Step 11:

[0316] Server: Sends the generated analysis result report to the terminal.

[0317] Step 12:

[0318] Terminal: Decodes the received analysis results and displays them on the user interface. Formats the information so that the user can easily check the results.

[0319] Step 13:

[0320] User: Review the analysis results and take any necessary follow-up actions, such as reporting to a manager or relevant department, or providing additional training.

[0321] Example 2

[0322] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] In internal meetings and interviews, there is a problem in that it is not possible to properly detect harassment behavior and analyze the user's emotional state and notify them appropriately. In particular, there is a need for a system that can convert voice data into text and quickly and accurately evaluate the content and emotional state of the text.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0325] In this invention, the server includes means for recording voice data, means for converting the recorded voice data into text data, means for analyzing the text data and voice data and evaluating the user's emotions, means for transmitting the text data and emotion evaluation results to the server, means for extracting problematic terms and phrases from the text data using natural language processing technology and detecting harassment behavior, means for generating recommended countermeasures based on the evaluation results, and means for notifying the user of the analysis results and recommended countermeasures. This makes it possible to quickly and accurately detect harassment behavior and analyze the user's emotional state during meetings and interviews.

[0326] "Audio data" refers to data in which the contents of a meeting or interview are recorded as audio.

[0327] "Text data" is data obtained by converting voice data into a character string.

[0328] "Emotion assessment" is the process of analyzing and assessing the emotional state (e.g., anger, anxiety, joy, etc.) from the voice and words uttered by the user.

[0329] A "server" is a computer system that processes, stores, and analyzes data over a network.

[0330] "Natural language processing technology" is a technology for extracting specific keywords and phrases from text data and analyzing the content of that text.

[0331] "Harassment" means unjustifiably discriminating, threatening, abusive, discriminatory, or other inappropriate conduct against another person in the workplace or any other environment.

[0332] "Recommended countermeasures" are suggestions that outline specific actions and countermeasures that should be taken when harassment is detected.

[0333] A "report" is a document or digital file that contains the results of an analysis and is generated to inform a user.

[0334] A "user" is someone who uses the system to record and analyze meetings and interviews.

[0335] "Encryption" is the process of transforming data using a specific algorithm to protect its contents.

[0336] "Decoding" is the process of restoring encrypted data to its original form.

[0337] The present invention relates to a system for effectively detecting harassment behavior in meetings and interviews, analyzing the user's emotions, and notifying the user. Specific embodiments of the system will be described in detail below.

[0338] System configuration

[0339] This system consists of multiple components, including the user's device, a server, and an emotion engine. Users can record the contents of meetings and interviews using a dedicated application and analyze the data in real time. The analysis results are then notified to the user, allowing them to take any necessary follow-up actions.

[0340] Hardware and Software Configuration

[0341] Device: The device used by the user, such as a smartphone or tablet.

[0342] Built-in microphone: An audio input device built into the device.

[0343] Local storage: Memory for temporarily storing voice data within the device.

[0344] Speech Recognition Software: Software used to convert recorded voice data into text data, such as the Google Cloud Speech-to-Text API.

[0345] Emotion engine: Software that analyzes emotions based on the user's voice data, such as IBM Watson (registered trademark).

[0346] Server: A remote computer system responsible for storing and analyzing data.

[0347] Natural Language Processing (NLP) engines: Includes technologies such as the Google NLP API that analyze text data and extract specific keywords and phrases.

[0348] Specific implementation steps

[0349] First, the user launches the application and taps the "Start Recording" button at the beginning of a meeting or interview. The device's built-in microphone begins collecting audio data, which is temporarily saved in the device's local storage. Once recording is complete, the audio data is sent to the Google Cloud Speech-to-Text API and converted into text. At the same time, the audio data is input into the IBM Watson Emotion Engine to analyze the user's emotional state.

[0350] The converted text data and the emotion engine's analysis results are sent to a server using AES-256 encryption. The server then analyzes the text data using the Google NLP API to extract specific keywords and phrases. For example, a phrase such as "It's your fault that we didn't achieve our goals" is detected and evaluated to determine whether it constitutes harassment.

[0351] The analysis results of the emotion engine are also used as supplementary data to further refine the evaluation results of harassment behavior. Finally, if harassment behavior is detected based on the output of the NLP engine and emotion engine, a report containing detailed information and recommended countermeasures is generated. This report is sent from the server to the device and displayed on the user interface. The user can review the report and take any necessary follow-up actions.

[0352] Specific Examples

[0353] Example 1: Meeting recording, harassment detection, and emotion analysis

[0354] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0355] 2. Device: Collects the recorded data and converts it into text using the Google Cloud Speech-to-Text API, while simultaneously analyzing the user's emotions using the IBM Watson emotion engine.

[0356] 3. Terminal: The converted text data and the emotion engine analysis results are sent to the server using AES-256 encryption.

[0357] 4. Server: Analyzes the text data and detects phrases such as "If we fail to achieve our goal, it's your fault." Using sentiment analysis, the server adds the user's strong anxiety to the score.

[0358] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0359] 6. Terminal: Receives the report and notifies the user.

[0360] 7. User: Review the report and take any necessary follow-up actions.

[0361] Example 2: Individual consultation minutes, harassment detection, and sentiment analysis

[0362] 1. User: Enter the details of the individual consultation into the app as minutes.

[0363] 2. Terminal: Sends the minutes data to the server. Sentiment analysis is also performed by the emotion engine in parallel.

[0364] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0365] 4. Server: Based on the results of sentiment analysis, the user indicates strong disappointment and adds it to the rating.

[0366] 5. Server: Determines that this constitutes harassment and generates a report with recommended action.

[0367] 6. Terminal: Receives the report and notifies the user.

[0368] 7. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[0369] This system will improve the quality of internal communication, detect harassment at an early stage, and take appropriate measures. Furthermore, by analyzing the user's emotional state, more accurate evaluations and countermeasures can be realized.

[0370] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0371] Step 1:

[0372] Start recording audio data

[0373] Users simply launch the application and tap the "Start Recording" button at the start of a meeting or interview.

[0374] Input: Launch the application and tap the "Start Recording" button.

[0375] Output: Begin recording audio data.

[0376] The device uses the built-in microphone to collect audio data and temporarily stores it in the device's local storage.

[0377] What it does: When the user presses the "Start Recording" button, the device's built-in microphone captures the sound and saves it as an audio file.

[0378] Step 2:

[0379] Converting audio data to text

[0380] After the user presses the stop recording button, the device sends the collected voice data to the Google Cloud Speech-to-Text API and converts it into text data.

[0381] Input: Recorded audio data.

[0382] Output: The converted text data.

[0383] Specific operation: Voice data is sent to voice recognition software in real time, the voice waveform is analyzed, and text data is generated as an appropriate string of characters.

[0384] Step 3:

[0385] Emotion Analysis

[0386] The device sends the voice data to the IBM Watson emotion engine to analyze the user's emotional state.

[0387] Input: Audio data.

[0388] Output: Emotion evaluation results.

[0389] The emotion engine analyzes the emotional expressions contained in the voice data and evaluates the user's emotional state.

[0390] Specific operation: The voice data is sent to the emotion engine, where the emotional state (e.g., anger, anxiety, joy, etc.) is analyzed. The analyzed emotional state is recorded as the calculation result.

[0391] Step 4:

[0392] Sending data

[0393] The device then sends the converted text data and the results of sentiment analysis to the server using AES-256 encryption.

[0394] Input: Text data and emotion evaluation results.

[0395] Output: Sending encrypted data.

[0396] How it works: Text data and sentiment analysis results are encrypted and sent securely over the internet to a server. The encryption process is typically done using the OpenSSL library.

[0397] Step 5:

[0398] Text data analysis

[0399] The server inputs the received text data into the Google NLP API, extracts specific keywords and phrases from the text data, and detects harassment behavior.

[0400] Input: Encrypted text data.

[0401] Output: Keyword / phrase extraction results and harassment rating.

[0402] The server uses the results of the emotion analysis as supplementary data to refine the assessment of harassment behavior.

[0403] How it works: Text data is sent to an NLP engine, which extracts specific keywords and phrases. For example, phrases like "If we fail to achieve our goal, it's your fault" are detected, and an overall evaluation is made taking into account the results of sentiment analysis.

[0404] Step 6:

[0405] Generation and notification of analysis results

[0406] Based on the output of the NLP engine and emotion engine, the server generates a report containing detailed information and recommended countermeasures if any harassment behavior is detected.

[0407] Input: Harassment behavior assessment results and sentiment analysis results.

[0408] Output: Reports are generated and sent to the terminal.

[0409] The server then sends the generated report to the device.

[0410] The terminal decodes the received report and displays it in a user interface, formatting the information so that the user can easily view the results.

[0411] Specific actions: A report is created on the server and sent to the device. The report contains specific information, such as "A superior's comment that 'You have no room for improvement' constitutes harassment." The user can review the report and take any necessary follow-up actions.

[0412] (Application example 2)

[0413] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] Communication between workers and machines is an important element in factories. However, harassment can occur not only between workers but also between workers and robots, which can lead to a deterioration of the work environment and a drop in productivity. It is also necessary to properly analyze the emotional state of workers on-site and respond quickly to maintain a safe and healthy work environment. Therefore, a system is needed to detect such harassment at an early stage and take appropriate measures.

[0415] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording voice data, means for converting the recorded voice data into text data, means for analyzing the emotional state from the recorded voice data and text data, means for analyzing the text data to detect harassment behavior, means for generating recommended countermeasures, and means for notifying the user of the analysis results and the countermeasures. This makes it possible to detect harassment behavior by workers in a factory early and take appropriate measures, thereby maintaining a safe and healthy working environment.

[0416] "Recording voice data" is the process of storing human voice as digital data using the device's internal microphone.

[0417] "Text conversion" is the process of converting recorded audio data into written information, often using speech recognition software.

[0418] "Emotional state analysis" is the process of detecting emotional expressions from audio and text data and assessing individual emotional states (e.g., anger, anxiety, joy, etc.).

[0419] "Harassment detection" is the process of analyzing specific keywords and phrases contained in text data or audio data using natural language processing technology to determine whether they constitute harassment.

[0420] "Sending analysis results" is the process of securely sending the analyzed data and results to the server.

[0421] "Generating recommended countermeasures" is the process of generating information that suggests appropriate actions and countermeasures based on the analysis results.

[0422] "Notifying the user" is the process of communicating the generated analysis results and countermeasures to the user using means such as voice or a display.

[0423] The present invention relates to a system that can detect harassment occurring in communication between workers and robots in a factory at an early stage, analyze the emotional state of the workers, and take appropriate measures. Specific embodiments of the system are described in detail below.

[0424] System configuration

[0425] This system consists of a factory robot equipped with a built-in microphone, a server that analyzes the data, and a terminal that notifies the user. The main functions of the system are as follows:

[0426] 1. Recording audio data

[0427] The robot uses a built-in microphone to record conversations with workers in real time, and this audio data is temporarily stored in local storage.

[0428] 2. Converting voice data to text

[0429] Once the recording is complete, the robot uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the audio data into text data, which is then used for subsequent analysis.

[0430] 3. Emotion Analysis

[0431] The robot inputs the generated text and voice data into an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze its emotional state. Emotion analysis evaluates emotional states such as anger, anxiety, and joy.

[0432] 4. Data submission and analysis

[0433] The robot sends the converted text data and sentiment analysis results to a server using encryption (e.g., OpenSSL), which then analyzes the text data using a natural language processing (NLP) engine (e.g., Spacy, BERT model) to assess whether specific keywords or phrases constitute harassing behavior.

[0434] 5. Generating recommended countermeasures

[0435] If harassment behavior is detected, the server generates detailed information and recommended countermeasures based on the output of the NLP engine and emotion engine, creating a report that combines the analysis results and recommended countermeasures.

[0436] 6. Notification of analysis results

[0437] The report generated by the server is sent to the robot, which then notifies the worker of the received report via voice or display, allowing the worker to immediately understand the problem and take appropriate follow-up action.

[0438] Hardware and software used

[0439] Robots: Factory robots with built-in microphones (e.g., Pepper robots)

[0440] Speech recognition software: Google Cloud Speech-to-Text API

[0441] Emotion engine: IBM Watson Tone Analyzer

[0442] Encryption: OpenSSL

[0443] Natural language processing engine: Spacy, BERT model

[0444] Specific examples

[0445] Here is an example of a working system:

[0446] 1. Example 1: Production line instructions

[0447] A worker on a production line tells a robot, "If we don't meet our goal, it's all your fault."

[0448] The robot records this speech, converts it into text data, and sends it to a server.

[0449] The server analyzes this text, determines it is harassment, and uses the sentiment analysis results to generate recommended countermeasures.

[0450] The generated report is sent to the robot and notifies the worker.

[0451] 2. Example 2: Comments on the production line

[0452] A worker says to a robot, "You're doing your job really badly. There's no room for improvement."

[0453] The robot records this speech, converts it into text data, and sends it to a server.

[0454] The server analyzes this text, determines it is harassment, and uses the sentiment analysis results to generate recommended countermeasures.

[0455] The generated report is sent to the robot and notifies the worker.

[0456] This system makes it possible to maintain a safe and healthy working environment within the factory.

[0457] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0458] Step 1:

[0459] When the robot starts to interact with a worker, it automatically starts recording voice data. The voice data collected using the built-in microphone is temporarily saved in local storage. The input is human voice. The output is digital voice data.

[0460] Step 2:

[0461] The robot uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the recorded voice data into text data. The input is voice data, and the output is text data converted into text information. In this conversion process, the waveform of the voice data is analyzed and converted into the most appropriate string of characters.

[0462] Step 3:

[0463] The robot inputs text and voice data into an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the worker's emotional state. The input is text and voice data, and the output is emotional state information such as anger, anxiety, and joy. The emotion analysis process analyzes the tone of the text and the intonation of the voice.

[0464] Step 4:

[0465] The robot sends the converted text data and the emotion analysis results to the server using encryption (e.g., OpenSSL). The input is text data and emotional state information, and the output is securely encrypted transmission data. The encryption process securely processes the data so that it cannot be deciphered by third parties.

[0466] Step 5:

[0467] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., Spacy, BERT model) and evaluates whether specific keywords and phrases constitute harassment. The input is the text data, and the output is a list of problematic terms and phrases and the evaluation results. This analysis process involves grammatically analyzing the text data to detect harmful expressions.

[0468] Step 6:

[0469] If harassment behavior is detected, the server generates recommended countermeasures based on the results of the NLP engine and emotion engine. The inputs are the evaluation results and emotional state information, and the output is countermeasures and detailed analysis results. The countermeasure generation process refers to past data and case studies to determine the optimal response method.

[0470] Step 7:

[0471] The server sends the generated countermeasure proposals and a report of the analysis results to the robot. The robot notifies the worker of the received report by voice or display. The inputs are countermeasure proposals and analysis results, and the output is a notification to the worker. In the notification process, the information in the report is conveyed to the worker in an easy-to-understand format.

[0472] In this way, the system can detect harassment in factories at an early stage and take appropriate measures.

[0473] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0474] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0475] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0476] [Second embodiment]

[0477] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0478] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0479] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0480] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0481] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0482] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0483] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0484] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0485] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0486] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0487] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0488] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0489] The present invention relates to a system for detecting harassment behavior during internal company meetings and interviews. An embodiment of the system will be described in detail below.

[0490] This system consists of a user-owned device, an associated server, and data communications between them. Users use an application to record audio from meetings and interviews in real time. The device converts the recorded audio data into text data and sends it to the server.

[0491] System Overview

[0492] 1. Recording audio data

[0493] User: At the start of a meeting or interview, launch the dedicated application and tap the "Start Recording" button.

[0494] Device: Uses the built-in microphone to begin collecting audio data, which is temporarily stored in local storage.

[0495] 2. Converting voice data to text

[0496] Terminal: Once the recording is complete, the collected voice data is converted into text data using speech recognition software, which analyzes the voice waveform and generates the most appropriate string of characters.

[0497] 3. Data transmission

[0498] Terminal: The converted text data, and manually entered minutes data if necessary, are sent to the server. Before being sent, the data is encrypted to ensure its security.

[0499] 4. Text Data Analysis

[0500] Server: The received text data is passed through a natural language processing (NLP) engine, which extracts specific keywords and phrases from the text and evaluates whether they constitute harassment.

[0501] Server: Based on the analysis results, if harassment is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0502] 5. Notification of analysis results

[0503] Server: Organizes the analysis results and generates them in report format.

[0504] Server: Sends the generated report to the terminal.

[0505] Terminal: Notifies the user of the received analysis results and displays them so that the user can easily check them.

[0506] Specific examples

[0507] Example 1: Meeting recording and harassment detection

[0508] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0509] 2. Terminal: Collects recorded data and converts it into text data using speech recognition software after the meeting is over.

[0510] 3. Terminal: Sends the converted text data to the server.

[0511] 4. Server: The text data is analyzed and phrases such as "If we fail to achieve our goal, it's your fault" are detected.

[0512] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0513] 6. Terminal: Receives the report and notifies the user.

[0514] 7. User: Review the report and take any necessary follow-up actions.

[0515] Example 2: Minutes of individual consultations and harassment detection

[0516] 1. User: Enter the details of the individual consultation into the app as minutes.

[0517] 2. Terminal: Sends the minutes data to the server.

[0518] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0519] 4. Server: Determines that this constitutes harassment and generates a report with recommended actions.

[0520] 5. Terminal: Receives the report and notifies the user.

[0521] 6. User: Review the report and take action to resolve the issue with your manager if necessary.

[0522] This system can provide an effective means of improving the quality of business communication, early detection of harassment, and appropriate responses.

[0523] The processing flow will be explained below.

[0524] Step 1:

[0525] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[0526] Step 2:

[0527] Device: Begins collecting audio data using the built-in microphone, which is saved to local storage in real time.

[0528] Step 3:

[0529] User: When the meeting or interview is over, tap the "Stop Recording" button in the application.

[0530] Step 4:

[0531] Terminal: The recorded voice data is input into voice recognition software, which converts the voice data into text data. The voice recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[0532] Step 5:

[0533] Terminal: The converted text data and, if necessary, manually entered minutes data are sent to the server. The data is encrypted before being sent to ensure its security.

[0534] Step 6:

[0535] Server: Receives data packets sent from the device and decompresses and decrypts the data.

[0536] Step 7:

[0537] Server: Inputs the received text data into a natural language processing (NLP) engine to extract specific keywords and phrases from the text, and evaluates whether the extracted keywords and phrases constitute harassment.

[0538] Step 8:

[0539] Server: Based on the output of the NLP engine, if harassment is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0540] Step 9:

[0541] Server: Sends the generated analysis result report to the terminal.

[0542] Step 10:

[0543] Terminal: Decodes the received analysis results and displays them in the user interface, formatting them so that the user can easily view them.

[0544] Step 11:

[0545] User: Review the analysis results and take any necessary follow-up actions, such as reporting to a manager or relevant department, or providing additional training.

[0546] Example 1

[0547] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] There is a need to prevent harassment during company meetings and interviews, and to quickly detect it if it does occur and take appropriate measures. However, there is currently a lack of systems that can detect harassment in real time and quickly propose appropriate measures. This has led to problems with delays in the early detection of harassment and effective responses.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0550] In this invention, the server includes means for converting voice data into text data, means for encrypting and transmitting the text data to the server, means for analyzing the text data using natural language processing technology to detect harassment, means for generating recommended countermeasures based on the analysis results, and means for notifying the user of the analysis results and the countermeasures, thereby enabling early detection of harassment and prompt provision of appropriate countermeasures.

[0551] "Audio data" refers to digital data that records audio from meetings, interviews, etc.

[0552] "Text data" is character string data generated by analyzing voice data.

[0553] "Encryption" is a technology that converts data using a special algorithm to prevent it from being intercepted by third parties.

[0554] A "server" is a system that receives requests from clients via a network and processes and stores data.

[0555] A "user" is a person who operates the system and records audio data of meetings and interviews.

[0556] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0557] "Analysis" is the process of examining data in detail and understanding its meaning and structure.

[0558] "Harassment" is any act that causes discomfort and mental stress to an individual.

[0559] "Countermeasures" are specific methods or measures proposed to solve a problem.

[0560] "Notification" is the act of conveying information to a specific recipient.

[0561] A "generative AI model" is a model that uses artificial intelligence to automatically generate new information and suggestions from data.

[0562] A "prompt sentence" is a specific instruction sentence that is input to a generative AI model.

[0563] The present invention relates to a system for detecting harassment behavior during internal meetings and interviews, and an embodiment thereof is described in detail below. The system is composed of a terminal owned by a user, an associated server, and data communications between them.

[0564] First, the user installs a dedicated application on their device (such as a smartphone or PC). After launching the application and tapping the "Start Recording" button at the start of a meeting or interview, the device will use its built-in microphone to record audio data in real time. The recorded audio data is temporarily stored in the device's local storage.

[0565] Next, once the meeting or interview is over, the device converts the collected voice data into text data using speech recognition software (such as Google Speech-to-Text API or Amazon Transcribe). This speech recognition software analyzes the voice waveform and generates the most appropriate string of characters. The generated text data is then encrypted (for example, using TLS / SSL protocol) and sent to the server.

[0566] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., spaCy, Google Cloud Natural Language API, etc.). The NLP engine extracts specific keywords and phrases from the text and evaluates whether they constitute harassing behavior. In particular, it applies a set of rules based on past cases and laws to perform a detailed analysis of the text data.

[0567] Based on the analysis results, the server uses a generative AI model to generate recommended countermeasures, including specific measures (e.g., "early reporting of problems" and "provision of mental health care"). The generated analysis results and countermeasures are then organized into a report and sent to the device.

[0568] Finally, the device notifies the user of the received report and can display the report within the application, where the user can open the report, review the details, and take any necessary action.

[0569] For example, the following specific example is shown.

[0570] Example 1: Meeting recording and harassment detection

[0571] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0572] 2. Terminal: Collects recorded data and converts it into text data using voice recognition software after the meeting is over.

[0573] 3. Terminal: Sends the converted text data to the server.

[0574] 4. Server: The text data is analyzed and phrases such as "If we fail to achieve our goal, it's your fault" are detected.

[0575] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0576] 6. Terminal: Receives the report and notifies the user.

[0577] 7. User: Review the report and take any necessary follow-up actions.

[0578] Example 2: Minutes of individual consultations and harassment detection

[0579] 1. User: Enter the details of the individual consultation into the app as minutes.

[0580] 2. Terminal: Sends the minutes data to the server.

[0581] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0582] 4. Server: Determines that this constitutes harassment and generates a report with recommended actions.

[0583] 5. Terminal: Receives the report and notifies the user.

[0584] 6. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[0585] The following can be used as an example prompt:

[0586] Example prompt 1:

[0587] Please analyze the following conversation and rate whether it contains harassing behavior:

[0588] 'Boss: If you fail to achieve your goals, it's all your fault.'

[0589] Based on the analysis results, generate a report and propose appropriate countermeasures.

[0590] Example prompt 2:

[0591] Please analyze the following minutes and assess whether they contain any harassing behavior:

[0592] Subordinate: I think there is room for improvement in this proposal.

[0593] Boss: You have no room for improvement.'

[0594] Based on the analysis results, generate a report and propose appropriate countermeasures.

[0595] This system can improve the quality of business communication and provide an effective means of supporting the early detection of harassment and appropriate responses.

[0596] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0597] Step 1:

[0598] User: Launch the dedicated application on a device such as a smartphone or PC, and tap the "Start Recording" button at the start of the meeting or interview.

[0599] Input: Application launch and button tap.

[0600] Specific operation: When the application is launched and the "Start Recording" button is pressed, the built-in microphone starts working.

[0601] Output: Recording start trigger set by device.

[0602] Step 2:

[0603] Device: Uses the built-in microphone to collect voice data in real time and temporarily stores it in local storage.

[0604] Input: Audio during a meeting or interview.

[0605] What it does: The microphone captures ambient sounds and stores them as digital audio data in file formats such as WAV and MP3.

[0606] Output: Audio data file saved to local storage.

[0607] Step 3:

[0608] Device: After recording is complete, the collected voice data is sent to speech recognition software (such as Google Speech-to-Text API), which begins analyzing the voice waveform and converting it into text data.

[0609] Input: Saved audio data file.

[0610] What it does: Speech recognition software analyzes the audio data and converts the audio waveform into text.

[0611] Output: The generated text data.

[0612] Step 4:

[0613] Terminal: The converted text data and manually entered minutes data are encrypted and sent to the server.

[0614] Input: Text data and meeting minutes data.

[0615] What it does: Encodes data using the TLS / SSL protocol and sends it securely to the server.

[0616] Output: Encrypted data reaches the server.

[0617] Step 5:

[0618] Server: Decodes the received data and analyzes the text data using a natural language processing (NLP) engine (such as spaCy).

[0619] Input: Encrypted text data.

[0620] What it does: The decoded text data is analyzed by an NLP engine to extract specific keywords and phrases.

[0621] Output: A dataset of the analysis results.

[0622] Step 6:

[0623] Server: Based on the analysis results, a generative AI model is used to generate recommended countermeasures.

[0624] Input: NLP engine analysis result dataset.

[0625] How it works: The generative AI model takes the analysis results as input and generates appropriate countermeasures based on past data and case studies.

[0626] Output: A report with recommended actions.

[0627] Step 7:

[0628] Server: Sends the generated report to the terminal.

[0629] Input: A report with recommended actions.

[0630] Specific operation: Encodes the report data and transmits it securely to the terminal.

[0631] Output: The report arrives at the terminal.

[0632] Step 8:

[0633] Terminal: Notifies and displays received reports to the user.

[0634] Input: Report received from server.

[0635] Specific operation: The terminal decodes the report data and notifies the user through the user interface. The contents of the report are displayed on the screen.

[0636] Output: The report is available for the user to review.

[0637] Through the above processing steps, this system enables early detection of harassment behavior and provides appropriate countermeasures.

[0638] (Application example 1)

[0639] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] There is a need for a system that can quickly detect harassment that occurs in communication between workers in factories and take appropriate measures. However, existing systems lack real-time capabilities and can be slow to notify managers, making it difficult to respond quickly and effectively.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0642] In this invention, the server includes means for encrypting voice data and transmitting it to the server, means for analyzing text data using a natural language processing engine and generating recommended countermeasures when harassment is detected, and means for receiving the report generated by the server and notifying the administrator in PDF format. This makes it possible to detect harassment in real time and quickly notify the administrator.

[0643] "Audio Data" means audio information recorded in analog or digital format.

[0644] "Text data" is character string information obtained by processing voice data.

[0645] "Encryption" is the process of transforming data using a specific algorithm to ensure its security.

[0646] A "natural language processing engine" is a computer program that analyzes text data and understands its meaning and context.

[0647] "Recommended actions" indicate actions or countermeasures to be taken when a problem is detected.

[0648] A "report" is a document summarizing the analysis results and countermeasures.

[0649] A "server" is a central computer system for storing and processing data.

[0650] "Real-time" refers to a time frame in which data processing and information provision occurs immediately without delay.

[0651] This invention relates to a system for quickly detecting harassment behavior in communication between workers in a factory and taking appropriate measures. This system is composed of a robot assistant that patrols the factory, an associated server, and data communication between them.

[0652] System Overview

[0653] Recording audio data

[0654] The robot uses a built-in microphone to record conversations between workers in the factory in real time, and the recordings are temporarily saved in local storage.

[0655] Converting audio data to text

[0656] Once the recording is complete, the collected audio data is converted to text using speech recognition software (e.g., Google Cloud Speech-to-Text API), which analyzes the audio waveform and generates the most appropriate string of characters.

[0657] Sending data

[0658] The converted text data is encrypted using a data encryption tool (e.g., AES encryption) and sent to a server via a reliable communication (e.g., HTTPS communication).

[0659] Text data analysis

[0660] The server then runs the received text data through a natural language processing (NLP) engine (such as OpenAI's GPT-4). The NLP engine extracts specific keywords and phrases from the text and evaluates whether they constitute harassing behavior. Based on the evaluation results, if harassing behavior is detected, it generates detailed information (such as the words used, the speaker, and the context) along with recommended countermeasures.

[0661] Notification of analysis results

[0662] The server compiles the analysis results and generates a report. This report is output in PDF format and sent to the administrator, who can then take appropriate measures promptly based on the report.

[0663] Specific examples of hardware and software used

[0664] Hardware: Robot for patrolling the factory, built-in microphone, local storage, internet communication module

[0665] software:

[0666] Speech recognition software (Google Cloud Speech-to-Text API)

[0667] Natural language processing engine (OpenAI's GPT-4)

[0668] Data encryption tool (AES encryption)

[0669] Data transmission module (HTTPS communication)

[0670] Report generation tools (e.g. PDF generation libraries)

[0671] Specific examples

[0672] Prompt Sentence Examples

[0673] markdown

[0674] Analyze the following conversation text to detect harassing behavior, and if so, generate details and appropriate countermeasures.

[0675] Conversation text:

[0676] Worker A: "You caused the line to stop, and it's all your fault."

[0677] Worker B: "Sorry, I'll be more careful next time."

[0678] Worker A: "It's not enough to just be careful. If you do it again, you're fired."

[0679] result:

[0680] 1. Type of conduct: Harassment

[0681] 2. Key phrases detected: "It's your fault", "You're fired"

[0682] 3. Recommended measures:

[0683] Recommend that Worker A take harassment prevention training

[0684] Follow-up meetings held to improve the work environment

[0685] In this way, the system that is the subject of this patent is extremely effective in improving human relationships and the working environment within a factory.

[0686] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0687] Step 1:

[0688] The user activates the robot assistant in an area of ​​the factory where communication between workers takes place. The robot uses a built-in microphone to record conversations in real time. The recorded data (input) is temporarily saved in local storage (output).

[0689] Step 2:

[0690] The robot's terminal converts the temporarily stored voice data into text data using voice recognition software (Google Cloud Speech-to-Text API). The voice data (input) is converted into text data (output) as character string information through analysis of the voice waveform. Specifically, the voice file is sent via the cloud to the voice recognition service, and the returned text data is received.

[0691] Step 3:

[0692] The terminal encrypts the converted text data using the AES encryption tool. The text data (input) is encrypted using a specified key to produce secure ciphertext data (output). During the encryption process, the text data is transformed through an algorithm to make it unintelligible to outsiders.

[0693] Step 4:

[0694] The terminal sends encrypted text data to the server via HTTPS communication. At this time, the encrypted text data (input) is sent to the server (output) using a secure communication protocol. Specifically, SSL / TLS is used to prevent data interception.

[0695] Step 5:

[0696] The server decrypts the ciphertext data it receives. The server decrypts the ciphertext data (input) with the appropriate key to obtain the original text data (output). Decryption is performed by reversing the encryption process.

[0697] Step 6:

[0698] The server analyzes the text data using a natural language processing engine (OpenAI's GPT-4). Specific keywords and phrases are extracted from the text data (input) and evaluated to determine whether they constitute harassment. The natural language processing engine generates the analysis results, and the output indicates whether or not the behavior constitutes harassment.

[0699] Step 7:

[0700] The server generates recommended countermeasures based on the analysis results. The analysis results (input) are converted into report-format data (output) that includes countermeasures. Specifically, the NLP engine proposes candidate countermeasures and compiles them into a report.

[0701] Step 8:

[0702] The server generates a report in PDF format and notifies the administrator. This report data (input) is output as a PDF file (output) using a PDF generation library and sent to the administrator's terminal.

[0703] Step 9:

[0704] The administrator checks the received report and takes appropriate measures. The administrator's device displays the received PDF file (input) and allows the contents to be checked (output). Specifically, the administrator uses a PDF viewer to view the report and select and execute the necessary actions.

[0705] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0706] The present invention relates to a system for effectively detecting harassment behavior in internal meetings and interviews, analyzing the user's emotions, and notifying the user. Specific embodiments of the system will be described in detail below.

[0707] This system consists of multiple components, including the user's device, a server, and an emotion engine. Users can use the application to record the contents of meetings and interviews and analyze the data in real time.

[0708] System Overview

[0709] 1. Recording audio data

[0710] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[0711] Device: Begins collecting audio data using the built-in microphone, which is temporarily stored in local storage.

[0712] 2. Converting voice data to text

[0713] Terminal: Once the recording is complete, the voice data is input into speech recognition software, which converts it into text data. The speech recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[0714] 3. Emotion Analysis

[0715] Device: During and after recording, the voice data is input into an emotion engine to analyze the user's emotional state (e.g., anger, anxiety, joy, etc.).

[0716] Emotion engine: Analyzes emotional expressions contained in voice data and evaluates the user's emotional state.

[0717] 4. Data transmission

[0718] Terminal: The converted text data and the emotion engine analysis results are sent to the server. The data is encrypted before being sent to ensure its security.

[0719] 5. Analysis of Text Data

[0720] Server: Inputs the received text data into a natural language processing (NLP) engine to extract specific keywords and phrases from the text, and evaluates whether the extracted keywords and phrases constitute harassment.

[0721] Server: Uses the analysis results of the emotion engine as additional data to complement the assessment results of harassment behavior.

[0722] 6. Generation and notification of analysis results

[0723] Server: Based on the output of the NLP engine and emotion engine, if harassment behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0724] Server: Sends the generated analysis result report to the terminal.

[0725] Terminal: Decodes the received analysis results and displays them on the user interface. Formats the information so that the user can easily check the results.

[0726] Specific examples

[0727] Example 1: Meeting recording, harassment detection, and emotion analysis

[0728] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0729] 2. Device: Collects recorded data and converts it into text using speech recognition software, while simultaneously analyzing the user's emotions using an emotion engine.

[0730] 3. Terminal: The converted text data and the emotion engine analysis results are sent to the server.

[0731] 4. Server: Analyzes the text data and detects phrases such as "If we fail to achieve our goal, it's your fault." Using sentiment analysis, the server adds the user's strong anxiety to the score.

[0732] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0733] 6. Terminal: Receives the report and notifies the user.

[0734] 7. User: Review the report and take any necessary follow-up actions.

[0735] Example 2: Individual consultation minutes, harassment detection, and sentiment analysis

[0736] 1. User: Enter the details of the individual consultation into the app as minutes.

[0737] 2. Terminal: The minutes data is sent to the server. Emotion analysis is also performed by the emotion engine in parallel.

[0738] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0739] 4. Server: Based on the results of sentiment analysis, the user indicates strong disappointment and adds it to the rating.

[0740] 5. Server: Determines that this constitutes harassment and generates a report with recommended action.

[0741] 6. Terminal: Receives the report and notifies the user.

[0742] 7. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[0743] This system will improve the quality of internal communication, enable early detection of harassment, and enable appropriate responses. Furthermore, by analyzing the user's emotional state, more accurate evaluations and countermeasures can be achieved.

[0744] The processing flow will be explained below.

[0745] Step 1:

[0746] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[0747] Step 2:

[0748] Device: Begins collecting audio data using the built-in microphone, which is saved to local storage in real time.

[0749] Step 3:

[0750] On the device, the voice data collected at the same time is input into the emotion engine, which analyzes the user's emotional state in real time by analyzing voice characteristics such as tone, speed, and volume.

[0751] Step 4:

[0752] User: When the meeting or interview is over, tap the "Stop Recording" button in the application.

[0753] Step 5:

[0754] Terminal: The recorded voice data is input into voice recognition software, which converts the voice data into text data. The voice recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[0755] Step 6:

[0756] Device: The converted text data and the emotion engine analysis results are sent to the server. The data is encrypted before being sent to ensure its security.

[0757] Step 7:

[0758] Server: Receives data packets sent from the device and decompresses and decrypts the data.

[0759] Step 8:

[0760] Server: The received text data is input into a natural language processing (NLP) engine to extract specific keywords and phrases from the text. The NLP engine then analyzes the text data and evaluates whether it constitutes harassment.

[0761] Step 9:

[0762] Server: Use the results of the emotion engine analysis as additional data to complement the assessment of harassment behavior, paying particular attention to when the user has negative emotions.

[0763] Step 10:

[0764] Server: Based on the output of the NLP engine and emotion engine, if harassment behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0765] Step 11:

[0766] Server: Sends the generated analysis result report to the terminal.

[0767] Step 12:

[0768] Terminal: Decodes the received analysis results and displays them on the user interface. Formats the information so that the user can easily check the results.

[0769] Step 13:

[0770] User: Review the analysis results and take any necessary follow-up actions, such as reporting to a manager or relevant department, or providing additional training.

[0771] Example 2

[0772] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0773] In internal meetings and interviews, there is a problem in that it is not possible to properly detect harassment behavior and analyze the user's emotional state and notify them appropriately. In particular, there is a need for a system that can convert voice data into text and quickly and accurately evaluate the content and emotional state of the text.

[0774] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0775] In this invention, the server includes means for recording voice data, means for converting the recorded voice data into text data, means for analyzing the text data and voice data and evaluating the user's emotions, means for transmitting the text data and emotion evaluation results to the server, means for extracting problematic terms and phrases from the text data using natural language processing technology and detecting harassment behavior, means for generating recommended countermeasures based on the evaluation results, and means for notifying the user of the analysis results and recommended countermeasures. This makes it possible to quickly and accurately detect harassment behavior and analyze the user's emotional state during meetings and interviews.

[0776] "Audio data" refers to data in which the contents of a meeting or interview are recorded as audio.

[0777] "Text data" is data obtained by converting voice data into a character string.

[0778] "Emotion assessment" is the process of analyzing and assessing the emotional state (e.g., anger, anxiety, joy, etc.) from the voice and words uttered by the user.

[0779] A "server" is a computer system that processes, stores, and analyzes data over a network.

[0780] "Natural language processing technology" is a technology for extracting specific keywords and phrases from text data and analyzing the content of that text.

[0781] "Harassment" means unjustifiably discriminating, threatening, abusive, discriminatory, or other inappropriate conduct against another person in the workplace or any other environment.

[0782] "Recommended countermeasures" are suggestions that outline specific actions and countermeasures that should be taken when harassment is detected.

[0783] A "report" is a document or digital file that contains the results of an analysis and is generated to inform a user.

[0784] A "user" is someone who uses the system to record and analyze meetings and interviews.

[0785] "Encryption" is the process of transforming data using a specific algorithm to protect its contents.

[0786] "Decoding" is the process of restoring encrypted data to its original form.

[0787] The present invention relates to a system for effectively detecting harassment behavior in meetings and interviews, analyzing the user's emotions, and notifying the user. Specific embodiments of the system will be described in detail below.

[0788] System configuration

[0789] This system consists of multiple components, including the user's device, a server, and an emotion engine. Users can record the contents of meetings and interviews using a dedicated application and analyze the data in real time. The analysis results are then notified to the user, allowing them to take any necessary follow-up actions.

[0790] Hardware and Software Configuration

[0791] Device: The device used by the user, such as a smartphone or tablet.

[0792] Built-in microphone: An audio input device built into the device.

[0793] Local storage: Memory for temporarily storing voice data within the device.

[0794] Speech Recognition Software: Software used to convert recorded voice data into text data, such as the Google Cloud Speech-to-Text API.

[0795] Emotion engine: Software such as IBM Watson that analyzes emotions based on user voice data.

[0796] Server: A remote computer system responsible for storing and analyzing data.

[0797] Natural Language Processing (NLP) engines: Includes technologies such as the Google NLP API that analyze text data and extract specific keywords and phrases.

[0798] Specific implementation steps

[0799] First, the user launches the application and taps the "Start Recording" button at the beginning of a meeting or interview. The device's built-in microphone begins collecting audio data, which is temporarily saved in the device's local storage. Once recording is complete, the audio data is sent to the Google Cloud Speech-to-Text API and converted into text. At the same time, the audio data is input into the IBM Watson Emotion Engine to analyze the user's emotional state.

[0800] The converted text data and the emotion engine's analysis results are sent to a server using AES-256 encryption. The server then analyzes the text data using the Google NLP API to extract specific keywords and phrases. For example, a phrase such as "It's your fault that we didn't achieve our goals" is detected and evaluated to determine whether it constitutes harassment.

[0801] The analysis results of the emotion engine are also used as supplementary data to further refine the evaluation results of harassment behavior. Finally, if harassment behavior is detected based on the output of the NLP engine and emotion engine, a report containing detailed information and recommended countermeasures is generated. This report is sent from the server to the device and displayed on the user interface. The user can review the report and take any necessary follow-up actions.

[0802] Specific Examples

[0803] Example 1: Meeting recording, harassment detection, and emotion analysis

[0804] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0805] 2. Device: Collects the recorded data and converts it into text using the Google Cloud Speech-to-Text API, while simultaneously analyzing the user's emotions using the IBM Watson emotion engine.

[0806] 3. Terminal: The converted text data and the emotion engine analysis results are sent to the server using AES-256 encryption.

[0807] 4. Server: Analyzes the text data and detects phrases such as "If we fail to achieve our goal, it's your fault." Using sentiment analysis, the server adds the user's strong anxiety to the score.

[0808] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0809] 6. Terminal: Receives the report and notifies the user.

[0810] 7. User: Review the report and take any necessary follow-up actions.

[0811] Example 2: Individual consultation minutes, harassment detection, and sentiment analysis

[0812] 1. User: Enter the details of the individual consultation into the app as minutes.

[0813] 2. Terminal: Sends the minutes data to the server. Sentiment analysis is also performed by the emotion engine in parallel.

[0814] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0815] 4. Server: Based on the results of sentiment analysis, the user indicates strong disappointment and adds it to the rating.

[0816] 5. Server: Determines that this constitutes harassment and generates a report with recommended action.

[0817] 6. Terminal: Receives the report and notifies the user.

[0818] 7. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[0819] This system will improve the quality of internal communication, detect harassment at an early stage, and take appropriate measures. Furthermore, by analyzing the user's emotional state, more accurate evaluations and countermeasures can be realized.

[0820] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0821] Step 1:

[0822] Start recording audio data

[0823] Users simply launch the application and tap the "Start Recording" button at the start of a meeting or interview.

[0824] Input: Launch the application and tap the "Start Recording" button.

[0825] Output: Begin recording audio data.

[0826] The device uses the built-in microphone to collect audio data and temporarily stores it in the device's local storage.

[0827] What it does: When the user presses the "Start Recording" button, the device's built-in microphone captures the sound and saves it as an audio file.

[0828] Step 2:

[0829] Converting audio data to text

[0830] After the user presses the stop recording button, the device sends the collected voice data to the Google Cloud Speech-to-Text API and converts it into text data.

[0831] Input: Recorded audio data.

[0832] Output: The converted text data.

[0833] Specific operation: Voice data is sent to voice recognition software in real time, the voice waveform is analyzed, and text data is generated as an appropriate string of characters.

[0834] Step 3:

[0835] Emotion Analysis

[0836] The device sends the voice data to the IBM Watson emotion engine to analyze the user's emotional state.

[0837] Input: Audio data.

[0838] Output: Emotion evaluation results.

[0839] The emotion engine analyzes the emotional expressions contained in the voice data and evaluates the user's emotional state.

[0840] Specific operation: The voice data is sent to the emotion engine, where the emotional state (e.g., anger, anxiety, joy, etc.) is analyzed. The analyzed emotional state is recorded as the calculation result.

[0841] Step 4:

[0842] Sending data

[0843] The device then sends the converted text data and the results of sentiment analysis to the server using AES-256 encryption.

[0844] Input: Text data and emotion evaluation results.

[0845] Output: Sending encrypted data.

[0846] How it works: Text data and sentiment analysis results are encrypted and sent securely over the internet to a server. The encryption process is typically done using the OpenSSL library.

[0847] Step 5:

[0848] Text data analysis

[0849] The server inputs the received text data into the Google NLP API, extracts specific keywords and phrases from the text data, and detects harassment behavior.

[0850] Input: Encrypted text data.

[0851] Output: Keyword / phrase extraction results and harassment rating.

[0852] The server uses the results of the emotion analysis as supplementary data to refine the assessment of harassment behavior.

[0853] How it works: Text data is sent to an NLP engine, which extracts specific keywords and phrases. For example, phrases like "If we fail to achieve our goal, it's your fault" are detected, and an overall evaluation is made taking into account the results of sentiment analysis.

[0854] Step 6:

[0855] Generation and notification of analysis results

[0856] Based on the output of the NLP engine and emotion engine, the server generates a report containing detailed information and recommended countermeasures if any harassment behavior is detected.

[0857] Input: Harassment behavior assessment results and sentiment analysis results.

[0858] Output: Reports are generated and sent to the terminal.

[0859] The server then sends the generated report to the device.

[0860] The terminal decodes the received report and displays it in a user interface, formatting the information so that the user can easily view the results.

[0861] Specific actions: A report is created on the server and sent to the device. The report contains specific information, such as "A superior's comment that 'You have no room for improvement' constitutes harassment." The user can review the report and take any necessary follow-up actions.

[0862] (Application example 2)

[0863] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0864] Communication between workers and machines is an important element in factories. However, harassment can occur not only between workers but also between workers and robots, which can lead to a deterioration of the work environment and a drop in productivity. It is also necessary to properly analyze the emotional state of workers on-site and respond quickly to maintain a safe and healthy work environment. Therefore, a system is needed to detect such harassment at an early stage and take appropriate measures.

[0865] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording voice data, means for converting the recorded voice data into text data, means for analyzing the emotional state from the recorded voice data and text data, means for analyzing the text data to detect harassment behavior, means for generating recommended countermeasures, and means for notifying the user of the analysis results and the countermeasures. This makes it possible to detect harassment behavior by workers in a factory early and take appropriate measures, thereby maintaining a safe and healthy working environment.

[0866] "Recording voice data" is the process of storing human voice as digital data using the device's internal microphone.

[0867] "Text conversion" is the process of converting recorded audio data into written information, often using speech recognition software.

[0868] "Emotional state analysis" is the process of detecting emotional expressions from audio and text data and assessing individual emotional states (e.g., anger, anxiety, joy, etc.).

[0869] "Harassment detection" is the process of analyzing specific keywords and phrases contained in text data or audio data using natural language processing technology to determine whether they constitute harassment.

[0870] "Sending analysis results" is the process of securely sending the analyzed data and results to the server.

[0871] "Generating recommended countermeasures" is the process of generating information that suggests appropriate actions and countermeasures based on the analysis results.

[0872] "Notifying the user" is the process of communicating the generated analysis results and countermeasures to the user using means such as voice or a display.

[0873] The present invention relates to a system that can detect harassment occurring in communication between workers and robots in a factory at an early stage, analyze the emotional state of the workers, and take appropriate measures. Specific embodiments of the system are described in detail below.

[0874] System configuration

[0875] This system consists of a factory robot equipped with a built-in microphone, a server that analyzes the data, and a terminal that notifies the user. The main functions of the system are as follows:

[0876] 1. Recording audio data

[0877] The robot uses a built-in microphone to record conversations with workers in real time, and this audio data is temporarily stored in local storage.

[0878] 2. Converting voice data to text

[0879] Once the recording is complete, the robot uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the audio data into text data, which is then used for subsequent analysis.

[0880] 3. Emotion Analysis

[0881] The robot inputs the generated text and voice data into an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze its emotional state. Emotion analysis evaluates emotional states such as anger, anxiety, and joy.

[0882] 4. Data submission and analysis

[0883] The robot sends the converted text data and sentiment analysis results to a server using encryption (e.g., OpenSSL), which then analyzes the text data using a natural language processing (NLP) engine (e.g., Spacy, BERT model) to assess whether specific keywords or phrases constitute harassing behavior.

[0884] 5. Generating recommended countermeasures

[0885] If harassment behavior is detected, the server generates detailed information and recommended countermeasures based on the output of the NLP engine and emotion engine, creating a report that combines the analysis results and recommended countermeasures.

[0886] 6. Notification of analysis results

[0887] The report generated by the server is sent to the robot, which then notifies the worker of the received report via voice or display, allowing the worker to immediately understand the problem and take appropriate follow-up action.

[0888] Hardware and software used

[0889] Robots: Factory robots with built-in microphones (e.g., Pepper robots)

[0890] Speech recognition software: Google Cloud Speech-to-Text API

[0891] Emotion engine: IBM Watson Tone Analyzer

[0892] Encryption: OpenSSL

[0893] Natural language processing engine: Spacy, BERT model

[0894] Specific examples

[0895] Here is an example of a working system:

[0896] 1. Example 1: Production line instructions

[0897] A worker on a production line tells a robot, "If we don't meet our goal, it's all your fault."

[0898] The robot records this speech, converts it into text data, and sends it to a server.

[0899] The server analyzes this text, determines it is harassment, and uses the sentiment analysis results to generate recommended countermeasures.

[0900] The generated report is sent to the robot and notifies the worker.

[0901] 2. Example 2: Comments on the production line

[0902] A worker says to a robot, "You're doing your job really badly. There's no room for improvement."

[0903] The robot records this speech, converts it into text data, and sends it to a server.

[0904] The server analyzes this text, determines it is harassment, and uses the sentiment analysis results to generate recommended countermeasures.

[0905] The generated report is sent to the robot and notifies the worker.

[0906] This system makes it possible to maintain a safe and healthy working environment within the factory.

[0907] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0908] Step 1:

[0909] When the robot starts to interact with a worker, it automatically starts recording voice data. The voice data collected using the built-in microphone is temporarily saved in local storage. The input is human voice. The output is digital voice data.

[0910] Step 2:

[0911] The robot uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the recorded voice data into text data. The input is voice data, and the output is text data converted into text information. In this conversion process, the waveform of the voice data is analyzed and converted into the most appropriate string of characters.

[0912] Step 3:

[0913] The robot inputs text and voice data into an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the worker's emotional state. The input is text and voice data, and the output is emotional state information such as anger, anxiety, and joy. The emotion analysis process analyzes the tone of the text and the intonation of the voice.

[0914] Step 4:

[0915] The robot sends the converted text data and the emotion analysis results to the server using encryption (e.g., OpenSSL). The input is text data and emotional state information, and the output is securely encrypted transmission data. The encryption process securely processes the data so that it cannot be deciphered by third parties.

[0916] Step 5:

[0917] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., Spacy, BERT model) and evaluates whether specific keywords and phrases constitute harassment. The input is the text data, and the output is a list of problematic terms and phrases and the evaluation results. This analysis process involves grammatically analyzing the text data to detect harmful expressions.

[0918] Step 6:

[0919] If harassment behavior is detected, the server generates recommended countermeasures based on the results of the NLP engine and emotion engine. The inputs are the evaluation results and emotional state information, and the output is countermeasures and detailed analysis results. The countermeasure generation process refers to past data and case studies to determine the optimal response method.

[0920] Step 7:

[0921] The server sends the generated countermeasure proposals and a report of the analysis results to the robot. The robot notifies the worker of the received report by voice or display. The inputs are countermeasure proposals and analysis results, and the output is a notification to the worker. In the notification process, the information in the report is conveyed to the worker in an easy-to-understand format.

[0922] In this way, the system can detect harassment in factories at an early stage and take appropriate measures.

[0923] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0924] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0925] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0926] [Third embodiment]

[0927] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0928] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0929] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0930] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0931] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0932] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0933] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0934] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0935] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0936] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0937] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0938] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0939] The present invention relates to a system for detecting harassment behavior during internal company meetings and interviews. An embodiment of the system will be described in detail below.

[0940] This system consists of a user-owned device, an associated server, and data communications between them. Users use an application to record audio from meetings and interviews in real time. The device converts the recorded audio data into text data and sends it to the server.

[0941] System Overview

[0942] 1. Recording audio data

[0943] User: At the start of a meeting or interview, launch the dedicated application and tap the "Start Recording" button.

[0944] Device: Uses the built-in microphone to begin collecting audio data, which is temporarily stored in local storage.

[0945] 2. Converting voice data to text

[0946] Terminal: Once the recording is complete, the collected voice data is converted into text data using speech recognition software, which analyzes the voice waveform and generates the most appropriate string of characters.

[0947] 3. Data transmission

[0948] Terminal: The converted text data, and manually entered minutes data if necessary, are sent to the server. Before being sent, the data is encrypted to ensure its security.

[0949] 4. Text Data Analysis

[0950] Server: The received text data is passed through a natural language processing (NLP) engine, which extracts specific keywords and phrases from the text and evaluates whether they constitute harassment.

[0951] Server: Based on the analysis results, if harassment is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0952] 5. Notification of analysis results

[0953] Server: Organizes the analysis results and generates them in report format.

[0954] Server: Sends the generated report to the terminal.

[0955] Terminal: Notifies the user of the received analysis results and displays them so that the user can easily check them.

[0956] Specific examples

[0957] Example 1: Meeting recording and harassment detection

[0958] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[0959] 2. Terminal: Collects recorded data and converts it into text data using speech recognition software after the meeting is over.

[0960] 3. Terminal: Sends the converted text data to the server.

[0961] 4. Server: The text data is analyzed and phrases such as "If we fail to achieve our goal, it's your fault" are detected.

[0962] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[0963] 6. Terminal: Receives the report and notifies the user.

[0964] 7. User: Review the report and take any necessary follow-up actions.

[0965] Example 2: Minutes of individual consultations and harassment detection

[0966] 1. User: Enter the details of the individual consultation into the app as minutes.

[0967] 2. Terminal: Sends the minutes data to the server.

[0968] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[0969] 4. Server: Determines that this constitutes harassment and generates a report with recommended actions.

[0970] 5. Terminal: Receives the report and notifies the user.

[0971] 6. User: Review the report and take action to resolve the issue with your manager if necessary.

[0972] This system can provide an effective means of improving the quality of business communication, early detection of harassment, and appropriate responses.

[0973] The processing flow will be explained below.

[0974] Step 1:

[0975] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[0976] Step 2:

[0977] Device: Begins collecting audio data using the built-in microphone, which is saved to local storage in real time.

[0978] Step 3:

[0979] User: When the meeting or interview is over, tap the "Stop Recording" button in the application.

[0980] Step 4:

[0981] Terminal: The recorded voice data is input into voice recognition software, which converts the voice data into text data. The voice recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[0982] Step 5:

[0983] Terminal: The converted text data and, if necessary, manually entered minutes data are sent to the server. The data is encrypted before being sent to ensure its security.

[0984] Step 6:

[0985] Server: Receives data packets sent from the device and decompresses and decrypts the data.

[0986] Step 7:

[0987] Server: Inputs the received text data into a natural language processing (NLP) engine to extract specific keywords and phrases from the text, and evaluates whether the extracted keywords and phrases constitute harassment.

[0988] Step 8:

[0989] Server: Based on the output of the NLP engine, if harassment is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[0990] Step 9:

[0991] Server: Sends the generated analysis result report to the terminal.

[0992] Step 10:

[0993] Terminal: Decodes the received analysis results and displays them in the user interface, formatting them so that the user can easily view them.

[0994] Step 11:

[0995] User: Review the analysis results and take any necessary follow-up actions, such as reporting to a manager or relevant department, or providing additional training.

[0996] Example 1

[0997] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0998] There is a need to prevent harassment during company meetings and interviews, and to quickly detect it if it does occur and take appropriate measures. However, there is currently a lack of systems that can detect harassment in real time and quickly propose appropriate measures. This has led to problems with delays in the early detection of harassment and effective responses.

[0999] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1000] In this invention, the server includes means for converting voice data into text data, means for encrypting and transmitting the text data to the server, means for analyzing the text data using natural language processing technology to detect harassment, means for generating recommended countermeasures based on the analysis results, and means for notifying the user of the analysis results and the countermeasures, thereby enabling early detection of harassment and prompt provision of appropriate countermeasures.

[1001] "Audio data" refers to digital data that records audio from meetings, interviews, etc.

[1002] "Text data" is character string data generated by analyzing voice data.

[1003] "Encryption" is a technology that converts data using a special algorithm to prevent it from being intercepted by third parties.

[1004] A "server" is a system that receives requests from clients via a network and processes and stores data.

[1005] A "user" is a person who operates the system and records audio data of meetings and interviews.

[1006] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[1007] "Analysis" is the process of examining data in detail and understanding its meaning and structure.

[1008] "Harassment" is any act that causes discomfort and mental stress to an individual.

[1009] "Countermeasures" are specific methods or measures proposed to solve a problem.

[1010] "Notification" is the act of conveying information to a specific recipient.

[1011] A "generative AI model" is a model that uses artificial intelligence to automatically generate new information and suggestions from data.

[1012] A "prompt sentence" is a specific instruction sentence that is input to a generative AI model.

[1013] The present invention relates to a system for detecting harassment behavior during internal meetings and interviews, and an embodiment thereof is described in detail below. The system is composed of a terminal owned by a user, an associated server, and data communications between them.

[1014] First, the user installs a dedicated application on their device (such as a smartphone or PC). After launching the application and tapping the "Start Recording" button at the start of a meeting or interview, the device will use its built-in microphone to record audio data in real time. The recorded audio data is temporarily stored in the device's local storage.

[1015] Next, once the meeting or interview is over, the device converts the collected voice data into text data using speech recognition software (such as Google Speech-to-Text API or Amazon Transcribe). This speech recognition software analyzes the voice waveform and generates the most appropriate string of characters. The generated text data is then encrypted (for example, using TLS / SSL protocol) and sent to the server.

[1016] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., spaCy, Google Cloud Natural Language API, etc.). The NLP engine extracts specific keywords and phrases from the text and evaluates whether they constitute harassing behavior. In particular, it applies a set of rules based on past cases and laws to perform a detailed analysis of the text data.

[1017] Based on the analysis results, the server uses a generative AI model to generate recommended countermeasures, including specific measures (e.g., "early reporting of problems" and "provision of mental health care"). The generated analysis results and countermeasures are then organized into a report and sent to the device.

[1018] Finally, the device notifies the user of the received report and can display the report within the application, where the user can open the report, review the details, and take any necessary action.

[1019] For example, the following specific example is shown.

[1020] Example 1: Meeting recording and harassment detection

[1021] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[1022] 2. Terminal: Collects recorded data and converts it into text data using voice recognition software after the meeting is over.

[1023] 3. Terminal: Sends the converted text data to the server.

[1024] 4. Server: The text data is analyzed and phrases such as "If we fail to achieve our goal, it's your fault" are detected.

[1025] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[1026] 6. Terminal: Receives the report and notifies the user.

[1027] 7. User: Review the report and take any necessary follow-up actions.

[1028] Example 2: Minutes of individual consultations and harassment detection

[1029] 1. User: Enter the details of the individual consultation into the app as minutes.

[1030] 2. Terminal: Sends the minutes data to the server.

[1031] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[1032] 4. Server: Determines that this constitutes harassment and generates a report with recommended actions.

[1033] 5. Terminal: Receives the report and notifies the user.

[1034] 6. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[1035] The following can be used as an example prompt:

[1036] Example prompt 1:

[1037] Please analyze the following conversation and rate whether it contains harassing behavior:

[1038] 'Boss: If you fail to achieve your goals, it's all your fault.'

[1039] Based on the analysis results, generate a report and propose appropriate countermeasures.

[1040] Example prompt 2:

[1041] Please analyze the following minutes and assess whether they contain any harassing behavior:

[1042] Subordinate: I think there is room for improvement in this proposal.

[1043] Boss: You have no room for improvement.'

[1044] Based on the analysis results, generate a report and propose appropriate countermeasures.

[1045] This system can improve the quality of business communication and provide an effective means of supporting the early detection of harassment and appropriate responses.

[1046] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1047] Step 1:

[1048] User: Launch the dedicated application on a device such as a smartphone or PC, and tap the "Start Recording" button at the start of the meeting or interview.

[1049] Input: Application launch and button tap.

[1050] Specific operation: When the application is launched and the "Start Recording" button is pressed, the built-in microphone starts working.

[1051] Output: Recording start trigger set by device.

[1052] Step 2:

[1053] Device: Uses the built-in microphone to collect voice data in real time and temporarily stores it in local storage.

[1054] Input: Audio during a meeting or interview.

[1055] What it does: The microphone captures ambient sounds and stores them as digital audio data in file formats such as WAV and MP3.

[1056] Output: Audio data file saved to local storage.

[1057] Step 3:

[1058] Device: After recording is complete, the collected voice data is sent to speech recognition software (such as Google Speech-to-Text API), which begins analyzing the voice waveform and converting it into text data.

[1059] Input: Saved audio data file.

[1060] What it does: Speech recognition software analyzes the audio data and converts the audio waveform into text.

[1061] Output: The generated text data.

[1062] Step 4:

[1063] Terminal: The converted text data and manually entered minutes data are encrypted and sent to the server.

[1064] Input: Text data and meeting minutes data.

[1065] What it does: Encodes data using the TLS / SSL protocol and sends it securely to the server.

[1066] Output: Encrypted data reaches the server.

[1067] Step 5:

[1068] Server: Decodes the received data and analyzes the text data using a natural language processing (NLP) engine (such as spaCy).

[1069] Input: Encrypted text data.

[1070] What it does: The decoded text data is analyzed by an NLP engine to extract specific keywords and phrases.

[1071] Output: A dataset of the analysis results.

[1072] Step 6:

[1073] Server: Based on the analysis results, a generative AI model is used to generate recommended countermeasures.

[1074] Input: NLP engine analysis result dataset.

[1075] How it works: The generative AI model takes the analysis results as input and generates appropriate countermeasures based on past data and case studies.

[1076] Output: A report with recommended actions.

[1077] Step 7:

[1078] Server: Sends the generated report to the terminal.

[1079] Input: A report with recommended actions.

[1080] Specific operation: Encodes the report data and transmits it securely to the terminal.

[1081] Output: The report arrives at the terminal.

[1082] Step 8:

[1083] Terminal: Notifies and displays received reports to the user.

[1084] Input: Report received from server.

[1085] Specific operation: The terminal decodes the report data and notifies the user through the user interface. The contents of the report are displayed on the screen.

[1086] Output: The report is available for the user to review.

[1087] Through the above processing steps, this system enables early detection of harassment behavior and provides appropriate countermeasures.

[1088] (Application example 1)

[1089] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1090] There is a need for a system that can quickly detect harassment that occurs in communication between workers in factories and take appropriate measures. However, existing systems lack real-time capabilities and can be slow to notify managers, making it difficult to respond quickly and effectively.

[1091] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1092] In this invention, the server includes means for encrypting voice data and transmitting it to the server, means for analyzing text data using a natural language processing engine and generating recommended countermeasures when harassment is detected, and means for receiving the report generated by the server and notifying the administrator in PDF format. This makes it possible to detect harassment in real time and quickly notify the administrator.

[1093] "Audio Data" means audio information recorded in analog or digital format.

[1094] "Text data" is character string information obtained by processing voice data.

[1095] "Encryption" is the process of transforming data using a specific algorithm to ensure its security.

[1096] A "natural language processing engine" is a computer program that analyzes text data and understands its meaning and context.

[1097] "Recommended actions" indicate actions or countermeasures to be taken when a problem is detected.

[1098] A "report" is a document summarizing the analysis results and countermeasures.

[1099] A "server" is a central computer system for storing and processing data.

[1100] "Real-time" refers to a time frame in which data processing and information provision occurs immediately without delay.

[1101] This invention relates to a system for quickly detecting harassment behavior in communication between workers in a factory and taking appropriate measures. This system is composed of a robot assistant that patrols the factory, an associated server, and data communication between them.

[1102] System Overview

[1103] Recording audio data

[1104] The robot uses a built-in microphone to record conversations between workers in the factory in real time, and the recordings are temporarily saved in local storage.

[1105] Converting audio data to text

[1106] Once the recording is complete, the collected audio data is converted to text using speech recognition software (e.g., Google Cloud Speech-to-Text API), which analyzes the audio waveform and generates the most appropriate string of characters.

[1107] Sending data

[1108] The converted text data is encrypted using a data encryption tool (e.g., AES encryption) and sent to a server via a reliable communication (e.g., HTTPS communication).

[1109] Text data analysis

[1110] The server then runs the received text data through a natural language processing (NLP) engine (such as OpenAI's GPT-4). The NLP engine extracts specific keywords and phrases from the text and evaluates whether they constitute harassing behavior. Based on the evaluation results, if harassing behavior is detected, it generates detailed information (such as the words used, the speaker, and the context) along with recommended countermeasures.

[1111] Notification of analysis results

[1112] The server compiles the analysis results and generates a report. This report is output in PDF format and sent to the administrator, who can then take appropriate measures promptly based on the report.

[1113] Specific examples of hardware and software used

[1114] Hardware: Robot for patrolling the factory, built-in microphone, local storage, internet communication module

[1115] software:

[1116] Speech recognition software (Google Cloud Speech-to-Text API)

[1117] Natural language processing engine (OpenAI's GPT-4)

[1118] Data encryption tool (AES encryption)

[1119] Data transmission module (HTTPS communication)

[1120] Report generation tools (e.g. PDF generation libraries)

[1121] Specific examples

[1122] Prompt Sentence Examples

[1123] markdown

[1124] Analyze the following conversation text to detect harassing behavior, and if so, generate details and appropriate countermeasures.

[1125] Conversation text:

[1126] Worker A: "You caused the line to stop, and it's all your fault."

[1127] Worker B: "Sorry, I'll be more careful next time."

[1128] Worker A: "It's not enough to just be careful. If you do it again, you're fired."

[1129] result:

[1130] 1. Type of conduct: Harassment

[1131] 2. Key phrases detected: "It's your fault", "You're fired"

[1132] 3. Recommended measures:

[1133] Recommend that Worker A take harassment prevention training

[1134] Follow-up meetings held to improve the work environment

[1135] In this way, the system that is the subject of this patent is extremely effective in improving human relationships and the working environment within a factory.

[1136] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1137] Step 1:

[1138] The user activates the robot assistant in an area of ​​the factory where communication between workers takes place. The robot uses a built-in microphone to record conversations in real time. The recorded data (input) is temporarily saved in local storage (output).

[1139] Step 2:

[1140] The robot's terminal converts the temporarily stored voice data into text data using voice recognition software (Google Cloud Speech-to-Text API). The voice data (input) is converted into text data (output) as character string information through analysis of the voice waveform. Specifically, the voice file is sent via the cloud to the voice recognition service, and the returned text data is received.

[1141] Step 3:

[1142] The terminal encrypts the converted text data using the AES encryption tool. The text data (input) is encrypted using a specified key to produce secure ciphertext data (output). During the encryption process, the text data is transformed through an algorithm to make it unintelligible to outsiders.

[1143] Step 4:

[1144] The terminal sends encrypted text data to the server via HTTPS communication. At this time, the encrypted text data (input) is sent to the server (output) using a secure communication protocol. Specifically, SSL / TLS is used to prevent data interception.

[1145] Step 5:

[1146] The server decrypts the ciphertext data it receives. The server decrypts the ciphertext data (input) with the appropriate key to obtain the original text data (output). Decryption is performed by reversing the encryption process.

[1147] Step 6:

[1148] The server analyzes the text data using a natural language processing engine (OpenAI's GPT-4). Specific keywords and phrases are extracted from the text data (input) and evaluated to determine whether they constitute harassment. The natural language processing engine generates the analysis results, and the output indicates whether or not the behavior constitutes harassment.

[1149] Step 7:

[1150] The server generates recommended countermeasures based on the analysis results. The analysis results (input) are converted into report-format data (output) that includes countermeasures. Specifically, the NLP engine proposes candidate countermeasures and compiles them into a report.

[1151] Step 8:

[1152] The server generates a report in PDF format and notifies the administrator. This report data (input) is output as a PDF file (output) using a PDF generation library and sent to the administrator's terminal.

[1153] Step 9:

[1154] The administrator checks the received report and takes appropriate measures. The administrator's device displays the received PDF file (input) and allows the contents to be checked (output). Specifically, the administrator uses a PDF viewer to view the report and select and execute the necessary actions.

[1155] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1156] The present invention relates to a system for effectively detecting harassment behavior in internal meetings and interviews, analyzing the user's emotions, and notifying the user. Specific embodiments of the system will be described in detail below.

[1157] This system consists of multiple components, including the user's device, a server, and an emotion engine. Users can use the application to record the contents of meetings and interviews and analyze the data in real time.

[1158] System Overview

[1159] 1. Recording audio data

[1160] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[1161] Device: Begins collecting audio data using the built-in microphone, which is temporarily stored in local storage.

[1162] 2. Converting voice data to text

[1163] Terminal: Once the recording is complete, the voice data is input into speech recognition software, which converts it into text data. The speech recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[1164] 3. Emotion Analysis

[1165] Device: During and after recording, the voice data is input into an emotion engine to analyze the user's emotional state (e.g., anger, anxiety, joy, etc.).

[1166] Emotion engine: Analyzes emotional expressions contained in voice data and evaluates the user's emotional state.

[1167] 4. Data transmission

[1168] Terminal: The converted text data and the emotion engine analysis results are sent to the server. The data is encrypted before being sent to ensure its security.

[1169] 5. Analysis of Text Data

[1170] Server: Inputs the received text data into a natural language processing (NLP) engine to extract specific keywords and phrases from the text, and evaluates whether the extracted keywords and phrases constitute harassment.

[1171] Server: Uses the analysis results of the emotion engine as additional data to complement the assessment results of harassment behavior.

[1172] 6. Generation and notification of analysis results

[1173] Server: Based on the output of the NLP engine and emotion engine, if harassment behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[1174] Server: Sends the generated analysis result report to the terminal.

[1175] Terminal: Decodes the received analysis results and displays them on the user interface. Formats the information so that the user can easily check the results.

[1176] Specific examples

[1177] Example 1: Meeting recording, harassment detection, and emotion analysis

[1178] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[1179] 2. Device: Collects recorded data and converts it into text using speech recognition software, while simultaneously analyzing the user's emotions using an emotion engine.

[1180] 3. Terminal: The converted text data and the emotion engine analysis results are sent to the server.

[1181] 4. Server: Analyzes the text data and detects phrases such as "If we fail to achieve our goal, it's your fault." Using sentiment analysis, the server adds the user's strong anxiety to the score.

[1182] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[1183] 6. Terminal: Receives the report and notifies the user.

[1184] 7. User: Review the report and take any necessary follow-up actions.

[1185] Example 2: Individual consultation minutes, harassment detection, and sentiment analysis

[1186] 1. User: Enter the details of the individual consultation into the app as minutes.

[1187] 2. Terminal: The minutes data is sent to the server. Emotion analysis is also performed by the emotion engine in parallel.

[1188] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[1189] 4. Server: Based on the results of sentiment analysis, the user indicates strong disappointment and adds it to the rating.

[1190] 5. Server: Determines that this constitutes harassment and generates a report with recommended action.

[1191] 6. Terminal: Receives the report and notifies the user.

[1192] 7. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[1193] This system will improve the quality of internal communication, enable early detection of harassment, and enable appropriate responses. Furthermore, by analyzing the user's emotional state, more accurate evaluations and countermeasures can be achieved.

[1194] The processing flow will be explained below.

[1195] Step 1:

[1196] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[1197] Step 2:

[1198] Device: Begins collecting audio data using the built-in microphone, which is saved to local storage in real time.

[1199] Step 3:

[1200] On the device, the voice data collected at the same time is input into the emotion engine, which analyzes the user's emotional state in real time by analyzing voice characteristics such as tone, speed, and volume.

[1201] Step 4:

[1202] User: When the meeting or interview is over, tap the "Stop Recording" button in the application.

[1203] Step 5:

[1204] Terminal: The recorded voice data is input into voice recognition software, which converts the voice data into text data. The voice recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[1205] Step 6:

[1206] Device: The converted text data and the emotion engine analysis results are sent to the server. The data is encrypted before being sent to ensure its security.

[1207] Step 7:

[1208] Server: Receives data packets sent from the device and decompresses and decrypts the data.

[1209] Step 8:

[1210] Server: The received text data is input into a natural language processing (NLP) engine to extract specific keywords and phrases from the text. The NLP engine then analyzes the text data and evaluates whether it constitutes harassment.

[1211] Step 9:

[1212] Server: Use the results of the emotion engine analysis as additional data to complement the assessment of harassment behavior, paying particular attention to when the user has negative emotions.

[1213] Step 10:

[1214] Server: Based on the output of the NLP engine and emotion engine, if harassment behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[1215] Step 11:

[1216] Server: Sends the generated analysis result report to the terminal.

[1217] Step 12:

[1218] Terminal: Decodes the received analysis results and displays them on the user interface. Formats the information so that the user can easily check the results.

[1219] Step 13:

[1220] User: Review the analysis results and take any necessary follow-up actions, such as reporting to a manager or relevant department, or providing additional training.

[1221] Example 2

[1222] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1223] In internal meetings and interviews, there is a problem in that it is not possible to properly detect harassment behavior and analyze the user's emotional state and notify them appropriately. In particular, there is a need for a system that can convert voice data into text and quickly and accurately evaluate the content and emotional state of the text.

[1224] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1225] In this invention, the server includes means for recording voice data, means for converting the recorded voice data into text data, means for analyzing the text data and voice data and evaluating the user's emotions, means for transmitting the text data and emotion evaluation results to the server, means for extracting problematic terms and phrases from the text data using natural language processing technology and detecting harassment behavior, means for generating recommended countermeasures based on the evaluation results, and means for notifying the user of the analysis results and recommended countermeasures. This makes it possible to quickly and accurately detect harassment behavior and analyze the user's emotional state during meetings and interviews.

[1226] "Audio data" refers to data in which the contents of a meeting or interview are recorded as audio.

[1227] "Text data" is data obtained by converting voice data into a character string.

[1228] "Emotion assessment" is the process of analyzing and assessing the emotional state (e.g., anger, anxiety, joy, etc.) from the voice and words uttered by the user.

[1229] A "server" is a computer system that processes, stores, and analyzes data over a network.

[1230] "Natural language processing technology" is a technology for extracting specific keywords and phrases from text data and analyzing the content of that text.

[1231] "Harassment" means unjustifiably discriminating, threatening, abusive, discriminatory, or other inappropriate conduct against another person in the workplace or any other environment.

[1232] "Recommended countermeasures" are suggestions that outline specific actions and countermeasures that should be taken when harassment is detected.

[1233] A "report" is a document or digital file that contains the results of an analysis and is generated to inform a user.

[1234] A "user" is someone who uses the system to record and analyze meetings and interviews.

[1235] "Encryption" is the process of transforming data using a specific algorithm to protect its contents.

[1236] "Decoding" is the process of restoring encrypted data to its original form.

[1237] The present invention relates to a system for effectively detecting harassment behavior in meetings and interviews, analyzing the user's emotions, and notifying the user. Specific embodiments of the system will be described in detail below.

[1238] System configuration

[1239] This system consists of multiple components, including the user's device, a server, and an emotion engine. Users can record the contents of meetings and interviews using a dedicated application and analyze the data in real time. The analysis results are then notified to the user, allowing them to take any necessary follow-up actions.

[1240] Hardware and Software Configuration

[1241] Device: The device used by the user, such as a smartphone or tablet.

[1242] Built-in microphone: An audio input device built into the device.

[1243] Local storage: Memory for temporarily storing voice data within the device.

[1244] Speech Recognition Software: Software used to convert recorded voice data into text data, such as the Google Cloud Speech-to-Text API.

[1245] Emotion engine: Software such as IBM Watson that analyzes emotions based on user voice data.

[1246] Server: A remote computer system responsible for storing and analyzing data.

[1247] Natural Language Processing (NLP) engines: Includes technologies such as the Google NLP API that analyze text data and extract specific keywords and phrases.

[1248] Specific implementation steps

[1249] First, the user launches the application and taps the "Start Recording" button at the beginning of a meeting or interview. The device's built-in microphone begins collecting audio data, which is temporarily saved in the device's local storage. Once recording is complete, the audio data is sent to the Google Cloud Speech-to-Text API and converted into text. At the same time, the audio data is input into the IBM Watson Emotion Engine to analyze the user's emotional state.

[1250] The converted text data and the emotion engine's analysis results are sent to a server using AES-256 encryption. The server then analyzes the text data using the Google NLP API to extract specific keywords and phrases. For example, a phrase such as "It's your fault that we didn't achieve our goals" is detected and evaluated to determine whether it constitutes harassment.

[1251] The analysis results of the emotion engine are also used as supplementary data to further refine the evaluation results of harassment behavior. Finally, if harassment behavior is detected based on the output of the NLP engine and emotion engine, a report containing detailed information and recommended countermeasures is generated. This report is sent from the server to the device and displayed on the user interface. The user can review the report and take any necessary follow-up actions.

[1252] Specific Examples

[1253] Example 1: Meeting recording, harassment detection, and emotion analysis

[1254] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[1255] 2. Device: Collects the recorded data and converts it into text using the Google Cloud Speech-to-Text API, while simultaneously analyzing the user's emotions using the IBM Watson emotion engine.

[1256] 3. Terminal: The converted text data and the emotion engine analysis results are sent to the server using AES-256 encryption.

[1257] 4. Server: Analyzes the text data and detects phrases such as "If we fail to achieve our goal, it's your fault." Using sentiment analysis, the server adds the user's strong anxiety to the score.

[1258] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[1259] 6. Terminal: Receives the report and notifies the user.

[1260] 7. User: Review the report and take any necessary follow-up actions.

[1261] Example 2: Individual consultation minutes, harassment detection, and sentiment analysis

[1262] 1. User: Enter the details of the individual consultation into the app as minutes.

[1263] 2. Terminal: Sends the minutes data to the server. Sentiment analysis is also performed by the emotion engine in parallel.

[1264] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[1265] 4. Server: Based on the results of sentiment analysis, the user indicates strong disappointment and adds it to the rating.

[1266] 5. Server: Determines that this constitutes harassment and generates a report with recommended action.

[1267] 6. Terminal: Receives the report and notifies the user.

[1268] 7. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[1269] This system will improve the quality of internal communication, detect harassment at an early stage, and take appropriate measures. Furthermore, by analyzing the user's emotional state, more accurate evaluations and countermeasures can be realized.

[1270] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1271] Step 1:

[1272] Start recording audio data

[1273] Users simply launch the application and tap the "Start Recording" button at the start of a meeting or interview.

[1274] Input: Launch the application and tap the "Start Recording" button.

[1275] Output: Begin recording audio data.

[1276] The device uses the built-in microphone to collect audio data and temporarily stores it in the device's local storage.

[1277] What it does: When the user presses the "Start Recording" button, the device's built-in microphone captures the sound and saves it as an audio file.

[1278] Step 2:

[1279] Converting audio data to text

[1280] After the user presses the stop recording button, the device sends the collected voice data to the Google Cloud Speech-to-Text API and converts it into text data.

[1281] Input: Recorded audio data.

[1282] Output: The converted text data.

[1283] Specific operation: Voice data is sent to voice recognition software in real time, the voice waveform is analyzed, and text data is generated as an appropriate string of characters.

[1284] Step 3:

[1285] Emotion Analysis

[1286] The device sends the voice data to the IBM Watson emotion engine to analyze the user's emotional state.

[1287] Input: Audio data.

[1288] Output: Emotion evaluation results.

[1289] The emotion engine analyzes the emotional expressions contained in the voice data and evaluates the user's emotional state.

[1290] Specific operation: The voice data is sent to the emotion engine, where the emotional state (e.g., anger, anxiety, joy, etc.) is analyzed. The analyzed emotional state is recorded as the calculation result.

[1291] Step 4:

[1292] Sending data

[1293] The device then sends the converted text data and the results of sentiment analysis to the server using AES-256 encryption.

[1294] Input: Text data and emotion evaluation results.

[1295] Output: Sending encrypted data.

[1296] How it works: Text data and sentiment analysis results are encrypted and sent securely over the internet to a server. The encryption process is typically done using the OpenSSL library.

[1297] Step 5:

[1298] Text data analysis

[1299] The server inputs the received text data into the Google NLP API, extracts specific keywords and phrases from the text data, and detects harassment behavior.

[1300] Input: Encrypted text data.

[1301] Output: Keyword / phrase extraction results and harassment rating.

[1302] The server uses the results of the emotion analysis as supplementary data to refine the assessment of harassment behavior.

[1303] How it works: Text data is sent to an NLP engine, which extracts specific keywords and phrases. For example, phrases like "If we fail to achieve our goal, it's your fault" are detected, and an overall evaluation is made taking into account the results of sentiment analysis.

[1304] Step 6:

[1305] Generation and notification of analysis results

[1306] Based on the output of the NLP engine and emotion engine, the server generates a report containing detailed information and recommended countermeasures if any harassment behavior is detected.

[1307] Input: Harassment behavior assessment results and sentiment analysis results.

[1308] Output: Reports are generated and sent to the terminal.

[1309] The server then sends the generated report to the device.

[1310] The terminal decodes the received report and displays it in a user interface, formatting the information so that the user can easily view the results.

[1311] Specific actions: A report is created on the server and sent to the device. The report contains specific information, such as "A superior's comment that 'You have no room for improvement' constitutes harassment." The user can review the report and take any necessary follow-up actions.

[1312] (Application example 2)

[1313] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1314] Communication between workers and machines is an important element in factories. However, harassment can occur not only between workers but also between workers and robots, which can lead to a deterioration of the work environment and a drop in productivity. It is also necessary to properly analyze the emotional state of workers on-site and respond quickly to maintain a safe and healthy work environment. Therefore, a system is needed to detect such harassment at an early stage and take appropriate measures.

[1315] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording voice data, means for converting the recorded voice data into text data, means for analyzing the emotional state from the recorded voice data and text data, means for analyzing the text data to detect harassment behavior, means for generating recommended countermeasures, and means for notifying the user of the analysis results and the countermeasures. This makes it possible to detect harassment behavior by workers in a factory early and take appropriate measures, thereby maintaining a safe and healthy working environment.

[1316] "Recording voice data" is the process of storing human voice as digital data using the device's internal microphone.

[1317] "Text conversion" is the process of converting recorded audio data into written information, often using speech recognition software.

[1318] "Emotional state analysis" is the process of detecting emotional expressions from audio and text data and assessing individual emotional states (e.g., anger, anxiety, joy, etc.).

[1319] "Harassment detection" is the process of analyzing specific keywords and phrases contained in text data or audio data using natural language processing technology to determine whether they constitute harassment.

[1320] "Sending analysis results" is the process of securely sending the analyzed data and results to the server.

[1321] "Generating recommended countermeasures" is the process of generating information that suggests appropriate actions and countermeasures based on the analysis results.

[1322] "Notifying the user" is the process of communicating the generated analysis results and countermeasures to the user using means such as voice or a display.

[1323] The present invention relates to a system that can detect harassment occurring in communication between workers and robots in a factory at an early stage, analyze the emotional state of the workers, and take appropriate measures. Specific embodiments of the system are described in detail below.

[1324] System configuration

[1325] This system consists of a factory robot equipped with a built-in microphone, a server that analyzes the data, and a terminal that notifies the user. The main functions of the system are as follows:

[1326] 1. Recording audio data

[1327] The robot uses a built-in microphone to record conversations with workers in real time, and this audio data is temporarily stored in local storage.

[1328] 2. Converting voice data to text

[1329] Once the recording is complete, the robot uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the audio data into text data, which is then used for subsequent analysis.

[1330] 3. Emotion Analysis

[1331] The robot inputs the generated text and voice data into an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze its emotional state. Emotion analysis evaluates emotional states such as anger, anxiety, and joy.

[1332] 4. Data submission and analysis

[1333] The robot sends the converted text data and sentiment analysis results to a server using encryption (e.g., OpenSSL), which then analyzes the text data using a natural language processing (NLP) engine (e.g., Spacy, BERT model) to assess whether specific keywords or phrases constitute harassing behavior.

[1334] 5. Generating recommended countermeasures

[1335] If harassment behavior is detected, the server generates detailed information and recommended countermeasures based on the output of the NLP engine and emotion engine, creating a report that combines the analysis results and recommended countermeasures.

[1336] 6. Notification of analysis results

[1337] The report generated by the server is sent to the robot, which then notifies the worker of the received report via voice or display, allowing the worker to immediately understand the problem and take appropriate follow-up action.

[1338] Hardware and software used

[1339] Robots: Factory robots with built-in microphones (e.g., Pepper robots)

[1340] Speech recognition software: Google Cloud Speech-to-Text API

[1341] Emotion engine: IBM Watson Tone Analyzer

[1342] Encryption: OpenSSL

[1343] Natural language processing engine: Spacy, BERT model

[1344] Specific examples

[1345] Here is an example of a working system:

[1346] 1. Example 1: Production line instructions

[1347] A worker on a production line tells a robot, "If we don't meet our goal, it's all your fault."

[1348] The robot records this speech, converts it into text data, and sends it to a server.

[1349] The server analyzes this text, determines it is harassment, and uses the sentiment analysis results to generate recommended countermeasures.

[1350] The generated report is sent to the robot and notifies the worker.

[1351] 2. Example 2: Comments on the production line

[1352] A worker says to a robot, "You're doing your job really badly. There's no room for improvement."

[1353] The robot records this speech, converts it into text data, and sends it to a server.

[1354] The server analyzes this text, determines it is harassment, and uses the sentiment analysis results to generate recommended countermeasures.

[1355] The generated report is sent to the robot and notifies the worker.

[1356] This system makes it possible to maintain a safe and healthy working environment within the factory.

[1357] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1358] Step 1:

[1359] When the robot starts to interact with a worker, it automatically starts recording voice data. The voice data collected using the built-in microphone is temporarily saved in local storage. The input is human voice. The output is digital voice data.

[1360] Step 2:

[1361] The robot uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the recorded voice data into text data. The input is voice data, and the output is text data converted into text information. In this conversion process, the waveform of the voice data is analyzed and converted into the most appropriate string of characters.

[1362] Step 3:

[1363] The robot inputs text and voice data into an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the worker's emotional state. The input is text and voice data, and the output is emotional state information such as anger, anxiety, and joy. The emotion analysis process analyzes the tone of the text and the intonation of the voice.

[1364] Step 4:

[1365] The robot sends the converted text data and the emotion analysis results to the server using encryption (e.g., OpenSSL). The input is text data and emotional state information, and the output is securely encrypted transmission data. The encryption process securely processes the data so that it cannot be deciphered by third parties.

[1366] Step 5:

[1367] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., Spacy, BERT model) and evaluates whether specific keywords and phrases constitute harassment. The input is the text data, and the output is a list of problematic terms and phrases and the evaluation results. This analysis process involves grammatically analyzing the text data to detect harmful expressions.

[1368] Step 6:

[1369] If harassment behavior is detected, the server generates recommended countermeasures based on the results of the NLP engine and emotion engine. The inputs are the evaluation results and emotional state information, and the output is countermeasures and detailed analysis results. The countermeasure generation process refers to past data and case studies to determine the optimal response method.

[1370] Step 7:

[1371] The server sends the generated countermeasure proposals and a report of the analysis results to the robot. The robot notifies the worker of the received report by voice or display. The inputs are countermeasure proposals and analysis results, and the output is a notification to the worker. In the notification process, the information in the report is conveyed to the worker in an easy-to-understand format.

[1372] In this way, the system can detect harassment in factories at an early stage and take appropriate measures.

[1373] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1374] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1375] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1376] [Fourth embodiment]

[1377] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1378] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1379] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1380] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1381] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1382] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1383] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1384] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1385] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1386] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1387] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1388] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1389] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1390] The present invention relates to a system for detecting harassment behavior during internal company meetings and interviews. An embodiment of the system will be described in detail below.

[1391] This system consists of a user-owned device, an associated server, and data communications between them. Users use an application to record audio from meetings and interviews in real time. The device converts the recorded audio data into text data and sends it to the server.

[1392] System Overview

[1393] 1. Recording audio data

[1394] User: At the start of a meeting or interview, launch the dedicated application and tap the "Start Recording" button.

[1395] Device: Uses the built-in microphone to begin collecting audio data, which is temporarily stored in local storage.

[1396] 2. Converting voice data to text

[1397] Terminal: Once the recording is complete, the collected voice data is converted into text data using speech recognition software, which analyzes the voice waveform and generates the most appropriate string of characters.

[1398] 3. Data transmission

[1399] Terminal: The converted text data, and manually entered minutes data if necessary, are sent to the server. Before being sent, the data is encrypted to ensure its security.

[1400] 4. Text Data Analysis

[1401] Server: The received text data is passed through a natural language processing (NLP) engine, which extracts specific keywords and phrases from the text and evaluates whether they constitute harassment.

[1402] Server: Based on the analysis results, if harassment is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[1403] 5. Notification of analysis results

[1404] Server: Organizes the analysis results and generates them in report format.

[1405] Server: Sends the generated report to the terminal.

[1406] Terminal: Notifies the user of the received analysis results and displays them so that the user can easily check them.

[1407] Specific examples

[1408] Example 1: Meeting recording and harassment detection

[1409] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[1410] 2. Terminal: Collects recorded data and converts it into text data using speech recognition software after the meeting is over.

[1411] 3. Terminal: Sends the converted text data to the server.

[1412] 4. Server: The text data is analyzed and phrases such as "If we fail to achieve our goal, it's your fault" are detected.

[1413] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[1414] 6. Terminal: Receives the report and notifies the user.

[1415] 7. User: Review the report and take any necessary follow-up actions.

[1416] Example 2: Minutes of individual consultations and harassment detection

[1417] 1. User: Enter the details of the individual consultation into the app as minutes.

[1418] 2. Terminal: Sends the minutes data to the server.

[1419] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[1420] 4. Server: Determines that this constitutes harassment and generates a report with recommended actions.

[1421] 5. Terminal: Receives the report and notifies the user.

[1422] 6. User: Review the report and take action to resolve the issue with your manager if necessary.

[1423] This system can provide an effective means of improving the quality of business communication, early detection of harassment, and appropriate responses.

[1424] The processing flow will be explained below.

[1425] Step 1:

[1426] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[1427] Step 2:

[1428] Device: Begins collecting audio data using the built-in microphone, which is saved to local storage in real time.

[1429] Step 3:

[1430] User: When the meeting or interview is over, tap the "Stop Recording" button in the application.

[1431] Step 4:

[1432] Terminal: The recorded voice data is input into voice recognition software, which converts the voice data into text data. The voice recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[1433] Step 5:

[1434] Terminal: The converted text data and, if necessary, manually entered minutes data are sent to the server. The data is encrypted before being sent to ensure its security.

[1435] Step 6:

[1436] Server: Receives data packets sent from the device and decompresses and decrypts the data.

[1437] Step 7:

[1438] Server: Inputs the received text data into a natural language processing (NLP) engine to extract specific keywords and phrases from the text, and evaluates whether the extracted keywords and phrases constitute harassment.

[1439] Step 8:

[1440] Server: Based on the output of the NLP engine, if harassment is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[1441] Step 9:

[1442] Server: Sends the generated analysis result report to the terminal.

[1443] Step 10:

[1444] Terminal: Decodes the received analysis results and displays them in the user interface, formatting them so that the user can easily view them.

[1445] Step 11:

[1446] User: Review the analysis results and take any necessary follow-up actions, such as reporting to a manager or relevant department, or providing additional training.

[1447] Example 1

[1448] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1449] There is a need to prevent harassment during company meetings and interviews, and to quickly detect it if it does occur and take appropriate measures. However, there is currently a lack of systems that can detect harassment in real time and quickly propose appropriate measures. This has led to problems with delays in the early detection of harassment and effective responses.

[1450] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1451] In this invention, the server includes means for converting voice data into text data, means for encrypting and transmitting the text data to the server, means for analyzing the text data using natural language processing technology to detect harassment, means for generating recommended countermeasures based on the analysis results, and means for notifying the user of the analysis results and the countermeasures, thereby enabling early detection of harassment and prompt provision of appropriate countermeasures.

[1452] "Audio data" refers to digital data that records audio from meetings, interviews, etc.

[1453] "Text data" is character string data generated by analyzing voice data.

[1454] "Encryption" is a technology that converts data using a special algorithm to prevent it from being intercepted by third parties.

[1455] A "server" is a system that receives requests from clients via a network and processes and stores data.

[1456] A "user" is a person who operates the system and records audio data of meetings and interviews.

[1457] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[1458] "Analysis" is the process of examining data in detail and understanding its meaning and structure.

[1459] "Harassment" is any act that causes discomfort and mental stress to an individual.

[1460] "Countermeasures" are specific methods or measures proposed to solve a problem.

[1461] "Notification" is the act of conveying information to a specific recipient.

[1462] A "generative AI model" is a model that uses artificial intelligence to automatically generate new information and suggestions from data.

[1463] A "prompt sentence" is a specific instruction sentence that is input to a generative AI model.

[1464] The present invention relates to a system for detecting harassment behavior during internal meetings and interviews, and an embodiment thereof is described in detail below. The system is composed of a terminal owned by a user, an associated server, and data communications between them.

[1465] First, the user installs a dedicated application on their device (such as a smartphone or PC). After launching the application and tapping the "Start Recording" button at the start of a meeting or interview, the device will use its built-in microphone to record audio data in real time. The recorded audio data is temporarily stored in the device's local storage.

[1466] Next, once the meeting or interview is over, the device converts the collected voice data into text data using speech recognition software (such as Google Speech-to-Text API or Amazon Transcribe). This speech recognition software analyzes the voice waveform and generates the most appropriate string of characters. The generated text data is then encrypted (for example, using TLS / SSL protocol) and sent to the server.

[1467] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., spaCy, Google Cloud Natural Language API, etc.). The NLP engine extracts specific keywords and phrases from the text and evaluates whether they constitute harassing behavior. In particular, it applies a set of rules based on past cases and laws to perform a detailed analysis of the text data.

[1468] Based on the analysis results, the server uses a generative AI model to generate recommended countermeasures, including specific measures (e.g., "early reporting of problems" and "provision of mental health care"). The generated analysis results and countermeasures are then organized into a report and sent to the device.

[1469] Finally, the device notifies the user of the received report and can display the report within the application, where the user can open the report, review the details, and take any necessary action.

[1470] For example, the following specific example is shown.

[1471] Example 1: Meeting recording and harassment detection

[1472] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[1473] 2. Terminal: Collects recorded data and converts it into text data using voice recognition software after the meeting is over.

[1474] 3. Terminal: Sends the converted text data to the server.

[1475] 4. Server: The text data is analyzed and phrases such as "If we fail to achieve our goal, it's your fault" are detected.

[1476] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[1477] 6. Terminal: Receives the report and notifies the user.

[1478] 7. User: Review the report and take any necessary follow-up actions.

[1479] Example 2: Minutes of individual consultations and harassment detection

[1480] 1. User: Enter the details of the individual consultation into the app as minutes.

[1481] 2. Terminal: Sends the minutes data to the server.

[1482] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[1483] 4. Server: Determines that this constitutes harassment and generates a report with recommended actions.

[1484] 5. Terminal: Receives the report and notifies the user.

[1485] 6. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[1486] The following can be used as an example prompt:

[1487] Example prompt 1:

[1488] Please analyze the following conversation and rate whether it contains harassing behavior:

[1489] 'Boss: If you fail to achieve your goals, it's all your fault.'

[1490] Based on the analysis results, generate a report and propose appropriate countermeasures.

[1491] Example prompt 2:

[1492] Please analyze the following minutes and assess whether they contain any harassing behavior:

[1493] Subordinate: I think there is room for improvement in this proposal.

[1494] Boss: You have no room for improvement.'

[1495] Based on the analysis results, generate a report and propose appropriate countermeasures.

[1496] This system can improve the quality of business communication and provide an effective means of supporting the early detection of harassment and appropriate responses.

[1497] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1498] Step 1:

[1499] User: Launch the dedicated application on a device such as a smartphone or PC, and tap the "Start Recording" button at the start of the meeting or interview.

[1500] Input: Application launch and button tap.

[1501] Specific operation: When the application is launched and the "Start Recording" button is pressed, the built-in microphone starts working.

[1502] Output: Recording start trigger set by device.

[1503] Step 2:

[1504] Device: Uses the built-in microphone to collect voice data in real time and temporarily stores it in local storage.

[1505] Input: Audio during a meeting or interview.

[1506] What it does: The microphone captures ambient sounds and stores them as digital audio data in file formats such as WAV and MP3.

[1507] Output: Audio data file saved to local storage.

[1508] Step 3:

[1509] Device: After recording is complete, the collected voice data is sent to speech recognition software (such as Google Speech-to-Text API), which begins analyzing the voice waveform and converting it into text data.

[1510] Input: Saved audio data file.

[1511] What it does: Speech recognition software analyzes the audio data and converts the audio waveform into text.

[1512] Output: The generated text data.

[1513] Step 4:

[1514] Terminal: The converted text data and manually entered minutes data are encrypted and sent to the server.

[1515] Input: Text data and meeting minutes data.

[1516] What it does: Encodes data using the TLS / SSL protocol and sends it securely to the server.

[1517] Output: Encrypted data reaches the server.

[1518] Step 5:

[1519] Server: Decodes the received data and analyzes the text data using a natural language processing (NLP) engine (such as spaCy).

[1520] Input: Encrypted text data.

[1521] What it does: The decoded text data is analyzed by an NLP engine to extract specific keywords and phrases.

[1522] Output: A dataset of the analysis results.

[1523] Step 6:

[1524] Server: Based on the analysis results, a generative AI model is used to generate recommended countermeasures.

[1525] Input: NLP engine analysis result dataset.

[1526] How it works: The generative AI model takes the analysis results as input and generates appropriate countermeasures based on past data and case studies.

[1527] Output: A report with recommended actions.

[1528] Step 7:

[1529] Server: Sends the generated report to the terminal.

[1530] Input: A report with recommended actions.

[1531] Specific operation: Encodes the report data and transmits it securely to the terminal.

[1532] Output: The report arrives at the terminal.

[1533] Step 8:

[1534] Terminal: Notifies and displays received reports to the user.

[1535] Input: Report received from server.

[1536] Specific operation: The terminal decodes the report data and notifies the user through the user interface. The contents of the report are displayed on the screen.

[1537] Output: The report is available for the user to review.

[1538] Through the above processing steps, this system enables early detection of harassment behavior and provides appropriate countermeasures.

[1539] (Application example 1)

[1540] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1541] There is a need for a system that can quickly detect harassment that occurs in communication between workers in factories and take appropriate measures. However, existing systems lack real-time capabilities and can be slow to notify managers, making it difficult to respond quickly and effectively.

[1542] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1543] In this invention, the server includes means for encrypting voice data and transmitting it to the server, means for analyzing text data using a natural language processing engine and generating recommended countermeasures when harassment is detected, and means for receiving the report generated by the server and notifying the administrator in PDF format. This makes it possible to detect harassment in real time and quickly notify the administrator.

[1544] "Audio Data" means audio information recorded in analog or digital format.

[1545] "Text data" is character string information obtained by processing voice data.

[1546] "Encryption" is the process of transforming data using a specific algorithm to ensure its security.

[1547] A "natural language processing engine" is a computer program that analyzes text data and understands its meaning and context.

[1548] "Recommended actions" indicate actions or countermeasures to be taken when a problem is detected.

[1549] A "report" is a document summarizing the analysis results and countermeasures.

[1550] A "server" is a central computer system for storing and processing data.

[1551] "Real-time" refers to a time frame in which data processing and information provision occurs immediately without delay.

[1552] This invention relates to a system for quickly detecting harassment behavior in communication between workers in a factory and taking appropriate measures. This system is composed of a robot assistant that patrols the factory, an associated server, and data communication between them.

[1553] System Overview

[1554] Recording audio data

[1555] The robot uses a built-in microphone to record conversations between workers in the factory in real time, and the recordings are temporarily saved in local storage.

[1556] Converting audio data to text

[1557] Once the recording is complete, the collected audio data is converted to text using speech recognition software (e.g., Google Cloud Speech-to-Text API), which analyzes the audio waveform and generates the most appropriate string of characters.

[1558] Sending data

[1559] The converted text data is encrypted using a data encryption tool (e.g., AES encryption) and sent to a server via a reliable communication (e.g., HTTPS communication).

[1560] Text data analysis

[1561] The server then runs the received text data through a natural language processing (NLP) engine (such as OpenAI's GPT-4). The NLP engine extracts specific keywords and phrases from the text and evaluates whether they constitute harassing behavior. Based on the evaluation results, if harassing behavior is detected, it generates detailed information (such as the words used, the speaker, and the context) along with recommended countermeasures.

[1562] Notification of analysis results

[1563] The server compiles the analysis results and generates a report. This report is output in PDF format and sent to the administrator, who can then take appropriate measures promptly based on the report.

[1564] Specific examples of hardware and software used

[1565] Hardware: Robot for patrolling the factory, built-in microphone, local storage, internet communication module

[1566] software:

[1567] Speech recognition software (Google Cloud Speech-to-Text API)

[1568] Natural language processing engine (OpenAI's GPT-4)

[1569] Data encryption tool (AES encryption)

[1570] Data transmission module (HTTPS communication)

[1571] Report generation tools (e.g. PDF generation libraries)

[1572] Specific examples

[1573] Prompt Sentence Examples

[1574] markdown

[1575] Analyze the following conversation text to detect harassing behavior, and if so, generate details and appropriate countermeasures.

[1576] Conversation text:

[1577] Worker A: "You caused the line to stop, and it's all your fault."

[1578] Worker B: "Sorry, I'll be more careful next time."

[1579] Worker A: "It's not enough to just be careful. If you do it again, you're fired."

[1580] result:

[1581] 1. Type of conduct: Harassment

[1582] 2. Key phrases detected: "It's your fault", "You're fired"

[1583] 3. Recommended measures:

[1584] Recommend that Worker A take harassment prevention training

[1585] Follow-up meetings held to improve the work environment

[1586] In this way, the system that is the subject of this patent is extremely effective in improving human relationships and the working environment within a factory.

[1587] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1588] Step 1:

[1589] The user activates the robot assistant in an area of ​​the factory where communication between workers takes place. The robot uses a built-in microphone to record conversations in real time. The recorded data (input) is temporarily saved in local storage (output).

[1590] Step 2:

[1591] The robot's terminal converts the temporarily stored voice data into text data using voice recognition software (Google Cloud Speech-to-Text API). The voice data (input) is converted into text data (output) as character string information through analysis of the voice waveform. Specifically, the voice file is sent via the cloud to the voice recognition service, and the returned text data is received.

[1592] Step 3:

[1593] The terminal encrypts the converted text data using the AES encryption tool. The text data (input) is encrypted using a specified key to produce secure ciphertext data (output). During the encryption process, the text data is transformed through an algorithm to make it unintelligible to outsiders.

[1594] Step 4:

[1595] The terminal sends encrypted text data to the server via HTTPS communication. At this time, the encrypted text data (input) is sent to the server (output) using a secure communication protocol. Specifically, SSL / TLS is used to prevent data interception.

[1596] Step 5:

[1597] The server decrypts the ciphertext data it receives. The server decrypts the ciphertext data (input) with the appropriate key to obtain the original text data (output). Decryption is performed by reversing the encryption process.

[1598] Step 6:

[1599] The server analyzes the text data using a natural language processing engine (OpenAI's GPT-4). Specific keywords and phrases are extracted from the text data (input) and evaluated to determine whether they constitute harassment. The natural language processing engine generates the analysis results, and the output indicates whether or not the behavior constitutes harassment.

[1600] Step 7:

[1601] The server generates recommended countermeasures based on the analysis results. The analysis results (input) are converted into report-format data (output) that includes countermeasures. Specifically, the NLP engine proposes candidate countermeasures and compiles them into a report.

[1602] Step 8:

[1603] The server generates a report in PDF format and notifies the administrator. This report data (input) is output as a PDF file (output) using a PDF generation library and sent to the administrator's terminal.

[1604] Step 9:

[1605] The administrator checks the received report and takes appropriate measures. The administrator's device displays the received PDF file (input) and allows the contents to be checked (output). Specifically, the administrator uses a PDF viewer to view the report and select and execute the necessary actions.

[1606] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1607] The present invention relates to a system for effectively detecting harassment behavior in internal meetings and interviews, analyzing the user's emotions, and notifying the user. Specific embodiments of the system will be described in detail below.

[1608] This system consists of multiple components, including the user's device, a server, and an emotion engine. Users can use the application to record the contents of meetings and interviews and analyze the data in real time.

[1609] System Overview

[1610] 1. Recording audio data

[1611] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[1612] Device: Begins collecting audio data using the built-in microphone, which is temporarily stored in local storage.

[1613] 2. Converting voice data to text

[1614] Terminal: Once the recording is complete, the voice data is input into speech recognition software, which converts it into text data. The speech recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[1615] 3. Emotion Analysis

[1616] Device: During and after recording, the voice data is input into an emotion engine to analyze the user's emotional state (e.g., anger, anxiety, joy, etc.).

[1617] Emotion engine: Analyzes emotional expressions contained in voice data and evaluates the user's emotional state.

[1618] 4. Data transmission

[1619] Terminal: The converted text data and the emotion engine analysis results are sent to the server. The data is encrypted before being sent to ensure its security.

[1620] 5. Analysis of Text Data

[1621] Server: Inputs the received text data into a natural language processing (NLP) engine to extract specific keywords and phrases from the text, and evaluates whether the extracted keywords and phrases constitute harassment.

[1622] Server: Uses the analysis results of the emotion engine as additional data to complement the assessment results of harassment behavior.

[1623] 6. Generation and notification of analysis results

[1624] Server: Based on the output of the NLP engine and emotion engine, if harassment behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[1625] Server: Sends the generated analysis result report to the terminal.

[1626] Terminal: Decodes the received analysis results and displays them on the user interface. Formats the information so that the user can easily check the results.

[1627] Specific examples

[1628] Example 1: Meeting recording, harassment detection, and emotion analysis

[1629] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[1630] 2. Device: Collects recorded data and converts it into text using speech recognition software, while simultaneously analyzing the user's emotions using an emotion engine.

[1631] 3. Terminal: The converted text data and the emotion engine analysis results are sent to the server.

[1632] 4. Server: Analyzes the text data and detects phrases such as "If we fail to achieve our goal, it's your fault." Using sentiment analysis, the server adds the user's strong anxiety to the score.

[1633] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[1634] 6. Terminal: Receives the report and notifies the user.

[1635] 7. User: Review the report and take any necessary follow-up actions.

[1636] Example 2: Individual consultation minutes, harassment detection, and sentiment analysis

[1637] 1. User: Enter the details of the individual consultation into the app as minutes.

[1638] 2. Terminal: The minutes data is sent to the server. Emotion analysis is also performed by the emotion engine in parallel.

[1639] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[1640] 4. Server: Based on the results of sentiment analysis, the user indicates strong disappointment and adds it to the rating.

[1641] 5. Server: Determines that this constitutes harassment and generates a report with recommended action.

[1642] 6. Terminal: Receives the report and notifies the user.

[1643] 7. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[1644] This system will improve the quality of internal communication, enable early detection of harassment, and enable appropriate responses. Furthermore, by analyzing the user's emotional state, more accurate evaluations and countermeasures can be achieved.

[1645] The processing flow will be explained below.

[1646] Step 1:

[1647] User: Launches the application and taps the "Start Recording" button at the beginning of a meeting or interview.

[1648] Step 2:

[1649] Device: Begins collecting audio data using the built-in microphone, which is saved to local storage in real time.

[1650] Step 3:

[1651] On the device, the voice data collected at the same time is input into the emotion engine, which analyzes the user's emotional state in real time by analyzing voice characteristics such as tone, speed, and volume.

[1652] Step 4:

[1653] User: When the meeting or interview is over, tap the "Stop Recording" button in the application.

[1654] Step 5:

[1655] Terminal: The recorded voice data is input into voice recognition software, which converts the voice data into text data. The voice recognition software analyzes the voice waveform and generates the most appropriate string of characters.

[1656] Step 6:

[1657] Device: The converted text data and the emotion engine analysis results are sent to the server. The data is encrypted before being sent to ensure its security.

[1658] Step 7:

[1659] Server: Receives data packets sent from the device and decompresses and decrypts the data.

[1660] Step 8:

[1661] Server: The received text data is input into a natural language processing (NLP) engine to extract specific keywords and phrases from the text. The NLP engine then analyzes the text data and evaluates whether it constitutes harassment.

[1662] Step 9:

[1663] Server: Use the results of the emotion engine analysis as additional data to complement the assessment of harassment behavior, paying particular attention to when the user has negative emotions.

[1664] Step 10:

[1665] Server: Based on the output of the NLP engine and emotion engine, if harassment behavior is detected, it generates recommended countermeasures along with detailed information (such as the words used, the speaker, and the context).

[1666] Step 11:

[1667] Server: Sends the generated analysis result report to the terminal.

[1668] Step 12:

[1669] Terminal: Decodes the received analysis results and displays them on the user interface. Formats the information so that the user can easily check the results.

[1670] Step 13:

[1671] User: Review the analysis results and take any necessary follow-up actions, such as reporting to a manager or relevant department, or providing additional training.

[1672] Example 2

[1673] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1674] In internal meetings and interviews, there is a problem in that it is not possible to properly detect harassment behavior and analyze the user's emotional state and notify them appropriately. In particular, there is a need for a system that can convert voice data into text and quickly and accurately evaluate the content and emotional state of the text.

[1675] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1676] In this invention, the server includes means for recording voice data, means for converting the recorded voice data into text data, means for analyzing the text data and voice data and evaluating the user's emotions, means for transmitting the text data and emotion evaluation results to the server, means for extracting problematic terms and phrases from the text data using natural language processing technology and detecting harassment behavior, means for generating recommended countermeasures based on the evaluation results, and means for notifying the user of the analysis results and recommended countermeasures. This makes it possible to quickly and accurately detect harassment behavior and analyze the user's emotional state during meetings and interviews.

[1677] "Audio data" refers to data in which the contents of a meeting or interview are recorded as audio.

[1678] "Text data" is data obtained by converting voice data into a character string.

[1679] "Emotion assessment" is the process of analyzing and assessing the emotional state (e.g., anger, anxiety, joy, etc.) from the voice and words uttered by the user.

[1680] A "server" is a computer system that processes, stores, and analyzes data over a network.

[1681] "Natural language processing technology" is a technology for extracting specific keywords and phrases from text data and analyzing the content of that text.

[1682] "Harassment" means unjustifiably discriminating, threatening, abusive, discriminatory, or other inappropriate conduct against another person in the workplace or any other environment.

[1683] "Recommended countermeasures" are suggestions that outline specific actions and countermeasures that should be taken when harassment is detected.

[1684] A "report" is a document or digital file that contains the results of an analysis and is generated to inform a user.

[1685] A "user" is someone who uses the system to record and analyze meetings and interviews.

[1686] "Encryption" is the process of transforming data using a specific algorithm to protect its contents.

[1687] "Decoding" is the process of restoring encrypted data to its original form.

[1688] The present invention relates to a system for effectively detecting harassment behavior in meetings and interviews, analyzing the user's emotions, and notifying the user. Specific embodiments of the system will be described in detail below.

[1689] System configuration

[1690] This system consists of multiple components, including the user's device, a server, and an emotion engine. Users can record the contents of meetings and interviews using a dedicated application and analyze the data in real time. The analysis results are then notified to the user, allowing them to take any necessary follow-up actions.

[1691] Hardware and Software Configuration

[1692] Device: The device used by the user, such as a smartphone or tablet.

[1693] Built-in microphone: An audio input device built into the device.

[1694] Local storage: Memory for temporarily storing voice data within the device.

[1695] Speech Recognition Software: Software used to convert recorded voice data into text data, such as the Google Cloud Speech-to-Text API.

[1696] Emotion engine: Software such as IBM Watson that analyzes emotions based on user voice data.

[1697] Server: A remote computer system responsible for storing and analyzing data.

[1698] Natural Language Processing (NLP) engines: Includes technologies such as the Google NLP API that analyze text data and extract specific keywords and phrases.

[1699] Specific implementation steps

[1700] First, the user launches the application and taps the "Start Recording" button at the beginning of a meeting or interview. The device's built-in microphone begins collecting audio data, which is temporarily saved in the device's local storage. Once recording is complete, the audio data is sent to the Google Cloud Speech-to-Text API and converted into text. At the same time, the audio data is input into the IBM Watson Emotion Engine to analyze the user's emotional state.

[1701] The converted text data and the emotion engine's analysis results are sent to a server using AES-256 encryption. The server then analyzes the text data using the Google NLP API to extract specific keywords and phrases. For example, a phrase such as "It's your fault that we didn't achieve our goals" is detected and evaluated to determine whether it constitutes harassment.

[1702] The analysis results of the emotion engine are also used as supplementary data to further refine the evaluation results of harassment behavior. Finally, if harassment behavior is detected based on the output of the NLP engine and emotion engine, a report containing detailed information and recommended countermeasures is generated. This report is sent from the server to the device and displayed on the user interface. The user can review the report and take any necessary follow-up actions.

[1703] Specific Examples

[1704] Example 1: Meeting recording, harassment detection, and emotion analysis

[1705] 1. User: When a meeting between a subordinate and a superior begins, the user launches the app and starts recording.

[1706] 2. Device: Collects the recorded data and converts it into text using the Google Cloud Speech-to-Text API, while simultaneously analyzing the user's emotions using the IBM Watson emotion engine.

[1707] 3. Terminal: The converted text data and the emotion engine analysis results are sent to the server using AES-256 encryption.

[1708] 4. Server: Analyzes the text data and detects phrases such as "If we fail to achieve our goal, it's your fault." Using sentiment analysis, the server adds the user's strong anxiety to the score.

[1709] 5. Server: Determine that this is harassment and generate a report with appropriate action suggestions.

[1710] 6. Terminal: Receives the report and notifies the user.

[1711] 7. User: Review the report and take any necessary follow-up actions.

[1712] Example 2: Individual consultation minutes, harassment detection, and sentiment analysis

[1713] 1. User: Enter the details of the individual consultation into the app as minutes.

[1714] 2. Terminal: Sends the minutes data to the server. Sentiment analysis is also performed by the emotion engine in parallel.

[1715] 3. Server: Analyzes the text data and detects non-constructive feedback such as "You have no room for improvement."

[1716] 4. Server: Based on the results of sentiment analysis, the user indicates strong disappointment and adds it to the rating.

[1717] 5. Server: Determines that this constitutes harassment and generates a report with recommended action.

[1718] 6. Terminal: Receives the report and notifies the user.

[1719] 7. User: Review the report and, if necessary, take action with your manager to resolve the issue.

[1720] This system will improve the quality of internal communication, detect harassment at an early stage, and take appropriate measures. Furthermore, by analyzing the user's emotional state, more accurate evaluations and countermeasures can be realized.

[1721] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1722] Step 1:

[1723] Start recording audio data

[1724] Users simply launch the application and tap the "Start Recording" button at the start of a meeting or interview.

[1725] Input: Launch the application and tap the "Start Recording" button.

[1726] Output: Begin recording audio data.

[1727] The device uses the built-in microphone to collect audio data and temporarily stores it in the device's local storage.

[1728] What it does: When the user presses the "Start Recording" button, the device's built-in microphone captures the sound and saves it as an audio file.

[1729] Step 2:

[1730] Converting audio data to text

[1731] After the user presses the stop recording button, the device sends the collected voice data to the Google Cloud Speech-to-Text API and converts it into text data.

[1732] Input: Recorded audio data.

[1733] Output: The converted text data.

[1734] Specific operation: Voice data is sent to voice recognition software in real time, the voice waveform is analyzed, and text data is generated as an appropriate string of characters.

[1735] Step 3:

[1736] Emotion Analysis

[1737] The device sends the voice data to the IBM Watson emotion engine to analyze the user's emotional state.

[1738] Input: Audio data.

[1739] Output: Emotion evaluation results.

[1740] The emotion engine analyzes the emotional expressions contained in the voice data and evaluates the user's emotional state.

[1741] Specific operation: The voice data is sent to the emotion engine, where the emotional state (e.g., anger, anxiety, joy, etc.) is analyzed. The analyzed emotional state is recorded as the calculation result.

[1742] Step 4:

[1743] Sending data

[1744] The device then sends the converted text data and the results of sentiment analysis to the server using AES-256 encryption.

[1745] Input: Text data and emotion evaluation results.

[1746] Output: Sending encrypted data.

[1747] How it works: Text data and sentiment analysis results are encrypted and sent securely over the internet to a server. The encryption process is typically done using the OpenSSL library.

[1748] Step 5:

[1749] Text data analysis

[1750] The server inputs the received text data into the Google NLP API, extracts specific keywords and phrases from the text data, and detects harassment behavior.

[1751] Input: Encrypted text data.

[1752] Output: Keyword / phrase extraction results and harassment rating.

[1753] The server uses the results of the emotion analysis as supplementary data to refine the assessment of harassment behavior.

[1754] How it works: Text data is sent to an NLP engine, which extracts specific keywords and phrases. For example, phrases like "If we fail to achieve our goal, it's your fault" are detected, and an overall evaluation is made taking into account the results of sentiment analysis.

[1755] Step 6:

[1756] Generation and notification of analysis results

[1757] Based on the output of the NLP engine and emotion engine, the server generates a report containing detailed information and recommended countermeasures if any harassment behavior is detected.

[1758] Input: Harassment behavior assessment results and sentiment analysis results.

[1759] Output: Reports are generated and sent to the terminal.

[1760] The server then sends the generated report to the device.

[1761] The terminal decodes the received report and displays it in a user interface, formatting the information so that the user can easily view the results.

[1762] Specific actions: A report is created on the server and sent to the device. The report contains specific information, such as "A superior's comment that 'You have no room for improvement' constitutes harassment." The user can review the report and take any necessary follow-up actions.

[1763] (Application example 2)

[1764] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1765] Communication between workers and machines is an important element in factories. However, harassment can occur not only between workers but also between workers and robots, which can lead to a deterioration of the work environment and a drop in productivity. It is also necessary to properly analyze the emotional state of workers on-site and respond quickly to maintain a safe and healthy work environment. Therefore, a system is needed to detect such harassment at an early stage and take appropriate measures.

[1766] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording voice data, means for converting the recorded voice data into text data, means for analyzing the emotional state from the recorded voice data and text data, means for analyzing the text data to detect harassment behavior, means for generating recommended countermeasures, and means for notifying the user of the analysis results and the countermeasures. This makes it possible to detect harassment behavior by workers in a factory early and take appropriate measures, thereby maintaining a safe and healthy working environment.

[1767] "Recording voice data" is the process of storing human voice as digital data using the device's internal microphone.

[1768] "Text conversion" is the process of converting recorded audio data into written information, often using speech recognition software.

[1769] "Emotional state analysis" is the process of detecting emotional expressions from audio and text data and assessing individual emotional states (e.g., anger, anxiety, joy, etc.).

[1770] "Harassment detection" is the process of analyzing specific keywords and phrases contained in text data or audio data using natural language processing technology to determine whether they constitute harassment.

[1771] "Sending analysis results" is the process of securely sending the analyzed data and results to the server.

[1772] "Generating recommended countermeasures" is the process of generating information that suggests appropriate actions and countermeasures based on the analysis results.

[1773] "Notifying the user" is the process of communicating the generated analysis results and countermeasures to the user using means such as voice or a display.

[1774] The present invention relates to a system that can detect harassment occurring in communication between workers and robots in a factory at an early stage, analyze the emotional state of the workers, and take appropriate measures. Specific embodiments of the system are described in detail below.

[1775] System configuration

[1776] This system consists of a factory robot equipped with a built-in microphone, a server that analyzes the data, and a terminal that notifies the user. The main functions of the system are as follows:

[1777] 1. Recording audio data

[1778] The robot uses a built-in microphone to record conversations with workers in real time, and this audio data is temporarily stored in local storage.

[1779] 2. Converting voice data to text

[1780] Once the recording is complete, the robot uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the audio data into text data, which is then used for subsequent analysis.

[1781] 3. Emotion Analysis

[1782] The robot inputs the generated text and voice data into an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze its emotional state. Emotion analysis evaluates emotional states such as anger, anxiety, and joy.

[1783] 4. Data submission and analysis

[1784] The robot sends the converted text data and sentiment analysis results to a server using encryption (e.g., OpenSSL), which then analyzes the text data using a natural language processing (NLP) engine (e.g., Spacy, BERT model) to assess whether specific keywords or phrases constitute harassing behavior.

[1785] 5. Generating recommended countermeasures

[1786] If harassment behavior is detected, the server generates detailed information and recommended countermeasures based on the output of the NLP engine and emotion engine, creating a report that combines the analysis results and recommended countermeasures.

[1787] 6. Notification of analysis results

[1788] The report generated by the server is sent to the robot, which then notifies the worker of the received report via voice or display, allowing the worker to immediately understand the problem and take appropriate follow-up action.

[1789] Hardware and software used

[1790] Robots: Factory robots with built-in microphones (e.g., Pepper robots)

[1791] Speech recognition software: Google Cloud Speech-to-Text API

[1792] Emotion engine: IBM Watson Tone Analyzer

[1793] Encryption: OpenSSL

[1794] Natural language processing engine: Spacy, BERT model

[1795] Specific examples

[1796] Here is an example of a working system:

[1797] 1. Example 1: Production line instructions

[1798] A worker on a production line tells a robot, "If we don't meet our goal, it's all your fault."

[1799] The robot records this speech, converts it into text data, and sends it to a server.

[1800] The server analyzes this text, determines it is harassment, and uses the sentiment analysis results to generate recommended countermeasures.

[1801] The generated report is sent to the robot and notifies the worker.

[1802] 2. Example 2: Comments on the production line

[1803] A worker says to a robot, "You're doing your job really badly. There's no room for improvement."

[1804] The robot records this speech, converts it into text data, and sends it to a server.

[1805] The server analyzes this text, determines it is harassment, and uses the sentiment analysis results to generate recommended countermeasures.

[1806] The generated report is sent to the robot and notifies the worker.

[1807] This system makes it possible to maintain a safe and healthy working environment within the factory.

[1808] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1809] Step 1:

[1810] When the robot starts to interact with a worker, it automatically starts recording voice data. The voice data collected using the built-in microphone is temporarily saved in local storage. The input is human voice. The output is digital voice data.

[1811] Step 2:

[1812] The robot uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the recorded voice data into text data. The input is voice data, and the output is text data converted into text information. In this conversion process, the waveform of the voice data is analyzed and converted into the most appropriate string of characters.

[1813] Step 3:

[1814] The robot inputs text and voice data into an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the worker's emotional state. The input is text and voice data, and the output is emotional state information such as anger, anxiety, and joy. The emotion analysis process analyzes the tone of the text and the intonation of the voice.

[1815] Step 4:

[1816] The robot sends the converted text data and the emotion analysis results to the server using encryption (e.g., OpenSSL). The input is text data and emotional state information, and the output is securely encrypted transmission data. The encryption process securely processes the data so that it cannot be deciphered by third parties.

[1817] Step 5:

[1818] The server analyzes the received text data using a natural language processing (NLP) engine (e.g., Spacy, BERT model) and evaluates whether specific keywords and phrases constitute harassment. The input is the text data, and the output is a list of problematic terms and phrases and the evaluation results. This analysis process involves grammatically analyzing the text data to detect harmful expressions.

[1819] Step 6:

[1820] If harassment behavior is detected, the server generates recommended countermeasures based on the results of the NLP engine and emotion engine. The inputs are the evaluation results and emotional state information, and the output is countermeasures and detailed analysis results. The countermeasure generation process refers to past data and case studies to determine the optimal response method.

[1821] Step 7:

[1822] The server sends the generated countermeasure proposals and a report of the analysis results to the robot. The robot notifies the worker of the received report by voice or display. The inputs are countermeasure proposals and analysis results, and the output is a notification to the worker. In the notification process, the information in the report is conveyed to the worker in an easy-to-understand format.

[1823] In this way, the system can detect harassment in factories at an early stage and take appropriate measures.

[1824] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1825] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1826] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1827] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1828] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1829] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1830] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1831] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1832] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1833] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1834] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1835] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1836] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1837] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1838] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1839] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1840] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1841] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1842] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1843] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1844] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1845] The following is further disclosed regarding the above embodiment.

[1846] (Claim 1)

[1847] A means of recording audio data from meetings and interviews,

[1848] means for converting the recorded voice data into text data;

[1849] A means for analyzing text data and detecting harassment behavior;

[1850] means for notifying the user of the analysis results;

[1851] A system including:

[1852] (Claim 2)

[1853] means for transmitting the manually inputted text data together with the recorded voice data to a server;

[1854] means for receiving the analysis results from the server;

[1855] 10. The system of claim 1, comprising:

[1856] (Claim 3)

[1857] A means of using natural language processing technology to extract problematic terms and phrases from text data and evaluate harassment behavior;

[1858] a means for generating recommended countermeasures based on the evaluation results;

[1859] 10. The system of claim 1, comprising:

[1860] "Example 1"

[1861] (Claim 1)

[1862] means for a user to record audio data;

[1863] means for converting the recorded voice data into text data;

[1864] means for encrypting text data and transmitting the data to a server;

[1865] A means for analyzing text data using natural language processing technology on a server and detecting harassment behavior;

[1866] a means for generating recommended countermeasures based on the analysis results;

[1867] A means for notifying the user of the analysis results and proposed countermeasures;

[1868] A system including:

[1869] (Claim 2)

[1870] means for transmitting the recorded voice data and manually entered text data to a server;

[1871] A means for receiving the analysis results and countermeasures from the server;

[1872] 10. The system of claim 1, comprising:

[1873] (Claim 3)

[1874] A means of using natural language processing technology to extract problematic terms and phrases from text data and evaluate harassment behavior;

[1875] A means for generating recommended countermeasures using a generative AI model based on the evaluation results;

[1876] 10. The system of claim 1, comprising:

[1877] "Application Example 1"

[1878] (Claim 1)

[1879] A means of recording audio data from meetings and interviews,

[1880] means for converting the recorded voice data into text data;

[1881] A means for analyzing text data and detecting harassment behavior;

[1882] means for notifying the user of the analysis results;

[1883] means for encrypting the audio data and transmitting it to a server;

[1884] A means for analyzing text data using a natural language processing engine and generating recommended countermeasures when harassment behavior is detected;

[1885] A system including:

[1886] (Claim 2)

[1887] means for transmitting the manually inputted text data together with the recorded voice data to a server;

[1888] means for receiving the analysis results from the server;

[1889] A means for receiving the server-generated report and notifying the administrator in PDF format;

[1890] 10. The system of claim 1, comprising:

[1891] (Claim 3)

[1892] A means of using natural language processing technology to extract problematic terms and phrases from text data and evaluate harassment behavior;

[1893] a means for generating recommended countermeasures based on the evaluation results;

[1894] a means for automatically generating a report including the evaluation results and recommended actions;

[1895] 10. The system of claim 1, comprising:

[1896] "Example 2: Combining Emotion Engines"

[1897] (Claim 1)

[1898] means for recording audio data;

[1899] means for converting the recorded voice data into text data;

[1900] means for analyzing the text data and the voice data and assessing the user's emotions;

[1901] means for transmitting the text data and the emotion evaluation result to a server;

[1902] A means of detecting harassment by extracting problematic terms and phrases from text data using natural language processing technology;

[1903] a means for generating recommended countermeasures based on the evaluation results;

[1904] A means of notifying users of the analysis results and recommended countermeasures,

[1905] A system including:

[1906] (Claim 2)

[1907] means for encrypting the converted text data and the emotion evaluation result and transmitting them to a server;

[1908] means for decoding the analysis results when received from the server;

[1909] 10. The system of claim 1, comprising:

[1910] (Claim 3)

[1911] A means for the server to generate a report of the analysis results and countermeasures and display it on a user interface;

[1912] 10. The system of claim 1, comprising:

[1913] "Application example 2 when combining emotion engines"

[1914] (Claim 1)

[1915] means for recording audio data;

[1916] means for converting the recorded voice data into text data;

[1917] means for analyzing an emotional state from the text data and the corresponding voice data;

[1918] A means for analyzing text data and detecting harassment behavior;

[1919] A means for transmitting the analysis results to a server and generating recommended countermeasures;

[1920] a means for notifying the user of the generated countermeasures and analysis results;

[1921] A system including:

[1922] (Claim 2)

[1923] means for transmitting the manually inputted text data together with the recorded voice data to a server;

[1924] means for receiving the analysis results from the server;

[1925] A means for detecting harassment based on the analysis results;

[1926] 10. The system of claim 1, comprising:

[1927] (Claim 3)

[1928] A means of using natural language processing technology to extract problematic terms and phrases from text data and evaluate harassment behavior;

[1929] a means for generating recommended countermeasures based on the evaluation results;

[1930] a means for notifying the evaluation results and proposed measures to devices in the factory;

[1931] 10. The system of claim 1, comprising: [Explanation of symbols]

[1932] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of recording audio data from meetings and interviews, means for converting the recorded voice data into text data; A means for analyzing text data and detecting harassment behavior; a means for notifying the user of the analysis results; A system including:

2. means for transmitting the manually inputted text data together with the recorded voice data to a server; means for receiving the analysis results from the server; The system of claim 1 , comprising:

3. A means of using natural language processing technology to extract problematic terms and phrases from text data and evaluate harassment behavior; a means for generating recommended countermeasures based on the evaluation results; The system of claim 1 , comprising:

Images