System

Abstract

A system is provided.SOLUTION: A system includes a voice collection unit provided in a communication terminal used by a child, a voice transmission unit configured to transmit voice data collected by the voice collection unit to a server, a voice conversion unit configured to convert the voice data into text data by using a voice recognition technology, a bullying detection unit configured to analyze the text data and detect a word having a possibility of bullying, and a notification unit configured to transmit a notification to a parent when the bullying detection unit determines that the possibility of bullying is high.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today's children use communication devices, and effective methods are needed to monitor their safety. Bullying in schools and public places is a major social problem, making it difficult for parents to notice abnormalities in their children early on. Conventional methods lack a mechanism for detecting signs of bullying in real time and immediately notifying parents. Therefore, there is a need for a system that can quickly and accurately detect bullying and notify parents. [Means for solving the problem]

[0005] The present invention provides a system including a voice collection means provided in a communication terminal used by a child, a voice transmission means for transmitting the voice data collected by the voice collection means to a server, a voice conversion means for the server to convert the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, and a notification means for sending a notification to parents if the bullying detection means determines that there is a high possibility of bullying. This allows signs of bullying to be detected in real time and parents to be immediately notified, enabling early action to be taken.

[0006] "Children" refers to people who are the target of using this system, and primarily refers to minors.

[0007] "Communication device" refers to a mobile phone, smartphone, or other voice communication device used by a child.

[0008] "Audio collection means" refers to a microphone or audio recording function built into a communication terminal, and refers to a device or function for collecting surrounding audio.

[0009] "Audio data" refers to audio signals in analog or digital form captured by an audio collection means.

[0010] "Server" refers to the computer system that receives and analyzes the audio data.

[0011] "Audio transmission means" refers to a function or device for transmitting collected audio data from a communication terminal to a server.

[0012] "Speech conversion means" refers to software that uses speech recognition technology and runs on a server, and is used to convert voice data into text data.

[0013] "Text data" refers to digital data obtained by converting voice data into character information using a voice conversion means.

[0014] "Bullying detection means" refers to algorithms or programs that analyze text data and determine whether it contains the possibility of bullying.

[0015] "High possibility of bullying" refers to cases where bullying detection measures suggest that bullying behavior exceeds a certain standard.

[0016] "Notification means" refers to a device or function for issuing a warning or notification to parents or guardians when it is determined that there is a high possibility of bullying.

[0017] "Parent" refers to the person to whom the system sends notifications, who is the guardian or guardian of the child.

[0018] "Push notifications" refer to notification messages sent in real time to smartphones and other devices. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention describes a system for collecting voice data and detecting bullying behavior in communication terminals used by children. The implementation of this system is carried out through the following steps.

[0041] Communication terminal operation

[0042] The device is equipped with an audio collection means for collecting the child's audio data. Specifically, it uses a microphone to collect surrounding sounds. This audio data is transmitted to a server in real time.

[0043] Once the device collects the voice data, it compresses it and sends it to the server as packets for efficient data transmission. Appropriate protocols must be used to ensure data security and reliability.

[0044] Server Operation

[0045] The server receives the voice data sent from the terminal. The received voice data is converted into text data by a voice conversion means. For this purpose, a voice recognition technology such as Google (registered trademark) Cloud Speech-to-Text API is used.

[0046] The server then analyzes the converted text data using a bullying detection method that utilizes natural language processing technology and machine learning algorithms to analyze the text data and determine whether its content indicates bullying behavior.

[0047] Notification method operation

[0048] If the bullying detection means analyzes the text data and determines that there is a high possibility of bullying, the server will use the notification means to send a warning to parents via push notifications on smartphones or email.

[0049] For example, if the text "No one cares if you're not around" is converted from speech, the bullying detection method will determine this as bullying, and a push notification will be sent to the parent's smartphone stating, "Your child may be being bullied."

[0050] Specific examples

[0051] Example 1: Classroom conversation

[0052] scenario:

[0053] Imagine a child is having a conversation with a friend in class and the friend says to them, "You're so stupid! You can't even figure out such a simple problem?"

[0054] 1. The device collects audio.

[0055] 2. The device sends the audio data to the server.

[0056] 3. The server converts the audio data into text data.

[0057] 4. The server analyzes the text data and detects possible bullying.

[0058] 5. The server sends a push notification to the parent's smartphone.

[0059] In this way, the system operates in real time, instantly notifying parents when their child is being bullied, allowing for prompt action.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The device collects ambient sounds in real time through a microphone, for example capturing the sound of a child talking with a friend in the classroom.

[0063] Step 2:

[0064] The terminal compresses the collected voice data to improve the efficiency of data transmission, and it is important to use an appropriate compression algorithm.

[0065] Step 3:

[0066] The terminal converts the compressed voice data into a packet format and sends it to the server using a secure data transmission protocol, which ensures the authenticity and security of the data.

[0067] Step 4:

[0068] The server decompresses the audio data received from the device and converts it into text using speech recognition technology. Commercially available speech recognition software, such as the Google Cloud Speech-to-Text API, can be used.

[0069] Step 5:

[0070] The server analyzes the converted text data with a bullying detection tool that uses natural language processing techniques and machine learning algorithms to detect keywords and patterns in the text that indicate bullying behavior.

[0071] Step 6:

[0072] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method, along with details of the bullying act, such as the time and location.

[0073] Step 7:

[0074] The server sends real-time push notifications to parents' smartphones or tablet devices, informing them of potential bullying incidents and providing additional information.

[0075] Step 8:

[0076] The user (parent) can check the details and take necessary action according to the push notification received. Parents can also view more detailed information and history through the application.

[0077] Step 9:

[0078] If the parent takes any action after receiving the notification, the server will check the response status again and send a follow-up notification if necessary. This series of processes enables early detection and prompt response when a child is being bullied.

[0079] Example 1

[0080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0081] The present invention relates to a system for detecting bullying behavior by collecting voice data using communication devices used by children. Conventional technologies have inefficient processes for collecting voice data, detecting bullying behavior, and notifying parents, making it difficult to respond in real time. Furthermore, the accuracy of determining bullying behavior is low, resulting in frequent false positives. There is a need to solve these issues and more quickly and accurately detect bullying behavior and notify parents.

[0082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0083] In this invention, the server includes an audio collection means, an audio transmission means, an audio conversion means, a bullying detection means, a notification generation means, a notification content generation means, and a notification transmission means. This allows for real-time collection of audio around a child and secure transmission of compressed audio data to the server. The server also quickly converts the audio data into text data and uses machine learning algorithms and natural language processing technology to analyze bullying behavior with high accuracy. Furthermore, it automatically generates appropriate notification content using a generative AI model, enabling prompt and appropriate warnings to be sent to parents.

[0084] The "voice collecting means" is a voice input device such as a microphone provided in the communication terminal used by the child, and is a device that has the function of collecting surrounding voices.

[0085] The "audio transmission means" is a communication means having the function of compressing collected audio data and transmitting it safely to the server.

[0086] The "voice conversion means" is a means having a function of converting voice data collected in the server into text data using voice recognition technology.

[0087] The "bullying detection means" is a means that has the function of analyzing the converted text data and detecting words that may be bullying.

[0088] The "notification means" is a means that has the function of sending a notification to parents when it is determined that there is a high possibility of bullying.

[0089] The "notification generating means" is a means having a function of generating the content of a notification to be sent to parents when bullying behavior is detected.

[0090] "Notification content generation means" means a means having the function of generating appropriate notification content based on predefined templates and analysis results using a generative AI model.

[0091] The "notification sending means" is a means having a function of sending the generated notification message to the parent communication terminal.

[0092] A "machine learning algorithm" is a technology that allows computers to learn from data and make predictions and classifications.

[0093] "Natural language processing technology" is a computational technology for understanding, generating, and analyzing human language.

[0094] A "generative AI model" is a pre-trained artificial intelligence model that has the ability to generate appropriate outputs based on specific inputs.

[0095] The present invention relates to a system for detecting bullying by collecting voice data using a communication terminal used by a child. The system includes a communication terminal, a server, a bullying detection unit, and a notification unit.

[0096] Configuration and operation of communication terminal

[0097] The device is equipped with an audio capture unit to capture sounds around the child. Specifically, a microphone captures the sound, and a dedicated application manages the sound data in real time. The collected sound data is compressed into MP3 format using a LAME MP3 encoder or similar and securely transmitted to a server using the HTTPS protocol.

[0098] Server configuration and operation

[0099] The server receives the voice data sent from the device and temporarily stores it in storage. The received voice data is converted into text data using speech recognition technology such as the Google Cloud Speech-to-Text API. The server then analyzes the text data using scikit-learn and TENSORFLOW (registered trademark) to detect bullying behavior. This analysis process utilizes machine learning algorithms and natural language processing technology.

[0100] Bullying detection and notification

[0101] The bullying detection means analyzes the text data to determine whether certain keywords or phrases correspond to bullying behavior. If bullying behavior is detected, the notification generation means uses a generative AI model to generate notification content to send to parents, which may utilize pre-defined templates and the generative AI model.

[0102] Example prompt sentence:

[0103] Audio analysis has detected the following potential bullying behavior:

[0104] "No one would care if you weren't here."

[0105] Parents should be notified. Please generate the notification content.

[0106] Parental Notification

[0107] The generated notification message is sent to the parent communication device using a notification sending means, such as a push notification using Firebase Cloud Messaging (FCM) or an email using the SMTP protocol.

[0108] Specific examples

[0109] For example, if a child in a classroom is told, "You're so stupid, you can't even figure out such a simple problem?", the system will act as follows:

[0110] 1. The device collects the audio, compresses it into MP3 format, and sends it to the server.

[0111] 2. The server receives the voice data and converts it into text data using voice recognition technology.

[0112] 3. The server analyzes the text data using machine learning algorithms and natural language processing technology, and detects statements such as, "You're so stupid, you can't even figure out such a simple question?" as bullying.

[0113] 4. The server uses the generative AI model to generate appropriate notification content, creating a message such as "Your child may be being bullied."

[0114] 5. The server sends the generated notification message to the parent communication device as a push notification.

[0115] This system allows parents to be notified immediately if their child is being bullied, allowing them to take immediate action.

[0116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0117] Step 1:

[0118] The device collects audio data from the child's surroundings. Specifically, it records audio in real time using a built-in microphone. The input is the surrounding audio, and the output is the recorded audio data. A dedicated application manages this audio data and temporarily stores it on the device.

[0119] Step 2:

[0120] The device compresses the collected audio data. Specifically, it converts the audio data into MP3 format using software such as the LAME MP3 encoder. The input is the recorded audio data, and the output is a compressed MP3 file. This process reduces the data size and enables efficient data transfer.

[0121] Step 3:

[0122] The device sends the compressed audio data to the server. Specifically, the data is uploaded to the server using the HTTPS protocol to ensure data security. The input is a compressed MP3 file, and the output is the audio data sent to the server.

[0123] Step 4:

[0124] The server receives the audio data sent from the device. The received data is temporarily stored in the server's storage. The input is the sent MP3 file, and the output is the audio data stored in the server's storage.

[0125] Step 5:

[0126] The server uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the audio data into text. Specifically, the audio data is sent to the API, and the returned text data is stored in a database on the server. The input is an MP3 file, and the output is the converted text data.

[0127] Step 6:

[0128] The server analyzes the text data to detect possible bullying. It uses machine learning algorithms such as scikit-learn and TensorFlow and natural language processing techniques to analyze keywords and emotions that indicate bullying behavior contained in the text data. The input is the text data, and the output is a flag indicating bullying behavior and the analysis results.

[0129] Step 7:

[0130] When the server detects bullying behavior, it uses a generative AI model to generate a notification to be sent to parents. Specifically, the generative AI model creates an appropriate message based on a notification template. The input is a flag indicating bullying behavior and analysis results, and the output is the generated notification message.

[0131] Step 8:

[0132] The server sends the generated notification message to the parent communication device. Specifically, it sends push notifications using Firebase Cloud Messaging (FCM) or emails using the SMTP protocol. The input is the generated notification message, and the output is the notification sent to the parent communication device.

[0133] This series of processing steps enables the system to immediately notify parents in real time when their child is being bullied.

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] In modern society, the risk of children being bullied is increasing. Particularly in group settings such as schools, bullying often occurs without the adults around them noticing. It is difficult for parents to grasp the situation of their child being bullied, making it difficult to take appropriate measures quickly. To solve this problem, a system is needed that can detect bullying at an early stage and immediately notify parents.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server includes a voice collection means provided in the communication terminal used by the child, a voice transmission means for transmitting voice data collected by the voice collection means to the server, a voice conversion means by the server for converting the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, a notification means for sending a notification to the parent if the bullying detection means determines that there is a high possibility of bullying, and a notification means for sending a push notification to the parent's smartphone. This enables parents to understand in real time when their child is being bullied and to respond quickly.

[0139] "Child" refers to a minor who is still in the developmental stage.

[0140] A "communication terminal" refers to an electronic device that transmits and receives data via a network.

[0141] "Sound collection means" refers to a device or function for collecting surrounding sounds in a communication terminal.

[0142] "Audio transmission means" refers to a device or function for transmitting collected audio data to a server.

[0143] A "server" refers to a computer system that provides data processing and storage via a network.

[0144] "Speech recognition technology" refers to technology for converting voice data into text data.

[0145] "Speech conversion means" refers to a process or function for converting voice data into text data.

[0146] "Text data" refers to data in sentence format converted using voice recognition technology.

[0147] "Bullying detection means" refers to a device or function for analyzing text data and detecting bullying behavior.

[0148] "Notification means" refers to a device or function for notifying the user of specific information.

[0149] A "smartphone" refers to a multi-function mobile device that can access the Internet and run applications in addition to the functions of a mobile phone.

[0150] "Push notification" refers to a means of sending information from a server to a device such as a smartphone in real time.

[0151] The present invention describes a system for detecting bullying by collecting voice data from communication devices used by children. This system detects bullying occurring around children in real time and immediately notifies parents, enabling them to take early action. Specific embodiments for implementing the present invention are described in detail below.

[0152] Communication terminal operation

[0153] The device is equipped with an audio collection means for collecting sounds around the child. A microphone is used to collect audio data, which is temporarily compressed in the device and sent to a server as packets for efficient data transmission. An appropriate protocol is used to ensure data security and reliability.

[0154] Server Operation

[0155] The server receives the voice data sent from the device. The received voice data is converted into text data using a voice conversion means. For example, Google Cloud Speech-to-Text API is used as a voice recognition technology. The server then analyzes the converted text data using a bullying detection means. This analysis utilizes natural language processing technology and machine learning algorithms.

[0156] Detecting and notifying bullying behavior

[0157] If the bullying detection means analyzes the text data and determines that there is a high possibility of bullying, the server uses the notification means to send a notification to the parent. This notification is sent as a push notification to the parent's smartphone. For example, if the text "No one cares if you're not here" is detected, the bullying detection means determines this to be bullying and sends a warning message to the parent's smartphone.

[0158] Hardware and software used

[0159] Hardware:

[0160] Communication device (smartphone)

[0161] Microphone (built-in or external)

[0162] software:

[0163] A speech recognition library (such as Google Cloud Speech-to-Text API)

[0164] Audio processing library (such as pydub)

[0165] HTTP communication library (requests, etc.)

[0166] Natural language processing libraries (NLTK, spaCy, etc.)

[0167] Email sending library (smtplib, etc.)

[0168] Specific examples

[0169] Example 1: Classroom conversation

[0170] Suppose a child is having a conversation with a friend in class and the friend says, "You're so stupid! You can't even figure out such a simple problem?" The device collects the audio and sends this audio data to a server. The server converts the audio data into text data and analyzes it. If this analysis determines that there is a possibility of bullying, a push notification is sent to the parent's smartphone saying, "Your child may be being bullied."

[0171] Prompt Sentence Examples

[0172] "If your child says, 'You're not my friend!' this is likely bullying. Please check immediately."

[0173] This system allows parents to understand in real time when their children are being bullied and to take quick action.

[0174] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0175] Step 1:

[0176] The device collects audio. It uses the smartphone's built-in microphone to record the audio around the child in real time. The input is the surrounding audio, which is then captured as digital audio data.

[0177] Step 2:

[0178] The terminal compresses the collected voice data and sends it to the server. The voice data is converted into packets in a compressed format and sent to the server using the HTTP protocol. The input is the collected voice data and the output is the compressed voice data sent to the server.

[0179] Step 3:

[0180] The server converts the received voice data into text data using speech recognition technology. The Google Cloud Speech-to-Text API is used to output the voice data as text data. The input is compressed voice data, and the output is text data.

[0181] Step 4:

[0182] The server analyzes the converted text data to detect potentially bullying language, using natural language processing technology and machine learning algorithms to determine whether specific keywords or phrases are included. The input is text data, and the analysis results are the output.

[0183] Step 5:

[0184] If the server determines that there is a high possibility of bullying, it will use a notification method to send a notification to the parent. Specifically, a push notification is sent to the parent's smartphone. The input is the analysis result, and the output is a warning message sent to the parent's smartphone as a push notification.

[0185] Step 6:

[0186] Parents receive push notifications on their smartphones and check the contents. They can check their child's status based on the notification content and take necessary measures. The input is the notification message displayed on the smartphone, and the output is the parent's actions.

[0187] This process allows parents to be notified in real time when their child is being bullied, allowing them to take immediate action.

[0188] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0189] The present invention combines a system that collects voice data and detects bullying behavior in a communication device used by a child with an emotion engine that recognizes the user's emotions. An embodiment of this system will be described in detail below.

[0190] Communication terminal operation

[0191] The terminal is equipped with a voice collection means for collecting voice data. This means includes a function for collecting surrounding voices in real time using a microphone. For example, the voices of children talking in a classroom can be captured. In this case, the voice collection means includes an emotion recognition function to analyze the user's emotions.

[0192] The collected audio data is compressed and converted into a packet format for efficient data transmission to the server, using a secure data transmission protocol.

[0193] Server Operation

[0194] The server receives the voice data sent from the device. The received voice data is converted into text data using a voice conversion means. This process uses voice recognition technology such as the Google Cloud Speech-to-Text API.

[0195] The server further includes an emotion recognition unit that recognizes the user's emotion from the voice data. This allows the emotion information to be analyzed along with the text data. The emotion recognition unit detects emotions from the tone, pitch, intensity, etc. of the voice and generates emotion categories such as joy, anger, sadness, etc.

[0196] The server then analyzes the text data and emotional information using a bullying detection method that utilizes natural language processing technology and machine learning algorithms. Taking emotional information into account when detecting potentially bullying statements enables more accurate detection.

[0197] Notification method operation

[0198] If the server determines that bullying is likely, it stores the results and triggers a notification mechanism, which includes details based on text data and emotional information.

[0199] Specifically, for example, if the text "No one would care if you weren't here" is converted from speech and the user's emotion is determined to be "sadness," the server will use this result to send a push notification to the parent's smartphone stating, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0200] Specific examples

[0201] Example 1: Classroom conversations and sentiment analysis

[0202] scenario:

[0203] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0204] 1. The device collects audio and analyzes emotions using emotion recognition means.

[0205] 2. The device sends the collected voice data and emotional information to the server.

[0206] 3. The server converts the voice data into text data and stores the emotional information.

[0207] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0208] 5. The server sends a push notification to the parent's smartphone stating, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0209] In this way, the system operates in real time, instantly notifying parents when their child is being bullied, enabling them to take prompt action. In addition, by incorporating emotional information, a deeper understanding and response can be achieved.

[0210] The processing flow will be explained below.

[0211] Step 1:

[0212] The device collects surrounding sounds in real time through a microphone, using a sound collection means to capture, for example, conversations in a classroom.

[0213] Step 2:

[0214] The device compresses the collected voice data and prepares it for efficient transmission to the server, using an appropriate compression algorithm to improve data transmission efficiency.

[0215] Step 3:

[0216] The terminal converts the compressed voice data into a packet format and sends it to the server using a secure data transmission protocol (e.g., HTTPS).

[0217] Step 4:

[0218] The server decompresses the audio data received from the device and converts it into text data using speech recognition technology, for example, the Google Cloud Speech-to-Text API.

[0219] Step 5:

[0220] The server analyzes the user's emotions using the voice data and emotion recognition means, inferring emotions from the tone, pitch, and intensity of the voice and generating emotion categories (e.g., joy, anger, sadness, etc.).

[0221] Step 6:

[0222] The server analyzes the text data and emotional information with a bullying detection method that uses natural language processing techniques and machine learning algorithms to detect keywords and patterns in the text that indicate bullying behavior, while also taking emotional information into account.

[0223] Step 7:

[0224] If the server determines that there is a high possibility of bullying based on the analysis results, it saves the results and triggers a notification method, notifying parents of the details of the detection and detailed data including emotional information.

[0225] Step 8:

[0226] The server sends a push notification to the parent's smartphone or tablet device, containing detailed information such as, "Your child may be experiencing bullying. Your child also appears to be experiencing strong sadness."

[0227] Step 9:

[0228] The user (parent) can check the received push notification and view detailed information through the app, which allows them to understand their child's situation and take necessary action.

[0229] Step 10:

[0230] The server monitors the parent's response and sends additional follow-up notifications as needed, enabling early detection and prompt response when a child is being bullied.

[0231] Example 2

[0232] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0233] In recent years, bullying among children has become a social problem, and early detection and countermeasures are needed. However, current monitoring and reporting systems have difficulty detecting signs of bullying in real time or making highly accurate judgments that take the user's emotions into account. In this environment, a system is needed that can not only detect bullying early, but also confirm changes in the victim's emotions.

[0234] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice collection means provided in the communication terminal used by the child, a voice transmission means for transmitting collected voice data to the server, a voice conversion means for converting the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, a notification means for sending a notification to the parent when it is determined that there is a high possibility of bullying, an emotion recognition means for recognizing the user's emotion from the voice data, and a means for analyzing the emotion information in combination with the text data. This makes it possible to detect signs of bullying in real time, and to make a highly accurate determination and immediately notify the victim while also taking into account changes in the victim's emotions.

[0235] Children are young people who are still growing, generally those under the age of 18.

[0236] A communication terminal refers to an electronic device for sending and receiving digital information, and includes, for example, a smartphone or tablet.

[0237] An audio collection means is a technical device or function that picks up surrounding sounds and voices, typically a microphone.

[0238] The audio transmission means refers to a technical mechanism for transmitting collected audio data to another device or server, and transmits data via the Internet or wireless communication.

[0239] A server is a computer system that provides data and resources over a network, providing file storage and processing power.

[0240] Audio data refers to a digital representation of sound collected by an audio input device such as a microphone, including the conversion of sound waves into electrical signals.

[0241] Speech recognition technology is a technology for analyzing voice data and converting it into text data, using specific algorithms and machine learning models.

[0242] Text data refers to a string of characters or a collection of letters, and refers to data that is stored and processed in a digital format.

[0243] Bullying detection methods are technologies that analyze text data to find signs of bullying, and include natural language processing technology and machine learning algorithms.

[0244] A notification means refers to a mechanism for conveying information about a specific event to a user or other system when that event occurs, and examples include push notifications and emails.

[0245] Emotion recognition means refers to a technical means for analyzing voice data to identify a user's emotions, and is based on characteristics such as tone, pitch, and intensity of the voice.

[0246] Emotion information refers to data indicating the user's emotional state obtained by emotion recognition means, and includes categories such as joy, anger, and sadness.

[0247] Natural language processing technology is a technology that allows computers to understand, interpret, and generate human language, and includes text analysis and speech recognition.

[0248] A machine learning algorithm is an algorithm that builds a model based on data and detects specific patterns and trends, and is used for data analysis on a server.

[0249] Push notifications refer to notification messages that are actively sent from a server to a client device, and are a way to get a user's real-time attention.

[0250] The present invention combines a system that collects voice data and detects bullying behavior in a communication device used by a child with an emotion engine that recognizes the user's emotions. An embodiment of this system will be described in detail below.

[0251] Communication terminal operation

[0252] The terminal is equipped with an audio collection means for collecting audio data. This means includes a function for collecting ambient audio in real time using a microphone. For example, audio is captured when children are talking in a classroom. In this case, an emotion recognition function is included in the audio collection means to analyze the user's emotions. The emotion recognition function analyzes the tone, pitch, and intensity of the audio to generate emotion categories such as joy, anger, and sadness.

[0253] The collected audio data is compressed and converted into a packet format for efficient data transmission to the server, using a secure data transmission protocol (e.g., TLS).

[0254] Server Operation

[0255] The server receives the voice data sent from the device. It then uses speech recognition technology to convert the received voice data into text data. This process uses services such as the Google Cloud Speech-to-Text API. The converted text data is saved in the form of a message such as, "You're so stupid, you can't even answer such a simple question?"

[0256] The server also runs an emotion recognition algorithm to identify the user's emotions from the audio data. Emotions are detected as emotion categories such as joy, anger, sadness, etc. based on the tone, pitch, and intensity of the audio. For example, the emotion of sadness is detected for the text, "You're so stupid, you can't even figure out such a simple question?"

[0257] The server then analyzes this text data and emotional information to detect potentially bullying phrases. The analysis uses natural language processing technology and machine learning algorithms, making it possible to detect phrases that are likely to be bullying, as well as complex patterns that take emotional information into account.

[0258] How the notification system works

[0259] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method. For example, it sends a push notification to the parent's smartphone. The message sent is, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0260] Specific examples

[0261] Example 1: Classroom conversations and sentiment analysis

[0262] scenario:

[0263] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0264] Operation steps:

[0265] 1. The device collects voice and analyzes emotions using emotion recognition.

[0266] 2. The device sends the collected voice data and emotional information to the server.

[0267] 3. The server converts the voice data into text data and stores the emotional information.

[0268] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0269] 5. The server sends a push notification to the parent's smartphone stating, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0270] Example prompts for generative AI models

[0271] "Please tell us how to detect possible bullying from speech data and emotional information when a child is told by a friend in the classroom, 'You're so stupid! You can't understand such a simple problem?' and notify parents."

[0272] In this way, the system operates in real time, notifying parents immediately when their child is being bullied, enabling them to take immediate action. By incorporating emotional information, the system can also provide a deeper understanding and response.

[0273] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0274] Step 1: Collecting audio data

[0275] The device uses a microphone to collect ambient audio in real time while children are conversing in the classroom. For example, it captures the audio when a child says something like, "You're so stupid! You can't even figure out such a simple question!" This collected audio data is used as input. This audio data also includes information such as the tone, pitch, and intensity of the voice.

[0276] Step 2: Recognize emotions

[0277] The device analyzes the user's emotions from the collected voice data. Using emotion recognition, it analyzes the tone, pitch, and intensity of the voice and generates emotion categories such as joy, anger, and sadness. For example, the emotion "sadness" is detected from the collected voice. This emotional information is generated as output.

[0278] Step 3: Compress and convert audio data

[0279] The terminal compresses the collected voice data and emotion information and converts them into packet format using a secure data transmission protocol (e.g., TLS). The compressed voice data and emotion information are input and packet format data is output.

[0280] Step 4: Sending audio data

[0281] The device transmits the voice data and emotion information converted into packets to the server via the Internet. The input is the packetized data, and the output is the data transmitted to the server.

[0282] Step 5: Receiving audio data

[0283] The server receives packetized voice data and emotion information sent from the terminal. The received data is input, and the voice data and emotion information are output.

[0284] Step 6: Convert audio data to text

[0285] The server uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is voice data, and the output is text data. For example, the generated text might say, "You're so stupid! You can't even answer such a simple question?"

[0286] Step 7: Combining Emotional Information

[0287] The server combines the emotional information detected using an emotion recognition algorithm with the text data. The input is text data and emotional information, and the output is a combination of text and emotional categories. For example, the text "You're so stupid! You can't even figure out such a simple question?" is combined with the emotion "sadness."

[0288] Step 8: Detect bullying

[0289] The server uses natural language processing technology and machine learning algorithms to analyze text data and emotional information to detect potentially bullying statements. The input is a combination of text and emotional category data, and the output is a judgment result indicating a high probability of bullying. For example, based on the text "You're so stupid! You can't even figure out such a simple question?" and the emotional information of "sadness," it is judged that there is a high probability of bullying.

[0290] Step 9: Sending notifications

[0291] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method. Specifically, it sends a push notification to the parent's smartphone. The input is the bullying determination result, and the output is a notification message. For example, a message may be sent saying, "Your child may be being bullied. Also, your child appears to be feeling very sad."

[0292] (Application example 2)

[0293] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0294] Conventional bullying detection systems convert only audio data into text data and detect bullying behaviors based on analyzing only the text data. This makes it difficult to consider emotional nuances and make highly accurate judgments. Furthermore, because it is not possible to grasp a child's emotional state when bullying occurs, appropriate responses can be delayed. Therefore, there is a need for a system that can improve the accuracy of bullying detection, quickly grasp a child's emotional state, and notify parents.

[0295] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice conversion means that converts voice data into text data using voice recognition technology, an emotion recognition means that analyzes emotions from the voice data and generates emotion information, a bullying detection means that analyzes the text data and the emotion information and detects words that may be bullying, and a notification means that sends a notification to the parents when the bullying detection means determines that there is a high possibility of bullying. This improves the accuracy of detecting bullying behavior and makes it possible to quickly grasp a child's emotional state.

[0296] The "audio collection means" is a function provided in the communication terminal used by the child to collect surrounding sounds.

[0297] The "voice transmission means" is a function that transmits collected voice data to a server.

[0298] The "voice conversion means" is a function that converts voice data into text data using voice recognition technology.

[0299] The "emotion recognition means" is a function that analyzes emotions from voice data and generates emotional information.

[0300] The "bullying detection means" is a function that analyzes text data and emotional information to detect potentially bullying language.

[0301] "Notification means" is a function that sends a notification to parents if it is determined that there is a high possibility of bullying.

[0302] A "machine learning algorithm" is a technology that builds a model based on empirical data and makes predictions and classifications for unknown data.

[0303] "Natural language processing technology" is a technology that uses computers to understand, interpret, and generate natural language.

[0304] A "server" is a computer that receives data from a terminal and analyzes and processes it.

[0305] System configuration and operation

[0306] The system collects ambient sounds in real time using a sound collection means provided in the communication terminal used by the child. The sound collection means includes a built-in microphone. The terminal further includes a sound transmission means for transmitting the collected sound data to a server. The sound data is compressed and converted into a packet format for efficient data transmission to the server. A secure data transmission protocol is used for this purpose.

[0307] Server Processing

[0308] The server has the function of receiving voice data sent from the device. The received voice data is first converted into text data by a voice conversion means. This process uses Google Cloud Speech-to-Text API as the voice recognition technology. The server also has an emotion recognition means to analyze emotions from the voice data. To do this, the tone, pitch, intensity, etc. of the voice are evaluated, and emotion categories such as joy, anger, and sadness are generated.

[0309] The generated text data and emotional information are analyzed by the bullying detection means, which uses natural language processing technology and machine learning algorithms to make highly accurate judgments by taking into account potentially bullying-related phrases and emotional information contained in the text.

[0310] Notification method operation

[0311] If the server determines that bullying is likely, it saves the result and triggers a notification method. This notification includes details based on the analyzed text data and emotional information. Specifically, if the text "No one cares if you're not around" is converted from speech and the user's emotion is determined to be "sadness," the server will use this result to send a notification to the parent's smartphone stating, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0312] Examples and prompts

[0313] Specific examples

[0314] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0315] 1. The device collects audio and analyzes emotions using emotion recognition means.

[0316] 2. The device sends the collected voice data and emotional information to the server.

[0317] 3. The server converts the voice data into text data and stores the emotional information.

[0318] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0319] 5. The server sends a notification to the parent's smartphone saying, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0320] Prompt Sentence Examples

[0321] "Analyzing children's classroom conversations to detect bullying behavior and identify emotional states."

[0322] "Oh, you don't know that?" is analyzed and emotions are identified from the voice characteristics.

[0323] "The text 'No one cares if you're gone' is converted to speech and the emotion is determined to be sadness."

[0324] As described above, this system not only uses voice data and emotional information to detect bullying with high accuracy, but also helps protect children's safety and psychological health by promptly reporting the incident to parents.

[0325] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0326] Step 1:

[0327] The device uses a built-in microphone to collect ambient sound, which is then temporarily stored as raw audio data. At this stage, the audio data is not yet in a directly usable format.

[0328] Input: Surrounding environmental sound

[0329] Output: Raw audio data

[0330] Step 2:

[0331] The device compresses the collected raw voice data and converts it into a packet format for efficient data transmission using voice data format conversion and compression algorithms, and then transmits the compressed data to the server via a secure data transmission protocol.

[0332] Input: Raw audio data

[0333] Output: Compressed and packetized audio data

[0334] Step 3:

[0335] The server receives the compressed and packetized voice data sent from the terminal, decompresses and decodes the received data, and reconstructs it into raw voice data.

[0336] Input: Compressed and packetized voice data

[0337] Output: Decompressed and decoded raw audio data

[0338] Step 4:

[0339] The server converts the raw audio data into text using the Google Cloud Speech-to-Text API, which feeds the audio data into the API and converts it into text in real time.

[0340] Input: Raw audio data

[0341] Output: Text data

[0342] Step 5:

[0343] The server uses emotion recognition to analyze the emotions in the voice data. It analyzes features such as tone, pitch, and intensity of the voice and generates emotion categories (e.g., joy, anger, sadness, etc.). Again, technologies such as Google Cloud Natural Language API may be used.

[0344] Input: Raw audio data

[0345] Output: Sentiment category and sentiment score

[0346] Step 6:

[0347] The server analyzes the text data and emotion categories using a bullying detection method. It uses natural language processing techniques and machine learning algorithms to detect potentially bullying statements in the text. The emotion information is used to assist in the judgment.

[0348] Input: Text data and emotion categories

[0349] Output: Judgment result of possibility of bullying

[0350] Step 7:

[0351] If the server determines that there is a high possibility of bullying, it stores the result of the determination and emotional information, and triggers a notification method. A push notification is sent to the parent or guardian's smartphone informing them of the possibility that their child is being bullied and details of their emotional state. This notification includes details based on text data and emotional information.

[0352] Input: Judgment result of possibility of bullying and emotion score

[0353] Output: Push notification to parent

[0354] example:

[0355] Suppose a child is talking with a friend in the classroom and says, "No one would care if you weren't here." In this case, the server recognizes the emotion "sadness" from the voice data and determines that this is bullying. As a result, the server sends a notification to the parent's smartphone saying, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0356] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0357] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0358] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0359] [Second embodiment]

[0360] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0361] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0362] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0363] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0364] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0365] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0366] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0367] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0368] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0369] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0370] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0371] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0372] The present invention describes a system for collecting voice data and detecting bullying behavior in communication terminals used by children. The implementation of this system is carried out through the following steps.

[0373] Communication terminal operation

[0374] The device is equipped with an audio collection means for collecting the child's audio data. Specifically, it uses a microphone to collect surrounding sounds. This audio data is transmitted to a server in real time.

[0375] Once the device collects the voice data, it compresses it and sends it to the server as packets for efficient data transmission. Appropriate protocols must be used to ensure data security and reliability.

[0376] Server Operation

[0377] The server receives the voice data sent from the terminal. The received voice data is converted into text data by a voice conversion means. For example, a voice recognition technology such as Google Cloud Speech-to-Text API is used for this purpose.

[0378] The server then analyzes the converted text data using a bullying detection method that utilizes natural language processing technology and machine learning algorithms to analyze the text data and determine whether its content indicates bullying behavior.

[0379] Notification method operation

[0380] If the bullying detection means analyzes the text data and determines that there is a high possibility of bullying, the server will use the notification means to send a warning to parents via push notifications on smartphones or email.

[0381] For example, if the text "No one cares if you're not around" is converted from speech, the bullying detection method will determine this as bullying, and a push notification will be sent to the parent's smartphone stating, "Your child may be being bullied."

[0382] Specific examples

[0383] Example 1: Classroom conversation

[0384] scenario:

[0385] Imagine a child is having a conversation with a friend in class and the friend says to them, "You're so stupid! You can't even figure out such a simple problem?"

[0386] 1. The device collects audio.

[0387] 2. The device sends the audio data to the server.

[0388] 3. The server converts the audio data into text data.

[0389] 4. The server analyzes the text data and detects possible bullying.

[0390] 5. The server sends a push notification to the parent's smartphone.

[0391] In this way, the system operates in real time, instantly notifying parents when their child is being bullied, allowing for prompt action.

[0392] The processing flow will be explained below.

[0393] Step 1:

[0394] The device collects ambient sounds in real time through a microphone, for example capturing the sound of a child talking with a friend in the classroom.

[0395] Step 2:

[0396] The terminal compresses the collected voice data to improve the efficiency of data transmission, and it is important to use an appropriate compression algorithm.

[0397] Step 3:

[0398] The terminal converts the compressed voice data into a packet format and sends it to the server using a secure data transmission protocol, which ensures the authenticity and security of the data.

[0399] Step 4:

[0400] The server decompresses the audio data received from the device and converts it into text using speech recognition technology. Commercially available speech recognition software, such as the Google Cloud Speech-to-Text API, can be used.

[0401] Step 5:

[0402] The server analyzes the converted text data with a bullying detection tool that uses natural language processing techniques and machine learning algorithms to detect keywords and patterns in the text that indicate bullying behavior.

[0403] Step 6:

[0404] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method, along with details of the bullying act, such as the time and location.

[0405] Step 7:

[0406] The server sends real-time push notifications to parents' smartphones or tablet devices, informing them of potential bullying incidents and providing additional information.

[0407] Step 8:

[0408] The user (parent) can check the details and take necessary action according to the push notification received. Parents can also view more detailed information and history through the application.

[0409] Step 9:

[0410] If the parent takes any action after receiving the notification, the server will check the response status again and send a follow-up notification if necessary. This series of processes enables early detection and prompt response when a child is being bullied.

[0411] Example 1

[0412] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0413] The present invention relates to a system for detecting bullying behavior by collecting voice data using communication devices used by children. Conventional technologies have inefficient processes for collecting voice data, detecting bullying behavior, and notifying parents, making it difficult to respond in real time. Furthermore, the accuracy of determining bullying behavior is low, resulting in frequent false positives. There is a need to solve these issues and more quickly and accurately detect bullying behavior and notify parents.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0415] In this invention, the server includes an audio collection means, an audio transmission means, an audio conversion means, a bullying detection means, a notification generation means, a notification content generation means, and a notification transmission means. This allows for real-time collection of audio around a child and secure transmission of compressed audio data to the server. The server also quickly converts the audio data into text data and uses machine learning algorithms and natural language processing technology to analyze bullying behavior with high accuracy. Furthermore, it automatically generates appropriate notification content using a generative AI model, enabling prompt and appropriate warnings to be sent to parents.

[0416] The "voice collecting means" is a voice input device such as a microphone provided in the communication terminal used by the child, and is a device that has the function of collecting surrounding voices.

[0417] The "audio transmission means" is a communication means having the function of compressing collected audio data and transmitting it safely to the server.

[0418] The "voice conversion means" is a means having a function of converting voice data collected in the server into text data using voice recognition technology.

[0419] The "bullying detection means" is a means that has the function of analyzing the converted text data and detecting words that may be bullying.

[0420] The "notification means" is a means that has the function of sending a notification to parents when it is determined that there is a high possibility of bullying.

[0421] The "notification generating means" is a means having a function of generating the content of a notification to be sent to parents when bullying behavior is detected.

[0422] "Notification content generation means" means a means having the function of generating appropriate notification content based on predefined templates and analysis results using a generative AI model.

[0423] The "notification sending means" is a means having a function of sending the generated notification message to the parent communication terminal.

[0424] A "machine learning algorithm" is a technology that allows computers to learn from data and make predictions and classifications.

[0425] "Natural language processing technology" is a computational technology for understanding, generating, and analyzing human language.

[0426] A "generative AI model" is a pre-trained artificial intelligence model that has the ability to generate appropriate outputs based on specific inputs.

[0427] The present invention relates to a system for detecting bullying by collecting voice data using a communication terminal used by a child. The system includes a communication terminal, a server, a bullying detection unit, and a notification unit.

[0428] Configuration and operation of communication terminal

[0429] The device is equipped with an audio capture unit to capture sounds around the child. Specifically, a microphone captures the sound, and a dedicated application manages the sound data in real time. The collected sound data is compressed into MP3 format using a LAME MP3 encoder or similar and securely transmitted to a server using the HTTPS protocol.

[0430] Server configuration and operation

[0431] The server receives the voice data sent from the device and temporarily stores it in storage. The received voice data is converted into text data using speech recognition technology such as the Google Cloud Speech-to-Text API. The server then analyzes the text data using scikit-learn and TensorFlow to detect bullying behavior. This analysis process utilizes machine learning algorithms and natural language processing technology.

[0432] Bullying detection and notification

[0433] The bullying detection means analyzes the text data to determine whether certain keywords or phrases correspond to bullying behavior. If bullying behavior is detected, the notification generation means uses a generative AI model to generate notification content to send to parents, which may utilize pre-defined templates and the generative AI model.

[0434] Example prompt sentence:

[0435] Audio analysis has detected the following potential bullying behavior:

[0436] "No one would care if you weren't here."

[0437] Parents should be notified. Please generate the notification content.

[0438] Parental Notification

[0439] The generated notification message is sent to the parent communication device using a notification sending means, such as a push notification using Firebase Cloud Messaging (FCM) or an email using the SMTP protocol.

[0440] Specific examples

[0441] For example, if a child in a classroom is told, "You're so stupid, you can't even figure out such a simple problem?", the system will act as follows:

[0442] 1. The device collects the audio, compresses it into MP3 format, and sends it to the server.

[0443] 2. The server receives the voice data and converts it into text data using voice recognition technology.

[0444] 3. The server analyzes the text data using machine learning algorithms and natural language processing technology, and detects statements such as, "You're so stupid, you can't even figure out such a simple question?" as bullying.

[0445] 4. The server uses the generative AI model to generate appropriate notification content, creating a message such as "Your child may be being bullied."

[0446] 5. The server sends the generated notification message to the parent communication device as a push notification.

[0447] This system allows parents to be notified immediately if their child is being bullied, allowing them to take immediate action.

[0448] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0449] Step 1:

[0450] The device collects audio data from the child's surroundings. Specifically, it records audio in real time using a built-in microphone. The input is the surrounding audio, and the output is the recorded audio data. A dedicated application manages this audio data and temporarily stores it on the device.

[0451] Step 2:

[0452] The device compresses the collected audio data. Specifically, it converts the audio data into MP3 format using software such as the LAME MP3 encoder. The input is the recorded audio data, and the output is a compressed MP3 file. This process reduces the data size and enables efficient data transfer.

[0453] Step 3:

[0454] The device sends the compressed audio data to the server. Specifically, the data is uploaded to the server using the HTTPS protocol to ensure data security. The input is a compressed MP3 file, and the output is the audio data sent to the server.

[0455] Step 4:

[0456] The server receives the audio data sent from the device. The received data is temporarily stored in the server's storage. The input is the sent MP3 file, and the output is the audio data stored in the server's storage.

[0457] Step 5:

[0458] The server uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the audio data into text. Specifically, the audio data is sent to the API, and the returned text data is stored in a database on the server. The input is an MP3 file, and the output is the converted text data.

[0459] Step 6:

[0460] The server analyzes the text data to detect possible bullying. It uses machine learning algorithms such as scikit-learn and TensorFlow and natural language processing techniques to analyze keywords and emotions that indicate bullying behavior contained in the text data. The input is the text data, and the output is a flag indicating bullying behavior and the analysis results.

[0461] Step 7:

[0462] When the server detects bullying behavior, it uses a generative AI model to generate a notification to be sent to parents. Specifically, the generative AI model creates an appropriate message based on a notification template. The input is a flag indicating bullying behavior and analysis results, and the output is the generated notification message.

[0463] Step 8:

[0464] The server sends the generated notification message to the parent communication device. Specifically, it sends push notifications using Firebase Cloud Messaging (FCM) or emails using the SMTP protocol. The input is the generated notification message, and the output is the notification sent to the parent communication device.

[0465] This series of processing steps enables the system to immediately notify parents in real time when their child is being bullied.

[0466] (Application example 1)

[0467] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0468] In modern society, the risk of children being bullied is increasing. Particularly in group settings such as schools, bullying often occurs without the adults around them noticing. It is difficult for parents to grasp the situation of their child being bullied, making it difficult to take appropriate measures quickly. To solve this problem, a system is needed that can detect bullying at an early stage and immediately notify parents.

[0469] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0470] In this invention, the server includes a voice collection means provided in the communication terminal used by the child, a voice transmission means for transmitting voice data collected by the voice collection means to the server, a voice conversion means by the server for converting the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, a notification means for sending a notification to the parent if the bullying detection means determines that there is a high possibility of bullying, and a notification means for sending a push notification to the parent's smartphone. This enables parents to understand in real time when their child is being bullied and to respond quickly.

[0471] "Child" refers to a minor who is still in the developmental stage.

[0472] A "communication terminal" refers to an electronic device that transmits and receives data via a network.

[0473] "Sound collection means" refers to a device or function for collecting surrounding sounds in a communication terminal.

[0474] "Audio transmission means" refers to a device or function for transmitting collected audio data to a server.

[0475] A "server" refers to a computer system that provides data processing and storage via a network.

[0476] "Speech recognition technology" refers to technology for converting voice data into text data.

[0477] "Speech conversion means" refers to a process or function for converting voice data into text data.

[0478] "Text data" refers to data in sentence format converted using voice recognition technology.

[0479] "Bullying detection means" refers to a device or function for analyzing text data and detecting bullying behavior.

[0480] "Notification means" refers to a device or function for notifying the user of specific information.

[0481] A "smartphone" refers to a multi-function mobile device that can access the Internet and run applications in addition to the functions of a mobile phone.

[0482] "Push notification" refers to a means of sending information from a server to a device such as a smartphone in real time.

[0483] The present invention describes a system for detecting bullying by collecting voice data from communication devices used by children. This system detects bullying occurring around children in real time and immediately notifies parents, enabling them to take early action. Specific embodiments for implementing the present invention are described in detail below.

[0484] Communication terminal operation

[0485] The device is equipped with an audio collection means for collecting sounds around the child. A microphone is used to collect audio data, which is temporarily compressed in the device and sent to a server as packets for efficient data transmission. An appropriate protocol is used to ensure data security and reliability.

[0486] Server Operation

[0487] The server receives the voice data sent from the device. The received voice data is converted into text data using a voice conversion means. For example, Google Cloud Speech-to-Text API is used as a voice recognition technology. The server then analyzes the converted text data using a bullying detection means. This analysis utilizes natural language processing technology and machine learning algorithms.

[0488] Detecting and notifying bullying behavior

[0489] If the bullying detection means analyzes the text data and determines that there is a high possibility of bullying, the server uses the notification means to send a notification to the parent. This notification is sent as a push notification to the parent's smartphone. For example, if the text "No one cares if you're not here" is detected, the bullying detection means determines this to be bullying and sends a warning message to the parent's smartphone.

[0490] Hardware and software used

[0491] Hardware:

[0492] Communication device (smartphone)

[0493] Microphone (built-in or external)

[0494] software:

[0495] A speech recognition library (such as Google Cloud Speech-to-Text API)

[0496] Audio processing library (such as pydub)

[0497] HTTP communication library (requests, etc.)

[0498] Natural language processing libraries (NLTK, spaCy, etc.)

[0499] Email sending library (smtplib, etc.)

[0500] Specific examples

[0501] Example 1: Classroom conversation

[0502] Suppose a child is having a conversation with a friend in class and the friend says, "You're so stupid! You can't even figure out such a simple problem?" The device collects the audio and sends this audio data to a server. The server converts the audio data into text data and analyzes it. If this analysis determines that there is a possibility of bullying, a push notification is sent to the parent's smartphone saying, "Your child may be being bullied."

[0503] Prompt Sentence Examples

[0504] "If your child says, 'You're not my friend!' this is likely bullying. Please check immediately."

[0505] This system allows parents to understand in real time when their children are being bullied and to take quick action.

[0506] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0507] Step 1:

[0508] The device collects audio. It uses the smartphone's built-in microphone to record the audio around the child in real time. The input is the surrounding audio, which is then captured as digital audio data.

[0509] Step 2:

[0510] The terminal compresses the collected voice data and sends it to the server. The voice data is converted into packets in a compressed format and sent to the server using the HTTP protocol. The input is the collected voice data and the output is the compressed voice data sent to the server.

[0511] Step 3:

[0512] The server converts the received voice data into text data using speech recognition technology. The Google Cloud Speech-to-Text API is used to output the voice data as text data. The input is compressed voice data, and the output is text data.

[0513] Step 4:

[0514] The server analyzes the converted text data to detect potentially bullying language, using natural language processing technology and machine learning algorithms to determine whether specific keywords or phrases are included. The input is text data, and the analysis results are the output.

[0515] Step 5:

[0516] If the server determines that there is a high possibility of bullying, it will use a notification method to send a notification to the parent. Specifically, a push notification is sent to the parent's smartphone. The input is the analysis result, and the output is a warning message sent to the parent's smartphone as a push notification.

[0517] Step 6:

[0518] Parents receive push notifications on their smartphones and check the contents. They can check their child's status based on the notification content and take necessary measures. The input is the notification message displayed on the smartphone, and the output is the parent's actions.

[0519] This process allows parents to be notified in real time when their child is being bullied, allowing them to take immediate action.

[0520] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0521] The present invention combines a system that collects voice data and detects bullying behavior in a communication device used by a child with an emotion engine that recognizes the user's emotions. An embodiment of this system will be described in detail below.

[0522] Communication terminal operation

[0523] The terminal is equipped with a voice collection means for collecting voice data. This means includes a function for collecting surrounding voices in real time using a microphone. For example, the voices of children talking in a classroom can be captured. In this case, the voice collection means includes an emotion recognition function to analyze the user's emotions.

[0524] The collected audio data is compressed and converted into a packet format for efficient data transmission to the server, using a secure data transmission protocol.

[0525] Server Operation

[0526] The server receives the voice data sent from the device. The received voice data is converted into text data using a voice conversion means. This process uses voice recognition technology such as the Google Cloud Speech-to-Text API.

[0527] The server further includes an emotion recognition unit that recognizes the user's emotion from the voice data. This allows the emotion information to be analyzed along with the text data. The emotion recognition unit detects emotions from the tone, pitch, intensity, etc. of the voice and generates emotion categories such as joy, anger, sadness, etc.

[0528] The server then analyzes the text data and emotional information using a bullying detection method that utilizes natural language processing technology and machine learning algorithms. Taking emotional information into account when detecting potentially bullying statements enables more accurate detection.

[0529] Notification method operation

[0530] If the server determines that bullying is likely, it stores the results and triggers a notification mechanism, which includes details based on text data and emotional information.

[0531] Specifically, for example, if the text "No one would care if you weren't here" is converted from speech and the user's emotion is determined to be "sadness," the server will use this result to send a push notification to the parent's smartphone stating, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0532] Specific examples

[0533] Example 1: Classroom conversations and sentiment analysis

[0534] scenario:

[0535] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0536] 1. The device collects audio and analyzes emotions using emotion recognition means.

[0537] 2. The device sends the collected voice data and emotional information to the server.

[0538] 3. The server converts the voice data into text data and stores the emotional information.

[0539] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0540] 5. The server sends a push notification to the parent's smartphone stating, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0541] In this way, the system operates in real time, instantly notifying parents when their child is being bullied, enabling them to take prompt action. In addition, by incorporating emotional information, a deeper understanding and response can be achieved.

[0542] The processing flow will be explained below.

[0543] Step 1:

[0544] The device collects surrounding sounds in real time through a microphone, using a sound collection means to capture, for example, conversations in a classroom.

[0545] Step 2:

[0546] The device compresses the collected voice data and prepares it for efficient transmission to the server, using an appropriate compression algorithm to improve data transmission efficiency.

[0547] Step 3:

[0548] The terminal converts the compressed voice data into a packet format and sends it to the server using a secure data transmission protocol (e.g., HTTPS).

[0549] Step 4:

[0550] The server decompresses the audio data received from the device and converts it into text data using speech recognition technology, for example, the Google Cloud Speech-to-Text API.

[0551] Step 5:

[0552] The server analyzes the user's emotions using the voice data and emotion recognition means, inferring emotions from the tone, pitch, and intensity of the voice and generating emotion categories (e.g., joy, anger, sadness, etc.).

[0553] Step 6:

[0554] The server analyzes the text data and emotional information with a bullying detection method that uses natural language processing techniques and machine learning algorithms to detect keywords and patterns in the text that indicate bullying behavior, while also taking emotional information into account.

[0555] Step 7:

[0556] If the server determines that there is a high possibility of bullying based on the analysis results, it saves the results and triggers a notification method, notifying parents of the details of the detection and detailed data including emotional information.

[0557] Step 8:

[0558] The server sends a push notification to the parent's smartphone or tablet device, containing detailed information such as, "Your child may be experiencing bullying. Your child also appears to be experiencing strong sadness."

[0559] Step 9:

[0560] The user (parent) can check the received push notification and view detailed information through the app, which allows them to understand their child's situation and take necessary action.

[0561] Step 10:

[0562] The server monitors the parent's response and sends additional follow-up notifications as needed, enabling early detection and prompt response when a child is being bullied.

[0563] Example 2

[0564] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0565] In recent years, bullying among children has become a social problem, and early detection and countermeasures are needed. However, current monitoring and reporting systems have difficulty detecting signs of bullying in real time or making highly accurate judgments that take the user's emotions into account. In this environment, a system is needed that can not only detect bullying early, but also confirm changes in the victim's emotions.

[0566] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice collection means provided in the communication terminal used by the child, a voice transmission means for transmitting collected voice data to the server, a voice conversion means for converting the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, a notification means for sending a notification to the parent when it is determined that there is a high possibility of bullying, an emotion recognition means for recognizing the user's emotion from the voice data, and a means for analyzing the emotion information in combination with the text data. This makes it possible to detect signs of bullying in real time, and to make a highly accurate determination and immediately notify the victim while also taking into account changes in the victim's emotions.

[0567] Children are young people who are still growing, generally those under the age of 18.

[0568] A communication terminal refers to an electronic device for sending and receiving digital information, and includes, for example, a smartphone or tablet.

[0569] An audio collection means is a technical device or function that picks up surrounding sounds and voices, typically a microphone.

[0570] The audio transmission means refers to a technical mechanism for transmitting collected audio data to another device or server, and transmits data via the Internet or wireless communication.

[0571] A server is a computer system that provides data and resources over a network, providing file storage and processing power.

[0572] Audio data refers to a digital representation of sound collected by an audio input device such as a microphone, including the conversion of sound waves into electrical signals.

[0573] Speech recognition technology is a technology for analyzing voice data and converting it into text data, using specific algorithms and machine learning models.

[0574] Text data refers to a string of characters or a collection of letters, and refers to data that is stored and processed in a digital format.

[0575] Bullying detection methods are technologies that analyze text data to find signs of bullying, and include natural language processing technology and machine learning algorithms.

[0576] A notification means refers to a mechanism for conveying information about a specific event to a user or other system when that event occurs, and examples include push notifications and emails.

[0577] Emotion recognition means refers to a technical means for analyzing voice data to identify a user's emotions, and is based on characteristics such as tone, pitch, and intensity of the voice.

[0578] Emotion information refers to data indicating the user's emotional state obtained by emotion recognition means, and includes categories such as joy, anger, and sadness.

[0579] Natural language processing technology is a technology that allows computers to understand, interpret, and generate human language, and includes text analysis and speech recognition.

[0580] A machine learning algorithm is an algorithm that builds a model based on data and detects specific patterns and trends, and is used for data analysis on a server.

[0581] Push notifications refer to notification messages that are actively sent from a server to a client device, and are a way to get a user's real-time attention.

[0582] The present invention combines a system that collects voice data and detects bullying behavior in a communication device used by a child with an emotion engine that recognizes the user's emotions. An embodiment of this system will be described in detail below.

[0583] Communication terminal operation

[0584] The terminal is equipped with an audio collection means for collecting audio data. This means includes a function for collecting ambient audio in real time using a microphone. For example, audio is captured when children are talking in a classroom. In this case, an emotion recognition function is included in the audio collection means to analyze the user's emotions. The emotion recognition function analyzes the tone, pitch, and intensity of the audio to generate emotion categories such as joy, anger, and sadness.

[0585] The collected audio data is compressed and converted into a packet format for efficient data transmission to the server, using a secure data transmission protocol (e.g., TLS).

[0586] Server Operation

[0587] The server receives the voice data sent from the device. It then uses speech recognition technology to convert the received voice data into text data. This process uses services such as the Google Cloud Speech-to-Text API. The converted text data is saved in the form of a message such as, "You're so stupid, you can't even answer such a simple question?"

[0588] The server also runs an emotion recognition algorithm to identify the user's emotions from the audio data. Emotions are detected as emotion categories such as joy, anger, sadness, etc. based on the tone, pitch, and intensity of the audio. For example, the emotion of sadness is detected for the text, "You're so stupid, you can't even figure out such a simple question?"

[0589] The server then analyzes this text data and emotional information to detect potentially bullying phrases. The analysis uses natural language processing technology and machine learning algorithms, making it possible to detect phrases that are likely to be bullying, as well as complex patterns that take emotional information into account.

[0590] How the notification system works

[0591] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method. For example, it sends a push notification to the parent's smartphone. The message sent is, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0592] Specific examples

[0593] Example 1: Classroom conversations and sentiment analysis

[0594] scenario:

[0595] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0596] Operation steps:

[0597] 1. The device collects voice and analyzes emotions using emotion recognition.

[0598] 2. The device sends the collected voice data and emotional information to the server.

[0599] 3. The server converts the voice data into text data and stores the emotional information.

[0600] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0601] 5. The server sends a push notification to the parent's smartphone stating, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0602] Example prompts for generative AI models

[0603] "Please tell us how to detect possible bullying from speech data and emotional information when a child is told by a friend in the classroom, 'You're so stupid! You can't understand such a simple problem?' and notify parents."

[0604] In this way, the system operates in real time, notifying parents immediately when their child is being bullied, enabling them to take immediate action. By incorporating emotional information, the system can also provide a deeper understanding and response.

[0605] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0606] Step 1: Collecting audio data

[0607] The device uses a microphone to collect ambient audio in real time while children are conversing in the classroom. For example, it captures the audio when a child says something like, "You're so stupid! You can't even figure out such a simple question!" This collected audio data is used as input. This audio data also includes information such as the tone, pitch, and intensity of the voice.

[0608] Step 2: Recognize emotions

[0609] The device analyzes the user's emotions from the collected voice data. Using emotion recognition, it analyzes the tone, pitch, and intensity of the voice and generates emotion categories such as joy, anger, and sadness. For example, the emotion "sadness" is detected from the collected voice. This emotional information is generated as output.

[0610] Step 3: Compress and convert audio data

[0611] The terminal compresses the collected voice data and emotion information and converts them into packet format using a secure data transmission protocol (e.g., TLS). The compressed voice data and emotion information are input and packet format data is output.

[0612] Step 4: Sending audio data

[0613] The device transmits the voice data and emotion information converted into packets to the server via the Internet. The input is the packetized data, and the output is the data transmitted to the server.

[0614] Step 5: Receiving audio data

[0615] The server receives packetized voice data and emotion information sent from the terminal. The received data is input, and the voice data and emotion information are output.

[0616] Step 6: Convert audio data to text

[0617] The server uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is voice data, and the output is text data. For example, the generated text might say, "You're so stupid! You can't even answer such a simple question?"

[0618] Step 7: Combining Emotional Information

[0619] The server combines the emotional information detected using an emotion recognition algorithm with the text data. The input is text data and emotional information, and the output is a combination of text and emotional categories. For example, the text "You're so stupid! You can't even figure out such a simple question?" is combined with the emotion "sadness."

[0620] Step 8: Detect bullying

[0621] The server uses natural language processing technology and machine learning algorithms to analyze text data and emotional information to detect potentially bullying statements. The input is a combination of text and emotional category data, and the output is a judgment result indicating a high probability of bullying. For example, based on the text "You're so stupid! You can't even figure out such a simple question?" and the emotional information of "sadness," it is judged that there is a high probability of bullying.

[0622] Step 9: Sending notifications

[0623] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method. Specifically, it sends a push notification to the parent's smartphone. The input is the bullying determination result, and the output is a notification message. For example, a message may be sent saying, "Your child may be being bullied. Also, your child appears to be feeling very sad."

[0624] (Application example 2)

[0625] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0626] Conventional bullying detection systems convert only audio data into text data and detect bullying behaviors based on analyzing only the text data. This makes it difficult to consider emotional nuances and make highly accurate judgments. Furthermore, because it is not possible to grasp a child's emotional state when bullying occurs, appropriate responses can be delayed. Therefore, there is a need for a system that can improve the accuracy of bullying detection, quickly grasp a child's emotional state, and notify parents.

[0627] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice conversion means that converts voice data into text data using voice recognition technology, an emotion recognition means that analyzes emotions from the voice data and generates emotion information, a bullying detection means that analyzes the text data and the emotion information and detects words that may be bullying, and a notification means that sends a notification to the parents when the bullying detection means determines that there is a high possibility of bullying. This improves the accuracy of detecting bullying behavior and makes it possible to quickly grasp a child's emotional state.

[0628] The "audio collection means" is a function provided in the communication terminal used by the child to collect surrounding sounds.

[0629] The "voice transmission means" is a function that transmits collected voice data to a server.

[0630] The "voice conversion means" is a function that converts voice data into text data using voice recognition technology.

[0631] The "emotion recognition means" is a function that analyzes emotions from voice data and generates emotional information.

[0632] The "bullying detection means" is a function that analyzes text data and emotional information to detect potentially bullying language.

[0633] "Notification means" is a function that sends a notification to parents if it is determined that there is a high possibility of bullying.

[0634] A "machine learning algorithm" is a technology that builds a model based on empirical data and makes predictions and classifications for unknown data.

[0635] "Natural language processing technology" is a technology that uses computers to understand, interpret, and generate natural language.

[0636] A "server" is a computer that receives data from a terminal and analyzes and processes it.

[0637] System configuration and operation

[0638] The system collects ambient sounds in real time using a sound collection means provided in the communication terminal used by the child. The sound collection means includes a built-in microphone. The terminal further includes a sound transmission means for transmitting the collected sound data to a server. The sound data is compressed and converted into a packet format for efficient data transmission to the server. A secure data transmission protocol is used for this purpose.

[0639] Server Processing

[0640] The server has the function of receiving voice data sent from the device. The received voice data is first converted into text data by a voice conversion means. This process uses Google Cloud Speech-to-Text API as the voice recognition technology. The server also has an emotion recognition means to analyze emotions from the voice data. To do this, the tone, pitch, intensity, etc. of the voice are evaluated, and emotion categories such as joy, anger, and sadness are generated.

[0641] The generated text data and emotional information are analyzed by the bullying detection means, which uses natural language processing technology and machine learning algorithms to make highly accurate judgments by taking into account potentially bullying-related phrases and emotional information contained in the text.

[0642] Notification method operation

[0643] If the server determines that bullying is likely, it saves the result and triggers a notification method. This notification includes details based on the analyzed text data and emotional information. Specifically, if the text "No one cares if you're not around" is converted from speech and the user's emotion is determined to be "sadness," the server will use this result to send a notification to the parent's smartphone stating, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0644] Examples and prompts

[0645] Specific examples

[0646] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0647] 1. The device collects audio and analyzes emotions using emotion recognition means.

[0648] 2. The device sends the collected voice data and emotional information to the server.

[0649] 3. The server converts the voice data into text data and stores the emotional information.

[0650] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0651] 5. The server sends a notification to the parent's smartphone saying, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0652] Prompt Sentence Examples

[0653] "Analyzing children's classroom conversations to detect bullying behavior and identify emotional states."

[0654] "Oh, you don't know that?" is analyzed and emotions are identified from the voice characteristics.

[0655] "The text 'No one cares if you're gone' is converted to speech and the emotion is determined to be sadness."

[0656] As described above, this system not only uses voice data and emotional information to detect bullying with high accuracy, but also helps protect children's safety and psychological health by promptly reporting the incident to parents.

[0657] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0658] Step 1:

[0659] The device uses a built-in microphone to collect ambient sound, which is then temporarily stored as raw audio data. At this stage, the audio data is not yet in a directly usable format.

[0660] Input: Surrounding environmental sound

[0661] Output: Raw audio data

[0662] Step 2:

[0663] The device compresses the collected raw voice data and converts it into a packet format for efficient data transmission using voice data format conversion and compression algorithms, and then transmits the compressed data to the server via a secure data transmission protocol.

[0664] Input: Raw audio data

[0665] Output: Compressed and packetized audio data

[0666] Step 3:

[0667] The server receives the compressed and packetized voice data sent from the terminal, decompresses and decodes the received data, and reconstructs it into raw voice data.

[0668] Input: Compressed and packetized voice data

[0669] Output: Decompressed and decoded raw audio data

[0670] Step 4:

[0671] The server converts the raw audio data into text using the Google Cloud Speech-to-Text API, which feeds the audio data into the API and converts it into text in real time.

[0672] Input: Raw audio data

[0673] Output: Text data

[0674] Step 5:

[0675] The server uses emotion recognition to analyze the emotions in the voice data. It analyzes features such as tone, pitch, and intensity of the voice and generates emotion categories (e.g., joy, anger, sadness, etc.). Again, technologies such as Google Cloud Natural Language API may be used.

[0676] Input: Raw audio data

[0677] Output: Sentiment category and sentiment score

[0678] Step 6:

[0679] The server analyzes the text data and emotion categories using a bullying detection method. It uses natural language processing techniques and machine learning algorithms to detect potentially bullying statements in the text. The emotion information is used to assist in the judgment.

[0680] Input: Text data and emotion categories

[0681] Output: Judgment result of possibility of bullying

[0682] Step 7:

[0683] If the server determines that there is a high possibility of bullying, it stores the result of the determination and emotional information, and triggers a notification method. A push notification is sent to the parent or guardian's smartphone informing them of the possibility that their child is being bullied and details of their emotional state. This notification includes details based on text data and emotional information.

[0684] Input: Judgment result of possibility of bullying and emotion score

[0685] Output: Push notification to parent

[0686] example:

[0687] Suppose a child is talking with a friend in the classroom and says, "No one would care if you weren't here." In this case, the server recognizes the emotion "sadness" from the voice data and determines that this is bullying. As a result, the server sends a notification to the parent's smartphone saying, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0688] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0689] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0690] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0691] [Third embodiment]

[0692] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0693] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0694] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0695] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0696] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0697] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0698] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0699] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0700] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0701] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0702] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0703] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0704] The present invention describes a system for collecting voice data and detecting bullying behavior in communication terminals used by children. The implementation of this system is carried out through the following steps.

[0705] Communication terminal operation

[0706] The device is equipped with an audio collection means for collecting the child's audio data. Specifically, it uses a microphone to collect surrounding sounds. This audio data is transmitted to a server in real time.

[0707] Once the device collects the voice data, it compresses it and sends it to the server as packets for efficient data transmission. Appropriate protocols must be used to ensure data security and reliability.

[0708] Server Operation

[0709] The server receives the voice data sent from the terminal. The received voice data is converted into text data by a voice conversion means. For example, a voice recognition technology such as Google Cloud Speech-to-Text API is used for this purpose.

[0710] The server then analyzes the converted text data using a bullying detection method that utilizes natural language processing technology and machine learning algorithms to analyze the text data and determine whether its content indicates bullying behavior.

[0711] Notification method operation

[0712] If the bullying detection means analyzes the text data and determines that there is a high possibility of bullying, the server will use the notification means to send a warning to parents via push notifications on smartphones or email.

[0713] For example, if the text "No one cares if you're not around" is converted from speech, the bullying detection method will determine this as bullying, and a push notification will be sent to the parent's smartphone stating, "Your child may be being bullied."

[0714] Specific examples

[0715] Example 1: Classroom conversation

[0716] scenario:

[0717] Imagine a child is having a conversation with a friend in class and the friend says to them, "You're so stupid! You can't even figure out such a simple problem?"

[0718] 1. The device collects audio.

[0719] 2. The device sends the audio data to the server.

[0720] 3. The server converts the audio data into text data.

[0721] 4. The server analyzes the text data and detects possible bullying.

[0722] 5. The server sends a push notification to the parent's smartphone.

[0723] In this way, the system operates in real time, instantly notifying parents when their child is being bullied, allowing for prompt action.

[0724] The processing flow will be explained below.

[0725] Step 1:

[0726] The device collects ambient sounds in real time through a microphone, for example capturing the sound of a child talking with a friend in the classroom.

[0727] Step 2:

[0728] The terminal compresses the collected voice data to improve the efficiency of data transmission, and it is important to use an appropriate compression algorithm.

[0729] Step 3:

[0730] The terminal converts the compressed voice data into a packet format and sends it to the server using a secure data transmission protocol, which ensures the authenticity and security of the data.

[0731] Step 4:

[0732] The server decompresses the audio data received from the device and converts it into text using speech recognition technology. Commercially available speech recognition software, such as the Google Cloud Speech-to-Text API, can be used.

[0733] Step 5:

[0734] The server analyzes the converted text data with a bullying detection tool that uses natural language processing techniques and machine learning algorithms to detect keywords and patterns in the text that indicate bullying behavior.

[0735] Step 6:

[0736] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method, along with details of the bullying act, such as the time and location.

[0737] Step 7:

[0738] The server sends real-time push notifications to parents' smartphones or tablet devices, informing them of potential bullying incidents and providing additional information.

[0739] Step 8:

[0740] The user (parent) can check the details and take necessary action according to the push notification received. Parents can also view more detailed information and history through the application.

[0741] Step 9:

[0742] If the parent takes any action after receiving the notification, the server will check the response status again and send a follow-up notification if necessary. This series of processes enables early detection and prompt response when a child is being bullied.

[0743] Example 1

[0744] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0745] The present invention relates to a system for detecting bullying behavior by collecting voice data using communication devices used by children. Conventional technologies have inefficient processes for collecting voice data, detecting bullying behavior, and notifying parents, making it difficult to respond in real time. Furthermore, the accuracy of determining bullying behavior is low, resulting in frequent false positives. There is a need to solve these issues and more quickly and accurately detect bullying behavior and notify parents.

[0746] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0747] In this invention, the server includes an audio collection means, an audio transmission means, an audio conversion means, a bullying detection means, a notification generation means, a notification content generation means, and a notification transmission means. This allows for real-time collection of audio around a child and secure transmission of compressed audio data to the server. The server also quickly converts the audio data into text data and uses machine learning algorithms and natural language processing technology to analyze bullying behavior with high accuracy. Furthermore, it automatically generates appropriate notification content using a generative AI model, enabling prompt and appropriate warnings to be sent to parents.

[0748] The "voice collecting means" is a voice input device such as a microphone provided in the communication terminal used by the child, and is a device that has the function of collecting surrounding voices.

[0749] The "audio transmission means" is a communication means having the function of compressing collected audio data and transmitting it safely to the server.

[0750] The "voice conversion means" is a means having a function of converting voice data collected in the server into text data using voice recognition technology.

[0751] The "bullying detection means" is a means that has the function of analyzing the converted text data and detecting words that may be bullying.

[0752] The "notification means" is a means that has the function of sending a notification to parents when it is determined that there is a high possibility of bullying.

[0753] The "notification generating means" is a means having a function of generating the content of a notification to be sent to parents when bullying behavior is detected.

[0754] "Notification content generation means" means a means having the function of generating appropriate notification content based on predefined templates and analysis results using a generative AI model.

[0755] The "notification sending means" is a means having a function of sending the generated notification message to the parent communication terminal.

[0756] A "machine learning algorithm" is a technology that allows computers to learn from data and make predictions and classifications.

[0757] "Natural language processing technology" is a computational technology for understanding, generating, and analyzing human language.

[0758] A "generative AI model" is a pre-trained artificial intelligence model that has the ability to generate appropriate outputs based on specific inputs.

[0759] The present invention relates to a system for detecting bullying by collecting voice data using a communication terminal used by a child. The system includes a communication terminal, a server, a bullying detection unit, and a notification unit.

[0760] Configuration and operation of communication terminal

[0761] The device is equipped with an audio capture unit to capture sounds around the child. Specifically, a microphone captures the sound, and a dedicated application manages the sound data in real time. The collected sound data is compressed into MP3 format using a LAME MP3 encoder or similar and securely transmitted to a server using the HTTPS protocol.

[0762] Server configuration and operation

[0763] The server receives the voice data sent from the device and temporarily stores it in storage. The received voice data is converted into text data using speech recognition technology such as the Google Cloud Speech-to-Text API. The server then analyzes the text data using scikit-learn and TensorFlow to detect bullying behavior. This analysis process utilizes machine learning algorithms and natural language processing technology.

[0764] Bullying detection and notification

[0765] The bullying detection means analyzes the text data to determine whether certain keywords or phrases correspond to bullying behavior. If bullying behavior is detected, the notification generation means uses a generative AI model to generate notification content to send to parents, which may utilize pre-defined templates and the generative AI model.

[0766] Example prompt sentence:

[0767] Audio analysis has detected the following potential bullying behavior:

[0768] "No one would care if you weren't here."

[0769] Parents should be notified. Please generate the notification content.

[0770] Parental Notification

[0771] The generated notification message is sent to the parent communication device using a notification sending means, such as a push notification using Firebase Cloud Messaging (FCM) or an email using the SMTP protocol.

[0772] Specific examples

[0773] For example, if a child in a classroom is told, "You're so stupid, you can't even figure out such a simple problem?", the system will act as follows:

[0774] 1. The device collects the audio, compresses it into MP3 format, and sends it to the server.

[0775] 2. The server receives the voice data and converts it into text data using voice recognition technology.

[0776] 3. The server analyzes the text data using machine learning algorithms and natural language processing technology, and detects statements such as, "You're so stupid, you can't even figure out such a simple question?" as bullying.

[0777] 4. The server uses the generative AI model to generate appropriate notification content, creating a message such as "Your child may be being bullied."

[0778] 5. The server sends the generated notification message to the parent communication device as a push notification.

[0779] This system allows parents to be notified immediately if their child is being bullied, allowing them to take immediate action.

[0780] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0781] Step 1:

[0782] The device collects audio data from the child's surroundings. Specifically, it records audio in real time using a built-in microphone. The input is the surrounding audio, and the output is the recorded audio data. A dedicated application manages this audio data and temporarily stores it on the device.

[0783] Step 2:

[0784] The device compresses the collected audio data. Specifically, it converts the audio data into MP3 format using software such as the LAME MP3 encoder. The input is the recorded audio data, and the output is a compressed MP3 file. This process reduces the data size and enables efficient data transfer.

[0785] Step 3:

[0786] The device sends the compressed audio data to the server. Specifically, the data is uploaded to the server using the HTTPS protocol to ensure data security. The input is a compressed MP3 file, and the output is the audio data sent to the server.

[0787] Step 4:

[0788] The server receives the audio data sent from the device. The received data is temporarily stored in the server's storage. The input is the sent MP3 file, and the output is the audio data stored in the server's storage.

[0789] Step 5:

[0790] The server uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the audio data into text. Specifically, the audio data is sent to the API, and the returned text data is stored in a database on the server. The input is an MP3 file, and the output is the converted text data.

[0791] Step 6:

[0792] The server analyzes the text data to detect possible bullying. It uses machine learning algorithms such as scikit-learn and TensorFlow and natural language processing techniques to analyze keywords and emotions that indicate bullying behavior contained in the text data. The input is the text data, and the output is a flag indicating bullying behavior and the analysis results.

[0793] Step 7:

[0794] When the server detects bullying behavior, it uses a generative AI model to generate a notification to be sent to parents. Specifically, the generative AI model creates an appropriate message based on a notification template. The input is a flag indicating bullying behavior and analysis results, and the output is the generated notification message.

[0795] Step 8:

[0796] The server sends the generated notification message to the parent communication device. Specifically, it sends push notifications using Firebase Cloud Messaging (FCM) or emails using the SMTP protocol. The input is the generated notification message, and the output is the notification sent to the parent communication device.

[0797] This series of processing steps enables the system to immediately notify parents in real time when their child is being bullied.

[0798] (Application example 1)

[0799] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0800] In modern society, the risk of children being bullied is increasing. Particularly in group settings such as schools, bullying often occurs without the adults around them noticing. It is difficult for parents to grasp the situation of their child being bullied, making it difficult to take appropriate measures quickly. To solve this problem, a system is needed that can detect bullying at an early stage and immediately notify parents.

[0801] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0802] In this invention, the server includes a voice collection means provided in the communication terminal used by the child, a voice transmission means for transmitting voice data collected by the voice collection means to the server, a voice conversion means by the server for converting the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, a notification means for sending a notification to the parent if the bullying detection means determines that there is a high possibility of bullying, and a notification means for sending a push notification to the parent's smartphone. This enables parents to understand in real time when their child is being bullied and to respond quickly.

[0803] "Child" refers to a minor who is still in the developmental stage.

[0804] A "communication terminal" refers to an electronic device that transmits and receives data via a network.

[0805] "Sound collection means" refers to a device or function for collecting surrounding sounds in a communication terminal.

[0806] "Audio transmission means" refers to a device or function for transmitting collected audio data to a server.

[0807] A "server" refers to a computer system that provides data processing and storage via a network.

[0808] "Speech recognition technology" refers to technology for converting voice data into text data.

[0809] "Speech conversion means" refers to a process or function for converting voice data into text data.

[0810] "Text data" refers to data in sentence format converted using voice recognition technology.

[0811] "Bullying detection means" refers to a device or function for analyzing text data and detecting bullying behavior.

[0812] "Notification means" refers to a device or function for notifying the user of specific information.

[0813] A "smartphone" refers to a multi-function mobile device that can access the Internet and run applications in addition to the functions of a mobile phone.

[0814] "Push notification" refers to a means of sending information from a server to a device such as a smartphone in real time.

[0815] The present invention describes a system for detecting bullying by collecting voice data from communication devices used by children. This system detects bullying occurring around children in real time and immediately notifies parents, enabling them to take early action. Specific embodiments for implementing the present invention are described in detail below.

[0816] Communication terminal operation

[0817] The device is equipped with an audio collection means for collecting sounds around the child. A microphone is used to collect audio data, which is temporarily compressed in the device and sent to a server as packets for efficient data transmission. An appropriate protocol is used to ensure data security and reliability.

[0818] Server Operation

[0819] The server receives the voice data sent from the device. The received voice data is converted into text data using a voice conversion means. For example, Google Cloud Speech-to-Text API is used as a voice recognition technology. The server then analyzes the converted text data using a bullying detection means. This analysis utilizes natural language processing technology and machine learning algorithms.

[0820] Detecting and notifying bullying behavior

[0821] If the bullying detection means analyzes the text data and determines that there is a high possibility of bullying, the server uses the notification means to send a notification to the parent. This notification is sent as a push notification to the parent's smartphone. For example, if the text "No one cares if you're not here" is detected, the bullying detection means determines this to be bullying and sends a warning message to the parent's smartphone.

[0822] Hardware and software used

[0823] Hardware:

[0824] Communication device (smartphone)

[0825] Microphone (built-in or external)

[0826] software:

[0827] A speech recognition library (such as Google Cloud Speech-to-Text API)

[0828] Audio processing library (such as pydub)

[0829] HTTP communication library (requests, etc.)

[0830] Natural language processing libraries (NLTK, spaCy, etc.)

[0831] Email sending library (smtplib, etc.)

[0832] Specific examples

[0833] Example 1: Classroom conversation

[0834] Suppose a child is having a conversation with a friend in class and the friend says, "You're so stupid! You can't even figure out such a simple problem?" The device collects the audio and sends this audio data to a server. The server converts the audio data into text data and analyzes it. If this analysis determines that there is a possibility of bullying, a push notification is sent to the parent's smartphone saying, "Your child may be being bullied."

[0835] Prompt Sentence Examples

[0836] "If your child says, 'You're not my friend!' this is likely bullying. Please check immediately."

[0837] This system allows parents to understand in real time when their children are being bullied and to take quick action.

[0838] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0839] Step 1:

[0840] The device collects audio. It uses the smartphone's built-in microphone to record the audio around the child in real time. The input is the surrounding audio, which is then captured as digital audio data.

[0841] Step 2:

[0842] The terminal compresses the collected voice data and sends it to the server. The voice data is converted into packets in a compressed format and sent to the server using the HTTP protocol. The input is the collected voice data and the output is the compressed voice data sent to the server.

[0843] Step 3:

[0844] The server converts the received voice data into text data using speech recognition technology. The Google Cloud Speech-to-Text API is used to output the voice data as text data. The input is compressed voice data, and the output is text data.

[0845] Step 4:

[0846] The server analyzes the converted text data to detect potentially bullying language, using natural language processing technology and machine learning algorithms to determine whether specific keywords or phrases are included. The input is text data, and the analysis results are the output.

[0847] Step 5:

[0848] If the server determines that there is a high possibility of bullying, it will use a notification method to send a notification to the parent. Specifically, a push notification is sent to the parent's smartphone. The input is the analysis result, and the output is a warning message sent to the parent's smartphone as a push notification.

[0849] Step 6:

[0850] Parents receive push notifications on their smartphones and check the contents. They can check their child's status based on the notification content and take necessary measures. The input is the notification message displayed on the smartphone, and the output is the parent's actions.

[0851] This process allows parents to be notified in real time when their child is being bullied, allowing them to take immediate action.

[0852] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0853] The present invention combines a system that collects voice data and detects bullying behavior in a communication device used by a child with an emotion engine that recognizes the user's emotions. An embodiment of this system will be described in detail below.

[0854] Communication terminal operation

[0855] The terminal is equipped with a voice collection means for collecting voice data. This means includes a function for collecting surrounding voices in real time using a microphone. For example, the voices of children talking in a classroom can be captured. In this case, the voice collection means includes an emotion recognition function to analyze the user's emotions.

[0856] The collected audio data is compressed and converted into a packet format for efficient data transmission to the server, using a secure data transmission protocol.

[0857] Server Operation

[0858] The server receives the voice data sent from the device. The received voice data is converted into text data using a voice conversion means. This process uses voice recognition technology such as the Google Cloud Speech-to-Text API.

[0859] The server further includes an emotion recognition unit that recognizes the user's emotion from the voice data. This allows the emotion information to be analyzed along with the text data. The emotion recognition unit detects emotions from the tone, pitch, intensity, etc. of the voice and generates emotion categories such as joy, anger, sadness, etc.

[0860] The server then analyzes the text data and emotional information using a bullying detection method that utilizes natural language processing technology and machine learning algorithms. Taking emotional information into account when detecting potentially bullying statements enables more accurate detection.

[0861] Notification method operation

[0862] If the server determines that bullying is likely, it stores the results and triggers a notification mechanism, which includes details based on text data and emotional information.

[0863] Specifically, for example, if the text "No one would care if you weren't here" is converted from speech and the user's emotion is determined to be "sadness," the server will use this result to send a push notification to the parent's smartphone stating, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0864] Specific examples

[0865] Example 1: Classroom conversations and sentiment analysis

[0866] scenario:

[0867] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0868] 1. The device collects audio and analyzes emotions using emotion recognition means.

[0869] 2. The device sends the collected voice data and emotional information to the server.

[0870] 3. The server converts the voice data into text data and stores the emotional information.

[0871] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0872] 5. The server sends a push notification to the parent's smartphone stating, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0873] In this way, the system operates in real time, instantly notifying parents when their child is being bullied, enabling them to take prompt action. In addition, by incorporating emotional information, a deeper understanding and response can be achieved.

[0874] The processing flow will be explained below.

[0875] Step 1:

[0876] The device collects surrounding sounds in real time through a microphone, using a sound collection means to capture, for example, conversations in a classroom.

[0877] Step 2:

[0878] The device compresses the collected voice data and prepares it for efficient transmission to the server, using an appropriate compression algorithm to improve data transmission efficiency.

[0879] Step 3:

[0880] The terminal converts the compressed voice data into a packet format and sends it to the server using a secure data transmission protocol (e.g., HTTPS).

[0881] Step 4:

[0882] The server decompresses the audio data received from the device and converts it into text data using speech recognition technology, for example, the Google Cloud Speech-to-Text API.

[0883] Step 5:

[0884] The server analyzes the user's emotions using the voice data and emotion recognition means, inferring emotions from the tone, pitch, and intensity of the voice and generating emotion categories (e.g., joy, anger, sadness, etc.).

[0885] Step 6:

[0886] The server analyzes the text data and emotional information with a bullying detection method that uses natural language processing techniques and machine learning algorithms to detect keywords and patterns in the text that indicate bullying behavior, while also taking emotional information into account.

[0887] Step 7:

[0888] If the server determines that there is a high possibility of bullying based on the analysis results, it saves the results and triggers a notification method, notifying parents of the details of the detection and detailed data including emotional information.

[0889] Step 8:

[0890] The server sends a push notification to the parent's smartphone or tablet device, containing detailed information such as, "Your child may be experiencing bullying. Your child also appears to be experiencing strong sadness."

[0891] Step 9:

[0892] The user (parent) can check the received push notification and view detailed information through the app, which allows them to understand their child's situation and take necessary action.

[0893] Step 10:

[0894] The server monitors the parent's response and sends additional follow-up notifications as needed, enabling early detection and prompt response when a child is being bullied.

[0895] Example 2

[0896] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0897] In recent years, bullying among children has become a social problem, and early detection and countermeasures are needed. However, current monitoring and reporting systems have difficulty detecting signs of bullying in real time or making highly accurate judgments that take the user's emotions into account. In this environment, a system is needed that can not only detect bullying early, but also confirm changes in the victim's emotions.

[0898] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice collection means provided in the communication terminal used by the child, a voice transmission means for transmitting collected voice data to the server, a voice conversion means for converting the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, a notification means for sending a notification to the parent when it is determined that there is a high possibility of bullying, an emotion recognition means for recognizing the user's emotion from the voice data, and a means for analyzing the emotion information in combination with the text data. This makes it possible to detect signs of bullying in real time, and to make a highly accurate determination and immediately notify the victim while also taking into account changes in the victim's emotions.

[0899] Children are young people who are still growing, generally those under the age of 18.

[0900] A communication terminal refers to an electronic device for sending and receiving digital information, and includes, for example, a smartphone or tablet.

[0901] An audio collection means is a technical device or function that picks up surrounding sounds and voices, typically a microphone.

[0902] The audio transmission means refers to a technical mechanism for transmitting collected audio data to another device or server, and transmits data via the Internet or wireless communication.

[0903] A server is a computer system that provides data and resources over a network, providing file storage and processing power.

[0904] Audio data refers to a digital representation of sound collected by an audio input device such as a microphone, including the conversion of sound waves into electrical signals.

[0905] Speech recognition technology is a technology for analyzing voice data and converting it into text data, using specific algorithms and machine learning models.

[0906] Text data refers to a string of characters or a collection of letters, and refers to data that is stored and processed in a digital format.

[0907] Bullying detection methods are technologies that analyze text data to find signs of bullying, and include natural language processing technology and machine learning algorithms.

[0908] A notification means refers to a mechanism for conveying information about a specific event to a user or other system when that event occurs, and examples include push notifications and emails.

[0909] Emotion recognition means refers to a technical means for analyzing voice data to identify a user's emotions, and is based on characteristics such as tone, pitch, and intensity of the voice.

[0910] Emotion information refers to data indicating the user's emotional state obtained by emotion recognition means, and includes categories such as joy, anger, and sadness.

[0911] Natural language processing technology is a technology that allows computers to understand, interpret, and generate human language, and includes text analysis and speech recognition.

[0912] A machine learning algorithm is an algorithm that builds a model based on data and detects specific patterns and trends, and is used for data analysis on a server.

[0913] Push notifications refer to notification messages that are actively sent from a server to a client device, and are a way to get a user's real-time attention.

[0914] The present invention combines a system that collects voice data and detects bullying behavior in a communication device used by a child with an emotion engine that recognizes the user's emotions. An embodiment of this system will be described in detail below.

[0915] Communication terminal operation

[0916] The terminal is equipped with an audio collection means for collecting audio data. This means includes a function for collecting ambient audio in real time using a microphone. For example, audio is captured when children are talking in a classroom. In this case, an emotion recognition function is included in the audio collection means to analyze the user's emotions. The emotion recognition function analyzes the tone, pitch, and intensity of the audio to generate emotion categories such as joy, anger, and sadness.

[0917] The collected audio data is compressed and converted into a packet format for efficient data transmission to the server, using a secure data transmission protocol (e.g., TLS).

[0918] Server Operation

[0919] The server receives the voice data sent from the device. It then uses speech recognition technology to convert the received voice data into text data. This process uses services such as the Google Cloud Speech-to-Text API. The converted text data is saved in the form of a message such as, "You're so stupid, you can't even answer such a simple question?"

[0920] The server also runs an emotion recognition algorithm to identify the user's emotions from the audio data. Emotions are detected as emotion categories such as joy, anger, sadness, etc. based on the tone, pitch, and intensity of the audio. For example, the emotion of sadness is detected for the text, "You're so stupid, you can't even figure out such a simple question?"

[0921] The server then analyzes this text data and emotional information to detect potentially bullying phrases. The analysis uses natural language processing technology and machine learning algorithms, making it possible to detect phrases that are likely to be bullying, as well as complex patterns that take emotional information into account.

[0922] How the notification system works

[0923] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method. For example, it sends a push notification to the parent's smartphone. The message sent is, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0924] Specific examples

[0925] Example 1: Classroom conversations and sentiment analysis

[0926] scenario:

[0927] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0928] Operation steps:

[0929] 1. The device collects voice and analyzes emotions using emotion recognition.

[0930] 2. The device sends the collected voice data and emotional information to the server.

[0931] 3. The server converts the voice data into text data and stores the emotional information.

[0932] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0933] 5. The server sends a push notification to the parent's smartphone stating, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0934] Example prompts for generative AI models

[0935] "Please tell us how to detect possible bullying from speech data and emotional information when a child is told by a friend in the classroom, 'You're so stupid! You can't understand such a simple problem?' and notify parents."

[0936] In this way, the system operates in real time, notifying parents immediately when their child is being bullied, enabling them to take immediate action. By incorporating emotional information, the system can also provide a deeper understanding and response.

[0937] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0938] Step 1: Collecting audio data

[0939] The device uses a microphone to collect ambient audio in real time while children are conversing in the classroom. For example, it captures the audio when a child says something like, "You're so stupid! You can't even figure out such a simple question!" This collected audio data is used as input. This audio data also includes information such as the tone, pitch, and intensity of the voice.

[0940] Step 2: Recognize emotions

[0941] The device analyzes the user's emotions from the collected voice data. Using emotion recognition, it analyzes the tone, pitch, and intensity of the voice and generates emotion categories such as joy, anger, and sadness. For example, the emotion "sadness" is detected from the collected voice. This emotional information is generated as output.

[0942] Step 3: Compress and convert audio data

[0943] The terminal compresses the collected voice data and emotion information and converts them into packet format using a secure data transmission protocol (e.g., TLS). The compressed voice data and emotion information are input and packet format data is output.

[0944] Step 4: Sending audio data

[0945] The device transmits the voice data and emotion information converted into packets to the server via the Internet. The input is the packetized data, and the output is the data transmitted to the server.

[0946] Step 5: Receiving audio data

[0947] The server receives packetized voice data and emotion information sent from the terminal. The received data is input, and the voice data and emotion information are output.

[0948] Step 6: Convert audio data to text

[0949] The server uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is voice data, and the output is text data. For example, the generated text might say, "You're so stupid! You can't even answer such a simple question?"

[0950] Step 7: Combining Emotional Information

[0951] The server combines the emotional information detected using an emotion recognition algorithm with the text data. The input is text data and emotional information, and the output is a combination of text and emotional categories. For example, the text "You're so stupid! You can't even figure out such a simple question?" is combined with the emotion "sadness."

[0952] Step 8: Detect bullying

[0953] The server uses natural language processing technology and machine learning algorithms to analyze text data and emotional information to detect potentially bullying statements. The input is a combination of text and emotional category data, and the output is a judgment result indicating a high probability of bullying. For example, based on the text "You're so stupid! You can't even figure out such a simple question?" and the emotional information of "sadness," it is judged that there is a high probability of bullying.

[0954] Step 9: Sending notifications

[0955] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method. Specifically, it sends a push notification to the parent's smartphone. The input is the bullying determination result, and the output is a notification message. For example, a message may be sent saying, "Your child may be being bullied. Also, your child appears to be feeling very sad."

[0956] (Application example 2)

[0957] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0958] Conventional bullying detection systems convert only audio data into text data and detect bullying behaviors based on analyzing only the text data. This makes it difficult to consider emotional nuances and make highly accurate judgments. Furthermore, because it is not possible to grasp a child's emotional state when bullying occurs, appropriate responses can be delayed. Therefore, there is a need for a system that can improve the accuracy of bullying detection, quickly grasp a child's emotional state, and notify parents.

[0959] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice conversion means that converts voice data into text data using voice recognition technology, an emotion recognition means that analyzes emotions from the voice data and generates emotion information, a bullying detection means that analyzes the text data and the emotion information and detects words that may be bullying, and a notification means that sends a notification to the parents when the bullying detection means determines that there is a high possibility of bullying. This improves the accuracy of detecting bullying behavior and makes it possible to quickly grasp a child's emotional state.

[0960] The "audio collection means" is a function provided in the communication terminal used by the child to collect surrounding sounds.

[0961] The "voice transmission means" is a function that transmits collected voice data to a server.

[0962] The "voice conversion means" is a function that converts voice data into text data using voice recognition technology.

[0963] The "emotion recognition means" is a function that analyzes emotions from voice data and generates emotional information.

[0964] The "bullying detection means" is a function that analyzes text data and emotional information to detect potentially bullying language.

[0965] "Notification means" is a function that sends a notification to parents if it is determined that there is a high possibility of bullying.

[0966] A "machine learning algorithm" is a technology that builds a model based on empirical data and makes predictions and classifications for unknown data.

[0967] "Natural language processing technology" is a technology that uses computers to understand, interpret, and generate natural language.

[0968] A "server" is a computer that receives data from a terminal and analyzes and processes it.

[0969] System configuration and operation

[0970] The system collects ambient sounds in real time using a sound collection means provided in the communication terminal used by the child. The sound collection means includes a built-in microphone. The terminal further includes a sound transmission means for transmitting the collected sound data to a server. The sound data is compressed and converted into a packet format for efficient data transmission to the server. A secure data transmission protocol is used for this purpose.

[0971] Server Processing

[0972] The server has the function of receiving voice data sent from the device. The received voice data is first converted into text data by a voice conversion means. This process uses Google Cloud Speech-to-Text API as the voice recognition technology. The server also has an emotion recognition means to analyze emotions from the voice data. To do this, the tone, pitch, intensity, etc. of the voice are evaluated, and emotion categories such as joy, anger, and sadness are generated.

[0973] The generated text data and emotional information are analyzed by the bullying detection means, which uses natural language processing technology and machine learning algorithms to make highly accurate judgments by taking into account potentially bullying-related phrases and emotional information contained in the text.

[0974] Notification method operation

[0975] If the server determines that bullying is likely, it saves the result and triggers a notification method. This notification includes details based on the analyzed text data and emotional information. Specifically, if the text "No one cares if you're not around" is converted from speech and the user's emotion is determined to be "sadness," the server will use this result to send a notification to the parent's smartphone stating, "Your child may be being bullied. It also appears that your child is feeling very sad."

[0976] Examples and prompts

[0977] Specific examples

[0978] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[0979] 1. The device collects audio and analyzes emotions using emotion recognition means.

[0980] 2. The device sends the collected voice data and emotional information to the server.

[0981] 3. The server converts the voice data into text data and stores the emotional information.

[0982] 4. The server analyzes the text data and emotional information to detect possible bullying.

[0983] 5. The server sends a notification to the parent's smartphone saying, "Your child may be being bullied. Your child also appears to be feeling very sad."

[0984] Prompt Sentence Examples

[0985] "Analyzing children's classroom conversations to detect bullying behavior and identify emotional states."

[0986] "Oh, you don't know that?" is analyzed and emotions are identified from the voice characteristics.

[0987] "The text 'No one cares if you're gone' is converted to speech and the emotion is determined to be sadness."

[0988] As described above, this system not only uses voice data and emotional information to detect bullying with high accuracy, but also helps protect children's safety and psychological health by promptly reporting the incident to parents.

[0989] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0990] Step 1:

[0991] The device uses a built-in microphone to collect ambient sound, which is then temporarily stored as raw audio data. At this stage, the audio data is not yet in a directly usable format.

[0992] Input: Surrounding environmental sound

[0993] Output: Raw audio data

[0994] Step 2:

[0995] The device compresses the collected raw voice data and converts it into a packet format for efficient data transmission using voice data format conversion and compression algorithms, and then transmits the compressed data to the server via a secure data transmission protocol.

[0996] Input: Raw audio data

[0997] Output: Compressed and packetized audio data

[0998] Step 3:

[0999] The server receives the compressed and packetized voice data sent from the terminal, decompresses and decodes the received data, and reconstructs it into raw voice data.

[1000] Input: Compressed and packetized voice data

[1001] Output: Decompressed and decoded raw audio data

[1002] Step 4:

[1003] The server converts the raw audio data into text using the Google Cloud Speech-to-Text API, which feeds the audio data into the API and converts it into text in real time.

[1004] Input: Raw audio data

[1005] Output: Text data

[1006] Step 5:

[1007] The server uses emotion recognition to analyze the emotions in the voice data. It analyzes features such as tone, pitch, and intensity of the voice and generates emotion categories (e.g., joy, anger, sadness, etc.). Again, technologies such as Google Cloud Natural Language API may be used.

[1008] Input: Raw audio data

[1009] Output: Sentiment category and sentiment score

[1010] Step 6:

[1011] The server analyzes the text data and emotion categories using a bullying detection method. It uses natural language processing techniques and machine learning algorithms to detect potentially bullying statements in the text. The emotion information is used to assist in the judgment.

[1012] Input: Text data and emotion categories

[1013] Output: Judgment result of possibility of bullying

[1014] Step 7:

[1015] If the server determines that there is a high possibility of bullying, it stores the result of the determination and emotional information, and triggers a notification method. A push notification is sent to the parent or guardian's smartphone informing them of the possibility that their child is being bullied and details of their emotional state. This notification includes details based on text data and emotional information.

[1016] Input: Judgment result of possibility of bullying and emotion score

[1017] Output: Push notification to parent

[1018] example:

[1019] Suppose a child is talking with a friend in the classroom and says, "No one would care if you weren't here." In this case, the server recognizes the emotion "sadness" from the voice data and determines that this is bullying. As a result, the server sends a notification to the parent's smartphone saying, "Your child may be being bullied. It also appears that your child is feeling very sad."

[1020] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1021] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1022] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1023] [Fourth embodiment]

[1024] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1025] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1027] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1028] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1029] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1031] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1032] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1033] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1035] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1037] The present invention describes a system for collecting voice data and detecting bullying behavior in communication terminals used by children. The implementation of this system is carried out through the following steps.

[1038] Communication terminal operation

[1039] The device is equipped with an audio collection means for collecting the child's audio data. Specifically, it uses a microphone to collect surrounding sounds. This audio data is transmitted to a server in real time.

[1040] Once the device collects the voice data, it compresses it and sends it to the server as packets for efficient data transmission. Appropriate protocols must be used to ensure data security and reliability.

[1041] Server Operation

[1042] The server receives the voice data sent from the terminal. The received voice data is converted into text data by a voice conversion means. For example, a voice recognition technology such as Google Cloud Speech-to-Text API is used for this purpose.

[1043] The server then analyzes the converted text data using a bullying detection method that utilizes natural language processing technology and machine learning algorithms to analyze the text data and determine whether its content indicates bullying behavior.

[1044] Notification method operation

[1045] If the bullying detection means analyzes the text data and determines that there is a high possibility of bullying, the server will use the notification means to send a warning to parents via push notifications on smartphones or email.

[1046] For example, if the text "No one cares if you're not around" is converted from speech, the bullying detection method will determine this as bullying, and a push notification will be sent to the parent's smartphone stating, "Your child may be being bullied."

[1047] Specific examples

[1048] Example 1: Classroom conversation

[1049] scenario:

[1050] Imagine a child is having a conversation with a friend in class and the friend says to them, "You're so stupid! You can't even figure out such a simple problem?"

[1051] 1. The device collects audio.

[1052] 2. The device sends the audio data to the server.

[1053] 3. The server converts the audio data into text data.

[1054] 4. The server analyzes the text data and detects possible bullying.

[1055] 5. The server sends a push notification to the parent's smartphone.

[1056] In this way, the system operates in real time, instantly notifying parents when their child is being bullied, allowing for prompt action.

[1057] The processing flow will be explained below.

[1058] Step 1:

[1059] The device collects ambient sounds in real time through a microphone, for example capturing the sound of a child talking with a friend in the classroom.

[1060] Step 2:

[1061] The terminal compresses the collected voice data to improve the efficiency of data transmission, and it is important to use an appropriate compression algorithm.

[1062] Step 3:

[1063] The terminal converts the compressed voice data into a packet format and sends it to the server using a secure data transmission protocol, which ensures the authenticity and security of the data.

[1064] Step 4:

[1065] The server decompresses the audio data received from the device and converts it into text using speech recognition technology. Commercially available speech recognition software, such as the Google Cloud Speech-to-Text API, can be used.

[1066] Step 5:

[1067] The server analyzes the converted text data with a bullying detection tool that uses natural language processing techniques and machine learning algorithms to detect keywords and patterns in the text that indicate bullying behavior.

[1068] Step 6:

[1069] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method, along with details of the bullying act, such as the time and location.

[1070] Step 7:

[1071] The server sends real-time push notifications to parents' smartphones or tablet devices, informing them of potential bullying incidents and providing additional information.

[1072] Step 8:

[1073] The user (parent) can check the details and take necessary action according to the push notification received. Parents can also view more detailed information and history through the application.

[1074] Step 9:

[1075] If the parent takes any action after receiving the notification, the server will check the response status again and send a follow-up notification if necessary. This series of processes enables early detection and prompt response when a child is being bullied.

[1076] Example 1

[1077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1078] The present invention relates to a system for detecting bullying behavior by collecting voice data using communication devices used by children. Conventional technologies have inefficient processes for collecting voice data, detecting bullying behavior, and notifying parents, making it difficult to respond in real time. Furthermore, the accuracy of determining bullying behavior is low, resulting in frequent false positives. There is a need to solve these issues and more quickly and accurately detect bullying behavior and notify parents.

[1079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1080] In this invention, the server includes an audio collection means, an audio transmission means, an audio conversion means, a bullying detection means, a notification generation means, a notification content generation means, and a notification transmission means. This allows for real-time collection of audio around a child and secure transmission of compressed audio data to the server. The server also quickly converts the audio data into text data and uses machine learning algorithms and natural language processing technology to analyze bullying behavior with high accuracy. Furthermore, it automatically generates appropriate notification content using a generative AI model, enabling prompt and appropriate warnings to be sent to parents.

[1081] The "voice collecting means" is a voice input device such as a microphone provided in the communication terminal used by the child, and is a device that has the function of collecting surrounding voices.

[1082] The "audio transmission means" is a communication means having the function of compressing collected audio data and transmitting it safely to the server.

[1083] The "voice conversion means" is a means having a function of converting voice data collected in the server into text data using voice recognition technology.

[1084] The "bullying detection means" is a means that has the function of analyzing the converted text data and detecting words that may be bullying.

[1085] The "notification means" is a means that has the function of sending a notification to parents when it is determined that there is a high possibility of bullying.

[1086] The "notification generating means" is a means having a function of generating the content of a notification to be sent to parents when bullying behavior is detected.

[1087] "Notification content generation means" means a means having the function of generating appropriate notification content based on predefined templates and analysis results using a generative AI model.

[1088] The "notification sending means" is a means having a function of sending the generated notification message to the parent communication terminal.

[1089] A "machine learning algorithm" is a technology that allows computers to learn from data and make predictions and classifications.

[1090] "Natural language processing technology" is a computational technology for understanding, generating, and analyzing human language.

[1091] A "generative AI model" is a pre-trained artificial intelligence model that has the ability to generate appropriate outputs based on specific inputs.

[1092] The present invention relates to a system for detecting bullying by collecting voice data using a communication terminal used by a child. The system includes a communication terminal, a server, a bullying detection unit, and a notification unit.

[1093] Configuration and operation of communication terminal

[1094] The device is equipped with an audio capture unit to capture sounds around the child. Specifically, a microphone captures the sound, and a dedicated application manages the sound data in real time. The collected sound data is compressed into MP3 format using a LAME MP3 encoder or similar and securely transmitted to a server using the HTTPS protocol.

[1095] Server configuration and operation

[1096] The server receives the voice data sent from the device and temporarily stores it in storage. The received voice data is converted into text data using speech recognition technology such as the Google Cloud Speech-to-Text API. The server then analyzes the text data using scikit-learn and TensorFlow to detect bullying behavior. This analysis process utilizes machine learning algorithms and natural language processing technology.

[1097] Bullying detection and notification

[1098] The bullying detection means analyzes the text data to determine whether certain keywords or phrases correspond to bullying behavior. If bullying behavior is detected, the notification generation means uses a generative AI model to generate notification content to send to parents, which may utilize pre-defined templates and the generative AI model.

[1099] Example prompt sentence:

[1100] Audio analysis has detected the following potential bullying behavior:

[1101] "No one would care if you weren't here."

[1102] Parents should be notified. Please generate the notification content.

[1103] Parental Notification

[1104] The generated notification message is sent to the parent communication device using a notification sending means, such as a push notification using Firebase Cloud Messaging (FCM) or an email using the SMTP protocol.

[1105] Specific examples

[1106] For example, if a child in a classroom is told, "You're so stupid, you can't even figure out such a simple problem?", the system will act as follows:

[1107] 1. The device collects the audio, compresses it into MP3 format, and sends it to the server.

[1108] 2. The server receives the voice data and converts it into text data using voice recognition technology.

[1109] 3. The server analyzes the text data using machine learning algorithms and natural language processing technology, and detects statements such as, "You're so stupid, you can't even figure out such a simple question?" as bullying.

[1110] 4. The server uses the generative AI model to generate appropriate notification content, creating a message such as "Your child may be being bullied."

[1111] 5. The server sends the generated notification message to the parent communication device as a push notification.

[1112] This system allows parents to be notified immediately if their child is being bullied, allowing them to take immediate action.

[1113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1114] Step 1:

[1115] The device collects audio data from the child's surroundings. Specifically, it records audio in real time using a built-in microphone. The input is the surrounding audio, and the output is the recorded audio data. A dedicated application manages this audio data and temporarily stores it on the device.

[1116] Step 2:

[1117] The device compresses the collected audio data. Specifically, it converts the audio data into MP3 format using software such as the LAME MP3 encoder. The input is the recorded audio data, and the output is a compressed MP3 file. This process reduces the data size and enables efficient data transfer.

[1118] Step 3:

[1119] The device sends the compressed audio data to the server. Specifically, the data is uploaded to the server using the HTTPS protocol to ensure data security. The input is a compressed MP3 file, and the output is the audio data sent to the server.

[1120] Step 4:

[1121] The server receives the audio data sent from the device. The received data is temporarily stored in the server's storage. The input is the sent MP3 file, and the output is the audio data stored in the server's storage.

[1122] Step 5:

[1123] The server uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the audio data into text. Specifically, the audio data is sent to the API, and the returned text data is stored in a database on the server. The input is an MP3 file, and the output is the converted text data.

[1124] Step 6:

[1125] The server analyzes the text data to detect possible bullying. It uses machine learning algorithms such as scikit-learn and TensorFlow and natural language processing techniques to analyze keywords and emotions that indicate bullying behavior contained in the text data. The input is the text data, and the output is a flag indicating bullying behavior and the analysis results.

[1126] Step 7:

[1127] When the server detects bullying behavior, it uses a generative AI model to generate a notification to be sent to parents. Specifically, the generative AI model creates an appropriate message based on a notification template. The input is a flag indicating bullying behavior and analysis results, and the output is the generated notification message.

[1128] Step 8:

[1129] The server sends the generated notification message to the parent communication device. Specifically, it sends push notifications using Firebase Cloud Messaging (FCM) or emails using the SMTP protocol. The input is the generated notification message, and the output is the notification sent to the parent communication device.

[1130] This series of processing steps enables the system to immediately notify parents in real time when their child is being bullied.

[1131] (Application example 1)

[1132] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1133] In modern society, the risk of children being bullied is increasing. Particularly in group settings such as schools, bullying often occurs without the adults around them noticing. It is difficult for parents to grasp the situation of their child being bullied, making it difficult to take appropriate measures quickly. To solve this problem, a system is needed that can detect bullying at an early stage and immediately notify parents.

[1134] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1135] In this invention, the server includes a voice collection means provided in the communication terminal used by the child, a voice transmission means for transmitting voice data collected by the voice collection means to the server, a voice conversion means by the server for converting the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, a notification means for sending a notification to the parent if the bullying detection means determines that there is a high possibility of bullying, and a notification means for sending a push notification to the parent's smartphone. This enables parents to understand in real time when their child is being bullied and to respond quickly.

[1136] "Child" refers to a minor who is still in the developmental stage.

[1137] A "communication terminal" refers to an electronic device that transmits and receives data via a network.

[1138] "Sound collection means" refers to a device or function for collecting surrounding sounds in a communication terminal.

[1139] "Audio transmission means" refers to a device or function for transmitting collected audio data to a server.

[1140] A "server" refers to a computer system that provides data processing and storage via a network.

[1141] "Speech recognition technology" refers to technology for converting voice data into text data.

[1142] "Speech conversion means" refers to a process or function for converting voice data into text data.

[1143] "Text data" refers to data in sentence format converted using voice recognition technology.

[1144] "Bullying detection means" refers to a device or function for analyzing text data and detecting bullying behavior.

[1145] "Notification means" refers to a device or function for notifying the user of specific information.

[1146] A "smartphone" refers to a multi-function mobile device that can access the Internet and run applications in addition to the functions of a mobile phone.

[1147] "Push notification" refers to a means of sending information from a server to a device such as a smartphone in real time.

[1148] The present invention describes a system for detecting bullying by collecting voice data from communication devices used by children. This system detects bullying occurring around children in real time and immediately notifies parents, enabling them to take early action. Specific embodiments for implementing the present invention are described in detail below.

[1149] Communication terminal operation

[1150] The device is equipped with an audio collection means for collecting sounds around the child. A microphone is used to collect audio data, which is temporarily compressed in the device and sent to a server as packets for efficient data transmission. An appropriate protocol is used to ensure data security and reliability.

[1151] Server Operation

[1152] The server receives the voice data sent from the device. The received voice data is converted into text data using a voice conversion means. For example, Google Cloud Speech-to-Text API is used as a voice recognition technology. The server then analyzes the converted text data using a bullying detection means. This analysis utilizes natural language processing technology and machine learning algorithms.

[1153] Detecting and notifying bullying behavior

[1154] If the bullying detection means analyzes the text data and determines that there is a high possibility of bullying, the server uses the notification means to send a notification to the parent. This notification is sent as a push notification to the parent's smartphone. For example, if the text "No one cares if you're not here" is detected, the bullying detection means determines this to be bullying and sends a warning message to the parent's smartphone.

[1155] Hardware and software used

[1156] Hardware:

[1157] Communication device (smartphone)

[1158] Microphone (built-in or external)

[1159] software:

[1160] A speech recognition library (such as Google Cloud Speech-to-Text API)

[1161] Audio processing library (such as pydub)

[1162] HTTP communication library (requests, etc.)

[1163] Natural language processing libraries (NLTK, spaCy, etc.)

[1164] Email sending library (smtplib, etc.)

[1165] Specific examples

[1166] Example 1: Classroom conversation

[1167] Suppose a child is having a conversation with a friend in class and the friend says, "You're so stupid! You can't even figure out such a simple problem?" The device collects the audio and sends this audio data to a server. The server converts the audio data into text data and analyzes it. If this analysis determines that there is a possibility of bullying, a push notification is sent to the parent's smartphone saying, "Your child may be being bullied."

[1168] Prompt Sentence Examples

[1169] "If your child says, 'You're not my friend!' this is likely bullying. Please check immediately."

[1170] This system allows parents to understand in real time when their children are being bullied and to take quick action.

[1171] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1172] Step 1:

[1173] The device collects audio. It uses the smartphone's built-in microphone to record the audio around the child in real time. The input is the surrounding audio, which is then captured as digital audio data.

[1174] Step 2:

[1175] The terminal compresses the collected voice data and sends it to the server. The voice data is converted into packets in a compressed format and sent to the server using the HTTP protocol. The input is the collected voice data and the output is the compressed voice data sent to the server.

[1176] Step 3:

[1177] The server converts the received voice data into text data using speech recognition technology. The Google Cloud Speech-to-Text API is used to output the voice data as text data. The input is compressed voice data, and the output is text data.

[1178] Step 4:

[1179] The server analyzes the converted text data to detect potentially bullying language, using natural language processing technology and machine learning algorithms to determine whether specific keywords or phrases are included. The input is text data, and the analysis results are the output.

[1180] Step 5:

[1181] If the server determines that there is a high possibility of bullying, it will use a notification method to send a notification to the parent. Specifically, a push notification is sent to the parent's smartphone. The input is the analysis result, and the output is a warning message sent to the parent's smartphone as a push notification.

[1182] Step 6:

[1183] Parents receive push notifications on their smartphones and check the contents. They can check their child's status based on the notification content and take necessary measures. The input is the notification message displayed on the smartphone, and the output is the parent's actions.

[1184] This process allows parents to be notified in real time when their child is being bullied, allowing them to take immediate action.

[1185] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1186] The present invention combines a system that collects voice data and detects bullying behavior in a communication device used by a child with an emotion engine that recognizes the user's emotions. An embodiment of this system will be described in detail below.

[1187] Communication terminal operation

[1188] The terminal is equipped with a voice collection means for collecting voice data. This means includes a function for collecting surrounding voices in real time using a microphone. For example, the voices of children talking in a classroom can be captured. In this case, the voice collection means includes an emotion recognition function to analyze the user's emotions.

[1189] The collected audio data is compressed and converted into a packet format for efficient data transmission to the server, using a secure data transmission protocol.

[1190] Server Operation

[1191] The server receives the voice data sent from the device. The received voice data is converted into text data using a voice conversion means. This process uses voice recognition technology such as the Google Cloud Speech-to-Text API.

[1192] The server further includes an emotion recognition unit that recognizes the user's emotion from the voice data. This allows the emotion information to be analyzed along with the text data. The emotion recognition unit detects emotions from the tone, pitch, intensity, etc. of the voice and generates emotion categories such as joy, anger, sadness, etc.

[1193] The server then analyzes the text data and emotional information using a bullying detection method that utilizes natural language processing technology and machine learning algorithms. Taking emotional information into account when detecting potentially bullying statements enables more accurate detection.

[1194] Notification method operation

[1195] If the server determines that bullying is likely, it stores the results and triggers a notification mechanism, which includes details based on text data and emotional information.

[1196] Specifically, for example, if the text "No one would care if you weren't here" is converted from speech and the user's emotion is determined to be "sadness," the server will use this result to send a push notification to the parent's smartphone stating, "Your child may be being bullied. It also appears that your child is feeling very sad."

[1197] Specific examples

[1198] Example 1: Classroom conversations and sentiment analysis

[1199] scenario:

[1200] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[1201] 1. The device collects audio and analyzes emotions using emotion recognition means.

[1202] 2. The device sends the collected voice data and emotional information to the server.

[1203] 3. The server converts the voice data into text data and stores the emotional information.

[1204] 4. The server analyzes the text data and emotional information to detect possible bullying.

[1205] 5. The server sends a push notification to the parent's smartphone stating, "Your child may be being bullied. Your child also appears to be feeling very sad."

[1206] In this way, the system operates in real time, instantly notifying parents when their child is being bullied, enabling them to take prompt action. In addition, by incorporating emotional information, a deeper understanding and response can be achieved.

[1207] The processing flow will be explained below.

[1208] Step 1:

[1209] The device collects surrounding sounds in real time through a microphone, using a sound collection means to capture, for example, conversations in a classroom.

[1210] Step 2:

[1211] The device compresses the collected voice data and prepares it for efficient transmission to the server, using an appropriate compression algorithm to improve data transmission efficiency.

[1212] Step 3:

[1213] The terminal converts the compressed voice data into a packet format and sends it to the server using a secure data transmission protocol (e.g., HTTPS).

[1214] Step 4:

[1215] The server decompresses the audio data received from the device and converts it into text data using speech recognition technology, for example, the Google Cloud Speech-to-Text API.

[1216] Step 5:

[1217] The server analyzes the user's emotions using the voice data and emotion recognition means, inferring emotions from the tone, pitch, and intensity of the voice and generating emotion categories (e.g., joy, anger, sadness, etc.).

[1218] Step 6:

[1219] The server analyzes the text data and emotional information with a bullying detection method that uses natural language processing techniques and machine learning algorithms to detect keywords and patterns in the text that indicate bullying behavior, while also taking emotional information into account.

[1220] Step 7:

[1221] If the server determines that there is a high possibility of bullying based on the analysis results, it saves the results and triggers a notification method, notifying parents of the details of the detection and detailed data including emotional information.

[1222] Step 8:

[1223] The server sends a push notification to the parent's smartphone or tablet device, containing detailed information such as, "Your child may be experiencing bullying. Your child also appears to be experiencing strong sadness."

[1224] Step 9:

[1225] The user (parent) can check the received push notification and view detailed information through the app, which allows them to understand their child's situation and take necessary action.

[1226] Step 10:

[1227] The server monitors the parent's response and sends additional follow-up notifications as needed, enabling early detection and prompt response when a child is being bullied.

[1228] Example 2

[1229] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1230] In recent years, bullying among children has become a social problem, and early detection and countermeasures are needed. However, current monitoring and reporting systems have difficulty detecting signs of bullying in real time or making highly accurate judgments that take the user's emotions into account. In this environment, a system is needed that can not only detect bullying early, but also confirm changes in the victim's emotions.

[1231] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice collection means provided in the communication terminal used by the child, a voice transmission means for transmitting collected voice data to the server, a voice conversion means for converting the voice data into text data using voice recognition technology, a bullying detection means for analyzing the text data and detecting words that may be bullying, a notification means for sending a notification to the parent when it is determined that there is a high possibility of bullying, an emotion recognition means for recognizing the user's emotion from the voice data, and a means for analyzing the emotion information in combination with the text data. This makes it possible to detect signs of bullying in real time, and to make a highly accurate determination and immediately notify the victim while also taking into account changes in the victim's emotions.

[1232] Children are young people who are still growing, generally those under the age of 18.

[1233] A communication terminal refers to an electronic device for sending and receiving digital information, and includes, for example, a smartphone or tablet.

[1234] An audio collection means is a technical device or function that picks up surrounding sounds and voices, typically a microphone.

[1235] The audio transmission means refers to a technical mechanism for transmitting collected audio data to another device or server, and transmits data via the Internet or wireless communication.

[1236] A server is a computer system that provides data and resources over a network, providing file storage and processing power.

[1237] Audio data refers to a digital representation of sound collected by an audio input device such as a microphone, including the conversion of sound waves into electrical signals.

[1238] Speech recognition technology is a technology for analyzing voice data and converting it into text data, using specific algorithms and machine learning models.

[1239] Text data refers to a string of characters or a collection of letters, and refers to data that is stored and processed in a digital format.

[1240] Bullying detection methods are technologies that analyze text data to find signs of bullying, and include natural language processing technology and machine learning algorithms.

[1241] A notification means refers to a mechanism for conveying information about a specific event to a user or other system when that event occurs, and examples include push notifications and emails.

[1242] Emotion recognition means refers to a technical means for analyzing voice data to identify a user's emotions, and is based on characteristics such as tone, pitch, and intensity of the voice.

[1243] Emotion information refers to data indicating the user's emotional state obtained by emotion recognition means, and includes categories such as joy, anger, and sadness.

[1244] Natural language processing technology is a technology that allows computers to understand, interpret, and generate human language, and includes text analysis and speech recognition.

[1245] A machine learning algorithm is an algorithm that builds a model based on data and detects specific patterns and trends, and is used for data analysis on a server.

[1246] Push notifications refer to notification messages that are actively sent from a server to a client device, and are a way to get a user's real-time attention.

[1247] The present invention combines a system that collects voice data and detects bullying behavior in a communication device used by a child with an emotion engine that recognizes the user's emotions. An embodiment of this system will be described in detail below.

[1248] Communication terminal operation

[1249] The terminal is equipped with an audio collection means for collecting audio data. This means includes a function for collecting ambient audio in real time using a microphone. For example, audio is captured when children are talking in a classroom. In this case, an emotion recognition function is included in the audio collection means to analyze the user's emotions. The emotion recognition function analyzes the tone, pitch, and intensity of the audio to generate emotion categories such as joy, anger, and sadness.

[1250] The collected audio data is compressed and converted into a packet format for efficient data transmission to the server, using a secure data transmission protocol (e.g., TLS).

[1251] Server Operation

[1252] The server receives the voice data sent from the device. It then uses speech recognition technology to convert the received voice data into text data. This process uses services such as the Google Cloud Speech-to-Text API. The converted text data is saved in the form of a message such as, "You're so stupid, you can't even answer such a simple question?"

[1253] The server also runs an emotion recognition algorithm to identify the user's emotions from the audio data. Emotions are detected as emotion categories such as joy, anger, sadness, etc. based on the tone, pitch, and intensity of the audio. For example, the emotion of sadness is detected for the text, "You're so stupid, you can't even figure out such a simple question?"

[1254] The server then analyzes this text data and emotional information to detect potentially bullying phrases. The analysis uses natural language processing technology and machine learning algorithms, making it possible to detect phrases that are likely to be bullying, as well as complex patterns that take emotional information into account.

[1255] How the notification system works

[1256] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method. For example, it sends a push notification to the parent's smartphone. The message sent is, "Your child may be being bullied. It also appears that your child is feeling very sad."

[1257] Specific examples

[1258] Example 1: Classroom conversations and sentiment analysis

[1259] scenario:

[1260] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[1261] Operation steps:

[1262] 1. The device collects voice and analyzes emotions using emotion recognition.

[1263] 2. The device sends the collected voice data and emotional information to the server.

[1264] 3. The server converts the voice data into text data and stores the emotional information.

[1265] 4. The server analyzes the text data and emotional information to detect possible bullying.

[1266] 5. The server sends a push notification to the parent's smartphone stating, "Your child may be being bullied. Your child also appears to be feeling very sad."

[1267] Example prompts for generative AI models

[1268] "Please tell us how to detect possible bullying from speech data and emotional information when a child is told by a friend in the classroom, 'You're so stupid! You can't understand such a simple problem?' and notify parents."

[1269] In this way, the system operates in real time, notifying parents immediately when their child is being bullied, enabling them to take immediate action. By incorporating emotional information, the system can also provide a deeper understanding and response.

[1270] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1271] Step 1: Collecting audio data

[1272] The device uses a microphone to collect ambient audio in real time while children are conversing in the classroom. For example, it captures the audio when a child says something like, "You're so stupid! You can't even figure out such a simple question!" This collected audio data is used as input. This audio data also includes information such as the tone, pitch, and intensity of the voice.

[1273] Step 2: Recognize emotions

[1274] The device analyzes the user's emotions from the collected voice data. Using emotion recognition, it analyzes the tone, pitch, and intensity of the voice and generates emotion categories such as joy, anger, and sadness. For example, the emotion "sadness" is detected from the collected voice. This emotional information is generated as output.

[1275] Step 3: Compress and convert audio data

[1276] The terminal compresses the collected voice data and emotion information and converts them into packet format using a secure data transmission protocol (e.g., TLS). The compressed voice data and emotion information are input and packet format data is output.

[1277] Step 4: Sending audio data

[1278] The device transmits the voice data and emotion information converted into packets to the server via the Internet. The input is the packetized data, and the output is the data transmitted to the server.

[1279] Step 5: Receiving audio data

[1280] The server receives packetized voice data and emotion information sent from the terminal. The received data is input, and the voice data and emotion information are output.

[1281] Step 6: Convert audio data to text

[1282] The server uses speech recognition technology such as the Google Cloud Speech-to-Text API to convert the received voice data into text data. The input is voice data, and the output is text data. For example, the generated text might say, "You're so stupid! You can't even answer such a simple question?"

[1283] Step 7: Combining Emotional Information

[1284] The server combines the emotional information detected using an emotion recognition algorithm with the text data. The input is text data and emotional information, and the output is a combination of text and emotional categories. For example, the text "You're so stupid! You can't even figure out such a simple question?" is combined with the emotion "sadness."

[1285] Step 8: Detect bullying

[1286] The server uses natural language processing technology and machine learning algorithms to analyze text data and emotional information to detect potentially bullying statements. The input is a combination of text and emotional category data, and the output is a judgment result indicating a high probability of bullying. For example, based on the text "You're so stupid! You can't even figure out such a simple question?" and the emotional information of "sadness," it is judged that there is a high probability of bullying.

[1287] Step 9: Sending notifications

[1288] If the server determines that there is a high possibility of bullying, it saves the result and triggers a notification method. Specifically, it sends a push notification to the parent's smartphone. The input is the bullying determination result, and the output is a notification message. For example, a message may be sent saying, "Your child may be being bullied. Also, your child appears to be feeling very sad."

[1289] (Application example 2)

[1290] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1291] Conventional bullying detection systems convert only audio data into text data and detect bullying behaviors based on analyzing only the text data. This makes it difficult to consider emotional nuances and make highly accurate judgments. Furthermore, because it is not possible to grasp a child's emotional state when bullying occurs, appropriate responses can be delayed. Therefore, there is a need for a system that can improve the accuracy of bullying detection, quickly grasp a child's emotional state, and notify parents.

[1292] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice conversion means that converts voice data into text data using voice recognition technology, an emotion recognition means that analyzes emotions from the voice data and generates emotion information, a bullying detection means that analyzes the text data and the emotion information and detects words that may be bullying, and a notification means that sends a notification to the parents when the bullying detection means determines that there is a high possibility of bullying. This improves the accuracy of detecting bullying behavior and makes it possible to quickly grasp a child's emotional state.

[1293] The "audio collection means" is a function provided in the communication terminal used by the child to collect surrounding sounds.

[1294] The "voice transmission means" is a function that transmits collected voice data to a server.

[1295] The "voice conversion means" is a function that converts voice data into text data using voice recognition technology.

[1296] The "emotion recognition means" is a function that analyzes emotions from voice data and generates emotional information.

[1297] The "bullying detection means" is a function that analyzes text data and emotional information to detect potentially bullying language.

[1298] "Notification means" is a function that sends a notification to parents if it is determined that there is a high possibility of bullying.

[1299] A "machine learning algorithm" is a technology that builds a model based on empirical data and makes predictions and classifications for unknown data.

[1300] "Natural language processing technology" is a technology that uses computers to understand, interpret, and generate natural language.

[1301] A "server" is a computer that receives data from a terminal and analyzes and processes it.

[1302] System configuration and operation

[1303] The system collects ambient sounds in real time using a sound collection means provided in the communication terminal used by the child. The sound collection means includes a built-in microphone. The terminal further includes a sound transmission means for transmitting the collected sound data to a server. The sound data is compressed and converted into a packet format for efficient data transmission to the server. A secure data transmission protocol is used for this purpose.

[1304] Server Processing

[1305] The server has the function of receiving voice data sent from the device. The received voice data is first converted into text data by a voice conversion means. This process uses Google Cloud Speech-to-Text API as the voice recognition technology. The server also has an emotion recognition means to analyze emotions from the voice data. To do this, the tone, pitch, intensity, etc. of the voice are evaluated, and emotion categories such as joy, anger, and sadness are generated.

[1306] The generated text data and emotional information are analyzed by the bullying detection means, which uses natural language processing technology and machine learning algorithms to make highly accurate judgments by taking into account potentially bullying-related phrases and emotional information contained in the text.

[1307] Notification method operation

[1308] If the server determines that bullying is likely, it saves the result and triggers a notification method. This notification includes details based on the analyzed text data and emotional information. Specifically, if the text "No one cares if you're not around" is converted from speech and the user's emotion is determined to be "sadness," the server will use this result to send a notification to the parent's smartphone stating, "Your child may be being bullied. It also appears that your child is feeling very sad."

[1309] Examples and prompts

[1310] Specific examples

[1311] Suppose a child is talking to a friend in the classroom and says, "You're so stupid! You can't even figure out such a simple problem?" In this case, the emotion of sadness is detected from the child's voice.

[1312] 1. The device collects audio and analyzes emotions using emotion recognition means.

[1313] 2. The device sends the collected voice data and emotional information to the server.

[1314] 3. The server converts the voice data into text data and stores the emotional information.

[1315] 4. The server analyzes the text data and emotional information to detect possible bullying.

[1316] 5. The server sends a notification to the parent's smartphone saying, "Your child may be being bullied. Your child also appears to be feeling very sad."

[1317] Prompt Sentence Examples

[1318] "Analyzing children's classroom conversations to detect bullying behavior and identify emotional states."

[1319] "Oh, you don't know that?" is analyzed and emotions are identified from the voice characteristics.

[1320] "The text 'No one cares if you're gone' is converted to speech and the emotion is determined to be sadness."

[1321] As described above, this system not only uses voice data and emotional information to detect bullying with high accuracy, but also helps protect children's safety and psychological health by promptly reporting the incident to parents.

[1322] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1323] Step 1:

[1324] The device uses a built-in microphone to collect ambient sound, which is then temporarily stored as raw audio data. At this stage, the audio data is not yet in a directly usable format.

[1325] Input: Surrounding environmental sound

[1326] Output: Raw audio data

[1327] Step 2:

[1328] The device compresses the collected raw voice data and converts it into a packet format for efficient data transmission using voice data format conversion and compression algorithms, and then transmits the compressed data to the server via a secure data transmission protocol.

[1329] Input: Raw audio data

[1330] Output: Compressed and packetized audio data

[1331] Step 3:

[1332] The server receives the compressed and packetized voice data sent from the terminal, decompresses and decodes the received data, and reconstructs it into raw voice data.

[1333] Input: Compressed and packetized voice data

[1334] Output: Decompressed and decoded raw audio data

[1335] Step 4:

[1336] The server converts the raw audio data into text using the Google Cloud Speech-to-Text API, which feeds the audio data into the API and converts it into text in real time.

[1337] Input: Raw audio data

[1338] Output: Text data

[1339] Step 5:

[1340] The server uses emotion recognition to analyze the emotions in the voice data. It analyzes features such as tone, pitch, and intensity of the voice and generates emotion categories (e.g., joy, anger, sadness, etc.). Again, technologies such as Google Cloud Natural Language API may be used.

[1341] Input: Raw audio data

[1342] Output: Sentiment category and sentiment score

[1343] Step 6:

[1344] The server analyzes the text data and emotion categories using a bullying detection method. It uses natural language processing techniques and machine learning algorithms to detect potentially bullying statements in the text. The emotion information is used to assist in the judgment.

[1345] Input: Text data and emotion categories

[1346] Output: Judgment result of possibility of bullying

[1347] Step 7:

[1348] If the server determines that there is a high possibility of bullying, it stores the result of the determination and emotional information, and triggers a notification method. A push notification is sent to the parent or guardian's smartphone informing them of the possibility that their child is being bullied and details of their emotional state. This notification includes details based on text data and emotional information.

[1349] Input: Judgment result of possibility of bullying and emotion score

[1350] Output: Push notification to parent

[1351] example:

[1352] Suppose a child is talking with a friend in the classroom and says, "No one would care if you weren't here." In this case, the server recognizes the emotion "sadness" from the voice data and determines that this is bullying. As a result, the server sends a notification to the parent's smartphone saying, "Your child may be being bullied. It also appears that your child is feeling very sad."

[1353] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1354] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1355] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1356] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1357] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1358] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1359] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1360] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1361] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1362] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1363] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1364] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1365] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1366] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1367] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1368] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1369] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1370] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1371] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1372] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1373] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1374] The following is further disclosed regarding the above embodiment.

[1375] (Claim 1)

[1376] a voice collection means provided in a communication terminal used by a child;

[1377] a voice transmission means for transmitting the voice data collected by the voice collection means to a server;

[1378] a voice conversion means for converting the voice data into text data using a voice recognition technique;

[1379] a bullying detection means for analyzing the text data and detecting possible bullying statements;

[1380] a notification means for sending a notification to a parent when the bullying detection means determines that there is a high possibility of bullying;

[1381] A system including:

[1382] (Claim 2)

[1383] The system of claim 1 , wherein the speech recognition technology uses machine learning algorithms.

[1384] (Claim 3)

[1385] The system of claim 1, wherein the bullying detection means uses natural language processing techniques.

[1386] (Claim 4)

[1387] 2. The system according to claim 1, wherein the notification means transmits a push notification to the parent communication terminal.

[1388] (Claim 5)

[1389] 2. The system according to claim 1, wherein the sound collecting means has a function of collecting sound data and filtering out environmental noise.

[1390] "Example 1"

[1391] (Claim 1)

[1392] a voice collection means provided in a communication terminal used by a child;

[1393] a voice transmission means for compressing the voice data collected by the voice collection means and transmitting the compressed data to a server;

[1394] a voice conversion means for converting the voice data into text data using a voice recognition technique;

[1395] a bullying detection means for analyzing the text data and detecting possible bullying statements;

[1396] a notification generating means for sending a notification to a parent when the bullying detection means determines that there is a high possibility of bullying;

[1397] a notification content generation means for generating notification content using a generation AI model;

[1398] a notification sending means for sending the notification message generated by the notification content generating means to the parent communication terminal;

[1399] A system including:

[1400] (Claim 2)

[1401] The system of claim 1 , wherein the speech recognition technology uses machine learning algorithms.

[1402] (Claim 3)

[1403] The system of claim 1, wherein the bullying detection means uses natural language processing techniques.

[1404] "Application Example 1"

[1405] (Claim 1)

[1406] a voice collection means provided in a communication terminal used by a child;

[1407] a voice transmission means for transmitting the voice data collected by the voice collection means to a server;

[1408] a voice conversion means for converting the voice data into text data using a voice recognition technique;

[1409] a bullying detection means for analyzing the text data and detecting possible bullying statements;

[1410] a notification means for sending a notification to a parent when the bullying detection means determines that there is a high possibility of bullying;

[1411] A notification method that sends push notifications to the parent's smartphone,

[1412] A system including:

[1413] (Claim 2)

[1414] The system of claim 1 , wherein the speech recognition technology uses machine learning algorithms.

[1415] (Claim 3)

[1416] The system of claim 1, wherein the bullying detection means uses natural language processing techniques.

[1417] "Example 2: Combining Emotion Engines"

[1418] (Claim 1)

[1419] a voice collection means provided in a communication terminal used by a child;

[1420] a voice transmission means for transmitting the voice data collected by the voice collection means to a server;

[1421] a voice conversion means for converting the voice data into text data using a voice recognition technique;

[1422] a bullying detection means for analyzing the text data and detecting possible bullying statements;

[1423] a notification means for sending a notification to a parent when the bullying detection means determines that there is a high possibility of bullying;

[1424] emotion recognition means for recognizing an emotion of a user from the voice data;

[1425] means for analyzing emotion information in combination with the text data;

[1426] A system including:

[1427] (Claim 2)

[1428] The system of claim 1 , wherein the speech recognition technology uses machine learning algorithms.

[1429] (Claim 3)

[1430] The system of claim 1, wherein the bullying detection means uses natural language processing techniques.

[1431] "Application example 2 when combining emotion engines"

[1432] (Claim 1)

[1433] a voice collection means provided in a communication terminal used by a child;

[1434] a voice transmission means for transmitting the voice data collected by the voice collection means to a server;

[1435] a voice conversion means for converting the voice data into text data using a voice recognition technique;

[1436] emotion recognition means for analyzing emotions from the text data and voice data and generating emotion information;

[1437] a bullying detection means for analyzing the text data and emotion information and detecting potentially bullying statements;

[1438] a notification means for sending a notification to a parent when the bullying detection means determines that there is a high possibility of bullying;

[1439] A system including:

[1440] (Claim 2)

[1441] The system of claim 1 , wherein the speech recognition technology and emotion recognition means use machine learning algorithms.

[1442] (Claim 3)

[1443] The system of claim 1, wherein the bullying detection means uses natural language processing techniques. [Explanation of symbols]

[1444] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice collection means provided in a communication terminal used by a child; a voice transmission means for transmitting the voice data collected by the voice collection means to a server; a voice conversion means for converting the voice data into text data using a voice recognition technique; a bullying detection means for analyzing the text data and detecting possible bullying statements; a notification means for sending a notification to a parent when the bullying detection means determines that there is a high possibility of bullying; A system including:

2. The system of claim 1 , wherein the speech recognition technology uses machine learning algorithms.

3. The system of claim 1 , wherein the bullying detection means uses natural language processing techniques.

4. 2. The system according to claim 1, wherein the notification means transmits a push notification to the parent communication terminal.

5. 2. The system according to claim 1, wherein said sound collecting means has a function of collecting sound data and filtering out environmental noise.

Images