System

Abstract

A system is provided.SOLUTION: A system including means for receiving data from a monitoring camera and a sensor in a school, means for analyzing a student's behavior pattern in real time based on the received data, means for detecting an abnormal behavior from an analysis result, means for scoring characteristics and relationships of a perpetrator and a victim based on the detected abnormal behavior, means for visualizing and displaying the scored information, and means for notifying an instructor of the detected abnormal behavior.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Bullying in schools has been a serious social issue for many years. It is necessary to prevent bullying in places where teachers cannot see, and to create an environment where students can live their school life with peace of mind. However, current bullying countermeasures are primarily reactive, making it difficult to detect bullying early or take prompt action. Furthermore, grasping the actual situation of bullying and assessing the relationship between the perpetrator and victim often rely on subjective judgment, making it difficult to respond based on objective data. For this reason, it is necessary to provide a system that can detect bullying before it becomes serious, and can take prompt and appropriate action. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for receiving data from surveillance cameras and sensors within a school, a means for analyzing student behavior patterns in real time based on the received data, a means for detecting abnormal behavior from the analysis results, a means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, a means for visualizing and displaying the scored information, and a means for reporting the detected abnormal behavior to instructors. Specifically, the system receives video and audio data from surveillance cameras and sensors in real time and analyzes student behavior using a machine learning algorithm to automatically detect abnormal behavior such as aggressive behavior or loud shouting. The detected abnormal behavior is scored based on the characteristics and relationships of the perpetrator and victim, and the information is visualized on a dashboard that updates in real time. Furthermore, when abnormal behavior is detected, the information is automatically reported to instructors' devices, enabling rapid response. This enables early detection and prevention of bullying and creates an environment where students can live their school life with peace of mind.

[0006] A "surveillance camera" is a camera device that collects video data in real time and records and analyzes the situation of the monitored object.

[0007] A "sensor" is a device that detects physical environmental information (sound, temperature, movement, etc.) and outputs it as a digital signal.

[0008] "Means for receiving data" refers to equipment and software that receives digital data transmitted from surveillance cameras, sensors, etc., and delivers it to a processing system.

[0009] "Means for analyzing behavioral patterns" refers to algorithms and software that extract and analyze students' movements and behaviors from digital data.

[0010] "Means for detecting abnormal behavior" refers to algorithms or software that identify and notify unusual or dangerous behavior from the analysis results.

[0011] A "trait scoring tool" is an algorithm or software that quantifies and evaluates the characteristics and behavioral tendencies of perpetrators and victims based on detected behaviors.

[0012] A "relationship scoring tool" is an algorithm or software that numerically evaluates the relationship and impact between the perpetrator and victim, and measures the strength and severity of the relationship.

[0013] "Visualization and display means" refers to a display device or software that graphically displays scored information and analysis results, and presents them in a visually easy-to-understand manner.

[0014] "Means of reporting" refers to systems or software that notify instructors and other relevant parties in real time of detected abnormal behavior, encouraging a prompt response.

[0015] A "mentor" is an educational professional who provides guidance and support to students within a school and is responsible for responding to bullying issues. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention relates to a system for detecting bullying in schools and responding quickly. This system receives and analyzes data from surveillance cameras and sensors installed in schools to understand students' behavioral patterns in real time and detect abnormal behavior early.

[0038] System Configuration

[0039] The system consists of the following components:

[0040] 1. Surveillance cameras and sensors:

[0041] Video and audio data is collected in real time from these devices installed in various locations throughout the school.

[0042] 2. Data receiving device (server):

[0043] The received digital data is managed centrally and any necessary pre-processing is performed.

[0044] 3. Behavioral analysis system (server):

[0045] Analyze data in real time to identify student behavior patterns.

[0046] 4. Abnormal behavior detection system (server):

[0047] Abnormal behavior is detected from the results of behavioral analysis and necessary processing is carried out.

[0048] 5. Scoring and visualization system (server):

[0049] Based on the results of abnormal behavior detection, the characteristics and relationships of the perpetrator and victim are scored and displayed graphically.

[0050] 6. Reporting system (server and terminal):

[0051] Any abnormal behavior detected is automatically reported to the instructor's device.

[0052] System program processing

[0053] Server-side processing

[0054] 1. Data Collection

[0055] The server receives real-time video and audio data from surveillance cameras and sensors located within the school, which is then stored with a timestamp and used for subsequent analysis.

[0056] 2. Behavioral pattern analysis

[0057] The server analyzes the student's movements and voice based on the received data, using machine learning algorithms to distinguish between normal and abnormal behavioral patterns.

[0058] 3. Detecting Abnormal Behavior

[0059] The server automatically detects abnormal behavior, such as aggressive behavior or loud shouting, based on the results of behavioral pattern analysis.

[0060] 4. Scoring and visualization

[0061] The server then scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, and this information is visualized on a dashboard that is updated in real time, making it easy for instructors to check.

[0062] 5. Automatic reporting

[0063] If the server detects any abnormal behavior, it sends that information to the instructor's device in real time. The report includes detailed information about the detected behavior, enabling a prompt response.

[0064] Terminal side processing

[0065] 1. Warning display

[0066] The terminal receives the notification message sent from the server in real time and displays it as a pop-up notification, allowing the instructor to immediately know when a problem has occurred.

[0067] 2. Check the situation

[0068] The device displays detailed information about detected abnormal behaviors via a dashboard, allowing instructors to accurately grasp the situation by checking real-time video footage and scoring information.

[0069] 3. Instructions for response

[0070] Instructors can select response options from a dashboard on the device and issue instructions to implement necessary measures, enabling swift and appropriate responses.

[0071] User processing

[0072] 1. Check the warning

[0073] Instructors can check notifications on devices to obtain detailed information about behaviors that are likely to be bullying, and a dashboard makes it easy to determine the severity of the problem and identify those involved.

[0074] 2. Implementation of measures

[0075] Instructors can go to the scene of abnormal behavior and quickly provide guidance to the perpetrator and protection to the victim, thereby helping to resolve the problem early.

[0076] Specific examples

[0077] Example 1: Detecting bullying in the hallway

[0078] The server receives video data from surveillance cameras installed in the hallways. An analysis algorithm detects situations in which multiple students are surrounding a single student and pushing against each other. If this behavior is detected as abnormal, the server immediately generates a warning message and sends it to the instructor's device. The instructor checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[0079] Example 2: Detecting verbal abuse in the classroom

[0080] The server receives audio data from microphones installed in classrooms. The analysis system detects the sound of a student shouting at a high volume and determines that this is verbal violence against other students. When this abnormal behavior is detected, a warning message is immediately sent to the instructor's device, who is notified and checks the detailed situation in the classroom. They then head to the scene, provide appropriate guidance according to the situation, and protect the victim.

[0081] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[0082] The processing flow will be explained below.

[0083] Step 1:

[0084] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, and the data is stored in a database with a timestamp for subsequent analysis.

[0085] Step 2:

[0086] The server preprocesses the received video and audio data, removing noise, correcting low-resolution areas, and making the audio clearer. This improves the quality of the data and increases the accuracy of analysis.

[0087] Step 3:

[0088] The server then applies machine learning algorithms to the pre-processed data to analyze student behavior patterns, specifically identifying facial and body movements, voice tone and intensity, and identifying aggressive or unnatural behavior.

[0089] Step 4:

[0090] The server automatically detects abnormal behavior from the analysis results. In this step, it compares the results with normal behavior patterns that have been trained in advance, and marks any deviations as abnormal behavior.

[0091] Step 5:

[0092] When abnormal behavior is detected, the server collects information about the students involved in the behavior and assigns a score to the characteristics and relationships of the perpetrator and victim. This score is calculated based on the frequency of the behavior and past records.

[0093] Step 6:

[0094] The server visualizes the scoring information and detected abnormal behavior on a dashboard that is updated in real time, allowing instructors to check detailed information about the problem.

[0095] Step 7:

[0096] When abnormal behavior is detected, the server immediately generates a warning message and automatically sends it to the instructor's device. The message includes detailed information about the detected behavior (location, time, involved parties, etc.).

[0097] Step 8:

[0098] The device receives the notification message sent from the server and displays a pop-up notification to the instructor in real time, allowing the instructor to immediately know of any problems.

[0099] Step 9:

[0100] The user (instructor) checks the warning notification for abnormal behavior and checks detailed situation information on the dashboard. Based on the video and scoring information, the situation can be accurately understood.

[0101] Step 10:

[0102] The user (instructor) can select the appropriate action from the response options on the dashboard and respond quickly by heading to the scene or issuing instructions to other staff.

[0103] Through the above steps, this system can detect bullying within schools early and respond quickly and appropriately.

[0104] Example 1

[0105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] In recent years, bullying in schools has become a serious problem, necessitating early detection and rapid response. However, conventional monitoring devices and methods have issues in that they are unable to adequately detect abnormal behavior in real time, analyze behavior in detail, or respond quickly. Furthermore, it is difficult to simultaneously detect abnormal behavior and identify the specific characteristics and relationships of the perpetrator and victim.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0108] In this invention, the server includes a means for receiving data from monitoring devices and detectors within the school, a means for analyzing the behavioral patterns of monitored individuals in real time based on the received data, and a means for detecting abnormal behavior from the analysis results. This enables early detection of abnormal behavior, scoring the characteristics and relationships of perpetrators and victims, and visualizing and displaying them in real time. Furthermore, detected abnormal behavior can be reported to an administrator, enabling a prompt response.

[0109] "Monitoring equipment" refers to devices such as surveillance cameras and microphones within a school that collect video and audio data in real time.

[0110] A "detector" is a device that consists of a sensor, a data receiving device, etc., and receives data from a monitoring device and detects abnormal behavior.

[0111] The "means for receiving data" refers to a means by which the server receives video and audio data transmitted from a monitoring device or detector in real time.

[0112] "Means for analyzing behavioral patterns in real time" refers to a means for analyzing student movements and voices based on received data and distinguishing between normal and abnormal behavioral patterns.

[0113] The "means for detecting abnormal behavior" is a means for automatically detecting abnormal behavior such as specific aggressive behavior or loud shouting from the results of analyzing behavioral patterns.

[0114] The "scoring method" is a method for quantitatively evaluating and scoring the characteristics and relationships of actors and influencers based on detected abnormal behavior.

[0115] The "means for visualizing and displaying" refers to a means for displaying scored information in real time using a visualization tool such as a dashboard.

[0116] The "means of reporting" is a means of notifying the administrator's terminal in real time of detailed information about detected abnormal behavior, and encouraging a prompt response.

[0117] The present invention is a system for detecting bullying in schools and responding quickly. This system receives and analyzes data from monitoring devices and detectors installed in schools to understand students' behavioral patterns in real time and detect abnormal behavior early. Specific embodiments of this system are described below.

[0118] Hardware and Software Configuration

[0119] Monitoring and Detectors

[0120] Surveillance cameras and microphones will be installed throughout the school to collect video and audio data, for example in classrooms, hallways, and other areas where bullying is likely to occur.

[0121] Data receiving device (server)

[0122] The server receives real-time data transmitted from surveillance cameras and microphones throughout the school, which is then stored with a timestamp and used for subsequent analysis.

[0123] Behavioral analysis system (server)

[0124] The server analyzes the received video and audio data using machine learning algorithms (e.g., TensorFlow), which analyze students' movements and voices to identify normal and abnormal behavioral patterns.

[0125] Abnormal behavior detection system (server)

[0126] The server automatically detects abnormal behavior based on the results of behavioral analysis, such as multiple students surrounding one student or loud shouting.

[0127] Scoring and visualization system (server)

[0128] The server scores the characteristics and relationships of the perpetrator and victim based on the detected anomalous behavior, and this information is displayed on a dashboard (e.g., Grafana) that updates in real time so that stakeholders can easily view it.

[0129] Reporting system (server and terminal)

[0130] If the server detects any abnormal behavior, it will report the information to the administrator's device in real time. The reporting system uses a real-time notification service such as Twilio, which enables the immediate transmission of information.

[0131] Specific examples

[0132] Example 1: Detecting bullying in the hallway

[0133] The server receives video data from surveillance cameras installed in the hallways. An analysis algorithm detects situations in which multiple students are surrounding and pushing around one student. If this behavior is detected as abnormal, the server immediately generates a warning message and sends it to the administrator's device. The administrator checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[0134] Example 2: Detecting verbal abuse in the classroom

[0135] The server receives audio data from microphones installed in classrooms. The analysis system detects the sound of a student shouting at a high volume and determines that this is verbal violence against other students. When this abnormal behavior is detected, a warning message is immediately sent to the administrator's device, who is notified and checks the detailed situation in the classroom. They then head to the scene, provide appropriate guidance according to the situation, and protect the victim.

[0136] Prompt Sentence Examples

[0137] To test and improve this system, you can feed the generative AI model prompts like:

[0138] "Please explain the algorithm that will detect abnormal behavior when multiple students are pushing against one another in the hallway."

[0139] "How can we detect high-volume shouting in a classroom and identify it as verbal abuse?"

[0140] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1: Data collection

[0143] The server receives video and audio data in real time from monitoring devices and detectors installed within the school. The input data is video and audio from surveillance cameras and microphones. The received data is saved with a timestamp and used for subsequent analysis. Specifically, video captured by cameras installed in the hallways and audio collected by microphones in the classrooms are sent to the server.

[0144] Step 2: Preprocessing

[0145] The server performs preprocessing on the received raw data. The input data is the video and audio data collected in step 1. Specific preprocessing steps include noise removal, data normalization, and timestamp alignment. The output data is clean data ready to be input into the analysis system. For example, unnecessary frames are removed from the video data and background noise is removed from the audio data.

[0146] Step 3: Behavioral pattern analysis

[0147] The server analyzes behavioral patterns based on the preprocessed data. The input data is the clean video and audio data generated in step 2. A machine learning algorithm (e.g., TensorFlow) is used to analyze students' movements and audio patterns and distinguish between normal and abnormal behavioral patterns. The output data is digital features related to each student's behavioral pattern. Specifically, the server analyzes the student's position and movements for each frame of video data and detects shouts from the waveform of the audio data.

[0148] Step 4: Detecting Abnormal Behavior

[0149] The server detects abnormal behavior based on the analysis results. The input data are the digital features of the behavioral patterns generated in step 3. Abnormal behavior includes aggressive behavior and loud shouting. For example, if a specific student behaves aggressively toward another student within a specific time period, this is detected as abnormal behavior. The output data is the abnormal behavior detection results and detailed information. Specifically, information such as "Student B behaved aggressively toward Student C in classroom A at XX:YY" is detected.

[0150] Step 5: Scoring and visualization

[0151] The server scores the characteristics and relationships of the perpetrator and victim based on the data of detected abnormal behavior. The input data is the abnormal behavior detection results generated in step 4. The scored information is displayed on a dashboard that is updated in real time. The output data is visualized score information. Specifically, the behavioral history and relationships of the perpetrator and victim are displayed as graphs and charts.

[0152] Step 6: Automatic reporting

[0153] The server sends a real-time report to the administrator's device based on the information on the detected abnormal behavior. The input data is the abnormal behavior and its scoring data generated in steps 4 and 5. The reporting system uses a real-time notification service such as Twilio. The output data is a report message. This message contains detailed information about the abnormal behavior and is displayed on the administrator's device. For example, a message such as "Student B is behaving aggressively toward Student C in classroom A" is sent.

[0154] Step 7: Warning Display

[0155] The terminal receives the report message sent from the server in real time and displays a pop-up notification. The input data is the report message sent in step 6. The notification contains detailed information about the abnormal behavior, which the administrator can check immediately. The output data is a pop-up notification that is displayed so that the administrator can check it.

[0156] Step 8: Check the situation

[0157] The terminal displays detailed information about the detected abnormal behavior through a dashboard. The input data is the scored information generated in step 5. Real-time video and scored data are displayed, allowing the administrator to accurately grasp the situation based on this. The output data is detailed situation information that is displayed to the administrator. Specifically, a dashboard screen is provided that allows the location, time, and involved parties of the problem to be identified.

[0158] Step 9: Instructions

[0159] The user selects a response option from the device dashboard and issues instructions to implement the necessary measures. The input data is detailed information about the abnormal behavior confirmed in step 8. The output data is the selected response option. Specifically, options such as "visit the scene," "contact the parents," and "notify other instructors" can be selected, and a prompt response will be implemented.

[0160] Step 10: Implementing the measures

[0161] The user goes to the scene where the abnormal behavior is occurring and quickly instructs the perpetrator and protects the victim. The input data is the response option selected in step 9. The output data is the instructions actually taken and the results. Specifically, the user goes to classroom A and takes action to stop the bullying. Counseling and protective measures are also implemented for those involved.

[0162] (Application example 1)

[0163] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0164] Bullying in schools is a serious problem, requiring early detection and rapid response. However, comprehensive detection and immediate response to bullying are difficult using current manual monitoring and approaches alone. Monitoring and responding to bullying is particularly challenging in large schools and dispersed school environments, making it essential to implement a smooth system to ensure student safety. Furthermore, analyzing data obtained from surveillance cameras and sensors requires advanced technology, and an efficient system is needed to detect abnormal behavior in real time and respond quickly.

[0165] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0166] In this invention, the server includes means for receiving data from surveillance cameras and sensors within the school, means for analyzing student behavior patterns in real time based on the received data, means for detecting abnormal behavior from the analysis results, means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, means for visualizing and displaying the scored information, and means for displaying detailed information about the abnormal behavior on a display device worn by an instructor who receives a notification of the abnormal behavior. This makes it possible to detect bullying and abnormal behavior within the school in real time and respond immediately. Instructors can check detailed information about the abnormal behavior on the display device and take prompt and appropriate measures.

[0167] A "surveillance camera" is a device that is installed in a specific area and is used to capture video.

[0168] A "sensor" is a device that detects physical changes and converts them into electrical signals.

[0169] The "data receiving means" is a device or system for receiving data acquired from the surveillance cameras and sensors.

[0170] The "behavior pattern analysis means" is a system that has the function of analyzing student behavior based on received data and identifying specific patterns.

[0171] The "abnormal behavior detection means" is a system for automatically identifying behavior that is different from normal based on the results of analyzing behavior patterns.

[0172] The "scoring means" is a system that has the function of quantifying the characteristics and relationships of the perpetrator and victim based on detected abnormal behavior.

[0173] The "visualization display means" is a device or software for graphically displaying the scored information.

[0174] The "reporting means" is a device or function for notifying the trainer of information about detected abnormal behavior.

[0175] The "display device" is a device that displays detailed information about abnormal behavior so that the trainer can check it.

[0176] The present invention is a system for detecting bullying in schools and responding quickly, and includes a series of processes that receive data from surveillance cameras and sensors, perform behavior analysis, and detect abnormal behavior in real time. Specific embodiments for implementing this system are described below.

[0177] System construction

[0178] This system consists of the following main components:

[0179] 1. Surveillance cameras and sensors:

[0180] Video and audio data is collected in real time using surveillance cameras and audio sensors (e.g., Logitech C920 cameras and Blue Yeti microphones) installed throughout the school.

[0181] 2. Data receiving means:

[0182] A server is used to receive and centrally manage data sent from each surveillance camera and sensor.

[0183] 3. Behavioral pattern analysis methods:

[0184] The server uses deep learning frameworks such as TensorFlow to analyze the data it receives and identify student behavior patterns, using machine learning algorithms to distinguish between normal and abnormal behavior patterns.

[0185] 4. Abnormal behavior detection methods:

[0186] The server automatically detects abnormal behavior based on the analyzed behavioral patterns, such as aggressive behavior or loud shouting.

[0187] 5. Scoring Procedure:

[0188] When abnormal behavior is detected, the server scores the characteristics and relationships of the perpetrator and victim, clarifying the severity of the problem and the relationships between the parties involved.

[0189] 6. Visualization display means:

[0190] Scoring information is displayed on a real-time updated dashboard or display device (e.g., smart glasses), allowing trainers to view detailed information about abnormal behavior.

[0191] 7. Reporting methods:

[0192] When abnormal behavior is detected, the server sends the information to the instructor's device in real time. The report includes detailed information about the abnormal behavior, enabling a prompt response.

[0193] Explanation of program processing

[0194] The system collects data from surveillance cameras and audio sensors in real time and preprocesses the received data on a server. It uses TensorFlow to analyze video data and identify abnormal behavior, while Scikit-learn analyzes audio data. When abnormal behavior is detected, the information is immediately sent to the instructor's terminal and display device (such as smart glasses).

[0195] Specific examples

[0196] For example, if a surveillance camera captures multiple students surrounding one another in a hallway, the TensorFlow model will detect this abnormal behavior. At this point, the server will immediately generate an alert message and notify the instructor's smart glasses. The instructor can then view the situation in real time through the glasses and rush to the scene to take appropriate action.

[0197] Prompt Sentence Examples

[0198] You can use the following prompt for the generative AI model: "Train a video analytics model using a dataset for detecting anomalous behavior and generate Python code to detect suspicious behavior and emergencies in real time."

[0199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0200] Step 1:

[0201] Data collection

[0202] The server collects real-time video and audio data from surveillance cameras and sensors (e.g., Logitech C920 cameras, Blue Yeti microphones) installed within the school. The input data are video frames and audio samples, which are sent to the server along with timestamps. The output data are time-stamped video and audio data.

[0203] Step 2:

[0204] Data Preprocessing

[0205] The server performs preprocessing on the collected video and audio data. This processing includes noise removal and data normalization. The input is the collected raw data, and the output is preprocessed video and audio data. Specifically, OpenCV is used to resize the video frames and perform noise filtering, and Scipy is used to remove noise from the audio data.

[0206] Step 3:

[0207] Behavioral pattern analysis

[0208] The server analyzes behavioral patterns based on the preprocessed data. This analysis uses a deep learning model using TensorFlow. The input is preprocessed video and audio data, and the output is the analysis results of behavioral patterns. Specifically, the server extracts video data frame by frame and inputs it into the model to classify normal and abnormal behavior.

[0209] Step 4:

[0210] Abnormal behavior detection

[0211] The server detects anomalous behavior from the results of behavioral pattern analysis. This detection uses a machine learning algorithm to identify suspicious behavior. The input is the analyzed behavioral pattern data, and the output is a judgment result on whether anomalous behavior has been detected. Specifically, it uses Scikit-learn's classification algorithm to identify aggressive behavior and high-volume shouting.

[0212] Step 5:

[0213] Scoring abnormal behavior

[0214] The server scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior. The input is the abnormal behavior detection result, and the output is the scored data. Specifically, it applies an algorithm that quantifies information such as the degree of abnormal behavior, the number of people involved, and frequency.

[0215] Step 6:

[0216] Information Visualization

[0217] The scored information is visualized on a display device such as a dashboard or smart glasses that is updated in real time. The input is the scored data, and the output is the information displayed graphically. Specifically, a JavaScript framework (e.g., D3.js) is used to display the information in graphs and charts on the dashboard.

[0218] Step 7:

[0219] Reporting Abnormal Behavior

[0220] When abnormal behavior is detected, the server immediately notifies the instructor's terminal or display device. The input is detailed information about the abnormal behavior, and the output is a warning message displayed on the instructor's terminal. Specific operations include a function to display a pop-up notification on the smart glasses and an API to send a push notification to the smartphone.

[0221] Prompt Sentence Examples

[0222] You can use the following prompt for the generative AI model: "Train a video analytics model using a dataset for detecting anomalous behavior and generate Python code to detect suspicious behavior and emergencies in real time."

[0223] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0224] This invention relates to a system for detecting bullying in schools and responding quickly. In particular, this system analyzes students' behavioral patterns and emotions in real time, detects abnormal behavior and emotional changes, and takes necessary measures. This system is composed of the following components:

[0225] System Configuration

[0226] 1. Surveillance cameras and sensors:

[0227] They are installed in various locations throughout the school and collect video and audio data in real time, allowing students' behavior and the surrounding sound environment to be monitored.

[0228] 2. Data receiving device (server):

[0229] The data sent from surveillance cameras and sensors is centrally managed and the necessary pre-processing is performed.

[0230] 3. Behavioral analysis system (server):

[0231] The received data is analyzed in real time to identify student behavioral patterns and emotions.

[0232] 4. Emotion engine (server):

[0233] Based on video and audio data, the system analyzes students' facial expressions and tone of voice to recognize their emotions. The recognized emotional information is added to the analysis results of their behavioral patterns.

[0234] 5. Abnormal behavior detection system (server):

[0235] Abnormal behavior is detected based on the results of behavioral analysis and the emotion engine, and necessary processing is carried out.

[0236] 6. Scoring and visualization system (server):

[0237] Based on detected abnormal behavior and recognized emotions, the characteristics and relationships of the perpetrator and victim are scored and displayed graphically.

[0238] 7. Reporting system (server and terminal):

[0239] Detected abnormal behavior and emotional information is automatically reported to the instructor's device.

[0240] System program processing

[0241] Server-side processing

[0242] 1. Data Collection

[0243] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, which is then stored with a timestamp and used for analytical processing.

[0244] 2. Behavioral and emotional analysis

[0245] The server applies machine learning algorithms and emotion engines to the received data to identify students' behavioral patterns and emotions, specifically recognizing emotional states (such as joy, anger, sadness, and fear) through facial expression and voice analysis.

[0246] 3. Detecting Abnormal Behavior

[0247] The server automatically detects abnormal behavior based on the analysis of behavioral patterns and emotions. For example, it detects aggressive behavior or loud shouting and determines these as abnormal behavior.

[0248] 4. Scoring and visualization

[0249] The server scores the characteristics and relationships of the perpetrator and victim based on abnormal behavior and emotional information. This information is visualized in real time on a dashboard, allowing instructors to easily understand the situation.

[0250] 5. Automatic reporting

[0251] When abnormal behavior or specific emotions are detected, the server sends a real-time report to the instructor's device, including detailed information about the detected behavior (location, time, involved parties, emotional state, etc.).

[0252] Terminal side processing

[0253] 1. Warning display

[0254] The device receives the notification message sent from the server and displays a real-time warning as a pop-up notification, allowing the instructor to immediately know of any problems.

[0255] 2. Check the situation

[0256] The device displays detailed situational information (video, score, emotional information) via a dashboard, allowing the instructor to quickly grasp the situation and consider appropriate responses.

[0257] 3. Instructions for response

[0258] Instructors can select the appropriate action from the response options on the dashboard and either head to the scene or give instructions to other staff.

[0259] User processing

[0260] 1. Check the warning

[0261] Teachers can view notifications on devices and get detailed information about behaviors and emotions that may be indicative of bullying. A dashboard makes it easy to determine the severity of the problem and identify those involved.

[0262] 2. Implementation of measures

[0263] Instructors can go to the scene where abnormal behavior and emotional changes are occurring and quickly provide guidance to the perpetrator and protection to the victim, thereby helping to resolve the problem early.

[0264] Specific examples

[0265] Example 1: Detecting bullying and emotional changes in the hallway

[0266] The server receives video data from surveillance cameras installed in the hallways. Using an analysis algorithm and emotion engine, it detects the emotion of fear from the facial expression of the student being pushed, as multiple students surround and push against one another. If this behavior and emotional information is detected as abnormal, the server immediately generates a warning message and sends it to the instructor's device. The instructor checks the pop-up notification on their device, understands the situation on the dashboard, and rushes to the scene to take appropriate action.

[0267] Example 2: Detecting verbal abuse and emotional changes in the classroom

[0268] The server receives audio data from microphones installed in classrooms. The analysis system and emotion engine detect the high-pitched voice of a student shouting and the sadness of a student believed to be the victim. If this is recognized as abnormal behavior such as verbal violence, a warning message is immediately sent to the instructor's device. The instructor is notified and checks the details of the situation in the classroom, then heads to the scene to provide appropriate guidance according to the situation and protect the victim.

[0269] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[0270] The processing flow will be explained below.

[0271] Step 1:

[0272] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, which is then stored in a database with a timestamp for subsequent analysis.

[0273] Step 2:

[0274] The server preprocesses the received video and audio data, removing noise, correcting low-resolution areas, and making the audio clearer. This improves the quality of the data and increases the accuracy of analysis.

[0275] Step 3:

[0276] The server then applies machine learning algorithms to the pre-processed data to analyze student behavioral patterns, such as facial and body movements, and vocal tone and intensity, to identify aggressive or unnatural behavior.

[0277] Step 4:

[0278] The server analyzes behavioral patterns and uses an emotion engine to recognize emotions from students' facial expressions and tone of voice, categorizing them as joy, anger, sadness, fear, and so on.

[0279] Step 5:

[0280] The server automatically detects abnormal behavior by combining the results of behavioral pattern analysis and the emotion engine's recognition results. For example, if aggressive behavior and angry emotions are detected simultaneously, the server marks the behavior as abnormal.

[0281] Step 6:

[0282] When abnormal behavior is detected, the server collects information about the students related to the behavior and emotions, and then scores the characteristics and relationships of the perpetrator and victim. This score is calculated based on the frequency of the behavior, past records, type of emotion, etc.

[0283] Step 7:

[0284] The server visualizes the scoring information and detected abnormal behavior and emotional information on a dashboard that is updated in real time, allowing instructors to check detailed information about the problem.

[0285] Step 8:

[0286] When abnormal behavior or a specific emotion is detected, the server immediately generates a warning message and automatically sends it to the instructor's device. The message includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.).

[0287] Step 9:

[0288] The device receives the notification message sent from the server and displays a pop-up notification to the instructor in real time, allowing the instructor to immediately know of any problems.

[0289] Step 10:

[0290] The user (instructor) checks the warning notification for abnormal behavior and checks detailed situation information on the dashboard. Based on the video, score information, and emotional information, the situation can be accurately grasped.

[0291] Step 11:

[0292] The user (instructor) can select the appropriate action from the response options on the dashboard and respond quickly by heading to the scene or issuing instructions to other staff.

[0293] Through the above steps, this system can detect bullying within schools early and respond quickly and appropriately.

[0294] Example 2

[0295] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0296] Bullying in schools has long-term mental and physical effects on students, so early detection and rapid response are necessary. However, traditional methods tend to be slow to detect and address bullying behavior, making it difficult for relevant parties to actively intervene. For this reason, a system that can detect bullying behavior in real time and respond quickly is needed.

[0297] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving information from monitoring devices and sensors in the school, a means for analyzing the behavioral patterns of individuals in real time based on the received information, and a means for detecting abnormal behavior from the analysis results. This makes it possible to detect abnormal behavior in real time and take prompt and appropriate action.

[0298] "Surveillance equipment" refers to cameras and sensors installed to collect video data within a school.

[0299] "Sensors" refer to devices that collect sound and environmental data within the school.

[0300] "Information" refers to data such as student behavior and audio collected from monitoring devices and sensors.

[0301] "Individual" refers to a specific person, such as a student, who is the subject of monitoring within a school.

[0302] "Behavior patterns" refer to students' patterns of movement and behavior.

[0303] "Real-time" refers to information being processed and analyzed almost as soon as it is generated.

[0304] "Abnormal behavior" refers to inappropriate behavior, such as bullying or aggressive behavior, that deviates from normal patterns of behavior.

[0305] "Characteristics" refers to the behavioral and emotional characteristics of individual students.

[0306] "Relationship" refers to the human relationships and interactions between the perpetrator and the victim.

[0307] "Scoring" refers to quantifying behaviors and characteristics and converting them into numbers or ratings.

[0308] "Visualization" refers to visually displaying the scored information to make it easier to understand.

[0309] "Administrator" refers to faculty and staff members who are responsible for monitoring the system and dealing with abnormal behavior.

[0310] "Management screen" refers to the interface for displaying scored information and detected abnormal behavior.

[0311] The present invention relates to a system for realizing early detection of bullying in schools and a rapid response to the same. Specific embodiments of the system are described below.

[0312] System configuration

[0313] This system mainly uses the following hardware and software:

[0314] 1. Monitoring Devices and Sensors

[0315] Surveillance cameras and audio sensors will be installed throughout the school to collect student behavior and audio data in real time.

[0316] 2. Data receiving device (server)

[0317] The collected video and audio data is sent to a server via a dedicated communication protocol, where it is stored in a database with a timestamp for subsequent analysis.

[0318] 3. Behavioral analysis system (server)

[0319] The server applies machine learning algorithms to the collected data to identify students' behavioral patterns and emotions in real time, using libraries such as OpenCV and Dlib for facial expression analysis and librosa for voice analysis.

[0320] 4. Emotion engine (server)

[0321] Through facial expression analysis and voice analysis, it identifies students' emotional states (joy, anger, sadness, fear, etc.). The emotion engine analyzes the collected data and identifies emotional information.

[0322] 5. Abnormal behavior detection system (server)

[0323] Based on the results of behavioral analysis and the emotion engine, the system automatically detects abnormal behavior. Machine learning models identify anomalous behavior such as aggressive behavior or high-volume shouting.

[0324] 6. Scoring and visualization system (server)

[0325] Based on abnormal behavior and emotional information, the characteristics and relationships of the perpetrator and victim are scored. This scoring information is visualized in real time on a dashboard, allowing administrators to easily understand the situation.

[0326] 7. Reporting System (Server and Terminal)

[0327] Detected abnormal behavior and emotional information are automatically reported to the administrator's terminal. The report message includes detailed information about the abnormal behavior (location, time, involved parties, emotional state, etc.).

[0328] Specific examples

[0329] Example 1: Detecting bullying and emotional changes in the hallway

[0330] The server receives video data from surveillance cameras installed in the hallways. Using an analysis algorithm and emotion engine, it detects a group of students pushing around one another, and detects fear from the facial expression of the student being pushed. If this behavior and emotional information is detected as abnormal, the server immediately generates a warning message and sends it to the administrator's device. The administrator checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[0331] Example 2: Detecting verbal abuse and emotional changes in the classroom

[0332] The server receives audio data from microphones installed in classrooms. The analysis system and emotion engine detect the high-pitched voices of students shouting and the sadness of students believed to be victims. If this is recognized as abnormal behavior such as verbal violence, a warning message is immediately sent to the administrator's device. The administrator is notified and checks the details of the situation in the classroom, then heads to the scene to provide appropriate guidance according to the situation and protect the victim.

[0333] Prompt Sentence Examples

[0334] "If the system detects bullying behavior and emotional changes in the hallways, please explain specifically what data is collected and how it is analyzed."

[0335] This allows the generative AI model to provide a good explanation for the specific behavior of the system.

[0336] The present invention enables early detection of bullying and rapid response, thereby providing a safe learning environment for students.

[0337] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0338] Step 1: Data collection

[0339] The server receives real-time video and audio data from monitoring devices and sensors, and stores the data in a database with a timestamp.

[0340] Input: Video and audio data from surveillance cameras and audio sensors.

[0341] Output: Stored raw data with timestamp.

[0342] How it works: The surveillance camera captures video frames and the sensor collects audio waveforms. These data are then sent to the server via a dedicated communication protocol.

[0343] Step 2: Analyzing behavioral patterns and emotions

[0344] The server applies machine learning algorithms and emotion engines to the collected data to identify student behavioral patterns and emotions.

[0345] Input: Stored video and audio data.

[0346] Output: Identification of behavioral patterns and emotional state data.

[0347] How it works: The server uses OpenCV and Dlib libraries to analyze facial expressions, and librosa to recognize tone of voice and emotions from audio data.

[0348] Step 3: Detecting Abnormal Behavior

[0349] The server automatically detects abnormal behavior based on the results of behavioral analysis and the emotion engine.

[0350] Input: Behavioral patterns and emotional state data.

[0351] Output: Abnormal behavior detection result.

[0352] Specific behaviors: Apply machine learning models to identify anomalous behaviors, such as aggressive behavior or high-volume screaming, by comparing them with historical patterns of normal behavior.

[0353] Step 4: Scoring and visualization

[0354] The server scores the characteristics and relationships between the perpetrator and victim based on the detected abnormal behavior and emotional information, and visualizes the scored information in real time on a dashboard.

[0355] Input: Abnormal behavior detection results.

[0356] Output: Scored trait and relationship information, graphical display.

[0357] Specific actions: Comprehensively evaluate the frequency of abnormal behavior, the location of occurrence, and historical data of those involved, and display the results as heat maps and graphs on the dashboard.

[0358] Step 5: Automatic reporting

[0359] If the server detects abnormal behavior or a specific emotion, it reports the information to the administrator's terminal in real time.

[0360] Input: Abnormal behavior detection results, scored information.

[0361] Output: Notification message to administrator terminal.

[0362] Specific operation: Use the reporting system to send detailed information about abnormal behavior via SMS, email, or the notification function of a dedicated app.

[0363] Step 6: Warning display

[0364] The terminal receives the notification message sent from the server and displays a warning in a pop-up notification.

[0365] Input: Notification message from the server.

[0366] Output: Display of warning notification.

[0367] Specific operation: A warning message will pop up on the device screen and an audio alert will also be issued.

[0368] Step 7: Check the situation

[0369] The device allows the notified administrator to view detailed status information via a dashboard.

[0370] Input: Report message, scored information, video data.

[0371] Output: Display detailed status information.

[0372] How it works: Administrators can view real-time video streams and historical data on a dashboard.

[0373] Step 8: Instructions

[0374] Managers can select the appropriate action from the response options on the dashboard and either head to the scene or give instructions to other staff.

[0375] Input: Dashboard status information.

[0376] Output: Decision on response instructions.

[0377] Specific actions: The administrator presses buttons such as "Rush to the scene" or "Notify staff" on the dashboard and instructs the necessary response.

[0378] (Application example 2)

[0379] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0380] Conventional security systems have difficulty detecting suspicious behavior, particularly in analyzing emotional changes in real time and associating them with suspicious behavior. This makes it difficult to respond quickly and, as a result, to provide a safe and secure environment. Furthermore, because scoring and visualization are not performed, it is not possible to intuitively understand the characteristics and relationships of those involved.

[0381] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0382] In this invention, the server includes means for receiving data from the monitoring device and the audio sensing device, means for analyzing people's behavioral patterns and emotions in real time based on the received data, means for detecting abnormal behavior and changes in emotions from the analysis results, means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and changes in emotions, means for visualizing and displaying the scored information, and means for notifying security personnel of the detected abnormal behavior and changes in emotions. This allows abnormal behavior and changes in emotions to be quickly detected and notified to security personnel, enabling a prompt response. Furthermore, visualizing the scored information allows users to intuitively understand the characteristics and relationships of the people involved.

[0383] A "monitoring device" is a device that collects video data in real time and monitors specific locations or people.

[0384] An "audio sensing device" is a device that collects surrounding audio data in real time and uses it for analysis.

[0385] A "behavioral pattern" is a repetitive pattern of actions or behaviors that a particular person performs on a daily basis.

[0386] "Emotions" are psychological states detected from facial expressions and tone of voice, and include joy, anger, sadness, fear, etc.

[0387] "Means for real-time analysis" refers to a method for processing received data immediately and obtaining results without delay.

[0388] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior, including aggressive behavior and loud shouting.

[0389] "Emotional shift" refers to an abrupt change in the observed individual's emotions, especially including changes to stronger emotional states (e.g., fear, anger, etc.).

[0390] "Scoring" is a method for evaluating and visualizing specific characteristics and relationships using numbers or indicators.

[0391] "Visualization" is the act of displaying data or information in a graphical format (e.g., a dashboard) to make it intuitively easier to understand.

[0392] "Security personnel" are professional personnel who monitor and respond to suspicious or abnormal behavior in order to ensure safety within facilities and buildings.

[0393] "Reporting" is the act of immediately notifying security personnel about detected abnormal behavior or emotional changes.

[0394] This invention is a system for detecting suspicious behavior and emotional changes and responding quickly in office buildings, commercial facilities, etc. The system includes means for receiving and analyzing data from monitoring devices and audio sensing devices. It also includes means for detecting, scoring, and visualizing abnormal behavior and emotional changes. Finally, it includes means for reporting the detected information to security personnel.

[0395] System Program Overview

[0396] The server receives real-time video and audio data from surveillance and audio sensing devices installed in office buildings and commercial facilities. This data is stored with a timestamp and used for analytical processing. The server then uses machine learning algorithms and emotion engines (e.g., TensorFlow, Keras, etc.) to perform facial recognition and audio analysis on the received data and identify behavioral patterns and emotions.

[0397] Analyzing data and detecting abnormal behavior and emotional changes

[0398] The server applies analysis algorithms and emotion engines to identify the target person's behavioral patterns and emotional state. For example, it recognizes emotional states (such as joy, anger, sadness, and fear) through facial expression and voice analysis. It then automatically detects abnormal behavior and sudden emotional changes based on the behavioral pattern and emotion analysis results. Examples of abnormal behavior include aggressive behavior and loud shouting.

[0399] Scoring and visualization process

[0400] The server scores the characteristics and relationships of the perpetrator and victim based on abnormal behavior and emotional information. For example, it quantifies each person's behavior and emotional state and displays this information in real time on a dashboard. This allows security personnel to easily grasp the situation and respond quickly.

[0401] Reporting function

[0402] When abnormal behavior or specific emotions are detected, the server sends a real-time notification to the security officer's smartphone. The notification includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.). This notification function is realized using the Twilio API or other notification systems.

[0403] Specific examples

[0404] Example 1: Detecting suspicious behavior and emotional changes in the office

[0405] Video data is received from surveillance cameras in an office building, and an analysis algorithm detects suspicious behavior in which multiple people surround one person. Furthermore, a strong expression of fear is detected from the facial expressions of the surrounded individuals. This situation is determined to be abnormal behavior, and a report is immediately sent to security personnel. An example of a prompt statement is, "Several people have surrounded one individual in an office building, and expressions of fear have been detected. Please check the scene immediately."

[0406] This system allows for real-time monitoring and analysis of suspicious behavior and sudden emotional changes within facilities and buildings, enabling efficient and rapid response, thereby providing a safe and secure environment at all times.

[0407] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0408] Step 1: Receiving data

[0409] The server receives real-time video and audio data from the monitoring and audio sensing devices. This data is time-stamped to identify when each input was collected. The input data are video frames and audio waveforms, which are then ready for the server to use in the next processing step.

[0410] Step 2: Preprocessing the data

[0411] The server converts the received video data to grayscale and denoises the audio data, improving the accuracy of the analysis. The video frames and audio waveforms received as input are converted into clean video frames and denoised audio waveforms after preprocessing.

[0412] Step 3: Face and voice detection

[0413] The server detects face areas from preprocessed video frames and identifies the sound source from the audio data. It uses OpenCV's face detection algorithm for face detection and a sound source localization algorithm for audio source identification. It receives preprocessed video frames and audio waveforms as input data, and obtains face areas and sound source locations as output.

[0414] Step 4: Analyzing behavioral patterns and emotions

[0415] The server analyzes the emotions of the identified facial regions using machine learning models (using TensorFlow and Keras) and identifies the person's emotional state through audio analysis. The analysis recognizes emotions such as joy, anger, sadness, and fear from facial expressions and tone of voice. It receives facial regions and audio data as input and obtains the determined emotional state as output.

[0416] Step 5: Detecting abnormal behavior and emotional changes

[0417] The server detects abnormal behavior and sudden emotional changes based on the analysis results. For example, this includes aggressive behavior, loud shouting, and strong emotions such as fear. It receives the analyzed emotional state and behavioral patterns as input, and obtains the detection results of abnormal behavior and emotional changes as output.

[0418] Step 6: Scoring and visualization

[0419] The server scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and emotional information. The scored information is displayed graphically on a dashboard. The server receives the detection results of abnormal behavior and emotional changes as input, and obtains the scored information and a visualized graphical dashboard display as output.

[0420] Step 7: Report

[0421] The server notifies the security personnel of detected abnormal behaviors and emotional changes in real time via their smartphones. The notification information includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.). It receives the scored and visualized information as input and generates a notification message for the security personnel as output.

[0422] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0423] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0424] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0425] [Second embodiment]

[0426] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0427] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0428] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0429] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0430] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0431] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0432] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0433] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0434] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0435] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0436] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0437] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0438] This invention relates to a system for detecting bullying in schools and responding quickly. This system receives and analyzes data from surveillance cameras and sensors installed in schools to understand students' behavioral patterns in real time and detect abnormal behavior early.

[0439] System Configuration

[0440] The system consists of the following components:

[0441] 1. Surveillance cameras and sensors:

[0442] Video and audio data is collected in real time from these devices installed in various locations throughout the school.

[0443] 2. Data receiving device (server):

[0444] The received digital data is managed centrally and any necessary pre-processing is performed.

[0445] 3. Behavioral analysis system (server):

[0446] Analyze data in real time to identify student behavior patterns.

[0447] 4. Abnormal behavior detection system (server):

[0448] Abnormal behavior is detected from the results of behavioral analysis and necessary processing is carried out.

[0449] 5. Scoring and visualization system (server):

[0450] Based on the results of abnormal behavior detection, the characteristics and relationships of the perpetrator and victim are scored and displayed graphically.

[0451] 6. Reporting system (server and terminal):

[0452] Any abnormal behavior detected is automatically reported to the instructor's device.

[0453] System program processing

[0454] Server-side processing

[0455] 1. Data Collection

[0456] The server receives real-time video and audio data from surveillance cameras and sensors located within the school, which is then stored with a timestamp and used for subsequent analysis.

[0457] 2. Behavioral pattern analysis

[0458] The server analyzes the student's movements and voice based on the received data, using machine learning algorithms to distinguish between normal and abnormal behavioral patterns.

[0459] 3. Detecting Abnormal Behavior

[0460] The server automatically detects abnormal behavior, such as aggressive behavior or loud shouting, based on the results of behavioral pattern analysis.

[0461] 4. Scoring and visualization

[0462] The server then scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, and this information is visualized on a dashboard that is updated in real time, making it easy for instructors to check.

[0463] 5. Automatic reporting

[0464] If the server detects any abnormal behavior, it sends that information to the instructor's device in real time. The report includes detailed information about the detected behavior, enabling a prompt response.

[0465] Terminal side processing

[0466] 1. Warning display

[0467] The terminal receives the notification message sent from the server in real time and displays it as a pop-up notification, allowing the instructor to immediately know when a problem has occurred.

[0468] 2. Check the situation

[0469] The device displays detailed information about detected abnormal behaviors via a dashboard, allowing instructors to accurately grasp the situation by checking real-time video footage and scoring information.

[0470] 3. Instructions for response

[0471] Instructors can select response options from a dashboard on the device and issue instructions to implement necessary measures, enabling swift and appropriate responses.

[0472] User processing

[0473] 1. Check the warning

[0474] Instructors can check notifications on devices to obtain detailed information about behaviors that are likely to be bullying, and a dashboard makes it easy to determine the severity of the problem and identify those involved.

[0475] 2. Implementation of measures

[0476] Instructors can go to the scene of abnormal behavior and quickly provide guidance to the perpetrator and protection to the victim, thereby helping to resolve the problem early.

[0477] Specific examples

[0478] Example 1: Detecting bullying in the hallway

[0479] The server receives video data from surveillance cameras installed in the hallways. An analysis algorithm detects situations in which multiple students are surrounding a single student and pushing against each other. If this behavior is detected as abnormal, the server immediately generates a warning message and sends it to the instructor's device. The instructor checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[0480] Example 2: Detecting verbal abuse in the classroom

[0481] The server receives audio data from microphones installed in classrooms. The analysis system detects the sound of a student shouting at a high volume and determines that this is verbal violence against other students. When this abnormal behavior is detected, a warning message is immediately sent to the instructor's device, who is notified and checks the detailed situation in the classroom. They then head to the scene, provide appropriate guidance according to the situation, and protect the victim.

[0482] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[0483] The processing flow will be explained below.

[0484] Step 1:

[0485] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, and the data is stored in a database with a timestamp for subsequent analysis.

[0486] Step 2:

[0487] The server preprocesses the received video and audio data, removing noise, correcting low-resolution areas, and making the audio clearer. This improves the quality of the data and increases the accuracy of analysis.

[0488] Step 3:

[0489] The server then applies machine learning algorithms to the pre-processed data to analyze student behavior patterns, specifically identifying facial and body movements, voice tone and intensity, and identifying aggressive or unnatural behavior.

[0490] Step 4:

[0491] The server automatically detects abnormal behavior from the analysis results. In this step, it compares the results with normal behavior patterns that have been trained in advance, and marks any deviations as abnormal behavior.

[0492] Step 5:

[0493] When abnormal behavior is detected, the server collects information about the students involved in the behavior and assigns a score to the characteristics and relationships of the perpetrator and victim. This score is calculated based on the frequency of the behavior and past records.

[0494] Step 6:

[0495] The server visualizes the scoring information and detected abnormal behavior on a dashboard that is updated in real time, allowing instructors to check detailed information about the problem.

[0496] Step 7:

[0497] When abnormal behavior is detected, the server immediately generates a warning message and automatically sends it to the instructor's device. The message includes detailed information about the detected behavior (location, time, involved parties, etc.).

[0498] Step 8:

[0499] The device receives the notification message sent from the server and displays a pop-up notification to the instructor in real time, allowing the instructor to immediately know of any problems.

[0500] Step 9:

[0501] The user (instructor) checks the warning notification for abnormal behavior and checks detailed situation information on the dashboard. Based on the video and scoring information, the situation can be accurately understood.

[0502] Step 10:

[0503] The user (instructor) can select the appropriate action from the response options on the dashboard and respond quickly by heading to the scene or issuing instructions to other staff.

[0504] Through the above steps, this system can detect bullying within schools early and respond quickly and appropriately.

[0505] Example 1

[0506] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] In recent years, bullying in schools has become a serious problem, necessitating early detection and rapid response. However, conventional monitoring devices and methods have issues in that they are unable to adequately detect abnormal behavior in real time, analyze behavior in detail, or respond quickly. Furthermore, it is difficult to simultaneously detect abnormal behavior and identify the specific characteristics and relationships of the perpetrator and victim.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0509] In this invention, the server includes a means for receiving data from monitoring devices and detectors within the school, a means for analyzing the behavioral patterns of monitored individuals in real time based on the received data, and a means for detecting abnormal behavior from the analysis results. This enables early detection of abnormal behavior, scoring the characteristics and relationships of perpetrators and victims, and visualizing and displaying them in real time. Furthermore, detected abnormal behavior can be reported to an administrator, enabling a prompt response.

[0510] "Monitoring equipment" refers to devices such as surveillance cameras and microphones within a school that collect video and audio data in real time.

[0511] A "detector" is a device that consists of a sensor, a data receiving device, etc., and receives data from a monitoring device and detects abnormal behavior.

[0512] The "means for receiving data" refers to a means by which the server receives video and audio data transmitted from a monitoring device or detector in real time.

[0513] "Means for analyzing behavioral patterns in real time" refers to a means for analyzing student movements and voices based on received data and distinguishing between normal and abnormal behavioral patterns.

[0514] The "means for detecting abnormal behavior" is a means for automatically detecting abnormal behavior such as specific aggressive behavior or loud shouting from the results of analyzing behavioral patterns.

[0515] The "scoring method" is a method for quantitatively evaluating and scoring the characteristics and relationships of actors and influencers based on detected abnormal behavior.

[0516] The "means for visualizing and displaying" refers to a means for displaying scored information in real time using a visualization tool such as a dashboard.

[0517] The "means of reporting" is a means of notifying the administrator's terminal in real time of detailed information about detected abnormal behavior, and encouraging a prompt response.

[0518] The present invention is a system for detecting bullying in schools and responding quickly. This system receives and analyzes data from monitoring devices and detectors installed in schools to understand students' behavioral patterns in real time and detect abnormal behavior early. Specific embodiments of this system are described below.

[0519] Hardware and Software Configuration

[0520] Monitoring and Detectors

[0521] Surveillance cameras and microphones will be installed throughout the school to collect video and audio data, for example in classrooms, hallways, and other areas where bullying is likely to occur.

[0522] Data receiving device (server)

[0523] The server receives real-time data transmitted from surveillance cameras and microphones throughout the school, which is then stored with a timestamp and used for subsequent analysis.

[0524] Behavioral analysis system (server)

[0525] The server analyzes the received video and audio data using machine learning algorithms (e.g., TensorFlow), which analyze students' movements and voices to identify normal and abnormal behavioral patterns.

[0526] Abnormal behavior detection system (server)

[0527] The server automatically detects abnormal behavior based on the results of behavioral analysis, such as multiple students surrounding one student or loud shouting.

[0528] Scoring and visualization system (server)

[0529] The server scores the characteristics and relationships of the perpetrator and victim based on the detected anomalous behavior, and this information is displayed on a dashboard (e.g., Grafana) that updates in real time so that stakeholders can easily view it.

[0530] Reporting system (server and terminal)

[0531] If the server detects any abnormal behavior, it will report the information to the administrator's device in real time. The reporting system uses a real-time notification service such as Twilio, which enables the immediate transmission of information.

[0532] Specific examples

[0533] Example 1: Detecting bullying in the hallway

[0534] The server receives video data from surveillance cameras installed in the hallways. An analysis algorithm detects situations in which multiple students are surrounding and pushing around one student. If this behavior is detected as abnormal, the server immediately generates a warning message and sends it to the administrator's device. The administrator checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[0535] Example 2: Detecting verbal abuse in the classroom

[0536] The server receives audio data from microphones installed in classrooms. The analysis system detects the sound of a student shouting at a high volume and determines that this is verbal violence against other students. When this abnormal behavior is detected, a warning message is immediately sent to the administrator's device, who is notified and checks the detailed situation in the classroom. They then head to the scene, provide appropriate guidance according to the situation, and protect the victim.

[0537] Prompt Sentence Examples

[0538] To test and improve this system, you can feed the generative AI model prompts like:

[0539] "Please explain the algorithm that will detect abnormal behavior when multiple students are pushing against one another in the hallway."

[0540] "How can we detect high-volume shouting in a classroom and identify it as verbal abuse?"

[0541] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[0542] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0543] Step 1: Data collection

[0544] The server receives video and audio data in real time from monitoring devices and detectors installed within the school. The input data is video and audio from surveillance cameras and microphones. The received data is saved with a timestamp and used for subsequent analysis. Specifically, video captured by cameras installed in the hallways and audio collected by microphones in the classrooms are sent to the server.

[0545] Step 2: Preprocessing

[0546] The server performs preprocessing on the received raw data. The input data is the video and audio data collected in step 1. Specific preprocessing steps include noise removal, data normalization, and timestamp alignment. The output data is clean data ready to be input into the analysis system. For example, unnecessary frames are removed from the video data and background noise is removed from the audio data.

[0547] Step 3: Behavioral pattern analysis

[0548] The server analyzes behavioral patterns based on the preprocessed data. The input data is the clean video and audio data generated in step 2. A machine learning algorithm (e.g., TensorFlow) is used to analyze students' movements and audio patterns and distinguish between normal and abnormal behavioral patterns. The output data is digital features related to each student's behavioral pattern. Specifically, the server analyzes the student's position and movements for each frame of video data and detects shouts from the waveform of the audio data.

[0549] Step 4: Detecting Abnormal Behavior

[0550] The server detects abnormal behavior based on the analysis results. The input data are the digital features of the behavioral patterns generated in step 3. Abnormal behavior includes aggressive behavior and loud shouting. For example, if a specific student behaves aggressively toward another student within a specific time period, this is detected as abnormal behavior. The output data is the abnormal behavior detection results and detailed information. Specifically, information such as "Student B behaved aggressively toward Student C in classroom A at XX:YY" is detected.

[0551] Step 5: Scoring and visualization

[0552] The server scores the characteristics and relationships of the perpetrator and victim based on the data of detected abnormal behavior. The input data is the abnormal behavior detection results generated in step 4. The scored information is displayed on a dashboard that is updated in real time. The output data is visualized score information. Specifically, the behavioral history and relationships of the perpetrator and victim are displayed as graphs and charts.

[0553] Step 6: Automatic reporting

[0554] The server sends a real-time report to the administrator's device based on the information on the detected abnormal behavior. The input data is the abnormal behavior and its scoring data generated in steps 4 and 5. The reporting system uses a real-time notification service such as Twilio. The output data is a report message. This message contains detailed information about the abnormal behavior and is displayed on the administrator's device. For example, a message such as "Student B is behaving aggressively toward Student C in classroom A" is sent.

[0555] Step 7: Warning Display

[0556] The terminal receives the report message sent from the server in real time and displays a pop-up notification. The input data is the report message sent in step 6. The notification contains detailed information about the abnormal behavior, which the administrator can check immediately. The output data is a pop-up notification that is displayed so that the administrator can check it.

[0557] Step 8: Check the situation

[0558] The terminal displays detailed information about the detected abnormal behavior through a dashboard. The input data is the scored information generated in step 5. Real-time video and scored data are displayed, allowing the administrator to accurately grasp the situation based on this. The output data is detailed situation information that is displayed to the administrator. Specifically, a dashboard screen is provided that allows the location, time, and involved parties of the problem to be identified.

[0559] Step 9: Instructions

[0560] The user selects a response option from the device dashboard and issues instructions to implement the necessary measures. The input data is detailed information about the abnormal behavior confirmed in step 8. The output data is the selected response option. Specifically, options such as "visit the scene," "contact the parents," and "notify other instructors" can be selected, and a prompt response will be implemented.

[0561] Step 10: Implementing the measures

[0562] The user goes to the scene where the abnormal behavior is occurring and quickly instructs the perpetrator and protects the victim. The input data is the response option selected in step 9. The output data is the instructions actually taken and the results. Specifically, the user goes to classroom A and takes action to stop the bullying. Counseling and protective measures are also implemented for those involved.

[0563] (Application example 1)

[0564] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0565] Bullying in schools is a serious problem, requiring early detection and rapid response. However, comprehensive detection and immediate response to bullying are difficult using current manual monitoring and approaches alone. Monitoring and responding to bullying is particularly challenging in large schools and dispersed school environments, making it essential to implement a smooth system to ensure student safety. Furthermore, analyzing data obtained from surveillance cameras and sensors requires advanced technology, and an efficient system is needed to detect abnormal behavior in real time and respond quickly.

[0566] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0567] In this invention, the server includes means for receiving data from surveillance cameras and sensors within the school, means for analyzing student behavior patterns in real time based on the received data, means for detecting abnormal behavior from the analysis results, means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, means for visualizing and displaying the scored information, and means for displaying detailed information about the abnormal behavior on a display device worn by an instructor who receives a notification of the abnormal behavior. This makes it possible to detect bullying and abnormal behavior within the school in real time and respond immediately. Instructors can check detailed information about the abnormal behavior on the display device and take prompt and appropriate measures.

[0568] A "surveillance camera" is a device that is installed in a specific area and is used to capture video.

[0569] A "sensor" is a device that detects physical changes and converts them into electrical signals.

[0570] The "data receiving means" is a device or system for receiving data acquired from the surveillance cameras and sensors.

[0571] The "behavior pattern analysis means" is a system that has the function of analyzing student behavior based on received data and identifying specific patterns.

[0572] The "abnormal behavior detection means" is a system for automatically identifying behavior that is different from normal based on the results of analyzing behavior patterns.

[0573] The "scoring means" is a system that has the function of quantifying the characteristics and relationships of the perpetrator and victim based on detected abnormal behavior.

[0574] The "visualization display means" is a device or software for graphically displaying the scored information.

[0575] The "reporting means" is a device or function for notifying the trainer of information about detected abnormal behavior.

[0576] The "display device" is a device that displays detailed information about abnormal behavior so that the trainer can check it.

[0577] The present invention is a system for detecting bullying in schools and responding quickly, and includes a series of processes that receive data from surveillance cameras and sensors, perform behavior analysis, and detect abnormal behavior in real time. Specific embodiments for implementing this system are described below.

[0578] System construction

[0579] This system consists of the following main components:

[0580] 1. Surveillance cameras and sensors:

[0581] Video and audio data is collected in real time using surveillance cameras and audio sensors (e.g., Logitech C920 cameras and Blue Yeti microphones) installed throughout the school.

[0582] 2. Data receiving means:

[0583] A server is used to receive and centrally manage data sent from each surveillance camera and sensor.

[0584] 3. Behavioral pattern analysis methods:

[0585] The server uses deep learning frameworks such as TensorFlow to analyze the data it receives and identify student behavior patterns, using machine learning algorithms to distinguish between normal and abnormal behavior patterns.

[0586] 4. Abnormal behavior detection methods:

[0587] The server automatically detects abnormal behavior based on the analyzed behavioral patterns, such as aggressive behavior or loud shouting.

[0588] 5. Scoring Procedure:

[0589] When abnormal behavior is detected, the server scores the characteristics and relationships of the perpetrator and victim, clarifying the severity of the problem and the relationships between the parties involved.

[0590] 6. Visualization display means:

[0591] Scoring information is displayed on a real-time updated dashboard or display device (e.g., smart glasses), allowing trainers to view detailed information about abnormal behavior.

[0592] 7. Reporting methods:

[0593] When abnormal behavior is detected, the server sends the information to the instructor's device in real time. The report includes detailed information about the abnormal behavior, enabling a prompt response.

[0594] Explanation of program processing

[0595] The system collects data from surveillance cameras and audio sensors in real time and preprocesses the received data on a server. It uses TensorFlow to analyze video data and identify abnormal behavior, while Scikit-learn analyzes audio data. When abnormal behavior is detected, the information is immediately sent to the instructor's terminal and display device (such as smart glasses).

[0596] Specific examples

[0597] For example, if a surveillance camera captures multiple students surrounding one another in a hallway, the TensorFlow model will detect this abnormal behavior. At this point, the server will immediately generate an alert message and notify the instructor's smart glasses. The instructor can then view the situation in real time through the glasses and rush to the scene to take appropriate action.

[0598] Prompt Sentence Examples

[0599] You can use the following prompt for the generative AI model: "Train a video analytics model using a dataset for detecting anomalous behavior and generate Python code to detect suspicious behavior and emergencies in real time."

[0600] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0601] Step 1:

[0602] Data collection

[0603] The server collects real-time video and audio data from surveillance cameras and sensors (e.g., Logitech C920 cameras, Blue Yeti microphones) installed within the school. The input data are video frames and audio samples, which are sent to the server along with timestamps. The output data are time-stamped video and audio data.

[0604] Step 2:

[0605] Data Preprocessing

[0606] The server performs preprocessing on the collected video and audio data. This processing includes noise removal and data normalization. The input is the collected raw data, and the output is preprocessed video and audio data. Specifically, OpenCV is used to resize the video frames and perform noise filtering, and Scipy is used to remove noise from the audio data.

[0607] Step 3:

[0608] Behavioral pattern analysis

[0609] The server analyzes behavioral patterns based on the preprocessed data. This analysis uses a deep learning model using TensorFlow. The input is preprocessed video and audio data, and the output is the analysis results of behavioral patterns. Specifically, the server extracts video data frame by frame and inputs it into the model to classify normal and abnormal behavior.

[0610] Step 4:

[0611] Abnormal behavior detection

[0612] The server detects anomalous behavior from the results of behavioral pattern analysis. This detection uses a machine learning algorithm to identify suspicious behavior. The input is the analyzed behavioral pattern data, and the output is a judgment result on whether anomalous behavior has been detected. Specifically, it uses Scikit-learn's classification algorithm to identify aggressive behavior and high-volume shouting.

[0613] Step 5:

[0614] Scoring abnormal behavior

[0615] The server scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior. The input is the abnormal behavior detection result, and the output is the scored data. Specifically, it applies an algorithm that quantifies information such as the degree of abnormal behavior, the number of people involved, and frequency.

[0616] Step 6:

[0617] Information Visualization

[0618] The scored information is visualized on a display device such as a dashboard or smart glasses that is updated in real time. The input is the scored data, and the output is the information displayed graphically. Specifically, a JavaScript framework (e.g., D3.js) is used to display the information in graphs and charts on the dashboard.

[0619] Step 7:

[0620] Reporting Abnormal Behavior

[0621] When abnormal behavior is detected, the server immediately notifies the instructor's terminal or display device. The input is detailed information about the abnormal behavior, and the output is a warning message displayed on the instructor's terminal. Specific operations include a function to display a pop-up notification on the smart glasses and an API to send a push notification to the smartphone.

[0622] Prompt Sentence Examples

[0623] You can use the following prompt for the generative AI model: "Train a video analytics model using a dataset for detecting anomalous behavior and generate Python code to detect suspicious behavior and emergencies in real time."

[0624] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0625] This invention relates to a system for detecting bullying in schools and responding quickly. In particular, this system analyzes students' behavioral patterns and emotions in real time, detects abnormal behavior and emotional changes, and takes necessary measures. This system is composed of the following components:

[0626] System Configuration

[0627] 1. Surveillance cameras and sensors:

[0628] They are installed in various locations throughout the school and collect video and audio data in real time, allowing students' behavior and the surrounding sound environment to be monitored.

[0629] 2. Data receiving device (server):

[0630] The data sent from surveillance cameras and sensors is centrally managed and the necessary pre-processing is performed.

[0631] 3. Behavioral analysis system (server):

[0632] The received data is analyzed in real time to identify student behavioral patterns and emotions.

[0633] 4. Emotion engine (server):

[0634] Based on video and audio data, the system analyzes students' facial expressions and tone of voice to recognize their emotions. The recognized emotional information is added to the analysis results of their behavioral patterns.

[0635] 5. Abnormal behavior detection system (server):

[0636] Abnormal behavior is detected based on the results of behavioral analysis and the emotion engine, and necessary processing is carried out.

[0637] 6. Scoring and visualization system (server):

[0638] Based on detected abnormal behavior and recognized emotions, the characteristics and relationships of the perpetrator and victim are scored and displayed graphically.

[0639] 7. Reporting system (server and terminal):

[0640] Detected abnormal behavior and emotional information is automatically reported to the instructor's device.

[0641] System program processing

[0642] Server-side processing

[0643] 1. Data Collection

[0644] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, which is then stored with a timestamp and used for analytical processing.

[0645] 2. Behavioral and emotional analysis

[0646] The server applies machine learning algorithms and emotion engines to the received data to identify students' behavioral patterns and emotions, specifically recognizing emotional states (such as joy, anger, sadness, and fear) through facial expression and voice analysis.

[0647] 3. Detecting Abnormal Behavior

[0648] The server automatically detects abnormal behavior based on the analysis of behavioral patterns and emotions. For example, it detects aggressive behavior or loud shouting and determines these as abnormal behavior.

[0649] 4. Scoring and visualization

[0650] The server scores the characteristics and relationships of the perpetrator and victim based on abnormal behavior and emotional information. This information is visualized in real time on a dashboard, allowing instructors to easily understand the situation.

[0651] 5. Automatic reporting

[0652] When abnormal behavior or specific emotions are detected, the server sends a real-time report to the instructor's device, including detailed information about the detected behavior (location, time, involved parties, emotional state, etc.).

[0653] Terminal side processing

[0654] 1. Warning display

[0655] The device receives the notification message sent from the server and displays a real-time warning as a pop-up notification, allowing the instructor to immediately know of any problems.

[0656] 2. Check the situation

[0657] The device displays detailed situational information (video, score, emotional information) via a dashboard, allowing the instructor to quickly grasp the situation and consider appropriate responses.

[0658] 3. Instructions for response

[0659] Instructors can select the appropriate action from the response options on the dashboard and either head to the scene or give instructions to other staff.

[0660] User processing

[0661] 1. Check the warning

[0662] Teachers can view notifications on devices and get detailed information about behaviors and emotions that may be indicative of bullying. A dashboard makes it easy to determine the severity of the problem and identify those involved.

[0663] 2. Implementation of measures

[0664] Instructors can go to the scene where abnormal behavior and emotional changes are occurring and quickly provide guidance to the perpetrator and protection to the victim, thereby helping to resolve the problem early.

[0665] Specific examples

[0666] Example 1: Detecting bullying and emotional changes in the hallway

[0667] The server receives video data from surveillance cameras installed in the hallways. Using an analysis algorithm and emotion engine, it detects the emotion of fear from the facial expression of the student being pushed, as multiple students surround and push against one another. If this behavior and emotional information is detected as abnormal, the server immediately generates a warning message and sends it to the instructor's device. The instructor checks the pop-up notification on their device, understands the situation on the dashboard, and rushes to the scene to take appropriate action.

[0668] Example 2: Detecting verbal abuse and emotional changes in the classroom

[0669] The server receives audio data from microphones installed in classrooms. The analysis system and emotion engine detect the high-pitched voice of a student shouting and the sadness of a student believed to be the victim. If this is recognized as abnormal behavior such as verbal violence, a warning message is immediately sent to the instructor's device. The instructor is notified and checks the details of the situation in the classroom, then heads to the scene to provide appropriate guidance according to the situation and protect the victim.

[0670] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[0671] The processing flow will be explained below.

[0672] Step 1:

[0673] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, which is then stored in a database with a timestamp for subsequent analysis.

[0674] Step 2:

[0675] The server preprocesses the received video and audio data, removing noise, correcting low-resolution areas, and making the audio clearer. This improves the quality of the data and increases the accuracy of analysis.

[0676] Step 3:

[0677] The server then applies machine learning algorithms to the pre-processed data to analyze student behavioral patterns, such as facial and body movements, and vocal tone and intensity, to identify aggressive or unnatural behavior.

[0678] Step 4:

[0679] The server analyzes behavioral patterns and uses an emotion engine to recognize emotions from students' facial expressions and tone of voice, categorizing them as joy, anger, sadness, fear, and so on.

[0680] Step 5:

[0681] The server automatically detects abnormal behavior by combining the results of behavioral pattern analysis and the emotion engine's recognition results. For example, if aggressive behavior and angry emotions are detected simultaneously, the server marks the behavior as abnormal.

[0682] Step 6:

[0683] When abnormal behavior is detected, the server collects information about the students related to the behavior and emotions, and then scores the characteristics and relationships of the perpetrator and victim. This score is calculated based on the frequency of the behavior, past records, type of emotion, etc.

[0684] Step 7:

[0685] The server visualizes the scoring information and detected abnormal behavior and emotional information on a dashboard that is updated in real time, allowing instructors to check detailed information about the problem.

[0686] Step 8:

[0687] When abnormal behavior or a specific emotion is detected, the server immediately generates a warning message and automatically sends it to the instructor's device. The message includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.).

[0688] Step 9:

[0689] The device receives the notification message sent from the server and displays a pop-up notification to the instructor in real time, allowing the instructor to immediately know of any problems.

[0690] Step 10:

[0691] The user (instructor) checks the warning notification for abnormal behavior and checks detailed situation information on the dashboard. Based on the video, score information, and emotional information, the situation can be accurately grasped.

[0692] Step 11:

[0693] The user (instructor) can select the appropriate action from the response options on the dashboard and respond quickly by heading to the scene or issuing instructions to other staff.

[0694] Through the above steps, this system can detect bullying within schools early and respond quickly and appropriately.

[0695] Example 2

[0696] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0697] Bullying in schools has long-term mental and physical effects on students, so early detection and rapid response are necessary. However, traditional methods tend to be slow to detect and address bullying behavior, making it difficult for relevant parties to actively intervene. For this reason, a system that can detect bullying behavior in real time and respond quickly is needed.

[0698] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving information from monitoring devices and sensors in the school, a means for analyzing the behavioral patterns of individuals in real time based on the received information, and a means for detecting abnormal behavior from the analysis results. This makes it possible to detect abnormal behavior in real time and take prompt and appropriate action.

[0699] "Surveillance equipment" refers to cameras and sensors installed to collect video data within a school.

[0700] "Sensors" refer to devices that collect sound and environmental data within the school.

[0701] "Information" refers to data such as student behavior and audio collected from monitoring devices and sensors.

[0702] "Individual" refers to a specific person, such as a student, who is the subject of monitoring within a school.

[0703] "Behavior patterns" refer to students' patterns of movement and behavior.

[0704] "Real-time" refers to information being processed and analyzed almost as soon as it is generated.

[0705] "Abnormal behavior" refers to inappropriate behavior, such as bullying or aggressive behavior, that deviates from normal patterns of behavior.

[0706] "Characteristics" refers to the behavioral and emotional characteristics of individual students.

[0707] "Relationship" refers to the human relationships and interactions between the perpetrator and the victim.

[0708] "Scoring" refers to quantifying behaviors and characteristics and converting them into numbers or ratings.

[0709] "Visualization" refers to visually displaying the scored information to make it easier to understand.

[0710] "Administrator" refers to faculty and staff members who are responsible for monitoring the system and dealing with abnormal behavior.

[0711] "Management screen" refers to the interface for displaying scored information and detected abnormal behavior.

[0712] The present invention relates to a system for realizing early detection of bullying in schools and a rapid response to the same. Specific embodiments of the system are described below.

[0713] System configuration

[0714] This system mainly uses the following hardware and software:

[0715] 1. Monitoring Devices and Sensors

[0716] Surveillance cameras and audio sensors will be installed throughout the school to collect student behavior and audio data in real time.

[0717] 2. Data receiving device (server)

[0718] The collected video and audio data is sent to a server via a dedicated communication protocol, where it is stored in a database with a timestamp for subsequent analysis.

[0719] 3. Behavioral analysis system (server)

[0720] The server applies machine learning algorithms to the collected data to identify students' behavioral patterns and emotions in real time, using libraries such as OpenCV and Dlib for facial expression analysis and librosa for voice analysis.

[0721] 4. Emotion engine (server)

[0722] Through facial expression analysis and voice analysis, it identifies students' emotional states (joy, anger, sadness, fear, etc.). The emotion engine analyzes the collected data and identifies emotional information.

[0723] 5. Abnormal behavior detection system (server)

[0724] Based on the results of behavioral analysis and the emotion engine, the system automatically detects abnormal behavior. Machine learning models identify anomalous behavior such as aggressive behavior or high-volume shouting.

[0725] 6. Scoring and visualization system (server)

[0726] Based on abnormal behavior and emotional information, the characteristics and relationships of the perpetrator and victim are scored. This scoring information is visualized in real time on a dashboard, allowing administrators to easily understand the situation.

[0727] 7. Reporting System (Server and Terminal)

[0728] Detected abnormal behavior and emotional information are automatically reported to the administrator's terminal. The report message includes detailed information about the abnormal behavior (location, time, involved parties, emotional state, etc.).

[0729] Specific examples

[0730] Example 1: Detecting bullying and emotional changes in the hallway

[0731] The server receives video data from surveillance cameras installed in the hallways. Using an analysis algorithm and emotion engine, it detects a group of students pushing around one another, and detects fear from the facial expression of the student being pushed. If this behavior and emotional information is detected as abnormal, the server immediately generates a warning message and sends it to the administrator's device. The administrator checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[0732] Example 2: Detecting verbal abuse and emotional changes in the classroom

[0733] The server receives audio data from microphones installed in classrooms. The analysis system and emotion engine detect the high-pitched voices of students shouting and the sadness of students believed to be victims. If this is recognized as abnormal behavior such as verbal violence, a warning message is immediately sent to the administrator's device. The administrator is notified and checks the details of the situation in the classroom, then heads to the scene to provide appropriate guidance according to the situation and protect the victim.

[0734] Prompt Sentence Examples

[0735] "If the system detects bullying behavior and emotional changes in the hallways, please explain specifically what data is collected and how it is analyzed."

[0736] This allows the generative AI model to provide a good explanation for the specific behavior of the system.

[0737] The present invention enables early detection of bullying and rapid response, thereby providing a safe learning environment for students.

[0738] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0739] Step 1: Data collection

[0740] The server receives real-time video and audio data from monitoring devices and sensors, and stores the data in a database with a timestamp.

[0741] Input: Video and audio data from surveillance cameras and audio sensors.

[0742] Output: Stored raw data with timestamp.

[0743] How it works: The surveillance camera captures video frames and the sensor collects audio waveforms. These data are then sent to the server via a dedicated communication protocol.

[0744] Step 2: Analyzing behavioral patterns and emotions

[0745] The server applies machine learning algorithms and emotion engines to the collected data to identify student behavioral patterns and emotions.

[0746] Input: Stored video and audio data.

[0747] Output: Identification of behavioral patterns and emotional state data.

[0748] How it works: The server uses OpenCV and Dlib libraries to analyze facial expressions, and librosa to recognize tone of voice and emotions from audio data.

[0749] Step 3: Detecting Abnormal Behavior

[0750] The server automatically detects abnormal behavior based on the results of behavioral analysis and the emotion engine.

[0751] Input: Behavioral patterns and emotional state data.

[0752] Output: Abnormal behavior detection result.

[0753] Specific behaviors: Apply machine learning models to identify anomalous behaviors, such as aggressive behavior or high-volume screaming, by comparing them with historical patterns of normal behavior.

[0754] Step 4: Scoring and visualization

[0755] The server scores the characteristics and relationships between the perpetrator and victim based on the detected abnormal behavior and emotional information, and visualizes the scored information in real time on a dashboard.

[0756] Input: Abnormal behavior detection results.

[0757] Output: Scored trait and relationship information, graphical display.

[0758] Specific actions: Comprehensively evaluate the frequency of abnormal behavior, the location of occurrence, and historical data of those involved, and display the results as heat maps and graphs on the dashboard.

[0759] Step 5: Automatic reporting

[0760] If the server detects abnormal behavior or a specific emotion, it reports the information to the administrator's terminal in real time.

[0761] Input: Abnormal behavior detection results, scored information.

[0762] Output: Notification message to administrator terminal.

[0763] Specific operation: Use the reporting system to send detailed information about abnormal behavior via SMS, email, or the notification function of a dedicated app.

[0764] Step 6: Warning display

[0765] The terminal receives the notification message sent from the server and displays a warning in a pop-up notification.

[0766] Input: Notification message from the server.

[0767] Output: Display of warning notification.

[0768] Specific operation: A warning message will pop up on the device screen and an audio alert will also be issued.

[0769] Step 7: Check the situation

[0770] The device allows the notified administrator to view detailed status information via a dashboard.

[0771] Input: Report message, scored information, video data.

[0772] Output: Display detailed status information.

[0773] How it works: Administrators can view real-time video streams and historical data on a dashboard.

[0774] Step 8: Instructions

[0775] Managers can select the appropriate action from the response options on the dashboard and either head to the scene or give instructions to other staff.

[0776] Input: Dashboard status information.

[0777] Output: Decision on response instructions.

[0778] Specific actions: The administrator presses buttons such as "Rush to the scene" or "Notify staff" on the dashboard and instructs the necessary response.

[0779] (Application example 2)

[0780] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0781] Conventional security systems have difficulty detecting suspicious behavior, particularly in analyzing emotional changes in real time and associating them with suspicious behavior. This makes it difficult to respond quickly and, as a result, to provide a safe and secure environment. Furthermore, because scoring and visualization are not performed, it is not possible to intuitively understand the characteristics and relationships of those involved.

[0782] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0783] In this invention, the server includes means for receiving data from the monitoring device and the audio sensing device, means for analyzing people's behavioral patterns and emotions in real time based on the received data, means for detecting abnormal behavior and changes in emotions from the analysis results, means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and changes in emotions, means for visualizing and displaying the scored information, and means for notifying security personnel of the detected abnormal behavior and changes in emotions. This allows abnormal behavior and changes in emotions to be quickly detected and notified to security personnel, enabling a prompt response. Furthermore, visualizing the scored information allows users to intuitively understand the characteristics and relationships of the people involved.

[0784] A "monitoring device" is a device that collects video data in real time and monitors specific locations or people.

[0785] An "audio sensing device" is a device that collects surrounding audio data in real time and uses it for analysis.

[0786] A "behavioral pattern" is a repetitive pattern of actions or behaviors that a particular person performs on a daily basis.

[0787] "Emotions" are psychological states detected from facial expressions and tone of voice, and include joy, anger, sadness, fear, etc.

[0788] "Means for real-time analysis" refers to a method for processing received data immediately and obtaining results without delay.

[0789] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior, including aggressive behavior and loud shouting.

[0790] "Emotional shift" refers to an abrupt change in the observed individual's emotions, especially including changes to stronger emotional states (e.g., fear, anger, etc.).

[0791] "Scoring" is a method for evaluating and visualizing specific characteristics and relationships using numbers or indicators.

[0792] "Visualization" is the act of displaying data or information in a graphical format (e.g., a dashboard) to make it intuitively easier to understand.

[0793] "Security personnel" are professional personnel who monitor and respond to suspicious or abnormal behavior in order to ensure safety within facilities and buildings.

[0794] "Reporting" is the act of immediately notifying security personnel about detected abnormal behavior or emotional changes.

[0795] This invention is a system for detecting suspicious behavior and emotional changes and responding quickly in office buildings, commercial facilities, etc. The system includes means for receiving and analyzing data from monitoring devices and audio sensing devices. It also includes means for detecting, scoring, and visualizing abnormal behavior and emotional changes. Finally, it includes means for reporting the detected information to security personnel.

[0796] System Program Overview

[0797] The server receives real-time video and audio data from surveillance and audio sensing devices installed in office buildings and commercial facilities. This data is stored with a timestamp and used for analytical processing. The server then uses machine learning algorithms and emotion engines (e.g., TensorFlow, Keras, etc.) to perform facial recognition and audio analysis on the received data and identify behavioral patterns and emotions.

[0798] Analyzing data and detecting abnormal behavior and emotional changes

[0799] The server applies analysis algorithms and emotion engines to identify the target person's behavioral patterns and emotional state. For example, it recognizes emotional states (such as joy, anger, sadness, and fear) through facial expression and voice analysis. It then automatically detects abnormal behavior and sudden emotional changes based on the behavioral pattern and emotion analysis results. Examples of abnormal behavior include aggressive behavior and loud shouting.

[0800] Scoring and visualization process

[0801] The server scores the characteristics and relationships of the perpetrator and victim based on abnormal behavior and emotional information. For example, it quantifies each person's behavior and emotional state and displays this information in real time on a dashboard. This allows security personnel to easily grasp the situation and respond quickly.

[0802] Reporting function

[0803] When abnormal behavior or specific emotions are detected, the server sends a real-time notification to the security officer's smartphone. The notification includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.). This notification function is realized using the Twilio API or other notification systems.

[0804] Specific examples

[0805] Example 1: Detecting suspicious behavior and emotional changes in the office

[0806] Video data is received from surveillance cameras in an office building, and an analysis algorithm detects suspicious behavior in which multiple people surround one person. Furthermore, a strong expression of fear is detected from the facial expressions of the surrounded individuals. This situation is determined to be abnormal behavior, and a report is immediately sent to security personnel. An example of a prompt statement is, "Several people have surrounded one individual in an office building, and expressions of fear have been detected. Please check the scene immediately."

[0807] This system allows for real-time monitoring and analysis of suspicious behavior and sudden emotional changes within facilities and buildings, enabling efficient and rapid response, thereby providing a safe and secure environment at all times.

[0808] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0809] Step 1: Receiving data

[0810] The server receives real-time video and audio data from the monitoring and audio sensing devices. This data is time-stamped to identify when each input was collected. The input data are video frames and audio waveforms, which are then ready for the server to use in the next processing step.

[0811] Step 2: Preprocessing the data

[0812] The server converts the received video data to grayscale and denoises the audio data, improving the accuracy of the analysis. The video frames and audio waveforms received as input are converted into clean video frames and denoised audio waveforms after preprocessing.

[0813] Step 3: Face and voice detection

[0814] The server detects face areas from preprocessed video frames and identifies the sound source from the audio data. It uses OpenCV's face detection algorithm for face detection and a sound source localization algorithm for audio source identification. It receives preprocessed video frames and audio waveforms as input data, and obtains face areas and sound source locations as output.

[0815] Step 4: Analyzing behavioral patterns and emotions

[0816] The server analyzes the emotions of the identified facial regions using machine learning models (using TensorFlow and Keras) and identifies the person's emotional state through audio analysis. The analysis recognizes emotions such as joy, anger, sadness, and fear from facial expressions and tone of voice. It receives facial regions and audio data as input and obtains the determined emotional state as output.

[0817] Step 5: Detecting abnormal behavior and emotional changes

[0818] The server detects abnormal behavior and sudden emotional changes based on the analysis results. For example, this includes aggressive behavior, loud shouting, and strong emotions such as fear. It receives the analyzed emotional state and behavioral patterns as input, and obtains the detection results of abnormal behavior and emotional changes as output.

[0819] Step 6: Scoring and visualization

[0820] The server scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and emotional information. The scored information is displayed graphically on a dashboard. The server receives the detection results of abnormal behavior and emotional changes as input, and obtains the scored information and a visualized graphical dashboard display as output.

[0821] Step 7: Report

[0822] The server notifies the security personnel of detected abnormal behaviors and emotional changes in real time via their smartphones. The notification information includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.). It receives the scored and visualized information as input and generates a notification message for the security personnel as output.

[0823] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0824] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0825] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0826] [Third embodiment]

[0827] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0828] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0829] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0830] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0831] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0832] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0833] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0834] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0835] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0836] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0837] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0838] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0839] This invention relates to a system for detecting bullying in schools and responding quickly. This system receives and analyzes data from surveillance cameras and sensors installed in schools to understand students' behavioral patterns in real time and detect abnormal behavior early.

[0840] System Configuration

[0841] The system consists of the following components:

[0842] 1. Surveillance cameras and sensors:

[0843] Video and audio data is collected in real time from these devices installed in various locations throughout the school.

[0844] 2. Data receiving device (server):

[0845] The received digital data is managed centrally and any necessary pre-processing is performed.

[0846] 3. Behavioral analysis system (server):

[0847] Analyze data in real time to identify student behavior patterns.

[0848] 4. Abnormal behavior detection system (server):

[0849] Abnormal behavior is detected from the results of behavioral analysis and necessary processing is carried out.

[0850] 5. Scoring and visualization system (server):

[0851] Based on the results of abnormal behavior detection, the characteristics and relationships of the perpetrator and victim are scored and displayed graphically.

[0852] 6. Reporting system (server and terminal):

[0853] Any abnormal behavior detected is automatically reported to the instructor's device.

[0854] System program processing

[0855] Server-side processing

[0856] 1. Data Collection

[0857] The server receives real-time video and audio data from surveillance cameras and sensors located within the school, which is then stored with a timestamp and used for subsequent analysis.

[0858] 2. Behavioral pattern analysis

[0859] The server analyzes the student's movements and voice based on the received data, using machine learning algorithms to distinguish between normal and abnormal behavioral patterns.

[0860] 3. Detecting Abnormal Behavior

[0861] The server automatically detects abnormal behavior, such as aggressive behavior or loud shouting, based on the results of behavioral pattern analysis.

[0862] 4. Scoring and visualization

[0863] The server then scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, and this information is visualized on a dashboard that is updated in real time, making it easy for instructors to check.

[0864] 5. Automatic reporting

[0865] If the server detects any abnormal behavior, it sends that information to the instructor's device in real time. The report includes detailed information about the detected behavior, enabling a prompt response.

[0866] Terminal side processing

[0867] 1. Warning display

[0868] The terminal receives the notification message sent from the server in real time and displays it as a pop-up notification, allowing the instructor to immediately know when a problem has occurred.

[0869] 2. Check the situation

[0870] The device displays detailed information about detected abnormal behaviors via a dashboard, allowing instructors to accurately grasp the situation by checking real-time video footage and scoring information.

[0871] 3. Instructions for response

[0872] Instructors can select response options from a dashboard on the device and issue instructions to implement necessary measures, enabling swift and appropriate responses.

[0873] User processing

[0874] 1. Check the warning

[0875] Instructors can check notifications on devices to obtain detailed information about behaviors that are likely to be bullying, and a dashboard makes it easy to determine the severity of the problem and identify those involved.

[0876] 2. Implementation of measures

[0877] Instructors can go to the scene of abnormal behavior and quickly provide guidance to the perpetrator and protection to the victim, thereby helping to resolve the problem early.

[0878] Specific examples

[0879] Example 1: Detecting bullying in the hallway

[0880] The server receives video data from surveillance cameras installed in the hallways. An analysis algorithm detects situations in which multiple students are surrounding a single student and pushing against each other. If this behavior is detected as abnormal, the server immediately generates a warning message and sends it to the instructor's device. The instructor checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[0881] Example 2: Detecting verbal abuse in the classroom

[0882] The server receives audio data from microphones installed in classrooms. The analysis system detects the sound of a student shouting at a high volume and determines that this is verbal violence against other students. When this abnormal behavior is detected, a warning message is immediately sent to the instructor's device, who is notified and checks the detailed situation in the classroom. They then head to the scene, provide appropriate guidance according to the situation, and protect the victim.

[0883] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[0884] The processing flow will be explained below.

[0885] Step 1:

[0886] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, and the data is stored in a database with a timestamp for subsequent analysis.

[0887] Step 2:

[0888] The server preprocesses the received video and audio data, removing noise, correcting low-resolution areas, and making the audio clearer. This improves the quality of the data and increases the accuracy of analysis.

[0889] Step 3:

[0890] The server then applies machine learning algorithms to the pre-processed data to analyze student behavior patterns, specifically identifying facial and body movements, voice tone and intensity, and identifying aggressive or unnatural behavior.

[0891] Step 4:

[0892] The server automatically detects abnormal behavior from the analysis results. In this step, it compares the results with normal behavior patterns that have been trained in advance, and marks any deviations as abnormal behavior.

[0893] Step 5:

[0894] When abnormal behavior is detected, the server collects information about the students involved in the behavior and assigns a score to the characteristics and relationships of the perpetrator and victim. This score is calculated based on the frequency of the behavior and past records.

[0895] Step 6:

[0896] The server visualizes the scoring information and detected abnormal behavior on a dashboard that is updated in real time, allowing instructors to check detailed information about the problem.

[0897] Step 7:

[0898] When abnormal behavior is detected, the server immediately generates a warning message and automatically sends it to the instructor's device. The message includes detailed information about the detected behavior (location, time, involved parties, etc.).

[0899] Step 8:

[0900] The device receives the notification message sent from the server and displays a pop-up notification to the instructor in real time, allowing the instructor to immediately know of any problems.

[0901] Step 9:

[0902] The user (instructor) checks the warning notification for abnormal behavior and checks detailed situation information on the dashboard. Based on the video and scoring information, the situation can be accurately understood.

[0903] Step 10:

[0904] The user (instructor) can select the appropriate action from the response options on the dashboard and respond quickly by heading to the scene or issuing instructions to other staff.

[0905] Through the above steps, this system can detect bullying within schools early and respond quickly and appropriately.

[0906] Example 1

[0907] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0908] In recent years, bullying in schools has become a serious problem, necessitating early detection and rapid response. However, conventional monitoring devices and methods have issues in that they are unable to adequately detect abnormal behavior in real time, analyze behavior in detail, or respond quickly. Furthermore, it is difficult to simultaneously detect abnormal behavior and identify the specific characteristics and relationships of the perpetrator and victim.

[0909] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0910] In this invention, the server includes a means for receiving data from monitoring devices and detectors within the school, a means for analyzing the behavioral patterns of monitored individuals in real time based on the received data, and a means for detecting abnormal behavior from the analysis results. This enables early detection of abnormal behavior, scoring the characteristics and relationships of perpetrators and victims, and visualizing and displaying them in real time. Furthermore, detected abnormal behavior can be reported to an administrator, enabling a prompt response.

[0911] "Monitoring equipment" refers to devices such as surveillance cameras and microphones within a school that collect video and audio data in real time.

[0912] A "detector" is a device that consists of a sensor, a data receiving device, etc., and receives data from a monitoring device and detects abnormal behavior.

[0913] The "means for receiving data" refers to a means by which the server receives video and audio data transmitted from a monitoring device or detector in real time.

[0914] "Means for analyzing behavioral patterns in real time" refers to a means for analyzing student movements and voices based on received data and distinguishing between normal and abnormal behavioral patterns.

[0915] The "means for detecting abnormal behavior" is a means for automatically detecting abnormal behavior such as specific aggressive behavior or loud shouting from the results of analyzing behavioral patterns.

[0916] The "scoring method" is a method for quantitatively evaluating and scoring the characteristics and relationships of actors and influencers based on detected abnormal behavior.

[0917] The "means for visualizing and displaying" refers to a means for displaying scored information in real time using a visualization tool such as a dashboard.

[0918] The "means of reporting" is a means of notifying the administrator's terminal in real time of detailed information about detected abnormal behavior, and encouraging a prompt response.

[0919] The present invention is a system for detecting bullying in schools and responding quickly. This system receives and analyzes data from monitoring devices and detectors installed in schools to understand students' behavioral patterns in real time and detect abnormal behavior early. Specific embodiments of this system are described below.

[0920] Hardware and Software Configuration

[0921] Monitoring and Detectors

[0922] Surveillance cameras and microphones will be installed throughout the school to collect video and audio data, for example in classrooms, hallways, and other areas where bullying is likely to occur.

[0923] Data receiving device (server)

[0924] The server receives real-time data transmitted from surveillance cameras and microphones throughout the school, which is then stored with a timestamp and used for subsequent analysis.

[0925] Behavioral analysis system (server)

[0926] The server analyzes the received video and audio data using machine learning algorithms (e.g., TensorFlow), which analyze students' movements and voices to identify normal and abnormal behavioral patterns.

[0927] Abnormal behavior detection system (server)

[0928] The server automatically detects abnormal behavior based on the results of behavioral analysis, such as multiple students surrounding one student or loud shouting.

[0929] Scoring and visualization system (server)

[0930] The server scores the characteristics and relationships of the perpetrator and victim based on the detected anomalous behavior, and this information is displayed on a dashboard (e.g., Grafana) that updates in real time so that stakeholders can easily view it.

[0931] Reporting system (server and terminal)

[0932] If the server detects any abnormal behavior, it will report the information to the administrator's device in real time. The reporting system uses a real-time notification service such as Twilio, which enables the immediate transmission of information.

[0933] Specific examples

[0934] Example 1: Detecting bullying in the hallway

[0935] The server receives video data from surveillance cameras installed in the hallways. An analysis algorithm detects situations in which multiple students are surrounding and pushing around one student. If this behavior is detected as abnormal, the server immediately generates a warning message and sends it to the administrator's device. The administrator checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[0936] Example 2: Detecting verbal abuse in the classroom

[0937] The server receives audio data from microphones installed in classrooms. The analysis system detects the sound of a student shouting at a high volume and determines that this is verbal violence against other students. When this abnormal behavior is detected, a warning message is immediately sent to the administrator's device, who is notified and checks the detailed situation in the classroom. They then head to the scene, provide appropriate guidance according to the situation, and protect the victim.

[0938] Prompt Sentence Examples

[0939] To test and improve this system, you can feed the generative AI model prompts like:

[0940] "Please explain the algorithm that will detect abnormal behavior when multiple students are pushing against one another in the hallway."

[0941] "How can we detect high-volume shouting in a classroom and identify it as verbal abuse?"

[0942] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[0943] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0944] Step 1: Data collection

[0945] The server receives video and audio data in real time from monitoring devices and detectors installed within the school. The input data is video and audio from surveillance cameras and microphones. The received data is saved with a timestamp and used for subsequent analysis. Specifically, video captured by cameras installed in the hallways and audio collected by microphones in the classrooms are sent to the server.

[0946] Step 2: Preprocessing

[0947] The server performs preprocessing on the received raw data. The input data is the video and audio data collected in step 1. Specific preprocessing steps include noise removal, data normalization, and timestamp alignment. The output data is clean data ready to be input into the analysis system. For example, unnecessary frames are removed from the video data and background noise is removed from the audio data.

[0948] Step 3: Behavioral pattern analysis

[0949] The server analyzes behavioral patterns based on the preprocessed data. The input data is the clean video and audio data generated in step 2. A machine learning algorithm (e.g., TensorFlow) is used to analyze students' movements and audio patterns and distinguish between normal and abnormal behavioral patterns. The output data is digital features related to each student's behavioral pattern. Specifically, the server analyzes the student's position and movements for each frame of video data and detects shouts from the waveform of the audio data.

[0950] Step 4: Detecting Abnormal Behavior

[0951] The server detects abnormal behavior based on the analysis results. The input data are the digital features of the behavioral patterns generated in step 3. Abnormal behavior includes aggressive behavior and loud shouting. For example, if a specific student behaves aggressively toward another student within a specific time period, this is detected as abnormal behavior. The output data is the abnormal behavior detection results and detailed information. Specifically, information such as "Student B behaved aggressively toward Student C in classroom A at XX:YY" is detected.

[0952] Step 5: Scoring and visualization

[0953] The server scores the characteristics and relationships of the perpetrator and victim based on the data of detected abnormal behavior. The input data is the abnormal behavior detection results generated in step 4. The scored information is displayed on a dashboard that is updated in real time. The output data is visualized score information. Specifically, the behavioral history and relationships of the perpetrator and victim are displayed as graphs and charts.

[0954] Step 6: Automatic reporting

[0955] The server sends a real-time report to the administrator's device based on the information on the detected abnormal behavior. The input data is the abnormal behavior and its scoring data generated in steps 4 and 5. The reporting system uses a real-time notification service such as Twilio. The output data is a report message. This message contains detailed information about the abnormal behavior and is displayed on the administrator's device. For example, a message such as "Student B is behaving aggressively toward Student C in classroom A" is sent.

[0956] Step 7: Warning Display

[0957] The terminal receives the report message sent from the server in real time and displays a pop-up notification. The input data is the report message sent in step 6. The notification contains detailed information about the abnormal behavior, which the administrator can check immediately. The output data is a pop-up notification that is displayed so that the administrator can check it.

[0958] Step 8: Check the situation

[0959] The terminal displays detailed information about the detected abnormal behavior through a dashboard. The input data is the scored information generated in step 5. Real-time video and scored data are displayed, allowing the administrator to accurately grasp the situation based on this. The output data is detailed situation information that is displayed to the administrator. Specifically, a dashboard screen is provided that allows the location, time, and involved parties of the problem to be identified.

[0960] Step 9: Instructions

[0961] The user selects a response option from the device dashboard and issues instructions to implement the necessary measures. The input data is detailed information about the abnormal behavior confirmed in step 8. The output data is the selected response option. Specifically, options such as "visit the scene," "contact the parents," and "notify other instructors" can be selected, and a prompt response will be implemented.

[0962] Step 10: Implementing the measures

[0963] The user goes to the scene where the abnormal behavior is occurring and quickly instructs the perpetrator and protects the victim. The input data is the response option selected in step 9. The output data is the instructions actually taken and the results. Specifically, the user goes to classroom A and takes action to stop the bullying. Counseling and protective measures are also implemented for those involved.

[0964] (Application example 1)

[0965] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0966] Bullying in schools is a serious problem, requiring early detection and rapid response. However, comprehensive detection and immediate response to bullying are difficult using current manual monitoring and approaches alone. Monitoring and responding to bullying is particularly challenging in large schools and dispersed school environments, making it essential to implement a smooth system to ensure student safety. Furthermore, analyzing data obtained from surveillance cameras and sensors requires advanced technology, and an efficient system is needed to detect abnormal behavior in real time and respond quickly.

[0967] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0968] In this invention, the server includes means for receiving data from surveillance cameras and sensors within the school, means for analyzing student behavior patterns in real time based on the received data, means for detecting abnormal behavior from the analysis results, means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, means for visualizing and displaying the scored information, and means for displaying detailed information about the abnormal behavior on a display device worn by an instructor who receives a notification of the abnormal behavior. This makes it possible to detect bullying and abnormal behavior within the school in real time and respond immediately. Instructors can check detailed information about the abnormal behavior on the display device and take prompt and appropriate measures.

[0969] A "surveillance camera" is a device that is installed in a specific area and is used to capture video.

[0970] A "sensor" is a device that detects physical changes and converts them into electrical signals.

[0971] The "data receiving means" is a device or system for receiving data acquired from the surveillance cameras and sensors.

[0972] The "behavior pattern analysis means" is a system that has the function of analyzing student behavior based on received data and identifying specific patterns.

[0973] The "abnormal behavior detection means" is a system for automatically identifying behavior that is different from normal based on the results of analyzing behavior patterns.

[0974] The "scoring means" is a system that has the function of quantifying the characteristics and relationships of the perpetrator and victim based on detected abnormal behavior.

[0975] The "visualization display means" is a device or software for graphically displaying the scored information.

[0976] The "reporting means" is a device or function for notifying the trainer of information about detected abnormal behavior.

[0977] The "display device" is a device that displays detailed information about abnormal behavior so that the trainer can check it.

[0978] The present invention is a system for detecting bullying in schools and responding quickly, and includes a series of processes that receive data from surveillance cameras and sensors, perform behavior analysis, and detect abnormal behavior in real time. Specific embodiments for implementing this system are described below.

[0979] System construction

[0980] This system consists of the following main components:

[0981] 1. Surveillance cameras and sensors:

[0982] Video and audio data is collected in real time using surveillance cameras and audio sensors (e.g., Logitech C920 cameras and Blue Yeti microphones) installed throughout the school.

[0983] 2. Data receiving means:

[0984] A server is used to receive and centrally manage data sent from each surveillance camera and sensor.

[0985] 3. Behavioral pattern analysis methods:

[0986] The server uses deep learning frameworks such as TensorFlow to analyze the data it receives and identify student behavior patterns, using machine learning algorithms to distinguish between normal and abnormal behavior patterns.

[0987] 4. Abnormal behavior detection methods:

[0988] The server automatically detects abnormal behavior based on the analyzed behavioral patterns, such as aggressive behavior or loud shouting.

[0989] 5. Scoring Procedure:

[0990] When abnormal behavior is detected, the server scores the characteristics and relationships of the perpetrator and victim, clarifying the severity of the problem and the relationships between the parties involved.

[0991] 6. Visualization display means:

[0992] Scoring information is displayed on a real-time updated dashboard or display device (e.g., smart glasses), allowing trainers to view detailed information about abnormal behavior.

[0993] 7. Reporting methods:

[0994] When abnormal behavior is detected, the server sends the information to the instructor's device in real time. The report includes detailed information about the abnormal behavior, enabling a prompt response.

[0995] Explanation of program processing

[0996] The system collects data from surveillance cameras and audio sensors in real time and preprocesses the received data on a server. It uses TensorFlow to analyze video data and identify abnormal behavior, while Scikit-learn analyzes audio data. When abnormal behavior is detected, the information is immediately sent to the instructor's terminal and display device (such as smart glasses).

[0997] Specific examples

[0998] For example, if a surveillance camera captures multiple students surrounding one another in a hallway, the TensorFlow model will detect this abnormal behavior. At this point, the server will immediately generate an alert message and notify the instructor's smart glasses. The instructor can then view the situation in real time through the glasses and rush to the scene to take appropriate action.

[0999] Prompt Sentence Examples

[1000] You can use the following prompt for the generative AI model: "Train a video analytics model using a dataset for detecting anomalous behavior and generate Python code to detect suspicious behavior and emergencies in real time."

[1001] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1002] Step 1:

[1003] Data collection

[1004] The server collects real-time video and audio data from surveillance cameras and sensors (e.g., Logitech C920 cameras, Blue Yeti microphones) installed within the school. The input data are video frames and audio samples, which are sent to the server along with timestamps. The output data are time-stamped video and audio data.

[1005] Step 2:

[1006] Data Preprocessing

[1007] The server performs preprocessing on the collected video and audio data. This processing includes noise removal and data normalization. The input is the collected raw data, and the output is preprocessed video and audio data. Specifically, OpenCV is used to resize the video frames and perform noise filtering, and Scipy is used to remove noise from the audio data.

[1008] Step 3:

[1009] Behavioral pattern analysis

[1010] The server analyzes behavioral patterns based on the preprocessed data. This analysis uses a deep learning model using TensorFlow. The input is preprocessed video and audio data, and the output is the analysis results of behavioral patterns. Specifically, the server extracts video data frame by frame and inputs it into the model to classify normal and abnormal behavior.

[1011] Step 4:

[1012] Abnormal behavior detection

[1013] The server detects anomalous behavior from the results of behavioral pattern analysis. This detection uses a machine learning algorithm to identify suspicious behavior. The input is the analyzed behavioral pattern data, and the output is a judgment result on whether anomalous behavior has been detected. Specifically, it uses Scikit-learn's classification algorithm to identify aggressive behavior and high-volume shouting.

[1014] Step 5:

[1015] Scoring abnormal behavior

[1016] The server scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior. The input is the abnormal behavior detection result, and the output is the scored data. Specifically, it applies an algorithm that quantifies information such as the degree of abnormal behavior, the number of people involved, and frequency.

[1017] Step 6:

[1018] Information Visualization

[1019] The scored information is visualized on a display device such as a dashboard or smart glasses that is updated in real time. The input is the scored data, and the output is the information displayed graphically. Specifically, a JavaScript framework (e.g., D3.js) is used to display the information in graphs and charts on the dashboard.

[1020] Step 7:

[1021] Reporting Abnormal Behavior

[1022] When abnormal behavior is detected, the server immediately notifies the instructor's terminal or display device. The input is detailed information about the abnormal behavior, and the output is a warning message displayed on the instructor's terminal. Specific operations include a function to display a pop-up notification on the smart glasses and an API to send a push notification to the smartphone.

[1023] Prompt Sentence Examples

[1024] You can use the following prompt for the generative AI model: "Train a video analytics model using a dataset for detecting anomalous behavior and generate Python code to detect suspicious behavior and emergencies in real time."

[1025] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1026] This invention relates to a system for detecting bullying in schools and responding quickly. In particular, this system analyzes students' behavioral patterns and emotions in real time, detects abnormal behavior and emotional changes, and takes necessary measures. This system is composed of the following components:

[1027] System Configuration

[1028] 1. Surveillance cameras and sensors:

[1029] They are installed in various locations throughout the school and collect video and audio data in real time, allowing students' behavior and the surrounding sound environment to be monitored.

[1030] 2. Data receiving device (server):

[1031] The data sent from surveillance cameras and sensors is centrally managed and the necessary pre-processing is performed.

[1032] 3. Behavioral analysis system (server):

[1033] The received data is analyzed in real time to identify student behavioral patterns and emotions.

[1034] 4. Emotion engine (server):

[1035] Based on video and audio data, the system analyzes students' facial expressions and tone of voice to recognize their emotions. The recognized emotional information is added to the analysis results of their behavioral patterns.

[1036] 5. Abnormal behavior detection system (server):

[1037] Abnormal behavior is detected based on the results of behavioral analysis and the emotion engine, and necessary processing is carried out.

[1038] 6. Scoring and visualization system (server):

[1039] Based on detected abnormal behavior and recognized emotions, the characteristics and relationships of the perpetrator and victim are scored and displayed graphically.

[1040] 7. Reporting system (server and terminal):

[1041] Detected abnormal behavior and emotional information is automatically reported to the instructor's device.

[1042] System program processing

[1043] Server-side processing

[1044] 1. Data Collection

[1045] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, which is then stored with a timestamp and used for analytical processing.

[1046] 2. Behavioral and emotional analysis

[1047] The server applies machine learning algorithms and emotion engines to the received data to identify students' behavioral patterns and emotions, specifically recognizing emotional states (such as joy, anger, sadness, and fear) through facial expression and voice analysis.

[1048] 3. Detecting Abnormal Behavior

[1049] The server automatically detects abnormal behavior based on the analysis of behavioral patterns and emotions. For example, it detects aggressive behavior or loud shouting and determines these as abnormal behavior.

[1050] 4. Scoring and visualization

[1051] The server scores the characteristics and relationships of the perpetrator and victim based on abnormal behavior and emotional information. This information is visualized in real time on a dashboard, allowing instructors to easily understand the situation.

[1052] 5. Automatic reporting

[1053] When abnormal behavior or specific emotions are detected, the server sends a real-time report to the instructor's device, including detailed information about the detected behavior (location, time, involved parties, emotional state, etc.).

[1054] Terminal side processing

[1055] 1. Warning display

[1056] The device receives the notification message sent from the server and displays a real-time warning as a pop-up notification, allowing the instructor to immediately know of any problems.

[1057] 2. Check the situation

[1058] The device displays detailed situational information (video, score, emotional information) via a dashboard, allowing the instructor to quickly grasp the situation and consider appropriate responses.

[1059] 3. Instructions for response

[1060] Instructors can select the appropriate action from the response options on the dashboard and either head to the scene or give instructions to other staff.

[1061] User processing

[1062] 1. Check the warning

[1063] Teachers can view notifications on devices and get detailed information about behaviors and emotions that may be indicative of bullying. A dashboard makes it easy to determine the severity of the problem and identify those involved.

[1064] 2. Implementation of measures

[1065] Instructors can go to the scene where abnormal behavior and emotional changes are occurring and quickly provide guidance to the perpetrator and protection to the victim, thereby helping to resolve the problem early.

[1066] Specific examples

[1067] Example 1: Detecting bullying and emotional changes in the hallway

[1068] The server receives video data from surveillance cameras installed in the hallways. Using an analysis algorithm and emotion engine, it detects the emotion of fear from the facial expression of the student being pushed, as multiple students surround and push against one another. If this behavior and emotional information is detected as abnormal, the server immediately generates a warning message and sends it to the instructor's device. The instructor checks the pop-up notification on their device, understands the situation on the dashboard, and rushes to the scene to take appropriate action.

[1069] Example 2: Detecting verbal abuse and emotional changes in the classroom

[1070] The server receives audio data from microphones installed in classrooms. The analysis system and emotion engine detect the high-pitched voice of a student shouting and the sadness of a student believed to be the victim. If this is recognized as abnormal behavior such as verbal violence, a warning message is immediately sent to the instructor's device. The instructor is notified and checks the details of the situation in the classroom, then heads to the scene to provide appropriate guidance according to the situation and protect the victim.

[1071] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[1072] The processing flow will be explained below.

[1073] Step 1:

[1074] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, which is then stored in a database with a timestamp for subsequent analysis.

[1075] Step 2:

[1076] The server preprocesses the received video and audio data, removing noise, correcting low-resolution areas, and making the audio clearer. This improves the quality of the data and increases the accuracy of analysis.

[1077] Step 3:

[1078] The server then applies machine learning algorithms to the pre-processed data to analyze student behavioral patterns, such as facial and body movements, and vocal tone and intensity, to identify aggressive or unnatural behavior.

[1079] Step 4:

[1080] The server analyzes behavioral patterns and uses an emotion engine to recognize emotions from students' facial expressions and tone of voice, categorizing them as joy, anger, sadness, fear, and so on.

[1081] Step 5:

[1082] The server automatically detects abnormal behavior by combining the results of behavioral pattern analysis and the emotion engine's recognition results. For example, if aggressive behavior and angry emotions are detected simultaneously, the server marks the behavior as abnormal.

[1083] Step 6:

[1084] When abnormal behavior is detected, the server collects information about the students related to the behavior and emotions, and then scores the characteristics and relationships of the perpetrator and victim. This score is calculated based on the frequency of the behavior, past records, type of emotion, etc.

[1085] Step 7:

[1086] The server visualizes the scoring information and detected abnormal behavior and emotional information on a dashboard that is updated in real time, allowing instructors to check detailed information about the problem.

[1087] Step 8:

[1088] When abnormal behavior or a specific emotion is detected, the server immediately generates a warning message and automatically sends it to the instructor's device. The message includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.).

[1089] Step 9:

[1090] The device receives the notification message sent from the server and displays a pop-up notification to the instructor in real time, allowing the instructor to immediately know of any problems.

[1091] Step 10:

[1092] The user (instructor) checks the warning notification for abnormal behavior and checks detailed situation information on the dashboard. Based on the video, score information, and emotional information, the situation can be accurately grasped.

[1093] Step 11:

[1094] The user (instructor) can select the appropriate action from the response options on the dashboard and respond quickly by heading to the scene or issuing instructions to other staff.

[1095] Through the above steps, this system can detect bullying within schools early and respond quickly and appropriately.

[1096] Example 2

[1097] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1098] Bullying in schools has long-term mental and physical effects on students, so early detection and rapid response are necessary. However, traditional methods tend to be slow to detect and address bullying behavior, making it difficult for relevant parties to actively intervene. For this reason, a system that can detect bullying behavior in real time and respond quickly is needed.

[1099] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving information from monitoring devices and sensors in the school, a means for analyzing the behavioral patterns of individuals in real time based on the received information, and a means for detecting abnormal behavior from the analysis results. This makes it possible to detect abnormal behavior in real time and take prompt and appropriate action.

[1100] "Surveillance equipment" refers to cameras and sensors installed to collect video data within a school.

[1101] "Sensors" refer to devices that collect sound and environmental data within the school.

[1102] "Information" refers to data such as student behavior and audio collected from monitoring devices and sensors.

[1103] "Individual" refers to a specific person, such as a student, who is the subject of monitoring within a school.

[1104] "Behavior patterns" refer to students' patterns of movement and behavior.

[1105] "Real-time" refers to information being processed and analyzed almost as soon as it is generated.

[1106] "Abnormal behavior" refers to inappropriate behavior, such as bullying or aggressive behavior, that deviates from normal patterns of behavior.

[1107] "Characteristics" refers to the behavioral and emotional characteristics of individual students.

[1108] "Relationship" refers to the human relationships and interactions between the perpetrator and the victim.

[1109] "Scoring" refers to quantifying behaviors and characteristics and converting them into numbers or ratings.

[1110] "Visualization" refers to visually displaying the scored information to make it easier to understand.

[1111] "Administrator" refers to faculty and staff members who are responsible for monitoring the system and dealing with abnormal behavior.

[1112] "Management screen" refers to the interface for displaying scored information and detected abnormal behavior.

[1113] The present invention relates to a system for realizing early detection of bullying in schools and a rapid response to the same. Specific embodiments of the system are described below.

[1114] System configuration

[1115] This system mainly uses the following hardware and software:

[1116] 1. Monitoring Devices and Sensors

[1117] Surveillance cameras and audio sensors will be installed throughout the school to collect student behavior and audio data in real time.

[1118] 2. Data receiving device (server)

[1119] The collected video and audio data is sent to a server via a dedicated communication protocol, where it is stored in a database with a timestamp for subsequent analysis.

[1120] 3. Behavioral analysis system (server)

[1121] The server applies machine learning algorithms to the collected data to identify students' behavioral patterns and emotions in real time, using libraries such as OpenCV and Dlib for facial expression analysis and librosa for voice analysis.

[1122] 4. Emotion engine (server)

[1123] Through facial expression analysis and voice analysis, it identifies students' emotional states (joy, anger, sadness, fear, etc.). The emotion engine analyzes the collected data and identifies emotional information.

[1124] 5. Abnormal behavior detection system (server)

[1125] Based on the results of behavioral analysis and the emotion engine, the system automatically detects abnormal behavior. Machine learning models identify anomalous behavior such as aggressive behavior or high-volume shouting.

[1126] 6. Scoring and visualization system (server)

[1127] Based on abnormal behavior and emotional information, the characteristics and relationships of the perpetrator and victim are scored. This scoring information is visualized in real time on a dashboard, allowing administrators to easily understand the situation.

[1128] 7. Reporting System (Server and Terminal)

[1129] Detected abnormal behavior and emotional information are automatically reported to the administrator's terminal. The report message includes detailed information about the abnormal behavior (location, time, involved parties, emotional state, etc.).

[1130] Specific examples

[1131] Example 1: Detecting bullying and emotional changes in the hallway

[1132] The server receives video data from surveillance cameras installed in the hallways. Using an analysis algorithm and emotion engine, it detects a group of students pushing around one another, and detects fear from the facial expression of the student being pushed. If this behavior and emotional information is detected as abnormal, the server immediately generates a warning message and sends it to the administrator's device. The administrator checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[1133] Example 2: Detecting verbal abuse and emotional changes in the classroom

[1134] The server receives audio data from microphones installed in classrooms. The analysis system and emotion engine detect the high-pitched voices of students shouting and the sadness of students believed to be victims. If this is recognized as abnormal behavior such as verbal violence, a warning message is immediately sent to the administrator's device. The administrator is notified and checks the details of the situation in the classroom, then heads to the scene to provide appropriate guidance according to the situation and protect the victim.

[1135] Prompt Sentence Examples

[1136] "If the system detects bullying behavior and emotional changes in the hallways, please explain specifically what data is collected and how it is analyzed."

[1137] This allows the generative AI model to provide a good explanation for the specific behavior of the system.

[1138] The present invention enables early detection of bullying and rapid response, thereby providing a safe learning environment for students.

[1139] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1140] Step 1: Data collection

[1141] The server receives real-time video and audio data from monitoring devices and sensors, and stores the data in a database with a timestamp.

[1142] Input: Video and audio data from surveillance cameras and audio sensors.

[1143] Output: Stored raw data with timestamp.

[1144] How it works: The surveillance camera captures video frames and the sensor collects audio waveforms. These data are then sent to the server via a dedicated communication protocol.

[1145] Step 2: Analyzing behavioral patterns and emotions

[1146] The server applies machine learning algorithms and emotion engines to the collected data to identify student behavioral patterns and emotions.

[1147] Input: Stored video and audio data.

[1148] Output: Identification of behavioral patterns and emotional state data.

[1149] How it works: The server uses OpenCV and Dlib libraries to analyze facial expressions, and librosa to recognize tone of voice and emotions from audio data.

[1150] Step 3: Detecting Abnormal Behavior

[1151] The server automatically detects abnormal behavior based on the results of behavioral analysis and the emotion engine.

[1152] Input: Behavioral patterns and emotional state data.

[1153] Output: Abnormal behavior detection result.

[1154] Specific behaviors: Apply machine learning models to identify anomalous behaviors, such as aggressive behavior or high-volume screaming, by comparing them with historical patterns of normal behavior.

[1155] Step 4: Scoring and visualization

[1156] The server scores the characteristics and relationships between the perpetrator and victim based on the detected abnormal behavior and emotional information, and visualizes the scored information in real time on a dashboard.

[1157] Input: Abnormal behavior detection results.

[1158] Output: Scored trait and relationship information, graphical display.

[1159] Specific actions: Comprehensively evaluate the frequency of abnormal behavior, the location of occurrence, and historical data of those involved, and display the results as heat maps and graphs on the dashboard.

[1160] Step 5: Automatic reporting

[1161] If the server detects abnormal behavior or a specific emotion, it reports the information to the administrator's terminal in real time.

[1162] Input: Abnormal behavior detection results, scored information.

[1163] Output: Notification message to administrator terminal.

[1164] Specific operation: Use the reporting system to send detailed information about abnormal behavior via SMS, email, or the notification function of a dedicated app.

[1165] Step 6: Warning display

[1166] The terminal receives the notification message sent from the server and displays a warning in a pop-up notification.

[1167] Input: Notification message from the server.

[1168] Output: Display of warning notification.

[1169] Specific operation: A warning message will pop up on the device screen and an audio alert will also be issued.

[1170] Step 7: Check the situation

[1171] The device allows the notified administrator to view detailed status information via a dashboard.

[1172] Input: Report message, scored information, video data.

[1173] Output: Display detailed status information.

[1174] How it works: Administrators can view real-time video streams and historical data on a dashboard.

[1175] Step 8: Instructions

[1176] Managers can select the appropriate action from the response options on the dashboard and either head to the scene or give instructions to other staff.

[1177] Input: Dashboard status information.

[1178] Output: Decision on response instructions.

[1179] Specific actions: The administrator presses buttons such as "Rush to the scene" or "Notify staff" on the dashboard and instructs the necessary response.

[1180] (Application example 2)

[1181] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1182] Conventional security systems have difficulty detecting suspicious behavior, particularly in analyzing emotional changes in real time and associating them with suspicious behavior. This makes it difficult to respond quickly and, as a result, to provide a safe and secure environment. Furthermore, because scoring and visualization are not performed, it is not possible to intuitively understand the characteristics and relationships of those involved.

[1183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1184] In this invention, the server includes means for receiving data from the monitoring device and the audio sensing device, means for analyzing people's behavioral patterns and emotions in real time based on the received data, means for detecting abnormal behavior and changes in emotions from the analysis results, means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and changes in emotions, means for visualizing and displaying the scored information, and means for notifying security personnel of the detected abnormal behavior and changes in emotions. This allows abnormal behavior and changes in emotions to be quickly detected and notified to security personnel, enabling a prompt response. Furthermore, visualizing the scored information allows users to intuitively understand the characteristics and relationships of the people involved.

[1185] A "monitoring device" is a device that collects video data in real time and monitors specific locations or people.

[1186] An "audio sensing device" is a device that collects surrounding audio data in real time and uses it for analysis.

[1187] A "behavioral pattern" is a repetitive pattern of actions or behaviors that a particular person performs on a daily basis.

[1188] "Emotions" are psychological states detected from facial expressions and tone of voice, and include joy, anger, sadness, fear, etc.

[1189] "Means for real-time analysis" refers to a method for processing received data immediately and obtaining results without delay.

[1190] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior, including aggressive behavior and loud shouting.

[1191] "Emotional shift" refers to an abrupt change in the observed individual's emotions, especially including changes to stronger emotional states (e.g., fear, anger, etc.).

[1192] "Scoring" is a method for evaluating and visualizing specific characteristics and relationships using numbers or indicators.

[1193] "Visualization" is the act of displaying data or information in a graphical format (e.g., a dashboard) to make it intuitively easier to understand.

[1194] "Security personnel" are professional personnel who monitor and respond to suspicious or abnormal behavior in order to ensure safety within facilities and buildings.

[1195] "Reporting" is the act of immediately notifying security personnel about detected abnormal behavior or emotional changes.

[1196] This invention is a system for detecting suspicious behavior and emotional changes and responding quickly in office buildings, commercial facilities, etc. The system includes means for receiving and analyzing data from monitoring devices and audio sensing devices. It also includes means for detecting, scoring, and visualizing abnormal behavior and emotional changes. Finally, it includes means for reporting the detected information to security personnel.

[1197] System Program Overview

[1198] The server receives real-time video and audio data from surveillance and audio sensing devices installed in office buildings and commercial facilities. This data is stored with a timestamp and used for analytical processing. The server then uses machine learning algorithms and emotion engines (e.g., TensorFlow, Keras, etc.) to perform facial recognition and audio analysis on the received data and identify behavioral patterns and emotions.

[1199] Analyzing data and detecting abnormal behavior and emotional changes

[1200] The server applies analysis algorithms and emotion engines to identify the target person's behavioral patterns and emotional state. For example, it recognizes emotional states (such as joy, anger, sadness, and fear) through facial expression and voice analysis. It then automatically detects abnormal behavior and sudden emotional changes based on the behavioral pattern and emotion analysis results. Examples of abnormal behavior include aggressive behavior and loud shouting.

[1201] Scoring and visualization process

[1202] The server scores the characteristics and relationships of the perpetrator and victim based on abnormal behavior and emotional information. For example, it quantifies each person's behavior and emotional state and displays this information in real time on a dashboard. This allows security personnel to easily grasp the situation and respond quickly.

[1203] Reporting function

[1204] When abnormal behavior or specific emotions are detected, the server sends a real-time notification to the security officer's smartphone. The notification includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.). This notification function is realized using the Twilio API or other notification systems.

[1205] Specific examples

[1206] Example 1: Detecting suspicious behavior and emotional changes in the office

[1207] Video data is received from surveillance cameras in an office building, and an analysis algorithm detects suspicious behavior in which multiple people surround one person. Furthermore, a strong expression of fear is detected from the facial expressions of the surrounded individuals. This situation is determined to be abnormal behavior, and a report is immediately sent to security personnel. An example of a prompt statement is, "Several people have surrounded one individual in an office building, and expressions of fear have been detected. Please check the scene immediately."

[1208] This system allows for real-time monitoring and analysis of suspicious behavior and sudden emotional changes within facilities and buildings, enabling efficient and rapid response, thereby providing a safe and secure environment at all times.

[1209] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1210] Step 1: Receiving data

[1211] The server receives real-time video and audio data from the monitoring and audio sensing devices. This data is time-stamped to identify when each input was collected. The input data are video frames and audio waveforms, which are then ready for the server to use in the next processing step.

[1212] Step 2: Preprocessing the data

[1213] The server converts the received video data to grayscale and denoises the audio data, improving the accuracy of the analysis. The video frames and audio waveforms received as input are converted into clean video frames and denoised audio waveforms after preprocessing.

[1214] Step 3: Face and voice detection

[1215] The server detects face areas from preprocessed video frames and identifies the sound source from the audio data. It uses OpenCV's face detection algorithm for face detection and a sound source localization algorithm for audio source identification. It receives preprocessed video frames and audio waveforms as input data, and obtains face areas and sound source locations as output.

[1216] Step 4: Analyzing behavioral patterns and emotions

[1217] The server analyzes the emotions of the identified facial regions using machine learning models (using TensorFlow and Keras) and identifies the person's emotional state through audio analysis. The analysis recognizes emotions such as joy, anger, sadness, and fear from facial expressions and tone of voice. It receives facial regions and audio data as input and obtains the determined emotional state as output.

[1218] Step 5: Detecting abnormal behavior and emotional changes

[1219] The server detects abnormal behavior and sudden emotional changes based on the analysis results. For example, this includes aggressive behavior, loud shouting, and strong emotions such as fear. It receives the analyzed emotional state and behavioral patterns as input, and obtains the detection results of abnormal behavior and emotional changes as output.

[1220] Step 6: Scoring and visualization

[1221] The server scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and emotional information. The scored information is displayed graphically on a dashboard. The server receives the detection results of abnormal behavior and emotional changes as input, and obtains the scored information and a visualized graphical dashboard display as output.

[1222] Step 7: Report

[1223] The server notifies the security personnel of detected abnormal behaviors and emotional changes in real time via their smartphones. The notification information includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.). It receives the scored and visualized information as input and generates a notification message for the security personnel as output.

[1224] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1225] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1226] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1227] [Fourth embodiment]

[1228] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1229] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1230] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1231] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1232] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1233] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1234] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1235] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1236] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1237] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1238] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1239] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1240] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1241] This invention relates to a system for detecting bullying in schools and responding quickly. This system receives and analyzes data from surveillance cameras and sensors installed in schools to understand students' behavioral patterns in real time and detect abnormal behavior early.

[1242] System Configuration

[1243] The system consists of the following components:

[1244] 1. Surveillance cameras and sensors:

[1245] Video and audio data is collected in real time from these devices installed in various locations throughout the school.

[1246] 2. Data receiving device (server):

[1247] The received digital data is managed centrally and any necessary pre-processing is performed.

[1248] 3. Behavioral analysis system (server):

[1249] Analyze data in real time to identify student behavior patterns.

[1250] 4. Abnormal behavior detection system (server):

[1251] Abnormal behavior is detected from the results of behavioral analysis and necessary processing is carried out.

[1252] 5. Scoring and visualization system (server):

[1253] Based on the results of abnormal behavior detection, the characteristics and relationships of the perpetrator and victim are scored and displayed graphically.

[1254] 6. Reporting system (server and terminal):

[1255] Any abnormal behavior detected is automatically reported to the instructor's device.

[1256] System program processing

[1257] Server-side processing

[1258] 1. Data Collection

[1259] The server receives real-time video and audio data from surveillance cameras and sensors located within the school, which is then stored with a timestamp and used for subsequent analysis.

[1260] 2. Behavioral pattern analysis

[1261] The server analyzes the student's movements and voice based on the received data, using machine learning algorithms to distinguish between normal and abnormal behavioral patterns.

[1262] 3. Detecting Abnormal Behavior

[1263] The server automatically detects abnormal behavior, such as aggressive behavior or loud shouting, based on the results of behavioral pattern analysis.

[1264] 4. Scoring and visualization

[1265] The server then scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, and this information is visualized on a dashboard that is updated in real time, making it easy for instructors to check.

[1266] 5. Automatic reporting

[1267] If the server detects any abnormal behavior, it sends that information to the instructor's device in real time. The report includes detailed information about the detected behavior, enabling a prompt response.

[1268] Terminal side processing

[1269] 1. Warning display

[1270] The terminal receives the notification message sent from the server in real time and displays it as a pop-up notification, allowing the instructor to immediately know when a problem has occurred.

[1271] 2. Check the situation

[1272] The device displays detailed information about detected abnormal behaviors via a dashboard, allowing instructors to accurately grasp the situation by checking real-time video footage and scoring information.

[1273] 3. Instructions for response

[1274] Instructors can select response options from a dashboard on the device and issue instructions to implement necessary measures, enabling swift and appropriate responses.

[1275] User processing

[1276] 1. Check the warning

[1277] Instructors can check notifications on devices to obtain detailed information about behaviors that are likely to be bullying, and a dashboard makes it easy to determine the severity of the problem and identify those involved.

[1278] 2. Implementation of measures

[1279] Instructors can go to the scene of abnormal behavior and quickly provide guidance to the perpetrator and protection to the victim, thereby helping to resolve the problem early.

[1280] Specific examples

[1281] Example 1: Detecting bullying in the hallway

[1282] The server receives video data from surveillance cameras installed in the hallways. An analysis algorithm detects situations in which multiple students are surrounding a single student and pushing against each other. If this behavior is detected as abnormal, the server immediately generates a warning message and sends it to the instructor's device. The instructor checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[1283] Example 2: Detecting verbal abuse in the classroom

[1284] The server receives audio data from microphones installed in classrooms. The analysis system detects the sound of a student shouting at a high volume and determines that this is verbal violence against other students. When this abnormal behavior is detected, a warning message is immediately sent to the instructor's device, who is notified and checks the detailed situation in the classroom. They then head to the scene, provide appropriate guidance according to the situation, and protect the victim.

[1285] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[1286] The processing flow will be explained below.

[1287] Step 1:

[1288] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, and the data is stored in a database with a timestamp for subsequent analysis.

[1289] Step 2:

[1290] The server preprocesses the received video and audio data, removing noise, correcting low-resolution areas, and making the audio clearer. This improves the quality of the data and increases the accuracy of analysis.

[1291] Step 3:

[1292] The server then applies machine learning algorithms to the pre-processed data to analyze student behavior patterns, specifically identifying facial and body movements, voice tone and intensity, and identifying aggressive or unnatural behavior.

[1293] Step 4:

[1294] The server automatically detects abnormal behavior from the analysis results. In this step, it compares the results with normal behavior patterns that have been trained in advance, and marks any deviations as abnormal behavior.

[1295] Step 5:

[1296] When abnormal behavior is detected, the server collects information about the students involved in the behavior and assigns a score to the characteristics and relationships of the perpetrator and victim. This score is calculated based on the frequency of the behavior and past records.

[1297] Step 6:

[1298] The server visualizes the scoring information and detected abnormal behavior on a dashboard that is updated in real time, allowing instructors to check detailed information about the problem.

[1299] Step 7:

[1300] When abnormal behavior is detected, the server immediately generates a warning message and automatically sends it to the instructor's device. The message includes detailed information about the detected behavior (location, time, involved parties, etc.).

[1301] Step 8:

[1302] The device receives the notification message sent from the server and displays a pop-up notification to the instructor in real time, allowing the instructor to immediately know of any problems.

[1303] Step 9:

[1304] The user (instructor) checks the warning notification for abnormal behavior and checks detailed situation information on the dashboard. Based on the video and scoring information, the situation can be accurately understood.

[1305] Step 10:

[1306] The user (instructor) can select the appropriate action from the response options on the dashboard and respond quickly by heading to the scene or issuing instructions to other staff.

[1307] Through the above steps, this system can detect bullying within schools early and respond quickly and appropriately.

[1308] Example 1

[1309] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1310] In recent years, bullying in schools has become a serious problem, necessitating early detection and rapid response. However, conventional monitoring devices and methods have issues in that they are unable to adequately detect abnormal behavior in real time, analyze behavior in detail, or respond quickly. Furthermore, it is difficult to simultaneously detect abnormal behavior and identify the specific characteristics and relationships of the perpetrator and victim.

[1311] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1312] In this invention, the server includes a means for receiving data from monitoring devices and detectors within the school, a means for analyzing the behavioral patterns of monitored individuals in real time based on the received data, and a means for detecting abnormal behavior from the analysis results. This enables early detection of abnormal behavior, scoring the characteristics and relationships of perpetrators and victims, and visualizing and displaying them in real time. Furthermore, detected abnormal behavior can be reported to an administrator, enabling a prompt response.

[1313] "Monitoring equipment" refers to devices such as surveillance cameras and microphones within a school that collect video and audio data in real time.

[1314] A "detector" is a device that consists of a sensor, a data receiving device, etc., and receives data from a monitoring device and detects abnormal behavior.

[1315] The "means for receiving data" refers to a means by which the server receives video and audio data transmitted from a monitoring device or detector in real time.

[1316] "Means for analyzing behavioral patterns in real time" refers to a means for analyzing student movements and voices based on received data and distinguishing between normal and abnormal behavioral patterns.

[1317] The "means for detecting abnormal behavior" is a means for automatically detecting abnormal behavior such as specific aggressive behavior or loud shouting from the results of analyzing behavioral patterns.

[1318] The "scoring method" is a method for quantitatively evaluating and scoring the characteristics and relationships of actors and influencers based on detected abnormal behavior.

[1319] The "means for visualizing and displaying" refers to a means for displaying scored information in real time using a visualization tool such as a dashboard.

[1320] The "means of reporting" is a means of notifying the administrator's terminal in real time of detailed information about detected abnormal behavior, and encouraging a prompt response.

[1321] The present invention is a system for detecting bullying in schools and responding quickly. This system receives and analyzes data from monitoring devices and detectors installed in schools to understand students' behavioral patterns in real time and detect abnormal behavior early. Specific embodiments of this system are described below.

[1322] Hardware and Software Configuration

[1323] Monitoring and Detectors

[1324] Surveillance cameras and microphones will be installed throughout the school to collect video and audio data, for example in classrooms, hallways, and other areas where bullying is likely to occur.

[1325] Data receiving device (server)

[1326] The server receives real-time data transmitted from surveillance cameras and microphones throughout the school, which is then stored with a timestamp and used for subsequent analysis.

[1327] Behavioral analysis system (server)

[1328] The server analyzes the received video and audio data using machine learning algorithms (e.g., TensorFlow), which analyze students' movements and voices to identify normal and abnormal behavioral patterns.

[1329] Abnormal behavior detection system (server)

[1330] The server automatically detects abnormal behavior based on the results of behavioral analysis, such as multiple students surrounding one student or loud shouting.

[1331] Scoring and visualization system (server)

[1332] The server scores the characteristics and relationships of the perpetrator and victim based on the detected anomalous behavior, and this information is displayed on a dashboard (e.g., Grafana) that updates in real time so that stakeholders can easily view it.

[1333] Reporting system (server and terminal)

[1334] If the server detects any abnormal behavior, it will report the information to the administrator's device in real time. The reporting system uses a real-time notification service such as Twilio, which enables the immediate transmission of information.

[1335] Specific examples

[1336] Example 1: Detecting bullying in the hallway

[1337] The server receives video data from surveillance cameras installed in the hallways. An analysis algorithm detects situations in which multiple students are surrounding and pushing around one student. If this behavior is detected as abnormal, the server immediately generates a warning message and sends it to the administrator's device. The administrator checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[1338] Example 2: Detecting verbal abuse in the classroom

[1339] The server receives audio data from microphones installed in classrooms. The analysis system detects the sound of a student shouting at a high volume and determines that this is verbal violence against other students. When this abnormal behavior is detected, a warning message is immediately sent to the administrator's device, who is notified and checks the detailed situation in the classroom. They then head to the scene, provide appropriate guidance according to the situation, and protect the victim.

[1340] Prompt Sentence Examples

[1341] To test and improve this system, you can feed the generative AI model prompts like:

[1342] "Please explain the algorithm that will detect abnormal behavior when multiple students are pushing against one another in the hallway."

[1343] "How can we detect high-volume shouting in a classroom and identify it as verbal abuse?"

[1344] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[1345] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1346] Step 1: Data collection

[1347] The server receives video and audio data in real time from monitoring devices and detectors installed within the school. The input data is video and audio from surveillance cameras and microphones. The received data is saved with a timestamp and used for subsequent analysis. Specifically, video captured by cameras installed in the hallways and audio collected by microphones in the classrooms are sent to the server.

[1348] Step 2: Preprocessing

[1349] The server performs preprocessing on the received raw data. The input data is the video and audio data collected in step 1. Specific preprocessing steps include noise removal, data normalization, and timestamp alignment. The output data is clean data ready to be input into the analysis system. For example, unnecessary frames are removed from the video data and background noise is removed from the audio data.

[1350] Step 3: Behavioral pattern analysis

[1351] The server analyzes behavioral patterns based on the preprocessed data. The input data is the clean video and audio data generated in step 2. A machine learning algorithm (e.g., TensorFlow) is used to analyze students' movements and audio patterns and distinguish between normal and abnormal behavioral patterns. The output data is digital features related to each student's behavioral pattern. Specifically, the server analyzes the student's position and movements for each frame of video data and detects shouts from the waveform of the audio data.

[1352] Step 4: Detecting Abnormal Behavior

[1353] The server detects abnormal behavior based on the analysis results. The input data are the digital features of the behavioral patterns generated in step 3. Abnormal behavior includes aggressive behavior and loud shouting. For example, if a specific student behaves aggressively toward another student within a specific time period, this is detected as abnormal behavior. The output data is the abnormal behavior detection results and detailed information. Specifically, information such as "Student B behaved aggressively toward Student C in classroom A at XX:YY" is detected.

[1354] Step 5: Scoring and visualization

[1355] The server scores the characteristics and relationships of the perpetrator and victim based on the data of detected abnormal behavior. The input data is the abnormal behavior detection results generated in step 4. The scored information is displayed on a dashboard that is updated in real time. The output data is visualized score information. Specifically, the behavioral history and relationships of the perpetrator and victim are displayed as graphs and charts.

[1356] Step 6: Automatic reporting

[1357] The server sends a real-time report to the administrator's device based on the information on the detected abnormal behavior. The input data is the abnormal behavior and its scoring data generated in steps 4 and 5. The reporting system uses a real-time notification service such as Twilio. The output data is a report message. This message contains detailed information about the abnormal behavior and is displayed on the administrator's device. For example, a message such as "Student B is behaving aggressively toward Student C in classroom A" is sent.

[1358] Step 7: Warning Display

[1359] The terminal receives the report message sent from the server in real time and displays a pop-up notification. The input data is the report message sent in step 6. The notification contains detailed information about the abnormal behavior, which the administrator can check immediately. The output data is a pop-up notification that is displayed so that the administrator can check it.

[1360] Step 8: Check the situation

[1361] The terminal displays detailed information about the detected abnormal behavior through a dashboard. The input data is the scored information generated in step 5. Real-time video and scored data are displayed, allowing the administrator to accurately grasp the situation based on this. The output data is detailed situation information that is displayed to the administrator. Specifically, a dashboard screen is provided that allows the location, time, and involved parties of the problem to be identified.

[1362] Step 9: Instructions

[1363] The user selects a response option from the device dashboard and issues instructions to implement the necessary measures. The input data is detailed information about the abnormal behavior confirmed in step 8. The output data is the selected response option. Specifically, options such as "visit the scene," "contact the parents," and "notify other instructors" can be selected, and a prompt response will be implemented.

[1364] Step 10: Implementing the measures

[1365] The user goes to the scene where the abnormal behavior is occurring and quickly instructs the perpetrator and protects the victim. The input data is the response option selected in step 9. The output data is the instructions actually taken and the results. Specifically, the user goes to classroom A and takes action to stop the bullying. Counseling and protective measures are also implemented for those involved.

[1366] (Application example 1)

[1367] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1368] Bullying in schools is a serious problem, requiring early detection and rapid response. However, comprehensive detection and immediate response to bullying are difficult using current manual monitoring and approaches alone. Monitoring and responding to bullying is particularly challenging in large schools and dispersed school environments, making it essential to implement a smooth system to ensure student safety. Furthermore, analyzing data obtained from surveillance cameras and sensors requires advanced technology, and an efficient system is needed to detect abnormal behavior in real time and respond quickly.

[1369] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1370] In this invention, the server includes means for receiving data from surveillance cameras and sensors within the school, means for analyzing student behavior patterns in real time based on the received data, means for detecting abnormal behavior from the analysis results, means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior, means for visualizing and displaying the scored information, and means for displaying detailed information about the abnormal behavior on a display device worn by an instructor who receives a notification of the abnormal behavior. This makes it possible to detect bullying and abnormal behavior within the school in real time and respond immediately. Instructors can check detailed information about the abnormal behavior on the display device and take prompt and appropriate measures.

[1371] A "surveillance camera" is a device that is installed in a specific area and is used to capture video.

[1372] A "sensor" is a device that detects physical changes and converts them into electrical signals.

[1373] The "data receiving means" is a device or system for receiving data acquired from the surveillance cameras and sensors.

[1374] The "behavior pattern analysis means" is a system that has the function of analyzing student behavior based on received data and identifying specific patterns.

[1375] The "abnormal behavior detection means" is a system for automatically identifying behavior that is different from normal based on the results of analyzing behavior patterns.

[1376] The "scoring means" is a system that has the function of quantifying the characteristics and relationships of the perpetrator and victim based on detected abnormal behavior.

[1377] The "visualization display means" is a device or software for graphically displaying the scored information.

[1378] The "reporting means" is a device or function for notifying the trainer of information about detected abnormal behavior.

[1379] The "display device" is a device that displays detailed information about abnormal behavior so that the trainer can check it.

[1380] The present invention is a system for detecting bullying in schools and responding quickly, and includes a series of processes that receive data from surveillance cameras and sensors, perform behavior analysis, and detect abnormal behavior in real time. Specific embodiments for implementing this system are described below.

[1381] System construction

[1382] This system consists of the following main components:

[1383] 1. Surveillance cameras and sensors:

[1384] Video and audio data is collected in real time using surveillance cameras and audio sensors (e.g., Logitech C920 cameras and Blue Yeti microphones) installed throughout the school.

[1385] 2. Data receiving means:

[1386] A server is used to receive and centrally manage data sent from each surveillance camera and sensor.

[1387] 3. Behavioral pattern analysis methods:

[1388] The server uses deep learning frameworks such as TensorFlow to analyze the data it receives and identify student behavior patterns, using machine learning algorithms to distinguish between normal and abnormal behavior patterns.

[1389] 4. Abnormal behavior detection methods:

[1390] The server automatically detects abnormal behavior based on the analyzed behavioral patterns, such as aggressive behavior or loud shouting.

[1391] 5. Scoring Procedure:

[1392] When abnormal behavior is detected, the server scores the characteristics and relationships of the perpetrator and victim, clarifying the severity of the problem and the relationships between the parties involved.

[1393] 6. Visualization display means:

[1394] Scoring information is displayed on a real-time updated dashboard or display device (e.g., smart glasses), allowing trainers to view detailed information about abnormal behavior.

[1395] 7. Reporting methods:

[1396] When abnormal behavior is detected, the server sends the information to the instructor's device in real time. The report includes detailed information about the abnormal behavior, enabling a prompt response.

[1397] Explanation of program processing

[1398] The system collects data from surveillance cameras and audio sensors in real time and preprocesses the received data on a server. It uses TensorFlow to analyze video data and identify abnormal behavior, while Scikit-learn analyzes audio data. When abnormal behavior is detected, the information is immediately sent to the instructor's terminal and display device (such as smart glasses).

[1399] Specific examples

[1400] For example, if a surveillance camera captures multiple students surrounding one another in a hallway, the TensorFlow model will detect this abnormal behavior. At this point, the server will immediately generate an alert message and notify the instructor's smart glasses. The instructor can then view the situation in real time through the glasses and rush to the scene to take appropriate action.

[1401] Prompt Sentence Examples

[1402] You can use the following prompt for the generative AI model: "Train a video analytics model using a dataset for detecting anomalous behavior and generate Python code to detect suspicious behavior and emergencies in real time."

[1403] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1404] Step 1:

[1405] Data collection

[1406] The server collects real-time video and audio data from surveillance cameras and sensors (e.g., Logitech C920 cameras, Blue Yeti microphones) installed within the school. The input data are video frames and audio samples, which are sent to the server along with timestamps. The output data are time-stamped video and audio data.

[1407] Step 2:

[1408] Data Preprocessing

[1409] The server performs preprocessing on the collected video and audio data. This processing includes noise removal and data normalization. The input is the collected raw data, and the output is preprocessed video and audio data. Specifically, OpenCV is used to resize the video frames and perform noise filtering, and Scipy is used to remove noise from the audio data.

[1410] Step 3:

[1411] Behavioral pattern analysis

[1412] The server analyzes behavioral patterns based on the preprocessed data. This analysis uses a deep learning model using TensorFlow. The input is preprocessed video and audio data, and the output is the analysis results of behavioral patterns. Specifically, the server extracts video data frame by frame and inputs it into the model to classify normal and abnormal behavior.

[1413] Step 4:

[1414] Abnormal behavior detection

[1415] The server detects anomalous behavior from the results of behavioral pattern analysis. This detection uses a machine learning algorithm to identify suspicious behavior. The input is the analyzed behavioral pattern data, and the output is a judgment result on whether anomalous behavior has been detected. Specifically, it uses Scikit-learn's classification algorithm to identify aggressive behavior and high-volume shouting.

[1416] Step 5:

[1417] Scoring abnormal behavior

[1418] The server scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior. The input is the abnormal behavior detection result, and the output is the scored data. Specifically, it applies an algorithm that quantifies information such as the degree of abnormal behavior, the number of people involved, and frequency.

[1419] Step 6:

[1420] Information Visualization

[1421] The scored information is visualized on a display device such as a dashboard or smart glasses that is updated in real time. The input is the scored data, and the output is the information displayed graphically. Specifically, a JavaScript framework (e.g., D3.js) is used to display the information in graphs and charts on the dashboard.

[1422] Step 7:

[1423] Reporting Abnormal Behavior

[1424] When abnormal behavior is detected, the server immediately notifies the instructor's terminal or display device. The input is detailed information about the abnormal behavior, and the output is a warning message displayed on the instructor's terminal. Specific operations include a function to display a pop-up notification on the smart glasses and an API to send a push notification to the smartphone.

[1425] Prompt Sentence Examples

[1426] You can use the following prompt for the generative AI model: "Train a video analytics model using a dataset for detecting anomalous behavior and generate Python code to detect suspicious behavior and emergencies in real time."

[1427] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1428] This invention relates to a system for detecting bullying in schools and responding quickly. In particular, this system analyzes students' behavioral patterns and emotions in real time, detects abnormal behavior and emotional changes, and takes necessary measures. This system is composed of the following components:

[1429] System Configuration

[1430] 1. Surveillance cameras and sensors:

[1431] They are installed in various locations throughout the school and collect video and audio data in real time, allowing students' behavior and the surrounding sound environment to be monitored.

[1432] 2. Data receiving device (server):

[1433] The data sent from surveillance cameras and sensors is centrally managed and the necessary pre-processing is performed.

[1434] 3. Behavioral analysis system (server):

[1435] The received data is analyzed in real time to identify student behavioral patterns and emotions.

[1436] 4. Emotion engine (server):

[1437] Based on video and audio data, the system analyzes students' facial expressions and tone of voice to recognize their emotions. The recognized emotional information is added to the analysis results of their behavioral patterns.

[1438] 5. Abnormal behavior detection system (server):

[1439] Abnormal behavior is detected based on the results of behavioral analysis and the emotion engine, and necessary processing is carried out.

[1440] 6. Scoring and visualization system (server):

[1441] Based on detected abnormal behavior and recognized emotions, the characteristics and relationships of the perpetrator and victim are scored and displayed graphically.

[1442] 7. Reporting system (server and terminal):

[1443] Detected abnormal behavior and emotional information is automatically reported to the instructor's device.

[1444] System program processing

[1445] Server-side processing

[1446] 1. Data Collection

[1447] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, which is then stored with a timestamp and used for analytical processing.

[1448] 2. Behavioral and emotional analysis

[1449] The server applies machine learning algorithms and emotion engines to the received data to identify students' behavioral patterns and emotions, specifically recognizing emotional states (such as joy, anger, sadness, and fear) through facial expression and voice analysis.

[1450] 3. Detecting Abnormal Behavior

[1451] The server automatically detects abnormal behavior based on the analysis of behavioral patterns and emotions. For example, it detects aggressive behavior or loud shouting and determines these as abnormal behavior.

[1452] 4. Scoring and visualization

[1453] The server scores the characteristics and relationships of the perpetrator and victim based on abnormal behavior and emotional information. This information is visualized in real time on a dashboard, allowing instructors to easily understand the situation.

[1454] 5. Automatic reporting

[1455] When abnormal behavior or specific emotions are detected, the server sends a real-time report to the instructor's device, including detailed information about the detected behavior (location, time, involved parties, emotional state, etc.).

[1456] Terminal side processing

[1457] 1. Warning display

[1458] The device receives the notification message sent from the server and displays a real-time warning as a pop-up notification, allowing the instructor to immediately know of any problems.

[1459] 2. Check the situation

[1460] The device displays detailed situational information (video, score, emotional information) via a dashboard, allowing the instructor to quickly grasp the situation and consider appropriate responses.

[1461] 3. Instructions for response

[1462] Instructors can select the appropriate action from the response options on the dashboard and either head to the scene or give instructions to other staff.

[1463] User processing

[1464] 1. Check the warning

[1465] Teachers can view notifications on devices and get detailed information about behaviors and emotions that may be indicative of bullying. A dashboard makes it easy to determine the severity of the problem and identify those involved.

[1466] 2. Implementation of measures

[1467] Instructors can go to the scene where abnormal behavior and emotional changes are occurring and quickly provide guidance to the perpetrator and protection to the victim, thereby helping to resolve the problem early.

[1468] Specific examples

[1469] Example 1: Detecting bullying and emotional changes in the hallway

[1470] The server receives video data from surveillance cameras installed in the hallways. Using an analysis algorithm and emotion engine, it detects the emotion of fear from the facial expression of the student being pushed, as multiple students surround and push against one another. If this behavior and emotional information is detected as abnormal, the server immediately generates a warning message and sends it to the instructor's device. The instructor checks the pop-up notification on their device, understands the situation on the dashboard, and rushes to the scene to take appropriate action.

[1471] Example 2: Detecting verbal abuse and emotional changes in the classroom

[1472] The server receives audio data from microphones installed in classrooms. The analysis system and emotion engine detect the high-pitched voice of a student shouting and the sadness of a student believed to be the victim. If this is recognized as abnormal behavior such as verbal violence, a warning message is immediately sent to the instructor's device. The instructor is notified and checks the details of the situation in the classroom, then heads to the scene to provide appropriate guidance according to the situation and protect the victim.

[1473] This system will enable early detection of bullying and prompt response, providing an environment in which students can live their school life with peace of mind.

[1474] The processing flow will be explained below.

[1475] Step 1:

[1476] The server receives real-time video and audio data from surveillance cameras and sensors installed within the school, which is then stored in a database with a timestamp for subsequent analysis.

[1477] Step 2:

[1478] The server preprocesses the received video and audio data, removing noise, correcting low-resolution areas, and making the audio clearer. This improves the quality of the data and increases the accuracy of analysis.

[1479] Step 3:

[1480] The server then applies machine learning algorithms to the pre-processed data to analyze student behavioral patterns, such as facial and body movements, and vocal tone and intensity, to identify aggressive or unnatural behavior.

[1481] Step 4:

[1482] The server analyzes behavioral patterns and uses an emotion engine to recognize emotions from students' facial expressions and tone of voice, categorizing them as joy, anger, sadness, fear, and so on.

[1483] Step 5:

[1484] The server automatically detects abnormal behavior by combining the results of behavioral pattern analysis and the emotion engine's recognition results. For example, if aggressive behavior and angry emotions are detected simultaneously, the server marks the behavior as abnormal.

[1485] Step 6:

[1486] When abnormal behavior is detected, the server collects information about the students related to the behavior and emotions, and then scores the characteristics and relationships of the perpetrator and victim. This score is calculated based on the frequency of the behavior, past records, type of emotion, etc.

[1487] Step 7:

[1488] The server visualizes the scoring information and detected abnormal behavior and emotional information on a dashboard that is updated in real time, allowing instructors to check detailed information about the problem.

[1489] Step 8:

[1490] When abnormal behavior or a specific emotion is detected, the server immediately generates a warning message and automatically sends it to the instructor's device. The message includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.).

[1491] Step 9:

[1492] The device receives the notification message sent from the server and displays a pop-up notification to the instructor in real time, allowing the instructor to immediately know of any problems.

[1493] Step 10:

[1494] The user (instructor) checks the warning notification for abnormal behavior and checks detailed situation information on the dashboard. Based on the video, score information, and emotional information, the situation can be accurately grasped.

[1495] Step 11:

[1496] The user (instructor) can select the appropriate action from the response options on the dashboard and respond quickly by heading to the scene or issuing instructions to other staff.

[1497] Through the above steps, this system can detect bullying within schools early and respond quickly and appropriately.

[1498] Example 2

[1499] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1500] Bullying in schools has long-term mental and physical effects on students, so early detection and rapid response are necessary. However, traditional methods tend to be slow to detect and address bullying behavior, making it difficult for relevant parties to actively intervene. For this reason, a system that can detect bullying behavior in real time and respond quickly is needed.

[1501] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving information from monitoring devices and sensors in the school, a means for analyzing the behavioral patterns of individuals in real time based on the received information, and a means for detecting abnormal behavior from the analysis results. This makes it possible to detect abnormal behavior in real time and take prompt and appropriate action.

[1502] "Surveillance equipment" refers to cameras and sensors installed to collect video data within a school.

[1503] "Sensors" refer to devices that collect sound and environmental data within the school.

[1504] "Information" refers to data such as student behavior and audio collected from monitoring devices and sensors.

[1505] "Individual" refers to a specific person, such as a student, who is the subject of monitoring within a school.

[1506] "Behavior patterns" refer to students' patterns of movement and behavior.

[1507] "Real-time" refers to information being processed and analyzed almost as soon as it is generated.

[1508] "Abnormal behavior" refers to inappropriate behavior, such as bullying or aggressive behavior, that deviates from normal patterns of behavior.

[1509] "Characteristics" refers to the behavioral and emotional characteristics of individual students.

[1510] "Relationship" refers to the human relationships and interactions between the perpetrator and the victim.

[1511] "Scoring" refers to quantifying behaviors and characteristics and converting them into numbers or ratings.

[1512] "Visualization" refers to visually displaying the scored information to make it easier to understand.

[1513] "Administrator" refers to faculty and staff members who are responsible for monitoring the system and dealing with abnormal behavior.

[1514] "Management screen" refers to the interface for displaying scored information and detected abnormal behavior.

[1515] The present invention relates to a system for realizing early detection of bullying in schools and a rapid response to the same. Specific embodiments of the system are described below.

[1516] System configuration

[1517] This system mainly uses the following hardware and software:

[1518] 1. Monitoring Devices and Sensors

[1519] Surveillance cameras and audio sensors will be installed throughout the school to collect student behavior and audio data in real time.

[1520] 2. Data receiving device (server)

[1521] The collected video and audio data is sent to a server via a dedicated communication protocol, where it is stored in a database with a timestamp for subsequent analysis.

[1522] 3. Behavioral analysis system (server)

[1523] The server applies machine learning algorithms to the collected data to identify students' behavioral patterns and emotions in real time, using libraries such as OpenCV and Dlib for facial expression analysis and librosa for voice analysis.

[1524] 4. Emotion engine (server)

[1525] Through facial expression analysis and voice analysis, it identifies students' emotional states (joy, anger, sadness, fear, etc.). The emotion engine analyzes the collected data and identifies emotional information.

[1526] 5. Abnormal behavior detection system (server)

[1527] Based on the results of behavioral analysis and the emotion engine, the system automatically detects abnormal behavior. Machine learning models identify anomalous behavior such as aggressive behavior or high-volume shouting.

[1528] 6. Scoring and visualization system (server)

[1529] Based on abnormal behavior and emotional information, the characteristics and relationships of the perpetrator and victim are scored. This scoring information is visualized in real time on a dashboard, allowing administrators to easily understand the situation.

[1530] 7. Reporting System (Server and Terminal)

[1531] Detected abnormal behavior and emotional information are automatically reported to the administrator's terminal. The report message includes detailed information about the abnormal behavior (location, time, involved parties, emotional state, etc.).

[1532] Specific examples

[1533] Example 1: Detecting bullying and emotional changes in the hallway

[1534] The server receives video data from surveillance cameras installed in the hallways. Using an analysis algorithm and emotion engine, it detects a group of students pushing around one another, and detects fear from the facial expression of the student being pushed. If this behavior and emotional information is detected as abnormal, the server immediately generates a warning message and sends it to the administrator's device. The administrator checks the pop-up notification on their device, understands the situation on the dashboard, and then rushes to the scene to take appropriate action.

[1535] Example 2: Detecting verbal abuse and emotional changes in the classroom

[1536] The server receives audio data from microphones installed in classrooms. The analysis system and emotion engine detect the high-pitched voices of students shouting and the sadness of students believed to be victims. If this is recognized as abnormal behavior such as verbal violence, a warning message is immediately sent to the administrator's device. The administrator is notified and checks the details of the situation in the classroom, then heads to the scene to provide appropriate guidance according to the situation and protect the victim.

[1537] Prompt Sentence Examples

[1538] "If the system detects bullying behavior and emotional changes in the hallways, please explain specifically what data is collected and how it is analyzed."

[1539] This allows the generative AI model to provide a good explanation for the specific behavior of the system.

[1540] The present invention enables early detection of bullying and rapid response, thereby providing a safe learning environment for students.

[1541] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1542] Step 1: Data collection

[1543] The server receives real-time video and audio data from monitoring devices and sensors, and stores the data in a database with a timestamp.

[1544] Input: Video and audio data from surveillance cameras and audio sensors.

[1545] Output: Stored raw data with timestamp.

[1546] How it works: The surveillance camera captures video frames and the sensor collects audio waveforms. These data are then sent to the server via a dedicated communication protocol.

[1547] Step 2: Analyzing behavioral patterns and emotions

[1548] The server applies machine learning algorithms and emotion engines to the collected data to identify student behavioral patterns and emotions.

[1549] Input: Stored video and audio data.

[1550] Output: Identification of behavioral patterns and emotional state data.

[1551] How it works: The server uses OpenCV and Dlib libraries to analyze facial expressions, and librosa to recognize tone of voice and emotions from audio data.

[1552] Step 3: Detecting Abnormal Behavior

[1553] The server automatically detects abnormal behavior based on the results of behavioral analysis and the emotion engine.

[1554] Input: Behavioral patterns and emotional state data.

[1555] Output: Abnormal behavior detection result.

[1556] Specific behaviors: Apply machine learning models to identify anomalous behaviors, such as aggressive behavior or high-volume screaming, by comparing them with historical patterns of normal behavior.

[1557] Step 4: Scoring and visualization

[1558] The server scores the characteristics and relationships between the perpetrator and victim based on the detected abnormal behavior and emotional information, and visualizes the scored information in real time on a dashboard.

[1559] Input: Abnormal behavior detection results.

[1560] Output: Scored trait and relationship information, graphical display.

[1561] Specific actions: Comprehensively evaluate the frequency of abnormal behavior, the location of occurrence, and historical data of those involved, and display the results as heat maps and graphs on the dashboard.

[1562] Step 5: Automatic reporting

[1563] If the server detects abnormal behavior or a specific emotion, it reports the information to the administrator's terminal in real time.

[1564] Input: Abnormal behavior detection results, scored information.

[1565] Output: Notification message to administrator terminal.

[1566] Specific operation: Use the reporting system to send detailed information about abnormal behavior via SMS, email, or the notification function of a dedicated app.

[1567] Step 6: Warning display

[1568] The terminal receives the notification message sent from the server and displays a warning in a pop-up notification.

[1569] Input: Notification message from the server.

[1570] Output: Display of warning notification.

[1571] Specific operation: A warning message will pop up on the device screen and an audio alert will also be issued.

[1572] Step 7: Check the situation

[1573] The device allows the notified administrator to view detailed status information via a dashboard.

[1574] Input: Report message, scored information, video data.

[1575] Output: Display detailed status information.

[1576] How it works: Administrators can view real-time video streams and historical data on a dashboard.

[1577] Step 8: Instructions

[1578] Managers can select the appropriate action from the response options on the dashboard and either head to the scene or give instructions to other staff.

[1579] Input: Dashboard status information.

[1580] Output: Decision on response instructions.

[1581] Specific actions: The administrator presses buttons such as "Rush to the scene" or "Notify staff" on the dashboard and instructs the necessary response.

[1582] (Application example 2)

[1583] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1584] Conventional security systems have difficulty detecting suspicious behavior, particularly in analyzing emotional changes in real time and associating them with suspicious behavior. This makes it difficult to respond quickly and, as a result, to provide a safe and secure environment. Furthermore, because scoring and visualization are not performed, it is not possible to intuitively understand the characteristics and relationships of those involved.

[1585] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1586] In this invention, the server includes means for receiving data from the monitoring device and the audio sensing device, means for analyzing people's behavioral patterns and emotions in real time based on the received data, means for detecting abnormal behavior and changes in emotions from the analysis results, means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and changes in emotions, means for visualizing and displaying the scored information, and means for notifying security personnel of the detected abnormal behavior and changes in emotions. This allows abnormal behavior and changes in emotions to be quickly detected and notified to security personnel, enabling a prompt response. Furthermore, visualizing the scored information allows users to intuitively understand the characteristics and relationships of the people involved.

[1587] A "monitoring device" is a device that collects video data in real time and monitors specific locations or people.

[1588] An "audio sensing device" is a device that collects surrounding audio data in real time and uses it for analysis.

[1589] A "behavioral pattern" is a repetitive pattern of actions or behaviors that a particular person performs on a daily basis.

[1590] "Emotions" are psychological states detected from facial expressions and tone of voice, and include joy, anger, sadness, fear, etc.

[1591] "Means for real-time analysis" refers to a method for processing received data immediately and obtaining results without delay.

[1592] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior, including aggressive behavior and loud shouting.

[1593] "Emotional shift" refers to an abrupt change in the observed individual's emotions, especially including changes to stronger emotional states (e.g., fear, anger, etc.).

[1594] "Scoring" is a method for evaluating and visualizing specific characteristics and relationships using numbers or indicators.

[1595] "Visualization" is the act of displaying data or information in a graphical format (e.g., a dashboard) to make it intuitively easier to understand.

[1596] "Security personnel" are professional personnel who monitor and respond to suspicious or abnormal behavior in order to ensure safety within facilities and buildings.

[1597] "Reporting" is the act of immediately notifying security personnel about detected abnormal behavior or emotional changes.

[1598] This invention is a system for detecting suspicious behavior and emotional changes and responding quickly in office buildings, commercial facilities, etc. The system includes means for receiving and analyzing data from monitoring devices and audio sensing devices. It also includes means for detecting, scoring, and visualizing abnormal behavior and emotional changes. Finally, it includes means for reporting the detected information to security personnel.

[1599] System Program Overview

[1600] The server receives real-time video and audio data from surveillance and audio sensing devices installed in office buildings and commercial facilities. This data is stored with a timestamp and used for analytical processing. The server then uses machine learning algorithms and emotion engines (e.g., TensorFlow, Keras, etc.) to perform facial recognition and audio analysis on the received data and identify behavioral patterns and emotions.

[1601] Analyzing data and detecting abnormal behavior and emotional changes

[1602] The server applies analysis algorithms and emotion engines to identify the target person's behavioral patterns and emotional state. For example, it recognizes emotional states (such as joy, anger, sadness, and fear) through facial expression and voice analysis. It then automatically detects abnormal behavior and sudden emotional changes based on the behavioral pattern and emotion analysis results. Examples of abnormal behavior include aggressive behavior and loud shouting.

[1603] Scoring and visualization process

[1604] The server scores the characteristics and relationships of the perpetrator and victim based on abnormal behavior and emotional information. For example, it quantifies each person's behavior and emotional state and displays this information in real time on a dashboard. This allows security personnel to easily grasp the situation and respond quickly.

[1605] Reporting function

[1606] When abnormal behavior or specific emotions are detected, the server sends a real-time notification to the security officer's smartphone. The notification includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.). This notification function is realized using the Twilio API or other notification systems.

[1607] Specific examples

[1608] Example 1: Detecting suspicious behavior and emotional changes in the office

[1609] Video data is received from surveillance cameras in an office building, and an analysis algorithm detects suspicious behavior in which multiple people surround one person. Furthermore, a strong expression of fear is detected from the facial expressions of the surrounded individuals. This situation is determined to be abnormal behavior, and a report is immediately sent to security personnel. An example of a prompt statement is, "Several people have surrounded one individual in an office building, and expressions of fear have been detected. Please check the scene immediately."

[1610] This system allows for real-time monitoring and analysis of suspicious behavior and sudden emotional changes within facilities and buildings, enabling efficient and rapid response, thereby providing a safe and secure environment at all times.

[1611] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1612] Step 1: Receiving data

[1613] The server receives real-time video and audio data from the monitoring and audio sensing devices. This data is time-stamped to identify when each input was collected. The input data are video frames and audio waveforms, which are then ready for the server to use in the next processing step.

[1614] Step 2: Preprocessing the data

[1615] The server converts the received video data to grayscale and denoises the audio data, improving the accuracy of the analysis. The video frames and audio waveforms received as input are converted into clean video frames and denoised audio waveforms after preprocessing.

[1616] Step 3: Face and voice detection

[1617] The server detects face areas from preprocessed video frames and identifies the sound source from the audio data. It uses OpenCV's face detection algorithm for face detection and a sound source localization algorithm for audio source identification. It receives preprocessed video frames and audio waveforms as input data, and obtains face areas and sound source locations as output.

[1618] Step 4: Analyzing behavioral patterns and emotions

[1619] The server analyzes the emotions of the identified facial regions using machine learning models (using TensorFlow and Keras) and identifies the person's emotional state through audio analysis. The analysis recognizes emotions such as joy, anger, sadness, and fear from facial expressions and tone of voice. It receives facial regions and audio data as input and obtains the determined emotional state as output.

[1620] Step 5: Detecting abnormal behavior and emotional changes

[1621] The server detects abnormal behavior and sudden emotional changes based on the analysis results. For example, this includes aggressive behavior, loud shouting, and strong emotions such as fear. It receives the analyzed emotional state and behavioral patterns as input, and obtains the detection results of abnormal behavior and emotional changes as output.

[1622] Step 6: Scoring and visualization

[1623] The server scores the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and emotional information. The scored information is displayed graphically on a dashboard. The server receives the detection results of abnormal behavior and emotional changes as input, and obtains the scored information and a visualized graphical dashboard display as output.

[1624] Step 7: Report

[1625] The server notifies the security personnel of detected abnormal behaviors and emotional changes in real time via their smartphones. The notification information includes detailed information about the detected behavior (location, time, involved parties, emotional state, etc.). It receives the scored and visualized information as input and generates a notification message for the security personnel as output.

[1626] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1627] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1628] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1629] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1630] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1631] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1632] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1633] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1634] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1635] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1636] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1637] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1638] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1639] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1640] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1641] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1642] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1643] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1644] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1645] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1646] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1647] The following is further disclosed regarding the above embodiment.

[1648] (Claim 1)

[1649] a means for receiving data from surveillance cameras and sensors within the school;

[1650] A means of analyzing student behavior patterns in real time based on the received data;

[1651] A means for detecting abnormal behavior from the analysis results;

[1652] A means for scoring the characteristics and relationships of perpetrators and victims based on detected abnormal behavior;

[1653] A means for visualizing and displaying the scored information;

[1654] a means for reporting detected abnormal behavior to an instructor;

[1655] A system including:

[1656] (Claim 2)

[1657] 2. The system of claim 1, wherein the abnormal behavior detection means uses an algorithm to detect aggressive behavior and loud shouting.

[1658] (Claim 3)

[1659] 2. The system according to claim 1, wherein the means for visualizing the scored information uses a dashboard that is updated in real time.

[1660] "Example 1"

[1661] (Claim 1)

[1662] means for receiving data from monitoring devices and detectors within the school;

[1663] A means for analyzing the behavioral patterns of the monitored person in real time based on the received data;

[1664] A means for detecting abnormal behavior from the analysis results;

[1665] A means for scoring the characteristics and relationships of actors and influencers based on detected anomalous behavior;

[1666] A means for visualizing and displaying the scored information;

[1667] a means for reporting detected abnormal behavior to an administrator;

[1668] A system including:

[1669] (Claim 2)

[1670] 2. The system of claim 1, wherein the abnormal behavior detection means uses an algorithm to detect aggressive behavior and loud shouting.

[1671] (Claim 3)

[1672] 2. The system according to claim 1, wherein the means for visualizing the scored information uses a visualization device that is updated in real time.

[1673] "Application Example 1"

[1674] (Claim 1)

[1675] a means for receiving data from surveillance cameras and sensors within the school;

[1676] A means of analyzing student behavior patterns in real time based on the received data;

[1677] A means for detecting abnormal behavior from the analysis results;

[1678] A means for scoring the characteristics and relationships of perpetrators and victims based on detected abnormal behavior;

[1679] A means for visualizing and displaying the scored information;

[1680] a means for reporting detected abnormal behavior to an instructor;

[1681] a means for displaying detailed information about the abnormal behavior on a display device worn by the instructor who has received the notification of the abnormal behavior;

[1682] A system including:

[1683] (Claim 2)

[1684] 2. The system of claim 1, wherein the abnormal behavior detection means uses an algorithm to detect aggressive behavior and loud shouting.

[1685] (Claim 3)

[1686] 2. The system of claim 1, wherein the means for visualizing the scored information uses a dashboard and display device that is updated in real time.

[1687] "Example 2: Combining Emotion Engines"

[1688] (Claim 1)

[1689] means for receiving information from monitoring devices and sensors within the school;

[1690] A means for analyzing the behavioral patterns of the individual in real time based on the received information;

[1691] A means for detecting abnormal behavior from the analysis results;

[1692] A means for scoring the characteristics and relationships of perpetrators and victims based on the detected abnormal behavior;

[1693] A means for visualizing and displaying the scored information;

[1694] means for reporting detected abnormal behavior to an administrator;

[1695] A system including:

[1696] (Claim 2)

[1697] 2. The system of claim 1, wherein the abnormal behavior detection means uses an algorithm to detect aggressive behavior and loud shouting.

[1698] (Claim 3)

[1699] 2. The system according to claim 1, wherein the means for visualizing the scored information uses a management screen that is updated in real time.

[1700] "Application example 2 when combining emotion engines"

[1701] Rewritten claims

[1702] (Claim 1)

[1703] means for receiving data from the monitoring device and the audio sensing device;

[1704] A means for analyzing a person's behavioral patterns and emotions in real time based on the received data;

[1705] A means for detecting abnormal behavior and emotional changes from the analysis results;

[1706] A means for scoring the characteristics and relationships of the perpetrator and victim based on the detected abnormal behavior and emotional changes;

[1707] A means for visualizing and displaying the scored information;

[1708] means for reporting detected abnormal behavior and emotional changes to security personnel;

[1709] A system including:

[1710] (Claim 2)

[1711] 2. The system of claim 1, wherein the abnormal behavior and emotional change detection means uses an algorithm that detects aggressive behavior, loud shouting, and strong emotions such as fear.

[1712] (Claim 3)

[1713] 2. The system according to claim 1, wherein the means for visualizing the scored information uses a dashboard that is updated in real time. [Explanation of symbols]

[1714] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for receiving data from surveillance cameras and sensors within the school; A means of analyzing student behavior patterns in real time based on the received data; A means for detecting abnormal behavior from the analysis results; A means for scoring the characteristics and relationships of perpetrators and victims based on detected abnormal behavior; A means for visualizing and displaying the scored information; a means for reporting detected abnormal behavior to an instructor; A system including:

2. 2. The system of claim 1, wherein the abnormal behavior detection means uses an algorithm to detect aggressive behavior and loud shouting.

3. The system according to claim 1 , wherein the means for visualizing the scored information uses a dashboard that is updated in real time.

Images