system

Abstract

Provide a system. The present invention includes a means for collecting video data in real time from multiple cameras installed in an educational institution; means for preprocessing the video data and preparing it for analysis; A means for analyzing the preprocessed video data to evaluate student behavior, facial expressions, voice, and positional relationships; A means of detecting abnormal behavior or abnormal voice and sending an alert to the relevant educator; A means of detecting sudden disasters and issuing evacuation instructions, A means of monitoring evacuation situations in real time and confirming safety, A means of notifying parents and relevant agencies that evacuation has been completed; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In educational institutions such as elementary and junior high schools, various problems such as bullying, accidents, and natural disasters are likely to occur. While prevention and rapid response to these problems are required, manual monitoring has its limitations. Monitoring student behavior and security across a large campus requires a great deal of time and effort, making it difficult to do so efficiently. Furthermore, in the event of a sudden disaster, prompt and accurate evacuation instructions and situational awareness are crucial to ensuring student safety, and a delayed response could result in significant damage. [Means for solving the problem]

[0005] To address the above-mentioned issues, the present invention provides a system for collecting video data in real time from multiple cameras installed in educational institutions. This system has the ability to preprocess and analyze the collected video data, evaluating students' behavior, facial expressions, voice, and positional relationships. It also has a means for detecting abnormal behavior or voice and sending alerts to relevant educators. It also has the ability to detect sudden disasters, issue evacuation instructions, and monitor evacuation situations in real time to confirm safety. It also has the ability to notify parents and relevant organizations when evacuation is complete. This ensures the safety of students and enables rapid and accurate response when problems occur.

[0006] "Educational institution" means a primary school, secondary school, or other learning facility, including any place where students live together.

[0007] A "camera" is a surveillance device that collects video data and is installed in classrooms, hallways, gymnasiums, schoolyards, etc.

[0008] "Real-time" refers to the state in which information can be processed immediately and a response can be made immediately.

[0009] "Video data" is a general term for images and video information captured by a camera.

[0010] "Preprocessing" refers to a series of data processing steps performed to check the quality of video data and prepare it for analysis.

[0011] "Analysis" refers to the process of using collected data to evaluate student behavior, facial expressions, voice, and positioning.

[0012] "Abnormal behavior" refers to behavior that deviates from normal behavior, such as bullying or trouble.

[0013] "Abnormal voice" refers to voices that deviate from normal voices, such as abusive language or cries for help.

[0014] An "alert" is a warning message sent to relevant educators when abnormal behavior or abnormal voice is detected.

[0015] "Disaster" refers to a sudden and serious event such as an earthquake or fire.

[0016] An "evacuation order" is an instruction message issued to ensure safety in the event of a disaster.

[0017] "Monitoring" is the process of continuously observing evacuation situations and other developments in real time.

[0018] "Safety check" is an action to confirm that all students have evacuated safely.

[0019] "Parent" means a student's parent or legal guardian.

[0020] "Related institutions" refers to public institutions such as fire departments and police stations. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention is a system for ensuring safety in educational institutions such as elementary and junior high schools, and for realizing a prompt and appropriate response when a problem occurs. A specific embodiment of this system is described below.

[0043] 1. Video data collection and preprocessing

[0044] The server collects video data in real time from multiple cameras installed within the educational institution. This video data is acquired from various areas such as classrooms, hallways, gymnasiums, and schoolyards. The server checks the quality of the collected video data and performs pre-processing such as noise reduction and image resolution correction.

[0045] 2. Video data analysis and problem detection

[0046] The server then inputs the preprocessed video data into behavioral and speech recognition algorithms for analysis. This allows the system to evaluate students' behavioral patterns and detect abnormal behavior or sounds. For example, if two students are making inappropriate contact in a hallway, the AI ​​will analyze this and recognize it as a sign of "bullying."

[0047] 3. Send real-time alerts to educators

[0048] If the server detects abnormal behavior or abnormal audio, it immediately generates an alert and sends it to the relevant educator's device. The device then displays the received alert on its screen and notifies the user via audio and visual means. The user (educator) can check the notification and rush to the scene to deal with the problematic behavior immediately.

[0049] 4. Disaster response

[0050] The server reads data from external devices such as earthquake sensors and fire alarms to detect sudden disasters. When a disaster occurs, the server immediately issues an emergency alert to the entire school and sends evacuation instructions to terminals in each area. For example, a terminal in a classroom might display a voice message saying, "Earthquake has occurred. Please evacuate," along with evacuation routes.

[0051] 5. Monitoring evacuation situations and sharing information

[0052] The server analyzes camera footage from each classroom and school in real time to monitor the evacuation status of students. It confirms that students have begun evacuation and issues another warning if any students are still unable to escape. Once safety is confirmed, the server automatically sends a notification that the evacuation is complete to parents and relevant organizations. The user (educator) receives the evacuation report and provides additional evacuation instructions or student care as necessary.

[0053] Specific examples

[0054] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0055] As described above, the present invention can ensure safety in educational institutions such as elementary and junior high schools, and realize efficient and prompt responses.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] The server collects video data in real time from multiple cameras installed throughout the school. Each camera is installed in a different area, such as a classroom, hallway, gymnasium, or schoolyard, and continuously monitors the behavior and situation of students.

[0059] Step 2:

[0060] The server checks the quality of the acquired video data and performs noise reduction and image resolution correction, resulting in clean data suitable for analysis. This pre-processing step is important to ensure data consistency and accuracy.

[0061] Step 3:

[0062] The preprocessed video data is input into an AI model integrated in the server. The AI ​​model evaluates students' behavior, facial expressions, and voice using behavioral and voice recognition algorithms to detect abnormal student behavior (e.g., bullying) and abnormal voice (e.g., shouting).

[0063] Step 4:

[0064] If any abnormal behavior or sound is detected, the server immediately generates an alert, which includes details of the problem and its location, and is sent to the device of the relevant educator.

[0065] Step 5:

[0066] The device will notify the educator of the received alert visually and audibly. The alert will include specific details of the abnormal behavior or abnormal sound, as well as recommended actions to take. The user (educator) will check the notification and immediately rush to the scene to address the problem.

[0067] Step 6:

[0068] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0069] Step 7:

[0070] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0071] Step 8:

[0072] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0073] Step 9:

[0074] As a follow-up after the evacuation, the server records all data for later analysis and reporting, allowing for improvements to be identified and future disaster response capabilities to be enhanced.

[0075] Through these steps, the system improves safety within educational institutions and enables quick and effective responses to emergencies such as bullying and disasters.

[0076] Example 1

[0077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0078] Conventional safety management systems in educational institutions have difficulty responding immediately to abnormal student behavior or sudden disasters. Detecting abnormal behavior, issuing evacuation instructions, and monitoring evacuation situations are all done manually, requiring rapid and accurate responses. This has led to problems with insufficient efforts to ensure student safety.

[0079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0080] In this invention, the server includes means for collecting video data in real time from multiple visual sensors installed in the educational institution, means for quality checking the video data, performing noise reduction and image resolution correction, means for analyzing the preprocessed video data using behavior recognition algorithms and voice recognition algorithms to evaluate behavioral patterns, means for detecting abnormal behavior and abnormal voices and sending alerts to relevant educators in real time, means for reading data from earthquake sensors and fire alarms to detect sudden disasters and issue evacuation instructions, means for analyzing camera images in each area in real time, monitoring the evacuation status of students, and confirming the completion of evacuation, and means for notifying parents and relevant organizations of the completion of evacuation. This enables quick and accurate response to abnormal student behavior and sudden disasters.

[0081] "Educational institutions" are facilities that provide various types of education, such as elementary and junior high schools, high schools, vocational schools, and universities.

[0082] A "visual sensor" is a device for capturing video and images, and typically operates as a camera.

[0083] "Video data" refers to image and video information acquired by a visual sensor and stored in digital format.

[0084] "Quality check" refers to checking the clarity and resolution of collected video data, as well as the presence or absence of blur and noise, and then processing the data to maintain appropriate quality.

[0085] "Noise reduction" is the process of reducing or removing unwanted noise contained in video data.

[0086] "Image resolution correction" is the process of adjusting the resolution of video data to an appropriate level.

[0087] An "action recognition algorithm" is an algorithm that analyzes the movements and actions of people in video data and identifies and classifies them.

[0088] A "voice recognition algorithm" is an algorithm that analyzes voice data and identifies and classifies the content of speech and the type of sound.

[0089] An "alert" is a warning message sent to relevant educators when abnormal behavior or abnormal voice is detected.

[0090] An "earthquake sensor" is a device that detects earthquake vibrations and shaking, and is a sensor that responds quickly.

[0091] A "fire alarm" is a device that detects the occurrence of a fire and sounds an alarm.

[0092] An "evacuation order" is a message that instructs people to evacuate to a safe place when a sudden disaster occurs.

[0093] "Monitoring" is the process of monitoring student behavior and evacuation status in real time and managing them appropriately.

[0094] "Evacuation completed" refers to the state in which all students have properly evacuated to a safe location.

[0095] "Related agencies" refers to public agencies and support organizations related to school operations and student safety.

[0096] MODE FOR CARRYING OUT THE INVENTION

[0097] The present invention is a safety management system for educational institutions, and more specifically, a system that can detect abnormal behavior of students and respond to sudden disasters in real time.

[0098] 1. Video data collection and preprocessing

[0099] The server collects video data in real time from multiple visual sensors installed within the educational institution. Specific hardware used is network cameras. For example, cameras placed in each classroom, hallway, gymnasium, schoolyard, etc. send video data to the server. The server checks the quality of this data, and performs noise reduction and image resolution correction using a software library called OpenCV.

[0100] 2. Video data analysis and problem detection

[0101] The server then analyzes the preprocessed video data using behavioral and speech recognition algorithms. This analysis uses behavioral recognition algorithms powered by Google®'s TENSORFLOW® and speech recognition algorithms powered by Amazon's Alexa Voice Service. For example, if two students are making inappropriate contact in a hallway, this is recognized as "bullying" and detected as an anomaly.

[0102] 3. Send real-time alerts to educators

[0103] If the server detects abnormal behavior or abnormal audio, it immediately generates an alert and sends it to the device of the relevant educator. The alert is displayed on the screen and notified through audio and visual means. For example, an alert stating "Inappropriate contact has occurred in classroom A" is displayed on the educator's device. The user (educator) can confirm this and quickly deal with the problematic behavior by rushing to the scene.

[0104] 4. Disaster response

[0105] The server reads data from earthquake sensors and fire alarms to detect sudden disasters. For example, if the earthquake sensor detects an earthquake, the server will send an emergency alert to the entire school, and a voice message saying "Please evacuate" and evacuation routes will be displayed on terminals in each area.

[0106] 5. Monitoring evacuation situations and sharing information

[0107] The server analyzes camera footage from each classroom and school in real time to monitor the evacuation situation. For example, it checks the cameras to see if students have begun evacuation, and issues another warning if any students are still unable to escape. Once safety is confirmed, the server automatically sends a notification that the evacuation is complete to parents and relevant organizations. The user (educator) receives the evacuation report and issues additional evacuation instructions or provides student care as necessary.

[0108] Specific examples

[0109] For example, if an earthquake occurs in a classroom, the server immediately detects it with the ShakeAlert sensor and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors camera footage in real time and confirms that all students have safely evacuated. Once evacuation is complete, parents and relevant organizations are notified and the specific situation is promptly shared.

[0110] Prompt Sentence Examples

[0111] Here are some example prompts for a generative AI model:

[0112] "Please explain in detail the system you use to detect abnormal behavior and sounds within your school. Please also provide details on the algorithms, hardware used, and notification methods."

[0113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0114] Step 1:

[0115] The server collects real-time video data sent by multiple visual sensors (network cameras) installed in educational institutions. Specifically, video data from each camera is input to the server. The server receives this video data as a stream. The input of this step is the video data from the cameras, and the output is the raw video data stored on the server.

[0116] Step 2:

[0117] The server performs a quality check on the received video data. Specifically, it uses the OpenCV library to evaluate the clarity, resolution, and presence or absence of blur and noise in the video data. During this process, the server analyzes each frame and extracts frames that do not meet the quality criteria. The input to this step is raw video data, and the output is video data with quality information added.

[0118] Step 3:

[0119] The server then performs noise reduction and image resolution correction on the quality-checked video data. Specifically, it uses OpenCV's fastNlMeansDenoisingColored function to remove noise and the resize function to adjust the resolution appropriately. The input to this step is the video data with quality information added, and the output is the preprocessed video data.

[0120] Step 4:

[0121] The server inputs the preprocessed video data into behavior recognition and speech recognition algorithms for analysis. Google's TensorFlow is used for behavior recognition, and Amazon's Alexa Voice Service is used for speech recognition. Specifically, the server analyzes the movements and postures of people in the video and the content of the audio data to detect abnormal behavior or audio. The input for this step is the preprocessed video data, and the output is the analysis results.

[0122] Step 5:

[0123] If the server detects abnormal behavior or abnormal audio based on the results of behavior and audio recognition, it immediately generates an alert. Specifically, it creates an alert message that includes details of the location, time, and circumstances of the abnormal behavior or audio. The input to this step is the analysis results, and the output is an alert message.

[0124] Step 6:

[0125] The server sends the generated alert to the device of the relevant educator. The device displays the received alert message on the screen and notifies the user by voice and visual means. As a specific example, an alert stating "Inappropriate contact has occurred in classroom A. Please rush to the scene" is displayed on the device. The input of this step is the alert message, and the output is a notification to the device.

[0126] Step 7:

[0127] The server reads data from earthquake sensors and fire alarms in real time to detect sudden disasters. Specifically, it analyzes the data sent from the sensors and immediately detects the occurrence of earthquakes or fires. The input to this step is data from the sensors, and the output is the disaster detection results.

[0128] Step 8:

[0129] When the server detects a disaster, it issues an emergency alert to the entire school and sends evacuation instructions to terminals in each area. Specifically, it displays a voice message such as "Earthquake has occurred. Please evacuate" along with evacuation routes. The input to this step is the disaster detection result, and the output is an emergency alert and evacuation instructions.

[0130] Step 9:

[0131] The server analyzes the camera footage from each classroom and school in real time and monitors the evacuation status of students. Specifically, it confirms that students have begun evacuation and issues another warning if any students are unable to escape in time. The input for this step is the real-time camera footage, and the output is the analysis result of the evacuation status.

[0132] Step 10:

[0133] Once the safe evacuation of all students is confirmed, the server sends a notification of evacuation completion to parents and relevant organizations. Specifically, it automatically sends a message such as, "The evacuation of all students has been completed. Please rest assured." The input for this step is the analysis result of the evacuation situation, and the output is a notification of evacuation completion.

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] In the past, educational institutions and industrial facilities lacked systems for detecting abnormal behavior or abnormal sounds in real time and responding quickly. Furthermore, evacuation instructions and management of evacuation situations during sudden disasters were also inefficient. Furthermore, while industrial facilities require ensuring safety and responding quickly to abnormal situations, existing technologies were unable to adequately resolve these issues. This resulted in a significant loss of safety and efficiency in educational institutions and industrial facilities.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server includes means for collecting video data in real time from multiple camera devices installed in educational institutions or industrial facilities, means for preprocessing the video data and preparing for analysis, means for analyzing the preprocessed video data and evaluating the behavior, facial expression, voice, and positional relationship of a target, means for detecting abnormal behavior or abnormal voice and sending an alert to relevant parties, means for detecting a sudden disaster and issuing evacuation instructions, means for monitoring the evacuation situation in real time and confirming safety, means for notifying relevant parties and organizations of the completion of evacuation, means for evaluating the operating status of machines and generating an alert if an abnormality is detected, and means for ensuring safety at industrial facilities and realizing efficient and rapid response to abnormalities, thereby enabling rapid and accurate response to abnormal behavior or disasters in educational institutions and industrial facilities.

[0139] A "camera" is a device for collecting video data in real time.

[0140] A "server" is a computer system that processes, stores, and analyzes data.

[0141] "Video data" refers to the video signal and its video information collected from the imaging device.

[0142] "Preprocessing" refers to the process of removing noise and correcting the resolution of collected video data to prepare it for analysis.

[0143] "Analysis" refers to the evaluation of processed video data to analyze subject behavior, facial expressions, voice, and positional relationships.

[0144] "Evaluation" refers to analyzing data and judging, recognizing, or evaluating it based on specific criteria.

[0145] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and affects safety and efficiency.

[0146] "Abnormal sounds" refer to sounds that deviate from normal sound patterns and that affect safety and efficiency.

[0147] An "alert" is a signal that warns an individual about detected abnormal behavior or sound.

[0148] "Disaster" refers to sudden natural phenomena or accidents such as earthquakes and fires.

[0149] An "evacuation order" is an instruction to guide people to a safe place in the event of a disaster.

[0150] "Monitoring" means observing the situation in real time and observing changes or abnormalities.

[0151] A "motion recognition algorithm" is a mathematical and statistical method for analyzing and recognizing the behavior and movements of a subject.

[0152] A "speech recognition algorithm" is a mathematical and statistical method for analyzing and recognizing a target's speech.

[0153] An "environmental sensor" is a sensor device used to detect environmental changes such as earthquakes and fires.

[0154] The present invention provides a system for ensuring safety in educational institutions and industrial facilities and for responding to abnormalities efficiently and quickly. Specific embodiments of this system are described below.

[0155] The server collects video data in real time from camera devices installed in educational or industrial facilities. This collected video data undergoes quality checks, noise reduction, image resolution correction, and other pre-processing. This pre-processing removes noise and blurred areas from the collected video data, making it ready for accurate analysis.

[0156] After preprocessing, the video data is analyzed to evaluate the target's behavior, voice, and positional relationship. The server uses motion and voice recognition algorithms to detect abnormal behavior or voice. If an abnormality is detected, an alert is immediately generated and sent to the relevant device, enabling a prompt response on-site.

[0157] In addition, the server reads data from environmental sensors such as earthquake sensors and fire alarms to detect the occurrence of sudden disasters. In the event of a disaster, the server immediately issues evacuation instructions and displays evacuation routes and instructions on terminals in each area.

[0158] After the evacuation has begun, the server analyzes the camera footage and monitors the evacuation situation in real time. If any individuals are unable to escape in time, it issues another warning, and once safety is confirmed, it notifies relevant parties and organizations that the evacuation has been completed.

[0159] In industrial facilities, the server also monitors the operation status of machines and generates alerts if an abnormality is detected, thereby preventing breakdowns and problems before they occur.

[0160] For example, if machine A in a factory makes an abnormal operating noise, the server will detect it and send a notification to the manager's terminal saying, "Machine A is making an abnormal noise." At this time, the manager can immediately rush to the site and take action.

[0161] The system also utilizes a generative AI model, allowing users to input prompts to detect abnormal behavior and speech, such as:

[0162] An AI algorithm that detects abnormal behavior and sounds in real time from camera footage within a factory and generates alerts. For example, it recognizes inappropriate contact between workers or abnormal sounds and immediately sends a notification to the manager's device.

[0163] This will significantly improve the safety of educational institutions and industrial facilities and enable efficient and rapid response.

[0164] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0165] Step 1:

[0166] The server collects video data in real time from multiple camera devices installed in educational or industrial facilities. It receives the video data sent by the camera devices (input) and stores it in a database (output). This collection process involves continuously capturing the video stream from each camera and adding necessary metadata (timestamp, camera location, etc.).

[0167] Step 2:

[0168] The server preprocesses the collected video data, specifically by reducing noise and correcting image resolution (input: video data, output: preprocessed video data). This preprocessing step uses filtering techniques and edge detection algorithms to reduce noise and improve image clarity.

[0169] Step 3:

[0170] The server analyzes the preprocessed video data and evaluates the subject's behavior, facial expressions, voice, and positional relationship (input: preprocessed video data, output: analysis results). Specifically, it uses motion recognition and voice recognition algorithms to detect abnormalities in behavior and voice. This involves analyzing the subject's behavioral patterns and identifying abnormalities.

[0171] Step 4:

[0172] If the server detects abnormal behavior or abnormal audio, it generates an alert and sends it to the device of the relevant person (input: analysis result, output: alert notification). Specifically, when an abnormality is detected, it creates a notification message and sends it to the device. This message includes the type of abnormality, the location where it was detected, and a timestamp.

[0173] Step 5:

[0174] The server reads data from earthquake sensors, fire alarm systems, etc. to detect sudden disasters (input: sensor data, output: disaster alerts). It periodically monitors data from the sensors and generates a disaster alert if an abnormal value is detected.

[0175] Step 6:

[0176] In the event of a disaster, the server sends evacuation instructions to terminals in each area (input: disaster alert, output: evacuation instructions). Specifically, it displays a message containing safe evacuation routes and instructions on the terminals and issues instructions via voice message.

[0177] Step 7:

[0178] After evacuation has begun, the server analyzes the camera footage and monitors the evacuation status in real time (input: camera footage, output: evacuation status report). The server detects moving objects in the footage and evaluates the progress of the evacuation.

[0179] Step 8:

[0180] Once safety is confirmed, the server notifies the relevant parties and organizations that the evacuation has been completed (input: evacuation status report, output: evacuation completion notification). Specifically, after confirming that everyone is safe, the server automatically sends notifications to the relevant parties.

[0181] Step 9:

[0182] The server monitors the operation status of machines in industrial facilities and generates an alert if an abnormality is detected (input: machine operation data, output: machine abnormality alert). It uses a behavior recognition algorithm to identify abnormal machine behavior and immediately notify relevant parties.

[0183] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0184] The present invention is a system for ensuring safety in educational institutions such as elementary and junior high schools, enabling quick and accurate responses when problems occur, and is also combined with an emotion engine that recognizes the user's emotions. A specific embodiment of this system is shown below.

[0185] 1. Video data collection and preprocessing

[0186] The server collects video data in real time from multiple cameras installed around the school. This video data is collected from various areas, such as classrooms, hallways, gymnasiums, and schoolyards. The server checks the quality of the collected video data and performs pre-processing such as noise reduction and image resolution correction.

[0187] 2. Analysis of video data

[0188] The server then inputs the preprocessed video data into behavioral and speech recognition algorithms for analysis. This evaluates students' behavioral patterns, facial expressions, and voice to detect abnormal behavior or voice. For example, if two students are making inappropriate contact in a hallway, the AI ​​will analyze this and recognize it as a sign of "bullying."

[0189] 3. Emotion Recognition by Emotion Engine

[0190] The server uses an emotion engine to analyze students' emotions from video and audio data. The emotion engine identifies emotions such as sadness, anger, and fear, improving the accuracy of detecting abnormal behavior. For example, if a student appears sad in the classroom, this could be a sign of a potential problem, allowing for early intervention.

[0191] 4. Send real-time alerts to educators

[0192] If the server detects abnormal behavior, abnormal voice, or abnormal emotion based on the emotion engine, it immediately generates an alert, which includes detailed information about the problem and its location, and is sent to the device of the relevant educator.

[0193] The device will notify the educator of the received alert visually and audibly. The alert content will include specific details of abnormal behavior, abnormal voice, or abnormal emotion, as well as recommended actions to take. The user (educator) can check the notification and immediately rush to the scene to address the problem.

[0194] 5. Disaster response

[0195] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0196] 6. Evacuation situation monitoring and information sharing

[0197] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0198] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0199] Specific examples

[0200] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0201] This system, combined with an emotion engine, will further improve safety within educational institutions and enable quick and effective responses to emergencies such as bullying and disasters.

[0202] The processing flow will be explained below.

[0203] Step 1:

[0204] The server collects video data in real time from multiple cameras installed throughout the school. Each camera is installed in a different area, such as a classroom, hallway, gymnasium, or schoolyard, and continuously monitors the behavior and situation of students.

[0205] Step 2:

[0206] The server preprocesses the acquired video data, specifically by performing noise reduction and image resolution correction to obtain clean data suitable for analysis. This preprocessing step is important to ensure data consistency and accuracy.

[0207] Step 3:

[0208] The preprocessed video data is input into an AI model integrated on the server. The AI ​​model evaluates students' behavior, facial expressions, and voice using behavior and voice recognition algorithms to detect abnormal student behavior (e.g., bullying) and abnormal voice (e.g., shouting).

[0209] Step 4:

[0210] The server analyzes students' emotions from the video and audio data using an emotion engine, which identifies emotions such as sadness, anger, and fear, thereby improving the accuracy of detecting abnormal behavior.

[0211] Step 5:

[0212] If abnormal behavior, voice, or emotion is detected, the server immediately generates an alert, which includes details of the problem and its location, and is sent to the device of the relevant educator.

[0213] Step 6:

[0214] The device will notify the educator of the received alert visually and audibly. The alert content will include specific details of abnormal behavior, abnormal voice, or abnormal emotion, as well as recommended actions to take. The user (educator) can check the notification and immediately rush to the scene to address the problem.

[0215] Step 7:

[0216] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," urging students and educators to evacuate.

[0217] Step 8:

[0218] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0219] Step 9:

[0220] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0221] Step 10:

[0222] As a follow-up after the evacuation, the server records all data for later analysis and reporting, allowing for improvements to be identified and future disaster response capabilities to be enhanced.

[0223] Example 2

[0224] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0225] Conventional safety management systems in educational institutions lack the functionality to detect abnormal student behavior or voice activity in real time and respond quickly. It is also difficult to issue evacuation instructions and monitor evacuation situations quickly during disasters, making it difficult to ensure student safety. Furthermore, they lack the technology to properly analyze student emotions and detect potential problems early. As a result, responses to issues such as bullying and violence are delayed, potentially leading to a decline in the quality of the educational environment.

[0226] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting video data in real time from multiple camera devices installed in the educational institution, means for preprocessing the video data to perform quality checks, noise reduction, and image resolution correction, means for analyzing the preprocessed video data and evaluating students' behavioral patterns, facial expressions, voice, and positional relationships, means for detecting abnormal behavior and voice and sending alerts to relevant educators, means for analyzing students' emotions (sadness, anger, fear, etc.) using an emotion recognition engine to improve the accuracy of detecting abnormal behavior, means for detecting sudden disasters and issuing evacuation instructions, means for monitoring the evacuation situation in real time and confirming safety, and means for notifying parents and relevant organizations of the completion of evacuation. This comprehensively ensures the safety of students and enables early detection and rapid response to problems.

[0227] "Capture equipment" refers to equipment such as cameras that are installed within educational institutions and are used to collect video data in real time.

[0228] "Video data" refers to digital video information that shows the students and facilities collected by a camera.

[0229] "Preprocessing" refers to the process of performing quality checks, noise reduction, image resolution correction, etc. on collected video data.

[0230] A "behavioral pattern" is a sequence of specific student actions or behaviors obtained from the analyzed video data.

[0231] "Facial expressions" refer to the emotions and reactions that can be read from the students' facial expressions.

[0232] "Speech" refers to acoustic information such as human voices and environmental sounds that are analyzed using a speech recognition algorithm.

[0233] "Positional relationship" is information that indicates the spatial arrangement and relationships of students and objects.

[0234] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and requires vigilance or action.

[0235] "Abnormal sounds" refer to sounds that are different from normal sounds, such as cries or calls for help, that require vigilance or a response.

[0236] An "alert" is a warning message sent to relevant educators when abnormal behavior or sound is detected.

[0237] An "emotion recognition engine" is a software technology that analyzes video and audio data to identify emotions such as sadness, anger, and fear.

[0238] "Sudden disasters" refer to natural disasters and accidents, such as earthquakes and fires, that occur unexpectedly and require a rapid response.

[0239] An "evacuation order" is an instruction to students and educators to move to a safe place in the event of a sudden disaster.

[0240] "Evacuation situation" refers to the progress of evacuation actions by students and educators after an evacuation order is issued.

[0241] "Safety check" is the act of making sure that students and educators are safe when evacuation is complete.

[0242] "Guardian" refers to the person registered as the parent or guardian of a student.

[0243] "Affiliated institutions" refers to external organizations or facilities that have a relationship with an educational institution (e.g., fire department, police department, etc.).

[0244] This invention is a comprehensive surveillance system that improves safety in educational institutions and enables rapid and appropriate response when problems occur. The system collects video data in real time, preprocesses and analyzes it, and detects abnormal behavior or abnormal sounds and notifies educators. Furthermore, it uses an emotion recognition engine to analyze student emotions and improve the accuracy of detecting abnormal behavior. Furthermore, it issues evacuation instructions in the event of a sudden disaster, monitors evacuation conditions, and confirms safety.

[0245] Hardware and Software Configuration

[0246] The system uses the following hardware and software:

[0247] Hardware: High-resolution IP cameras, earthquake sensors, fire alarms

[0248] Software: OpenCV (image processing library), TensorFlow (machine learning library), EmotionAPI (emotion recognition API)

[0249] Server Roles

[0250] The server collects video data in real time from multiple high-resolution IP cameras installed on campus. The collected video data undergoes preprocessing, such as noise reduction and image resolution correction. The preprocessed video data is then analyzed using behavior recognition algorithms (using OpenCV and TensorFlow) to evaluate students' behavioral patterns, facial expressions, and voice. In addition, an emotion recognition engine (EmotionAPI) is used to analyze students' emotions from the video and audio data.

[0251] When abnormal behavior, abnormal voice, or abnormal emotion is detected by the emotion recognition engine, an alert is immediately generated. The server sends this alert to the device of the relevant educator, enabling a prompt response.

[0252] Device Role

[0253] The device will notify the educator of the received alert visually and audibly. The alert will include details of the problem and the location where it occurred. For example, it could say, "Student A is sad in the classroom. Details: Student A was captured on video crying alone. Action is required." After the educator confirms this, they can immediately rush to the scene and take action.

[0254] User Roles

[0255] The user (educator) checks the alert from the device and immediately rushes to the scene to deal with the problem. Also, in the event of a sudden disaster, the user receives evacuation instructions from the server and instructs students to evacuate. The server constantly monitors data from earthquake sensors and fire alarms, and immediately issues evacuation instructions when it detects a disaster. Devices in each classroom and area display audio messages and visual displays such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0256] Specific examples

[0257] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0258] Prompt Sentence Examples

[0259] The prompt sentence to be input to the generative AI model is shown below.

[0260] "You are building a safety management system for an elementary school. Explain how you can collect video data from cameras in real time, use behavior recognition algorithms and an emotion engine to detect abnormal behaviors and emotions, and send alerts to educators."

[0261] This prompt allows the generative AI model to suggest relevant technical approaches and detailed processing methods, thus comprehensively ensuring safety within educational institutions and enabling early detection and rapid response to problems.

[0262] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0263] Step 1:

[0264] The server collects video data in real time from multiple camera devices installed on campus. The server receives the collected video data as input, checks the video quality, and performs pre-processing such as noise reduction and image resolution correction. The output of this pre-processing is improved video data. Specifically, the server continuously acquires the video stream from the cameras, applies noise reduction filters, and executes correction algorithms to improve image quality.

[0265] Step 2:

[0266] The server inputs the preprocessed video data and analyzes it using behavioral and speech recognition algorithms. Specifically, it evaluates students' behavioral patterns, facial expressions, and audio data from the video data and detects abnormal behavior and audio based on these. The output of this analysis is the detection results of abnormal behavior and audio. For example, the behavioral recognition algorithm identifies whether a student is performing a specific action in the video, and the speech recognition algorithm detects audio events such as screaming or calling for help.

[0267] Step 3:

[0268] The server then inputs the analyzed video and audio data using an emotion recognition engine to identify the student's emotions. The output of this emotion recognition is an emotion label, such as sadness, anger, or fear. At this stage, the student's facial expressions and voice tone captured on video are analyzed and mapped to specific emotions. For example, EmotionAPI analyzes a student's facial expression and determines whether it corresponds to "sadness."

[0269] Step 4:

[0270] The server generates an alert when it detects abnormal behavior, abnormal voice, or abnormal emotion based on the analysis results. This alert includes detailed information about the problem and the location where it occurred. The generated alert is sent to the educator's device. The input for this transmission is the analysis result, and the output is notification information that is displayed on the relevant educator's device. Specifically, the server classifies the analysis result, composes an appropriate alert message, and sends it to the educator in real time.

[0271] Step 5:

[0272] The device will notify the educator of the received alert visually and audibly. The input here is the alert information sent from the server, and the output is an alert message displayed on the teacher's device. For example, a message such as "Student A is crying in the classroom. Details: Student A has been captured on video crying alone. Action required" will be displayed.

[0273] Step 6:

[0274] The server constantly monitors data from earthquake sensors and fire alarms, and if it detects a disaster, it automatically issues an emergency alert and sends evacuation instructions to the entire school. The input here is data from the earthquake sensors and fire alarms, and the output is an alert message that sends evacuation instructions. For example, the moment an earthquake is detected, an audio and visual message such as "Earthquake has occurred. Please evacuate" is displayed on terminals in each classroom and area.

[0275] Step 7:

[0276] The server analyzes camera footage in real time and monitors the evacuation status from each area. It checks for any obstructions in the evacuation routes and automatically issues a second warning. The input is video data from the school cameras, and the output is real-time information on the evacuation status. If a obstruction is detected, an alert message ordering a second evacuation is generated and sent to each device.

[0277] Step 8:

[0278] After the evacuation is complete, the server notifies parents and related organizations of the completion of the evacuation. The input here is the detection result of the evacuation completion, and the output is a notification message sent to parents and related organizations. Specifically, once safety confirmation is complete, a message such as "Evacuation Complete Notification: All students and educators have evacuated safely" is sent to parents and related organizations.

[0279] In this way, the entire system can comprehensively ensure student safety and enable early detection and rapid response to problems.

[0280] (Application example 2)

[0281] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0282] The present invention is required to ensure safety in places where many people gather, such as educational institutions and office buildings, and to respond quickly to abnormal behavior and emergencies. However, current systems are unable to adequately detect abnormal behavior and emotions in real time, making it difficult to respond quickly and accurately. Furthermore, even in the event of a sudden disaster, the system is unable to monitor evacuation situations in real time and confirm overall safety, which may result in delayed response.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0284] In this invention, the server includes: means for collecting video data in real time from multiple cameras installed in educational institutions or other facilities; means for preprocessing the video data and preparing for analysis; means for analyzing the preprocessed video data and evaluating the behavior, facial expressions, voice, and positional relationship of subjects; means for detecting abnormal behavior or abnormal voice and sending alerts to relevant parties; means for analyzing the emotions of subjects from the video data and voice data using an emotion engine and detecting abnormal emotions; means for detecting sudden disasters and issuing evacuation instructions; means for monitoring the evacuation situation in real time and confirming safety; and means for notifying parents and relevant organizations of the completion of evacuation. This enables the detection of abnormal behavior or abnormal emotions in real time from multiple camera footage, enabling rapid and accurate response. Furthermore, it is possible to quickly issue evacuation instructions, monitor the evacuation situation, and confirm safety even in the event of a sudden disaster.

[0285] An "educational institution or other facility" is a place where people gather to carry out educational, business, or other activities, including, but not limited to, a school.

[0286] "Multiple cameras" refer to imaging devices that are installed at different locations and serve to collect video data from multiple viewpoints simultaneously.

[0287] "Means for collecting video data in real time" refers to a method and device for instantly importing video captured by a camera into a system such as a server.

[0288] "Means for preprocessing and preparing for analysis" refers to methods and devices for reducing noise in acquired video data, correcting resolution, etc., to make the data suitable for analysis.

[0289] "Means for analyzing preprocessed video data and assessing subject behavior, facial expressions, voice, and positional relationships" refers to a method and apparatus that uses preprocessed data to identify and assess a subject's behavioral patterns, emotional expressions, vocal content, and spatial relationships.

[0290] "Means for detecting abnormal behavior or abnormal sounds and sending alerts to relevant parties" refers to a method and device for detecting unusual behavior or sounds from analyzed data and notifying relevant parties of that information.

[0291] An "emotion engine" is software or algorithms that analyze video and audio data to identify a subject's emotional state, such as joy, anger, sadness, or fear.

[0292] The "means for analyzing emotions" refers to a method and apparatus for analyzing emotions from specific data using an emotion engine.

[0293] "Means for detecting sudden disasters and issuing evacuation instructions" refers to methods and devices for instantly recognizing emergencies such as earthquakes and fires and issuing appropriate evacuation instructions based on that information.

[0294] "Means for monitoring evacuation situations in real time and confirming safety" refers to a method and device for monitoring the movements of people during evacuation in real time and confirming whether the safety of everyone is ensured.

[0295] "Means for notifying parents and related organizations of the completion of evacuation" refers to methods and devices for confirming that evacuation has been completed safely and for quickly and accurately communicating the results to parents and related organizations.

[0296] The present invention provides a system for ensuring safety in educational institutions or other facilities and for responding quickly and appropriately to abnormal behavior or emergencies. This system is constructed using the hardware and software described below.

[0297] The system involves installing multiple cameras in designated locations and collecting video data in real time. For example, in educational institutions, cameras are placed in classrooms, hallways, gymnasiums, and schoolyards, while in office buildings, cameras are installed at the entrance and in shared spaces on each floor.

[0298] The server receives the collected video data and performs preprocessing, which includes noise reduction and image resolution correction. The preprocessed video data is then analyzed to evaluate the subject's behavior, facial expression, voice, and positional relationship using behavior and voice recognition algorithms.

[0299] The server uses an emotion engine to analyze the subject's emotions from video and audio data. The emotion engine identifies emotions such as sadness, anger, and fear, improving the accuracy of detecting abnormal behavior. If abnormal behavior, audio, or emotion is detected, an alert is sent to the relevant parties. The alert is immediately displayed on a notification terminal, allowing the relevant parties to respond quickly.

[0300] In addition, the server constantly monitors data from earthquake sensors and fire alarms to detect sudden disasters. If a disaster is detected, the server automatically issues an emergency alert and sends evacuation instructions to all facilities. During an evacuation, camera footage is analyzed, the evacuation situation is monitored in real time, and safety is confirmed. Once everyone has evacuated, the server notifies parents and relevant organizations that the evacuation is complete.

[0301] As a concrete example, consider the case where someone shouts at the entrance of an office building. The server analyzes the camera footage and audio to detect anger. If it determines that the behavior is abnormal, an alert is immediately sent to the relevant parties. The alert includes the location of the abnormality and how to respond, allowing the relevant parties to quickly rush to the scene and take appropriate action.

[0302] An example prompt is, "We would like to develop a security system for an office building. Please design a system that analyzes camera footage, performs emotion recognition, and sends an alert if there is any abnormal behavior."

[0303] In this way, abnormal behavior and abnormal emotions can be detected in real time from multiple camera footage, enabling a quick and appropriate response. It can also be used to quickly issue evacuation instructions, monitor evacuation situations, and confirm safety in the event of a sudden disaster.

[0304] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0305] Step 1:

[0306] The server collects video data in real time from multiple cameras installed in educational institutions or other facilities.

[0307] Input: Video feed from camera

[0308] Output: Raw video data

[0309] Specific operation: Continuously collects video data from multiple locations where cameras are installed and sends it to a server.

[0310] Step 2:

[0311] The server pre-processes the collected video data.

[0312] Input: Raw video data

[0313] Output: Pre-processed video data

[0314] Specific operation: Performs noise reduction and image resolution correction to make the image suitable for analysis. For example, improves the quality of the video data using OpenCV.

[0315] Step 3:

[0316] The server analyzes the pre-processed video data and evaluates the subject's behavior, facial expression, voice, and position.

[0317] Input: Preprocessed video data

[0318] Output: Evaluation results on the subject's behavior, facial expression, voice, and positional relationship

[0319] Specific operation: Using behavioral and voice recognition algorithms, the system analyzes video and audio to identify the target's behavioral patterns, facial expressions, vocalizations, etc.

[0320] Step 4:

[0321] The server uses an emotion engine to analyze the emotion of the target from the video data and audio data.

[0322] Input: Preprocessed video and audio data

[0323] Output: Analysis results regarding the subject's emotions

[0324] Specific operation: For example, the EmotionRecognizer library is used to analyze facial expressions and voice tones in video to identify emotions. The emotion engine identifies emotions such as sadness, anger, and fear.

[0325] Step 5:

[0326] If the server detects abnormal behavior, abnormal voice, or abnormal emotions, it will send an alert to relevant parties.

[0327] Input: Analysis results of behavior, voice, and emotions

[0328] Output: Alert message

[0329] Specific operation: If abnormal behavior or abnormal emotions are detected, an alert message will be generated and sent to the device or smartphone, for example, via email or push notification.

[0330] Step 6:

[0331] The server detects sudden disasters and issues evacuation instructions.

[0332] Input: Data from disaster sensors and fire alarms

[0333] Output: Evacuation instruction message

[0334] Specific operation: Data from earthquake sensors and fire alarms is constantly monitored, and if a disaster is detected, an emergency alert is automatically issued and evacuation instructions are sent to all facilities.

[0335] Step 7:

[0336] The server monitors the evacuation situation in real time and confirms safety.

[0337] Input: Camera footage during evacuation

[0338] Output: Monitoring results of evacuation situation

[0339] Specific operation: Analyzes camera footage and monitors in real time whether people are evacuating properly. If there is a problem, it sends another warning.

[0340] Step 8:

[0341] After the evacuation is completed, the server notifies the parents and related organizations of the completion of the evacuation.

[0342] Input: Confirmation data of evacuation completion

[0343] Output: Evacuation completion notification

[0344] Specific operations: After confirming that everyone has evacuated, an evacuation completion notification will be sent to parents and relevant organizations via email or message.

[0345] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0346] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0347] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0348] [Second embodiment]

[0349] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0350] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0351] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0352] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0353] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0354] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0355] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0356] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0357] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0358] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0359] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0360] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0361] The present invention is a system for ensuring safety in educational institutions such as elementary and junior high schools, and for realizing a prompt and appropriate response when a problem occurs. A specific embodiment of this system is described below.

[0362] 1. Video data collection and preprocessing

[0363] The server collects video data in real time from multiple cameras installed within the educational institution. This video data is acquired from various areas such as classrooms, hallways, gymnasiums, and schoolyards. The server checks the quality of the collected video data and performs pre-processing such as noise reduction and image resolution correction.

[0364] 2. Video data analysis and problem detection

[0365] The server then inputs the preprocessed video data into behavioral and speech recognition algorithms for analysis. This allows the system to evaluate students' behavioral patterns and detect abnormal behavior or sounds. For example, if two students are making inappropriate contact in a hallway, the AI ​​will analyze this and recognize it as a sign of "bullying."

[0366] 3. Send real-time alerts to educators

[0367] If the server detects abnormal behavior or abnormal audio, it immediately generates an alert and sends it to the relevant educator's device. The device then displays the received alert on its screen and notifies the user via audio and visual means. The user (educator) can check the notification and rush to the scene to deal with the problematic behavior immediately.

[0368] 4. Disaster response

[0369] The server reads data from external devices such as earthquake sensors and fire alarms to detect sudden disasters. When a disaster occurs, the server immediately issues an emergency alert to the entire school and sends evacuation instructions to terminals in each area. For example, a terminal in a classroom might display a voice message saying, "Earthquake has occurred. Please evacuate," along with evacuation routes.

[0370] 5. Monitoring evacuation situations and sharing information

[0371] The server analyzes camera footage from each classroom and school in real time to monitor the evacuation status of students. It confirms that students have begun evacuation and issues another warning if any students are still unable to escape. Once safety is confirmed, the server automatically sends a notification that the evacuation is complete to parents and relevant organizations. The user (educator) receives the evacuation report and provides additional evacuation instructions or student care as necessary.

[0372] Specific examples

[0373] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0374] As described above, the present invention can ensure safety in educational institutions such as elementary and junior high schools, and realize efficient and prompt responses.

[0375] The processing flow will be explained below.

[0376] Step 1:

[0377] The server collects video data in real time from multiple cameras installed throughout the school. Each camera is installed in a different area, such as a classroom, hallway, gymnasium, or schoolyard, and continuously monitors the behavior and situation of students.

[0378] Step 2:

[0379] The server checks the quality of the acquired video data and performs noise reduction and image resolution correction, resulting in clean data suitable for analysis. This pre-processing step is important to ensure data consistency and accuracy.

[0380] Step 3:

[0381] The preprocessed video data is input into an AI model integrated in the server. The AI ​​model evaluates students' behavior, facial expressions, and voice using behavioral and voice recognition algorithms to detect abnormal student behavior (e.g., bullying) and abnormal voice (e.g., shouting).

[0382] Step 4:

[0383] If any abnormal behavior or sound is detected, the server immediately generates an alert, which includes details of the problem and its location, and is sent to the device of the relevant educator.

[0384] Step 5:

[0385] The device will notify the educator of the received alert visually and audibly. The alert will include specific details of the abnormal behavior or abnormal sound, as well as recommended actions to take. The user (educator) will check the notification and immediately rush to the scene to address the problem.

[0386] Step 6:

[0387] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0388] Step 7:

[0389] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0390] Step 8:

[0391] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0392] Step 9:

[0393] As a follow-up after the evacuation, the server records all data for later analysis and reporting, allowing for improvements to be identified and future disaster response capabilities to be enhanced.

[0394] Through these steps, the system improves safety within educational institutions and enables quick and effective responses to emergencies such as bullying and disasters.

[0395] Example 1

[0396] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0397] Conventional safety management systems in educational institutions have difficulty responding immediately to abnormal student behavior or sudden disasters. Detecting abnormal behavior, issuing evacuation instructions, and monitoring evacuation situations are all done manually, requiring rapid and accurate responses. This has led to problems with insufficient efforts to ensure student safety.

[0398] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0399] In this invention, the server includes means for collecting video data in real time from multiple visual sensors installed in the educational institution, means for quality checking the video data, performing noise reduction and image resolution correction, means for analyzing the preprocessed video data using behavior recognition algorithms and voice recognition algorithms to evaluate behavioral patterns, means for detecting abnormal behavior and abnormal voices and sending alerts to relevant educators in real time, means for reading data from earthquake sensors and fire alarms to detect sudden disasters and issue evacuation instructions, means for analyzing camera images in each area in real time, monitoring the evacuation status of students, and confirming the completion of evacuation, and means for notifying parents and relevant organizations of the completion of evacuation. This enables quick and accurate response to abnormal student behavior and sudden disasters.

[0400] "Educational institutions" are facilities that provide various types of education, such as elementary and junior high schools, high schools, vocational schools, and universities.

[0401] A "visual sensor" is a device for capturing video and images, and typically operates as a camera.

[0402] "Video data" refers to image and video information acquired by a visual sensor and stored in digital format.

[0403] "Quality check" refers to checking the clarity and resolution of collected video data, as well as the presence or absence of blur and noise, and then processing the data to maintain appropriate quality.

[0404] "Noise reduction" is the process of reducing or removing unwanted noise contained in video data.

[0405] "Image resolution correction" is the process of adjusting the resolution of video data to an appropriate level.

[0406] An "action recognition algorithm" is an algorithm that analyzes the movements and actions of people in video data and identifies and classifies them.

[0407] A "voice recognition algorithm" is an algorithm that analyzes voice data and identifies and classifies the content of speech and the type of sound.

[0408] An "alert" is a warning message sent to relevant educators when abnormal behavior or abnormal voice is detected.

[0409] An "earthquake sensor" is a device that detects earthquake vibrations and shaking, and is a sensor that responds quickly.

[0410] A "fire alarm" is a device that detects the occurrence of a fire and sounds an alarm.

[0411] An "evacuation order" is a message that instructs people to evacuate to a safe place when a sudden disaster occurs.

[0412] "Monitoring" is the process of monitoring student behavior and evacuation status in real time and managing them appropriately.

[0413] "Evacuation completed" refers to the state in which all students have properly evacuated to a safe location.

[0414] "Related agencies" refers to public agencies and support organizations related to school operations and student safety.

[0415] MODE FOR CARRYING OUT THE INVENTION

[0416] The present invention is a safety management system for educational institutions, and more specifically, a system that can detect abnormal behavior of students and respond to sudden disasters in real time.

[0417] 1. Video data collection and preprocessing

[0418] The server collects video data in real time from multiple visual sensors installed within the educational institution. Specific hardware used is network cameras. For example, cameras placed in each classroom, hallway, gymnasium, schoolyard, etc. send video data to the server. The server checks the quality of this data, and performs noise reduction and image resolution correction using a software library called OpenCV.

[0419] 2. Video data analysis and problem detection

[0420] The server then analyzes the preprocessed video data using behavioral and speech recognition algorithms. For this analysis, behavioral recognition algorithms using Google's TensorFlow and speech recognition algorithms using Amazon's Alexa Voice Service are used. For example, if two students are making inappropriate contact in a hallway, this is recognized as "bullying" and detected as an anomaly.

[0421] 3. Send real-time alerts to educators

[0422] If the server detects abnormal behavior or abnormal audio, it immediately generates an alert and sends it to the device of the relevant educator. The alert is displayed on the screen and notified through audio and visual means. For example, an alert stating "Inappropriate contact has occurred in classroom A" is displayed on the educator's device. The user (educator) can confirm this and quickly deal with the problematic behavior by rushing to the scene.

[0423] 4. Disaster response

[0424] The server reads data from earthquake sensors and fire alarms to detect sudden disasters. For example, if the earthquake sensor detects an earthquake, the server will send an emergency alert to the entire school, and a voice message saying "Please evacuate" and evacuation routes will be displayed on terminals in each area.

[0425] 5. Monitoring evacuation situations and sharing information

[0426] The server analyzes camera footage from each classroom and school in real time to monitor the evacuation situation. For example, it checks the cameras to see if students have begun evacuation, and issues another warning if any students are still unable to escape. Once safety is confirmed, the server automatically sends a notification that the evacuation is complete to parents and relevant organizations. The user (educator) receives the evacuation report and issues additional evacuation instructions or provides student care as necessary.

[0427] Specific examples

[0428] For example, if an earthquake occurs in a classroom, the server immediately detects it with the ShakeAlert sensor and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors camera footage in real time and confirms that all students have safely evacuated. Once evacuation is complete, parents and relevant organizations are notified and the specific situation is promptly shared.

[0429] Prompt Sentence Examples

[0430] Here are some example prompts for a generative AI model:

[0431] "Please explain in detail the system you use to detect abnormal behavior and sounds within your school. Please also provide details on the algorithms, hardware used, and notification methods."

[0432] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0433] Step 1:

[0434] The server collects real-time video data sent by multiple visual sensors (network cameras) installed in educational institutions. Specifically, video data from each camera is input to the server. The server receives this video data as a stream. The input of this step is the video data from the cameras, and the output is the raw video data stored on the server.

[0435] Step 2:

[0436] The server performs a quality check on the received video data. Specifically, it uses the OpenCV library to evaluate the clarity, resolution, and presence or absence of blur and noise in the video data. During this process, the server analyzes each frame and extracts frames that do not meet the quality criteria. The input to this step is raw video data, and the output is video data with quality information added.

[0437] Step 3:

[0438] The server then performs noise reduction and image resolution correction on the quality-checked video data. Specifically, it uses OpenCV's fastNlMeansDenoisingColored function to remove noise and the resize function to adjust the resolution appropriately. The input to this step is the video data with quality information added, and the output is the preprocessed video data.

[0439] Step 4:

[0440] The server inputs the preprocessed video data into behavior recognition and speech recognition algorithms for analysis. Google's TensorFlow is used for behavior recognition, and Amazon's Alexa Voice Service is used for speech recognition. Specifically, the server analyzes the movements and postures of people in the video and the content of the audio data to detect abnormal behavior or audio. The input for this step is the preprocessed video data, and the output is the analysis results.

[0441] Step 5:

[0442] If the server detects abnormal behavior or abnormal audio based on the results of behavior and audio recognition, it immediately generates an alert. Specifically, it creates an alert message that includes details of the location, time, and circumstances of the abnormal behavior or audio. The input to this step is the analysis results, and the output is an alert message.

[0443] Step 6:

[0444] The server sends the generated alert to the device of the relevant educator. The device displays the received alert message on the screen and notifies the user by voice and visual means. As a specific example, an alert stating "Inappropriate contact has occurred in classroom A. Please rush to the scene" is displayed on the device. The input of this step is the alert message, and the output is a notification to the device.

[0445] Step 7:

[0446] The server reads data from earthquake sensors and fire alarms in real time to detect sudden disasters. Specifically, it analyzes the data sent from the sensors and immediately detects the occurrence of earthquakes or fires. The input to this step is data from the sensors, and the output is the disaster detection results.

[0447] Step 8:

[0448] When the server detects a disaster, it issues an emergency alert to the entire school and sends evacuation instructions to terminals in each area. Specifically, it displays a voice message such as "Earthquake has occurred. Please evacuate" along with evacuation routes. The input to this step is the disaster detection result, and the output is an emergency alert and evacuation instructions.

[0449] Step 9:

[0450] The server analyzes the camera footage from each classroom and school in real time and monitors the evacuation status of students. Specifically, it confirms that students have begun evacuation and issues another warning if any students are unable to escape in time. The input for this step is the real-time camera footage, and the output is the analysis result of the evacuation status.

[0451] Step 10:

[0452] Once the safe evacuation of all students is confirmed, the server sends a notification of evacuation completion to parents and relevant organizations. Specifically, it automatically sends a message such as, "The evacuation of all students has been completed. Please rest assured." The input for this step is the analysis result of the evacuation situation, and the output is a notification of evacuation completion.

[0453] (Application example 1)

[0454] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0455] In the past, educational institutions and industrial facilities lacked systems for detecting abnormal behavior or abnormal sounds in real time and responding quickly. Furthermore, evacuation instructions and management of evacuation situations during sudden disasters were also inefficient. Furthermore, while industrial facilities require ensuring safety and responding quickly to abnormal situations, existing technologies were unable to adequately resolve these issues. This resulted in a significant loss of safety and efficiency in educational institutions and industrial facilities.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0457] In this invention, the server includes means for collecting video data in real time from multiple camera devices installed in educational institutions or industrial facilities, means for preprocessing the video data and preparing for analysis, means for analyzing the preprocessed video data and evaluating the behavior, facial expression, voice, and positional relationship of a target, means for detecting abnormal behavior or abnormal voice and sending an alert to relevant parties, means for detecting a sudden disaster and issuing evacuation instructions, means for monitoring the evacuation situation in real time and confirming safety, means for notifying relevant parties and organizations of the completion of evacuation, means for evaluating the operating status of machines and generating an alert if an abnormality is detected, and means for ensuring safety at industrial facilities and realizing efficient and rapid response to abnormalities, thereby enabling rapid and accurate response to abnormal behavior or disasters in educational institutions and industrial facilities.

[0458] A "camera" is a device for collecting video data in real time.

[0459] A "server" is a computer system that processes, stores, and analyzes data.

[0460] "Video data" refers to the video signal and its video information collected from the imaging device.

[0461] "Preprocessing" refers to the process of removing noise and correcting the resolution of collected video data to prepare it for analysis.

[0462] "Analysis" refers to the evaluation of processed video data to analyze subject behavior, facial expressions, voice, and positional relationships.

[0463] "Evaluation" refers to analyzing data and judging, recognizing, or evaluating it based on specific criteria.

[0464] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and affects safety and efficiency.

[0465] "Abnormal sounds" refer to sounds that deviate from normal sound patterns and that affect safety and efficiency.

[0466] An "alert" is a signal that warns an individual about detected abnormal behavior or sound.

[0467] "Disaster" refers to sudden natural phenomena or accidents such as earthquakes and fires.

[0468] An "evacuation order" is an instruction to guide people to a safe place in the event of a disaster.

[0469] "Monitoring" means observing the situation in real time and observing changes or abnormalities.

[0470] A "motion recognition algorithm" is a mathematical and statistical method for analyzing and recognizing the behavior and movements of a subject.

[0471] A "speech recognition algorithm" is a mathematical and statistical method for analyzing and recognizing a target's speech.

[0472] An "environmental sensor" is a sensor device used to detect environmental changes such as earthquakes and fires.

[0473] The present invention provides a system for ensuring safety in educational institutions and industrial facilities and for responding to abnormalities efficiently and quickly. Specific embodiments of this system are described below.

[0474] The server collects video data in real time from camera devices installed in educational or industrial facilities. This collected video data undergoes quality checks, noise reduction, image resolution correction, and other pre-processing. This pre-processing removes noise and blurred areas from the collected video data, making it ready for accurate analysis.

[0475] After preprocessing, the video data is analyzed to evaluate the target's behavior, voice, and positional relationship. The server uses motion and voice recognition algorithms to detect abnormal behavior or voice. If an abnormality is detected, an alert is immediately generated and sent to the relevant device, enabling a prompt response on-site.

[0476] In addition, the server reads data from environmental sensors such as earthquake sensors and fire alarms to detect the occurrence of sudden disasters. In the event of a disaster, the server immediately issues evacuation instructions and displays evacuation routes and instructions on terminals in each area.

[0477] After the evacuation has begun, the server analyzes the camera footage and monitors the evacuation situation in real time. If any individuals are unable to escape in time, it issues another warning, and once safety is confirmed, it notifies relevant parties and organizations that the evacuation has been completed.

[0478] In industrial facilities, the server also monitors the operation status of machines and generates alerts if an abnormality is detected, thereby preventing breakdowns and problems before they occur.

[0479] For example, if machine A in a factory makes an abnormal operating noise, the server will detect it and send a notification to the manager's terminal saying, "Machine A is making an abnormal noise." At this time, the manager can immediately rush to the site and take action.

[0480] The system also utilizes a generative AI model, allowing users to input prompts to detect abnormal behavior and speech, such as:

[0481] An AI algorithm that detects abnormal behavior and sounds in real time from camera footage within a factory and generates alerts. For example, it recognizes inappropriate contact between workers or abnormal sounds and immediately sends a notification to the manager's device.

[0482] This will significantly improve the safety of educational institutions and industrial facilities and enable efficient and rapid response.

[0483] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0484] Step 1:

[0485] The server collects video data in real time from multiple camera devices installed in educational or industrial facilities. It receives the video data sent by the camera devices (input) and stores it in a database (output). This collection process involves continuously capturing the video stream from each camera and adding necessary metadata (timestamp, camera location, etc.).

[0486] Step 2:

[0487] The server preprocesses the collected video data, specifically by reducing noise and correcting image resolution (input: video data, output: preprocessed video data). This preprocessing step uses filtering techniques and edge detection algorithms to reduce noise and improve image clarity.

[0488] Step 3:

[0489] The server analyzes the preprocessed video data and evaluates the subject's behavior, facial expressions, voice, and positional relationship (input: preprocessed video data, output: analysis results). Specifically, it uses motion recognition and voice recognition algorithms to detect abnormalities in behavior and voice. This involves analyzing the subject's behavioral patterns and identifying abnormalities.

[0490] Step 4:

[0491] If the server detects abnormal behavior or abnormal audio, it generates an alert and sends it to the device of the relevant person (input: analysis result, output: alert notification). Specifically, when an abnormality is detected, it creates a notification message and sends it to the device. This message includes the type of abnormality, the location where it was detected, and a timestamp.

[0492] Step 5:

[0493] The server reads data from earthquake sensors, fire alarm systems, etc. to detect sudden disasters (input: sensor data, output: disaster alerts). It periodically monitors data from the sensors and generates a disaster alert if an abnormal value is detected.

[0494] Step 6:

[0495] In the event of a disaster, the server sends evacuation instructions to terminals in each area (input: disaster alert, output: evacuation instructions). Specifically, it displays a message containing safe evacuation routes and instructions on the terminals and issues instructions via voice message.

[0496] Step 7:

[0497] After evacuation has begun, the server analyzes the camera footage and monitors the evacuation status in real time (input: camera footage, output: evacuation status report). The server detects moving objects in the footage and evaluates the progress of the evacuation.

[0498] Step 8:

[0499] Once safety is confirmed, the server notifies the relevant parties and organizations that the evacuation has been completed (input: evacuation status report, output: evacuation completion notification). Specifically, after confirming that everyone is safe, the server automatically sends notifications to the relevant parties.

[0500] Step 9:

[0501] The server monitors the operation status of machines in industrial facilities and generates an alert if an abnormality is detected (input: machine operation data, output: machine abnormality alert). It uses a behavior recognition algorithm to identify abnormal machine behavior and immediately notify relevant parties.

[0502] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0503] The present invention is a system for ensuring safety in educational institutions such as elementary and junior high schools, enabling quick and accurate responses when problems occur, and is also combined with an emotion engine that recognizes the user's emotions. A specific embodiment of this system is shown below.

[0504] 1. Video data collection and preprocessing

[0505] The server collects video data in real time from multiple cameras installed around the school. This video data is collected from various areas, such as classrooms, hallways, gymnasiums, and schoolyards. The server checks the quality of the collected video data and performs pre-processing such as noise reduction and image resolution correction.

[0506] 2. Analysis of video data

[0507] The server then inputs the preprocessed video data into behavioral and speech recognition algorithms for analysis. This evaluates students' behavioral patterns, facial expressions, and voice to detect abnormal behavior or voice. For example, if two students are making inappropriate contact in a hallway, the AI ​​will analyze this and recognize it as a sign of "bullying."

[0508] 3. Emotion Recognition by Emotion Engine

[0509] The server uses an emotion engine to analyze students' emotions from video and audio data. The emotion engine identifies emotions such as sadness, anger, and fear, improving the accuracy of detecting abnormal behavior. For example, if a student appears sad in the classroom, this could be a sign of a potential problem, allowing for early intervention.

[0510] 4. Send real-time alerts to educators

[0511] If the server detects abnormal behavior, abnormal voice, or abnormal emotion based on the emotion engine, it immediately generates an alert, which includes detailed information about the problem and its location, and is sent to the device of the relevant educator.

[0512] The device will notify the educator of the received alert visually and audibly. The alert content will include specific details of abnormal behavior, abnormal voice, or abnormal emotion, as well as recommended actions to take. The user (educator) can check the notification and immediately rush to the scene to address the problem.

[0513] 5. Disaster response

[0514] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0515] 6. Evacuation situation monitoring and information sharing

[0516] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0517] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0518] Specific examples

[0519] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0520] This system, combined with an emotion engine, will further improve safety within educational institutions and enable quick and effective responses to emergencies such as bullying and disasters.

[0521] The processing flow will be explained below.

[0522] Step 1:

[0523] The server collects video data in real time from multiple cameras installed throughout the school. Each camera is installed in a different area, such as a classroom, hallway, gymnasium, or schoolyard, and continuously monitors the behavior and situation of students.

[0524] Step 2:

[0525] The server preprocesses the acquired video data, specifically by performing noise reduction and image resolution correction to obtain clean data suitable for analysis. This preprocessing step is important to ensure data consistency and accuracy.

[0526] Step 3:

[0527] The preprocessed video data is input into an AI model integrated on the server. The AI ​​model evaluates students' behavior, facial expressions, and voice using behavior and voice recognition algorithms to detect abnormal student behavior (e.g., bullying) and abnormal voice (e.g., shouting).

[0528] Step 4:

[0529] The server analyzes students' emotions from the video and audio data using an emotion engine, which identifies emotions such as sadness, anger, and fear, thereby improving the accuracy of detecting abnormal behavior.

[0530] Step 5:

[0531] If abnormal behavior, voice, or emotion is detected, the server immediately generates an alert, which includes details of the problem and its location, and is sent to the device of the relevant educator.

[0532] Step 6:

[0533] The device will notify the educator of the received alert visually and audibly. The alert content will include specific details of abnormal behavior, abnormal voice, or abnormal emotion, as well as recommended actions to take. The user (educator) can check the notification and immediately rush to the scene to address the problem.

[0534] Step 7:

[0535] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," urging students and educators to evacuate.

[0536] Step 8:

[0537] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0538] Step 9:

[0539] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0540] Step 10:

[0541] As a follow-up after the evacuation, the server records all data for later analysis and reporting, allowing for improvements to be identified and future disaster response capabilities to be enhanced.

[0542] Example 2

[0543] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0544] Conventional safety management systems in educational institutions lack the functionality to detect abnormal student behavior or voice activity in real time and respond quickly. It is also difficult to issue evacuation instructions and monitor evacuation situations quickly during disasters, making it difficult to ensure student safety. Furthermore, they lack the technology to properly analyze student emotions and detect potential problems early. As a result, responses to issues such as bullying and violence are delayed, potentially leading to a decline in the quality of the educational environment.

[0545] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting video data in real time from multiple camera devices installed in the educational institution, means for preprocessing the video data to perform quality checks, noise reduction, and image resolution correction, means for analyzing the preprocessed video data and evaluating students' behavioral patterns, facial expressions, voice, and positional relationships, means for detecting abnormal behavior and voice and sending alerts to relevant educators, means for analyzing students' emotions (sadness, anger, fear, etc.) using an emotion recognition engine to improve the accuracy of detecting abnormal behavior, means for detecting sudden disasters and issuing evacuation instructions, means for monitoring the evacuation situation in real time and confirming safety, and means for notifying parents and relevant organizations of the completion of evacuation. This comprehensively ensures the safety of students and enables early detection and rapid response to problems.

[0546] "Capture equipment" refers to equipment such as cameras that are installed within educational institutions and are used to collect video data in real time.

[0547] "Video data" refers to digital video information that shows the students and facilities collected by a camera.

[0548] "Preprocessing" refers to the process of performing quality checks, noise reduction, image resolution correction, etc. on collected video data.

[0549] A "behavioral pattern" is a sequence of specific student actions or behaviors obtained from the analyzed video data.

[0550] "Facial expressions" refer to the emotions and reactions that can be read from the students' facial expressions.

[0551] "Speech" refers to acoustic information such as human voices and environmental sounds that are analyzed using a speech recognition algorithm.

[0552] "Positional relationship" is information that indicates the spatial arrangement and relationships of students and objects.

[0553] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and requires vigilance or action.

[0554] "Abnormal sounds" refer to sounds that are different from normal sounds, such as cries or calls for help, that require vigilance or a response.

[0555] An "alert" is a warning message sent to relevant educators when abnormal behavior or sound is detected.

[0556] An "emotion recognition engine" is a software technology that analyzes video and audio data to identify emotions such as sadness, anger, and fear.

[0557] "Sudden disasters" refer to natural disasters and accidents, such as earthquakes and fires, that occur unexpectedly and require a rapid response.

[0558] An "evacuation order" is an instruction to students and educators to move to a safe place in the event of a sudden disaster.

[0559] "Evacuation situation" refers to the progress of evacuation actions by students and educators after an evacuation order is issued.

[0560] "Safety check" is the act of making sure that students and educators are safe when evacuation is complete.

[0561] "Guardian" refers to the person registered as the parent or guardian of a student.

[0562] "Affiliated institutions" refers to external organizations or facilities that have a relationship with an educational institution (e.g., fire department, police department, etc.).

[0563] This invention is a comprehensive surveillance system that improves safety in educational institutions and enables rapid and appropriate response when problems occur. The system collects video data in real time, preprocesses and analyzes it, and detects abnormal behavior or abnormal sounds and notifies educators. Furthermore, it uses an emotion recognition engine to analyze student emotions and improve the accuracy of detecting abnormal behavior. Furthermore, it issues evacuation instructions in the event of a sudden disaster, monitors evacuation conditions, and confirms safety.

[0564] Hardware and Software Configuration

[0565] The system uses the following hardware and software:

[0566] Hardware: High-resolution IP cameras, earthquake sensors, fire alarms

[0567] Software: OpenCV (image processing library), TensorFlow (machine learning library), EmotionAPI (emotion recognition API)

[0568] Server Roles

[0569] The server collects video data in real time from multiple high-resolution IP cameras installed on campus. The collected video data undergoes preprocessing, such as noise reduction and image resolution correction. The preprocessed video data is then analyzed using behavior recognition algorithms (using OpenCV and TensorFlow) to evaluate students' behavioral patterns, facial expressions, and voice. In addition, an emotion recognition engine (EmotionAPI) is used to analyze students' emotions from the video and audio data.

[0570] When abnormal behavior, abnormal voice, or abnormal emotion is detected by the emotion recognition engine, an alert is immediately generated. The server sends this alert to the device of the relevant educator, enabling a prompt response.

[0571] Device Role

[0572] The device will notify the educator of the received alert visually and audibly. The alert will include details of the problem and the location where it occurred. For example, it could say, "Student A is sad in the classroom. Details: Student A was captured on video crying alone. Action is required." After the educator confirms this, they can immediately rush to the scene and take action.

[0573] User Roles

[0574] The user (educator) checks the alert from the device and immediately rushes to the scene to deal with the problem. Also, in the event of a sudden disaster, the user receives evacuation instructions from the server and instructs students to evacuate. The server constantly monitors data from earthquake sensors and fire alarms, and immediately issues evacuation instructions when it detects a disaster. Devices in each classroom and area display audio messages and visual displays such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0575] Specific examples

[0576] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0577] Prompt Sentence Examples

[0578] The prompt sentence to be input to the generative AI model is shown below.

[0579] "You are building a safety management system for an elementary school. Explain how you can collect video data from cameras in real time, use behavior recognition algorithms and an emotion engine to detect abnormal behaviors and emotions, and send alerts to educators."

[0580] This prompt allows the generative AI model to suggest relevant technical approaches and detailed processing methods, thus comprehensively ensuring safety within educational institutions and enabling early detection and rapid response to problems.

[0581] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0582] Step 1:

[0583] The server collects video data in real time from multiple camera devices installed on campus. The server receives the collected video data as input, checks the video quality, and performs pre-processing such as noise reduction and image resolution correction. The output of this pre-processing is improved video data. Specifically, the server continuously acquires the video stream from the cameras, applies noise reduction filters, and executes correction algorithms to improve image quality.

[0584] Step 2:

[0585] The server inputs the preprocessed video data and analyzes it using behavioral and speech recognition algorithms. Specifically, it evaluates students' behavioral patterns, facial expressions, and audio data from the video data and detects abnormal behavior and audio based on these. The output of this analysis is the detection results of abnormal behavior and audio. For example, the behavioral recognition algorithm identifies whether a student is performing a specific action in the video, and the speech recognition algorithm detects audio events such as screaming or calling for help.

[0586] Step 3:

[0587] The server then inputs the analyzed video and audio data using an emotion recognition engine to identify the student's emotions. The output of this emotion recognition is an emotion label, such as sadness, anger, or fear. At this stage, the student's facial expressions and voice tone captured on video are analyzed and mapped to specific emotions. For example, EmotionAPI analyzes a student's facial expression and determines whether it corresponds to "sadness."

[0588] Step 4:

[0589] The server generates an alert when it detects abnormal behavior, abnormal voice, or abnormal emotion based on the analysis results. This alert includes detailed information about the problem and the location where it occurred. The generated alert is sent to the educator's device. The input for this transmission is the analysis result, and the output is notification information that is displayed on the relevant educator's device. Specifically, the server classifies the analysis result, composes an appropriate alert message, and sends it to the educator in real time.

[0590] Step 5:

[0591] The device will notify the educator of the received alert visually and audibly. The input here is the alert information sent from the server, and the output is an alert message displayed on the teacher's device. For example, a message such as "Student A is crying in the classroom. Details: Student A has been captured on video crying alone. Action required" will be displayed.

[0592] Step 6:

[0593] The server constantly monitors data from earthquake sensors and fire alarms, and if it detects a disaster, it automatically issues an emergency alert and sends evacuation instructions to the entire school. The input here is data from the earthquake sensors and fire alarms, and the output is an alert message that sends evacuation instructions. For example, the moment an earthquake is detected, an audio and visual message such as "Earthquake has occurred. Please evacuate" is displayed on terminals in each classroom and area.

[0594] Step 7:

[0595] The server analyzes camera footage in real time and monitors the evacuation status from each area. It checks for any obstructions in the evacuation routes and automatically issues a second warning. The input is video data from the school cameras, and the output is real-time information on the evacuation status. If a obstruction is detected, an alert message ordering a second evacuation is generated and sent to each device.

[0596] Step 8:

[0597] After the evacuation is complete, the server notifies parents and related organizations of the completion of the evacuation. The input here is the detection result of the evacuation completion, and the output is a notification message sent to parents and related organizations. Specifically, once safety confirmation is complete, a message such as "Evacuation Complete Notification: All students and educators have evacuated safely" is sent to parents and related organizations.

[0598] In this way, the entire system can comprehensively ensure student safety and enable early detection and rapid response to problems.

[0599] (Application example 2)

[0600] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0601] The present invention is required to ensure safety in places where many people gather, such as educational institutions and office buildings, and to respond quickly to abnormal behavior and emergencies. However, current systems are unable to adequately detect abnormal behavior and emotions in real time, making it difficult to respond quickly and accurately. Furthermore, even in the event of a sudden disaster, the system is unable to monitor evacuation situations in real time and confirm overall safety, which may result in delayed response.

[0602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0603] In this invention, the server includes: means for collecting video data in real time from multiple cameras installed in educational institutions or other facilities; means for preprocessing the video data and preparing for analysis; means for analyzing the preprocessed video data and evaluating the behavior, facial expressions, voice, and positional relationship of subjects; means for detecting abnormal behavior or abnormal voice and sending alerts to relevant parties; means for analyzing the emotions of subjects from the video data and voice data using an emotion engine and detecting abnormal emotions; means for detecting sudden disasters and issuing evacuation instructions; means for monitoring the evacuation situation in real time and confirming safety; and means for notifying parents and relevant organizations of the completion of evacuation. This enables the detection of abnormal behavior or abnormal emotions in real time from multiple camera footage, enabling rapid and accurate response. Furthermore, it is possible to quickly issue evacuation instructions, monitor the evacuation situation, and confirm safety even in the event of a sudden disaster.

[0604] An "educational institution or other facility" is a place where people gather to carry out educational, business, or other activities, including, but not limited to, a school.

[0605] "Multiple cameras" refer to imaging devices that are installed at different locations and serve to collect video data from multiple viewpoints simultaneously.

[0606] "Means for collecting video data in real time" refers to a method and device for instantly importing video captured by a camera into a system such as a server.

[0607] "Means for preprocessing and preparing for analysis" refers to methods and devices for reducing noise in acquired video data, correcting resolution, etc., to make the data suitable for analysis.

[0608] "Means for analyzing preprocessed video data and assessing subject behavior, facial expressions, voice, and positional relationships" refers to a method and apparatus that uses preprocessed data to identify and assess a subject's behavioral patterns, emotional expressions, vocal content, and spatial relationships.

[0609] "Means for detecting abnormal behavior or abnormal sounds and sending alerts to relevant parties" refers to a method and device for detecting unusual behavior or sounds from analyzed data and notifying relevant parties of that information.

[0610] An "emotion engine" is software or algorithms that analyze video and audio data to identify a subject's emotional state, such as joy, anger, sadness, or fear.

[0611] The "means for analyzing emotions" refers to a method and apparatus for analyzing emotions from specific data using an emotion engine.

[0612] "Means for detecting sudden disasters and issuing evacuation instructions" refers to methods and devices for instantly recognizing emergencies such as earthquakes and fires and issuing appropriate evacuation instructions based on that information.

[0613] "Means for monitoring evacuation situations in real time and confirming safety" refers to a method and device for monitoring the movements of people during evacuation in real time and confirming whether the safety of everyone is ensured.

[0614] "Means for notifying parents and related organizations of the completion of evacuation" refers to methods and devices for confirming that evacuation has been completed safely and for quickly and accurately communicating the results to parents and related organizations.

[0615] The present invention provides a system for ensuring safety in educational institutions or other facilities and for responding quickly and appropriately to abnormal behavior or emergencies. This system is constructed using the hardware and software described below.

[0616] The system involves installing multiple cameras in designated locations and collecting video data in real time. For example, in educational institutions, cameras are placed in classrooms, hallways, gymnasiums, and schoolyards, while in office buildings, cameras are installed at the entrance and in shared spaces on each floor.

[0617] The server receives the collected video data and performs preprocessing, which includes noise reduction and image resolution correction. The preprocessed video data is then analyzed to evaluate the subject's behavior, facial expression, voice, and positional relationship using behavior and voice recognition algorithms.

[0618] The server uses an emotion engine to analyze the subject's emotions from video and audio data. The emotion engine identifies emotions such as sadness, anger, and fear, improving the accuracy of detecting abnormal behavior. If abnormal behavior, audio, or emotion is detected, an alert is sent to the relevant parties. The alert is immediately displayed on a notification terminal, allowing the relevant parties to respond quickly.

[0619] In addition, the server constantly monitors data from earthquake sensors and fire alarms to detect sudden disasters. If a disaster is detected, the server automatically issues an emergency alert and sends evacuation instructions to all facilities. During an evacuation, camera footage is analyzed, the evacuation situation is monitored in real time, and safety is confirmed. Once everyone has evacuated, the server notifies parents and relevant organizations that the evacuation is complete.

[0620] As a concrete example, consider the case where someone shouts at the entrance of an office building. The server analyzes the camera footage and audio to detect anger. If it determines that the behavior is abnormal, an alert is immediately sent to the relevant parties. The alert includes the location of the abnormality and how to respond, allowing the relevant parties to quickly rush to the scene and take appropriate action.

[0621] An example prompt is, "We would like to develop a security system for an office building. Please design a system that analyzes camera footage, performs emotion recognition, and sends an alert if there is any abnormal behavior."

[0622] In this way, abnormal behavior and abnormal emotions can be detected in real time from multiple camera footage, enabling a quick and appropriate response. It can also be used to quickly issue evacuation instructions, monitor evacuation situations, and confirm safety in the event of a sudden disaster.

[0623] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0624] Step 1:

[0625] The server collects video data in real time from multiple cameras installed in educational institutions or other facilities.

[0626] Input: Video feed from camera

[0627] Output: Raw video data

[0628] Specific operation: Continuously collects video data from multiple locations where cameras are installed and sends it to a server.

[0629] Step 2:

[0630] The server pre-processes the collected video data.

[0631] Input: Raw video data

[0632] Output: Pre-processed video data

[0633] Specific operation: Performs noise reduction and image resolution correction to make the image suitable for analysis. For example, improves the quality of the video data using OpenCV.

[0634] Step 3:

[0635] The server analyzes the pre-processed video data and evaluates the subject's behavior, facial expression, voice, and position.

[0636] Input: Preprocessed video data

[0637] Output: Evaluation results on the subject's behavior, facial expression, voice, and positional relationship

[0638] Specific operation: Using behavioral and voice recognition algorithms, the system analyzes video and audio to identify the target's behavioral patterns, facial expressions, vocalizations, etc.

[0639] Step 4:

[0640] The server uses an emotion engine to analyze the emotion of the target from the video data and audio data.

[0641] Input: Preprocessed video and audio data

[0642] Output: Analysis results regarding the subject's emotions

[0643] Specific operation: For example, the EmotionRecognizer library is used to analyze facial expressions and voice tones in video to identify emotions. The emotion engine identifies emotions such as sadness, anger, and fear.

[0644] Step 5:

[0645] If the server detects abnormal behavior, abnormal voice, or abnormal emotions, it will send an alert to relevant parties.

[0646] Input: Analysis results of behavior, voice, and emotions

[0647] Output: Alert message

[0648] Specific operation: If abnormal behavior or abnormal emotions are detected, an alert message will be generated and sent to the device or smartphone, for example, via email or push notification.

[0649] Step 6:

[0650] The server detects sudden disasters and issues evacuation instructions.

[0651] Input: Data from disaster sensors and fire alarms

[0652] Output: Evacuation instruction message

[0653] Specific operation: Data from earthquake sensors and fire alarms is constantly monitored, and if a disaster is detected, an emergency alert is automatically issued and evacuation instructions are sent to all facilities.

[0654] Step 7:

[0655] The server monitors the evacuation situation in real time and confirms safety.

[0656] Input: Camera footage during evacuation

[0657] Output: Monitoring results of evacuation situation

[0658] Specific operation: Analyzes camera footage and monitors in real time whether people are evacuating properly. If there is a problem, it sends another warning.

[0659] Step 8:

[0660] After the evacuation is completed, the server notifies the parents and related organizations of the completion of the evacuation.

[0661] Input: Confirmation data of evacuation completion

[0662] Output: Evacuation completion notification

[0663] Specific operations: After confirming that everyone has evacuated, an evacuation completion notification will be sent to parents and relevant organizations via email or message.

[0664] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0665] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0666] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0667] [Third embodiment]

[0668] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0669] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0670] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0671] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0672] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0673] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0674] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0675] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0676] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0677] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0678] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0679] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0680] The present invention is a system for ensuring safety in educational institutions such as elementary and junior high schools, and for realizing a prompt and appropriate response when a problem occurs. A specific embodiment of this system is described below.

[0681] 1. Video data collection and preprocessing

[0682] The server collects video data in real time from multiple cameras installed within the educational institution. This video data is acquired from various areas such as classrooms, hallways, gymnasiums, and schoolyards. The server checks the quality of the collected video data and performs pre-processing such as noise reduction and image resolution correction.

[0683] 2. Video data analysis and problem detection

[0684] The server then inputs the preprocessed video data into behavioral and speech recognition algorithms for analysis. This allows the system to evaluate students' behavioral patterns and detect abnormal behavior or sounds. For example, if two students are making inappropriate contact in a hallway, the AI ​​will analyze this and recognize it as a sign of "bullying."

[0685] 3. Send real-time alerts to educators

[0686] If the server detects abnormal behavior or abnormal audio, it immediately generates an alert and sends it to the relevant educator's device. The device then displays the received alert on its screen and notifies the user via audio and visual means. The user (educator) can check the notification and rush to the scene to deal with the problematic behavior immediately.

[0687] 4. Disaster response

[0688] The server reads data from external devices such as earthquake sensors and fire alarms to detect sudden disasters. When a disaster occurs, the server immediately issues an emergency alert to the entire school and sends evacuation instructions to terminals in each area. For example, a terminal in a classroom might display a voice message saying, "Earthquake has occurred. Please evacuate," along with evacuation routes.

[0689] 5. Monitoring evacuation situations and sharing information

[0690] The server analyzes camera footage from each classroom and school in real time to monitor the evacuation status of students. It confirms that students have begun evacuation and issues another warning if any students are still unable to escape. Once safety is confirmed, the server automatically sends a notification that the evacuation is complete to parents and relevant organizations. The user (educator) receives the evacuation report and provides additional evacuation instructions or student care as necessary.

[0691] Specific examples

[0692] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0693] As described above, the present invention can ensure safety in educational institutions such as elementary and junior high schools, and realize efficient and prompt responses.

[0694] The processing flow will be explained below.

[0695] Step 1:

[0696] The server collects video data in real time from multiple cameras installed throughout the school. Each camera is installed in a different area, such as a classroom, hallway, gymnasium, or schoolyard, and continuously monitors the behavior and situation of students.

[0697] Step 2:

[0698] The server checks the quality of the acquired video data and performs noise reduction and image resolution correction, resulting in clean data suitable for analysis. This pre-processing step is important to ensure data consistency and accuracy.

[0699] Step 3:

[0700] The preprocessed video data is input into an AI model integrated in the server. The AI ​​model evaluates students' behavior, facial expressions, and voice using behavioral and voice recognition algorithms to detect abnormal student behavior (e.g., bullying) and abnormal voice (e.g., shouting).

[0701] Step 4:

[0702] If any abnormal behavior or sound is detected, the server immediately generates an alert, which includes details of the problem and its location, and is sent to the device of the relevant educator.

[0703] Step 5:

[0704] The device will notify the educator of the received alert visually and audibly. The alert will include specific details of the abnormal behavior or abnormal sound, as well as recommended actions to take. The user (educator) will check the notification and immediately rush to the scene to address the problem.

[0705] Step 6:

[0706] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0707] Step 7:

[0708] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0709] Step 8:

[0710] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0711] Step 9:

[0712] As a follow-up after the evacuation, the server records all data for later analysis and reporting, allowing for improvements to be identified and future disaster response capabilities to be enhanced.

[0713] Through these steps, the system improves safety within educational institutions and enables quick and effective responses to emergencies such as bullying and disasters.

[0714] Example 1

[0715] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0716] Conventional safety management systems in educational institutions have difficulty responding immediately to abnormal student behavior or sudden disasters. Detecting abnormal behavior, issuing evacuation instructions, and monitoring evacuation situations are all done manually, requiring rapid and accurate responses. This has led to problems with insufficient efforts to ensure student safety.

[0717] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0718] In this invention, the server includes means for collecting video data in real time from multiple visual sensors installed in the educational institution, means for quality checking the video data, performing noise reduction and image resolution correction, means for analyzing the preprocessed video data using behavior recognition algorithms and voice recognition algorithms to evaluate behavioral patterns, means for detecting abnormal behavior and abnormal voices and sending alerts to relevant educators in real time, means for reading data from earthquake sensors and fire alarms to detect sudden disasters and issue evacuation instructions, means for analyzing camera images in each area in real time, monitoring the evacuation status of students, and confirming the completion of evacuation, and means for notifying parents and relevant organizations of the completion of evacuation. This enables quick and accurate response to abnormal student behavior and sudden disasters.

[0719] "Educational institutions" are facilities that provide various types of education, such as elementary and junior high schools, high schools, vocational schools, and universities.

[0720] A "visual sensor" is a device for capturing video and images, and typically operates as a camera.

[0721] "Video data" refers to image and video information acquired by a visual sensor and stored in digital format.

[0722] "Quality check" refers to checking the clarity and resolution of collected video data, as well as the presence or absence of blur and noise, and then processing the data to maintain appropriate quality.

[0723] "Noise reduction" is the process of reducing or removing unwanted noise contained in video data.

[0724] "Image resolution correction" is the process of adjusting the resolution of video data to an appropriate level.

[0725] An "action recognition algorithm" is an algorithm that analyzes the movements and actions of people in video data and identifies and classifies them.

[0726] A "voice recognition algorithm" is an algorithm that analyzes voice data and identifies and classifies the content of speech and the type of sound.

[0727] An "alert" is a warning message sent to relevant educators when abnormal behavior or abnormal voice is detected.

[0728] An "earthquake sensor" is a device that detects earthquake vibrations and shaking, and is a sensor that responds quickly.

[0729] A "fire alarm" is a device that detects the occurrence of a fire and sounds an alarm.

[0730] An "evacuation order" is a message that instructs people to evacuate to a safe place when a sudden disaster occurs.

[0731] "Monitoring" is the process of monitoring student behavior and evacuation status in real time and managing them appropriately.

[0732] "Evacuation completed" refers to the state in which all students have properly evacuated to a safe location.

[0733] "Related agencies" refers to public agencies and support organizations related to school operations and student safety.

[0734] MODE FOR CARRYING OUT THE INVENTION

[0735] The present invention is a safety management system for educational institutions, and more specifically, a system that can detect abnormal behavior of students and respond to sudden disasters in real time.

[0736] 1. Video data collection and preprocessing

[0737] The server collects video data in real time from multiple visual sensors installed within the educational institution. Specific hardware used is network cameras. For example, cameras placed in each classroom, hallway, gymnasium, schoolyard, etc. send video data to the server. The server checks the quality of this data, and performs noise reduction and image resolution correction using a software library called OpenCV.

[0738] 2. Video data analysis and problem detection

[0739] The server then analyzes the preprocessed video data using behavioral and speech recognition algorithms. For this analysis, behavioral recognition algorithms using Google's TensorFlow and speech recognition algorithms using Amazon's Alexa Voice Service are used. For example, if two students are making inappropriate contact in a hallway, this is recognized as "bullying" and detected as an anomaly.

[0740] 3. Send real-time alerts to educators

[0741] If the server detects abnormal behavior or abnormal audio, it immediately generates an alert and sends it to the device of the relevant educator. The alert is displayed on the screen and notified through audio and visual means. For example, an alert stating "Inappropriate contact has occurred in classroom A" is displayed on the educator's device. The user (educator) can confirm this and quickly deal with the problematic behavior by rushing to the scene.

[0742] 4. Disaster response

[0743] The server reads data from earthquake sensors and fire alarms to detect sudden disasters. For example, if the earthquake sensor detects an earthquake, the server will send an emergency alert to the entire school, and a voice message saying "Please evacuate" and evacuation routes will be displayed on terminals in each area.

[0744] 5. Monitoring evacuation situations and sharing information

[0745] The server analyzes camera footage from each classroom and school in real time to monitor the evacuation situation. For example, it checks the cameras to see if students have begun evacuation, and issues another warning if any students are still unable to escape. Once safety is confirmed, the server automatically sends a notification that the evacuation is complete to parents and relevant organizations. The user (educator) receives the evacuation report and issues additional evacuation instructions or provides student care as necessary.

[0746] Specific examples

[0747] For example, if an earthquake occurs in a classroom, the server immediately detects it with the ShakeAlert sensor and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors camera footage in real time and confirms that all students have safely evacuated. Once evacuation is complete, parents and relevant organizations are notified and the specific situation is promptly shared.

[0748] Prompt Sentence Examples

[0749] Here are some example prompts for a generative AI model:

[0750] "Please explain in detail the system you use to detect abnormal behavior and sounds within your school. Please also provide details on the algorithms, hardware used, and notification methods."

[0751] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0752] Step 1:

[0753] The server collects real-time video data sent by multiple visual sensors (network cameras) installed in educational institutions. Specifically, video data from each camera is input to the server. The server receives this video data as a stream. The input of this step is the video data from the cameras, and the output is the raw video data stored on the server.

[0754] Step 2:

[0755] The server performs a quality check on the received video data. Specifically, it uses the OpenCV library to evaluate the clarity, resolution, and presence or absence of blur and noise in the video data. During this process, the server analyzes each frame and extracts frames that do not meet the quality criteria. The input to this step is raw video data, and the output is video data with quality information added.

[0756] Step 3:

[0757] The server then performs noise reduction and image resolution correction on the quality-checked video data. Specifically, it uses OpenCV's fastNlMeansDenoisingColored function to remove noise and the resize function to adjust the resolution appropriately. The input to this step is the video data with quality information added, and the output is the preprocessed video data.

[0758] Step 4:

[0759] The server inputs the preprocessed video data into behavior recognition and speech recognition algorithms for analysis. Google's TensorFlow is used for behavior recognition, and Amazon's Alexa Voice Service is used for speech recognition. Specifically, the server analyzes the movements and postures of people in the video and the content of the audio data to detect abnormal behavior or audio. The input for this step is the preprocessed video data, and the output is the analysis results.

[0760] Step 5:

[0761] If the server detects abnormal behavior or abnormal audio based on the results of behavior and audio recognition, it immediately generates an alert. Specifically, it creates an alert message that includes details of the location, time, and circumstances of the abnormal behavior or audio. The input to this step is the analysis results, and the output is an alert message.

[0762] Step 6:

[0763] The server sends the generated alert to the device of the relevant educator. The device displays the received alert message on the screen and notifies the user by voice and visual means. As a specific example, an alert stating "Inappropriate contact has occurred in classroom A. Please rush to the scene" is displayed on the device. The input of this step is the alert message, and the output is a notification to the device.

[0764] Step 7:

[0765] The server reads data from earthquake sensors and fire alarms in real time to detect sudden disasters. Specifically, it analyzes the data sent from the sensors and immediately detects the occurrence of earthquakes or fires. The input to this step is data from the sensors, and the output is the disaster detection results.

[0766] Step 8:

[0767] When the server detects a disaster, it issues an emergency alert to the entire school and sends evacuation instructions to terminals in each area. Specifically, it displays a voice message such as "Earthquake has occurred. Please evacuate" along with evacuation routes. The input to this step is the disaster detection result, and the output is an emergency alert and evacuation instructions.

[0768] Step 9:

[0769] The server analyzes the camera footage from each classroom and school in real time and monitors the evacuation status of students. Specifically, it confirms that students have begun evacuation and issues another warning if any students are unable to escape in time. The input for this step is the real-time camera footage, and the output is the analysis result of the evacuation status.

[0770] Step 10:

[0771] Once the safe evacuation of all students is confirmed, the server sends a notification of evacuation completion to parents and relevant organizations. Specifically, it automatically sends a message such as, "The evacuation of all students has been completed. Please rest assured." The input for this step is the analysis result of the evacuation situation, and the output is a notification of evacuation completion.

[0772] (Application example 1)

[0773] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0774] In the past, educational institutions and industrial facilities lacked systems for detecting abnormal behavior or abnormal sounds in real time and responding quickly. Furthermore, evacuation instructions and management of evacuation situations during sudden disasters were also inefficient. Furthermore, while industrial facilities require ensuring safety and responding quickly to abnormal situations, existing technologies were unable to adequately resolve these issues. This resulted in a significant loss of safety and efficiency in educational institutions and industrial facilities.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0776] In this invention, the server includes means for collecting video data in real time from multiple camera devices installed in educational institutions or industrial facilities, means for preprocessing the video data and preparing for analysis, means for analyzing the preprocessed video data and evaluating the behavior, facial expression, voice, and positional relationship of a target, means for detecting abnormal behavior or abnormal voice and sending an alert to relevant parties, means for detecting a sudden disaster and issuing evacuation instructions, means for monitoring the evacuation situation in real time and confirming safety, means for notifying relevant parties and organizations of the completion of evacuation, means for evaluating the operating status of machines and generating an alert if an abnormality is detected, and means for ensuring safety at industrial facilities and realizing efficient and rapid response to abnormalities, thereby enabling rapid and accurate response to abnormal behavior or disasters in educational institutions and industrial facilities.

[0777] A "camera" is a device for collecting video data in real time.

[0778] A "server" is a computer system that processes, stores, and analyzes data.

[0779] "Video data" refers to the video signal and its video information collected from the imaging device.

[0780] "Preprocessing" refers to the process of removing noise and correcting the resolution of collected video data to prepare it for analysis.

[0781] "Analysis" refers to the evaluation of processed video data to analyze subject behavior, facial expressions, voice, and positional relationships.

[0782] "Evaluation" refers to analyzing data and judging, recognizing, or evaluating it based on specific criteria.

[0783] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and affects safety and efficiency.

[0784] "Abnormal sounds" refer to sounds that deviate from normal sound patterns and that affect safety and efficiency.

[0785] An "alert" is a signal that warns an individual about detected abnormal behavior or sound.

[0786] "Disaster" refers to sudden natural phenomena or accidents such as earthquakes and fires.

[0787] An "evacuation order" is an instruction to guide people to a safe place in the event of a disaster.

[0788] "Monitoring" means observing the situation in real time and observing changes or abnormalities.

[0789] A "motion recognition algorithm" is a mathematical and statistical method for analyzing and recognizing the behavior and movements of a subject.

[0790] A "speech recognition algorithm" is a mathematical and statistical method for analyzing and recognizing a target's speech.

[0791] An "environmental sensor" is a sensor device used to detect environmental changes such as earthquakes and fires.

[0792] The present invention provides a system for ensuring safety in educational institutions and industrial facilities and for responding to abnormalities efficiently and quickly. Specific embodiments of this system are described below.

[0793] The server collects video data in real time from camera devices installed in educational or industrial facilities. This collected video data undergoes quality checks, noise reduction, image resolution correction, and other pre-processing. This pre-processing removes noise and blurred areas from the collected video data, making it ready for accurate analysis.

[0794] After preprocessing, the video data is analyzed to evaluate the target's behavior, voice, and positional relationship. The server uses motion and voice recognition algorithms to detect abnormal behavior or voice. If an abnormality is detected, an alert is immediately generated and sent to the relevant device, enabling a prompt response on-site.

[0795] In addition, the server reads data from environmental sensors such as earthquake sensors and fire alarms to detect the occurrence of sudden disasters. In the event of a disaster, the server immediately issues evacuation instructions and displays evacuation routes and instructions on terminals in each area.

[0796] After the evacuation has begun, the server analyzes the camera footage and monitors the evacuation situation in real time. If any individuals are unable to escape in time, it issues another warning, and once safety is confirmed, it notifies relevant parties and organizations that the evacuation has been completed.

[0797] In industrial facilities, the server also monitors the operation status of machines and generates alerts if an abnormality is detected, thereby preventing breakdowns and problems before they occur.

[0798] For example, if machine A in a factory makes an abnormal operating noise, the server will detect it and send a notification to the manager's terminal saying, "Machine A is making an abnormal noise." At this time, the manager can immediately rush to the site and take action.

[0799] The system also utilizes a generative AI model, allowing users to input prompts to detect abnormal behavior and speech, such as:

[0800] An AI algorithm that detects abnormal behavior and sounds in real time from camera footage within a factory and generates alerts. For example, it recognizes inappropriate contact between workers or abnormal sounds and immediately sends a notification to the manager's device.

[0801] This will significantly improve the safety of educational institutions and industrial facilities and enable efficient and rapid response.

[0802] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0803] Step 1:

[0804] The server collects video data in real time from multiple camera devices installed in educational or industrial facilities. It receives the video data sent by the camera devices (input) and stores it in a database (output). This collection process involves continuously capturing the video stream from each camera and adding necessary metadata (timestamp, camera location, etc.).

[0805] Step 2:

[0806] The server preprocesses the collected video data, specifically by reducing noise and correcting image resolution (input: video data, output: preprocessed video data). This preprocessing step uses filtering techniques and edge detection algorithms to reduce noise and improve image clarity.

[0807] Step 3:

[0808] The server analyzes the preprocessed video data and evaluates the subject's behavior, facial expressions, voice, and positional relationship (input: preprocessed video data, output: analysis results). Specifically, it uses motion recognition and voice recognition algorithms to detect abnormalities in behavior and voice. This involves analyzing the subject's behavioral patterns and identifying abnormalities.

[0809] Step 4:

[0810] If the server detects abnormal behavior or abnormal audio, it generates an alert and sends it to the device of the relevant person (input: analysis result, output: alert notification). Specifically, when an abnormality is detected, it creates a notification message and sends it to the device. This message includes the type of abnormality, the location where it was detected, and a timestamp.

[0811] Step 5:

[0812] The server reads data from earthquake sensors, fire alarm systems, etc. to detect sudden disasters (input: sensor data, output: disaster alerts). It periodically monitors data from the sensors and generates a disaster alert if an abnormal value is detected.

[0813] Step 6:

[0814] In the event of a disaster, the server sends evacuation instructions to terminals in each area (input: disaster alert, output: evacuation instructions). Specifically, it displays a message containing safe evacuation routes and instructions on the terminals and issues instructions via voice message.

[0815] Step 7:

[0816] After evacuation has begun, the server analyzes the camera footage and monitors the evacuation status in real time (input: camera footage, output: evacuation status report). The server detects moving objects in the footage and evaluates the progress of the evacuation.

[0817] Step 8:

[0818] Once safety is confirmed, the server notifies the relevant parties and organizations that the evacuation has been completed (input: evacuation status report, output: evacuation completion notification). Specifically, after confirming that everyone is safe, the server automatically sends notifications to the relevant parties.

[0819] Step 9:

[0820] The server monitors the operation status of machines in industrial facilities and generates an alert if an abnormality is detected (input: machine operation data, output: machine abnormality alert). It uses a behavior recognition algorithm to identify abnormal machine behavior and immediately notify relevant parties.

[0821] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0822] The present invention is a system for ensuring safety in educational institutions such as elementary and junior high schools, enabling quick and accurate responses when problems occur, and is also combined with an emotion engine that recognizes the user's emotions. A specific embodiment of this system is shown below.

[0823] 1. Video data collection and preprocessing

[0824] The server collects video data in real time from multiple cameras installed around the school. This video data is collected from various areas, such as classrooms, hallways, gymnasiums, and schoolyards. The server checks the quality of the collected video data and performs pre-processing such as noise reduction and image resolution correction.

[0825] 2. Analysis of video data

[0826] The server then inputs the preprocessed video data into behavioral and speech recognition algorithms for analysis. This evaluates students' behavioral patterns, facial expressions, and voice to detect abnormal behavior or voice. For example, if two students are making inappropriate contact in a hallway, the AI ​​will analyze this and recognize it as a sign of "bullying."

[0827] 3. Emotion Recognition by Emotion Engine

[0828] The server uses an emotion engine to analyze students' emotions from video and audio data. The emotion engine identifies emotions such as sadness, anger, and fear, improving the accuracy of detecting abnormal behavior. For example, if a student appears sad in the classroom, this could be a sign of a potential problem, allowing for early intervention.

[0829] 4. Send real-time alerts to educators

[0830] If the server detects abnormal behavior, abnormal voice, or abnormal emotion based on the emotion engine, it immediately generates an alert, which includes detailed information about the problem and its location, and is sent to the device of the relevant educator.

[0831] The device will notify the educator of the received alert visually and audibly. The alert content will include specific details of abnormal behavior, abnormal voice, or abnormal emotion, as well as recommended actions to take. The user (educator) can check the notification and immediately rush to the scene to address the problem.

[0832] 5. Disaster response

[0833] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0834] 6. Evacuation situation monitoring and information sharing

[0835] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0836] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0837] Specific examples

[0838] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0839] This system, combined with an emotion engine, will further improve safety within educational institutions and enable quick and effective responses to emergencies such as bullying and disasters.

[0840] The processing flow will be explained below.

[0841] Step 1:

[0842] The server collects video data in real time from multiple cameras installed throughout the school. Each camera is installed in a different area, such as a classroom, hallway, gymnasium, or schoolyard, and continuously monitors the behavior and situation of students.

[0843] Step 2:

[0844] The server preprocesses the acquired video data, specifically by performing noise reduction and image resolution correction to obtain clean data suitable for analysis. This preprocessing step is important to ensure data consistency and accuracy.

[0845] Step 3:

[0846] The preprocessed video data is input into an AI model integrated on the server. The AI ​​model evaluates students' behavior, facial expressions, and voice using behavior and voice recognition algorithms to detect abnormal student behavior (e.g., bullying) and abnormal voice (e.g., shouting).

[0847] Step 4:

[0848] The server analyzes students' emotions from the video and audio data using an emotion engine, which identifies emotions such as sadness, anger, and fear, thereby improving the accuracy of detecting abnormal behavior.

[0849] Step 5:

[0850] If abnormal behavior, voice, or emotion is detected, the server immediately generates an alert, which includes details of the problem and its location, and is sent to the device of the relevant educator.

[0851] Step 6:

[0852] The device will notify the educator of the received alert visually and audibly. The alert content will include specific details of abnormal behavior, abnormal voice, or abnormal emotion, as well as recommended actions to take. The user (educator) can check the notification and immediately rush to the scene to address the problem.

[0853] Step 7:

[0854] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," urging students and educators to evacuate.

[0855] Step 8:

[0856] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[0857] Step 9:

[0858] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[0859] Step 10:

[0860] As a follow-up after the evacuation, the server records all data for later analysis and reporting, allowing for improvements to be identified and future disaster response capabilities to be enhanced.

[0861] Example 2

[0862] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0863] Conventional safety management systems in educational institutions lack the functionality to detect abnormal student behavior or voice activity in real time and respond quickly. It is also difficult to issue evacuation instructions and monitor evacuation situations quickly during disasters, making it difficult to ensure student safety. Furthermore, they lack the technology to properly analyze student emotions and detect potential problems early. As a result, responses to issues such as bullying and violence are delayed, potentially leading to a decline in the quality of the educational environment.

[0864] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting video data in real time from multiple camera devices installed in the educational institution, means for preprocessing the video data to perform quality checks, noise reduction, and image resolution correction, means for analyzing the preprocessed video data and evaluating students' behavioral patterns, facial expressions, voice, and positional relationships, means for detecting abnormal behavior and voice and sending alerts to relevant educators, means for analyzing students' emotions (sadness, anger, fear, etc.) using an emotion recognition engine to improve the accuracy of detecting abnormal behavior, means for detecting sudden disasters and issuing evacuation instructions, means for monitoring the evacuation situation in real time and confirming safety, and means for notifying parents and relevant organizations of the completion of evacuation. This comprehensively ensures the safety of students and enables early detection and rapid response to problems.

[0865] "Capture equipment" refers to equipment such as cameras that are installed within educational institutions and are used to collect video data in real time.

[0866] "Video data" refers to digital video information that shows the students and facilities collected by a camera.

[0867] "Preprocessing" refers to the process of performing quality checks, noise reduction, image resolution correction, etc. on collected video data.

[0868] A "behavioral pattern" is a sequence of specific student actions or behaviors obtained from the analyzed video data.

[0869] "Facial expressions" refer to the emotions and reactions that can be read from the students' facial expressions.

[0870] "Speech" refers to acoustic information such as human voices and environmental sounds that are analyzed using a speech recognition algorithm.

[0871] "Positional relationship" is information that indicates the spatial arrangement and relationships of students and objects.

[0872] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and requires vigilance or action.

[0873] "Abnormal sounds" refer to sounds that are different from normal sounds, such as cries or calls for help, that require vigilance or a response.

[0874] An "alert" is a warning message sent to relevant educators when abnormal behavior or sound is detected.

[0875] An "emotion recognition engine" is a software technology that analyzes video and audio data to identify emotions such as sadness, anger, and fear.

[0876] "Sudden disasters" refer to natural disasters and accidents, such as earthquakes and fires, that occur unexpectedly and require a rapid response.

[0877] An "evacuation order" is an instruction to students and educators to move to a safe place in the event of a sudden disaster.

[0878] "Evacuation situation" refers to the progress of evacuation actions by students and educators after an evacuation order is issued.

[0879] "Safety check" is the act of making sure that students and educators are safe when evacuation is complete.

[0880] "Guardian" refers to the person registered as the parent or guardian of a student.

[0881] "Affiliated institutions" refers to external organizations or facilities that have a relationship with an educational institution (e.g., fire department, police department, etc.).

[0882] This invention is a comprehensive surveillance system that improves safety in educational institutions and enables rapid and appropriate response when problems occur. The system collects video data in real time, preprocesses and analyzes it, and detects abnormal behavior or abnormal sounds and notifies educators. Furthermore, it uses an emotion recognition engine to analyze student emotions and improve the accuracy of detecting abnormal behavior. Furthermore, it issues evacuation instructions in the event of a sudden disaster, monitors evacuation conditions, and confirms safety.

[0883] Hardware and Software Configuration

[0884] The system uses the following hardware and software:

[0885] Hardware: High-resolution IP cameras, earthquake sensors, fire alarms

[0886] Software: OpenCV (image processing library), TensorFlow (machine learning library), EmotionAPI (emotion recognition API)

[0887] Server Roles

[0888] The server collects video data in real time from multiple high-resolution IP cameras installed on campus. The collected video data undergoes preprocessing, such as noise reduction and image resolution correction. The preprocessed video data is then analyzed using behavior recognition algorithms (using OpenCV and TensorFlow) to evaluate students' behavioral patterns, facial expressions, and voice. In addition, an emotion recognition engine (EmotionAPI) is used to analyze students' emotions from the video and audio data.

[0889] When abnormal behavior, abnormal voice, or abnormal emotion is detected by the emotion recognition engine, an alert is immediately generated. The server sends this alert to the device of the relevant educator, enabling a prompt response.

[0890] Device Role

[0891] The device will notify the educator of the received alert visually and audibly. The alert will include details of the problem and the location where it occurred. For example, it could say, "Student A is sad in the classroom. Details: Student A was captured on video crying alone. Action is required." After the educator confirms this, they can immediately rush to the scene and take action.

[0892] User Roles

[0893] The user (educator) checks the alert from the device and immediately rushes to the scene to deal with the problem. Also, in the event of a sudden disaster, the user receives evacuation instructions from the server and instructs students to evacuate. The server constantly monitors data from earthquake sensors and fire alarms, and immediately issues evacuation instructions when it detects a disaster. Devices in each classroom and area display audio messages and visual displays such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[0894] Specific examples

[0895] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[0896] Prompt Sentence Examples

[0897] The prompt sentence to be input to the generative AI model is shown below.

[0898] "You are building a safety management system for an elementary school. Explain how you can collect video data from cameras in real time, use behavior recognition algorithms and an emotion engine to detect abnormal behaviors and emotions, and send alerts to educators."

[0899] This prompt allows the generative AI model to suggest relevant technical approaches and detailed processing methods, thus comprehensively ensuring safety within educational institutions and enabling early detection and rapid response to problems.

[0900] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0901] Step 1:

[0902] The server collects video data in real time from multiple camera devices installed on campus. The server receives the collected video data as input, checks the video quality, and performs pre-processing such as noise reduction and image resolution correction. The output of this pre-processing is improved video data. Specifically, the server continuously acquires the video stream from the cameras, applies noise reduction filters, and executes correction algorithms to improve image quality.

[0903] Step 2:

[0904] The server inputs the preprocessed video data and analyzes it using behavioral and speech recognition algorithms. Specifically, it evaluates students' behavioral patterns, facial expressions, and audio data from the video data and detects abnormal behavior and audio based on these. The output of this analysis is the detection results of abnormal behavior and audio. For example, the behavioral recognition algorithm identifies whether a student is performing a specific action in the video, and the speech recognition algorithm detects audio events such as screaming or calling for help.

[0905] Step 3:

[0906] The server then inputs the analyzed video and audio data using an emotion recognition engine to identify the student's emotions. The output of this emotion recognition is an emotion label, such as sadness, anger, or fear. At this stage, the student's facial expressions and voice tone captured on video are analyzed and mapped to specific emotions. For example, EmotionAPI analyzes a student's facial expression and determines whether it corresponds to "sadness."

[0907] Step 4:

[0908] The server generates an alert when it detects abnormal behavior, abnormal voice, or abnormal emotion based on the analysis results. This alert includes detailed information about the problem and the location where it occurred. The generated alert is sent to the educator's device. The input for this transmission is the analysis result, and the output is notification information that is displayed on the relevant educator's device. Specifically, the server classifies the analysis result, composes an appropriate alert message, and sends it to the educator in real time.

[0909] Step 5:

[0910] The device will notify the educator of the received alert visually and audibly. The input here is the alert information sent from the server, and the output is an alert message displayed on the teacher's device. For example, a message such as "Student A is crying in the classroom. Details: Student A has been captured on video crying alone. Action required" will be displayed.

[0911] Step 6:

[0912] The server constantly monitors data from earthquake sensors and fire alarms, and if it detects a disaster, it automatically issues an emergency alert and sends evacuation instructions to the entire school. The input here is data from the earthquake sensors and fire alarms, and the output is an alert message that sends evacuation instructions. For example, the moment an earthquake is detected, an audio and visual message such as "Earthquake has occurred. Please evacuate" is displayed on terminals in each classroom and area.

[0913] Step 7:

[0914] The server analyzes camera footage in real time and monitors the evacuation status from each area. It checks for any obstructions in the evacuation routes and automatically issues a second warning. The input is video data from the school cameras, and the output is real-time information on the evacuation status. If a obstruction is detected, an alert message ordering a second evacuation is generated and sent to each device.

[0915] Step 8:

[0916] After the evacuation is complete, the server notifies parents and related organizations of the completion of the evacuation. The input here is the detection result of the evacuation completion, and the output is a notification message sent to parents and related organizations. Specifically, once safety confirmation is complete, a message such as "Evacuation Complete Notification: All students and educators have evacuated safely" is sent to parents and related organizations.

[0917] In this way, the entire system can comprehensively ensure student safety and enable early detection and rapid response to problems.

[0918] (Application example 2)

[0919] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0920] The present invention is required to ensure safety in places where many people gather, such as educational institutions and office buildings, and to respond quickly to abnormal behavior and emergencies. However, current systems are unable to adequately detect abnormal behavior and emotions in real time, making it difficult to respond quickly and accurately. Furthermore, even in the event of a sudden disaster, the system is unable to monitor evacuation situations in real time and confirm overall safety, which may result in delayed response.

[0921] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0922] In this invention, the server includes: means for collecting video data in real time from multiple cameras installed in educational institutions or other facilities; means for preprocessing the video data and preparing for analysis; means for analyzing the preprocessed video data and evaluating the behavior, facial expressions, voice, and positional relationship of subjects; means for detecting abnormal behavior or abnormal voice and sending alerts to relevant parties; means for analyzing the emotions of subjects from the video data and voice data using an emotion engine and detecting abnormal emotions; means for detecting sudden disasters and issuing evacuation instructions; means for monitoring the evacuation situation in real time and confirming safety; and means for notifying parents and relevant organizations of the completion of evacuation. This enables the detection of abnormal behavior or abnormal emotions in real time from multiple camera footage, enabling rapid and accurate response. Furthermore, it is possible to quickly issue evacuation instructions, monitor the evacuation situation, and confirm safety even in the event of a sudden disaster.

[0923] An "educational institution or other facility" is a place where people gather to carry out educational, business, or other activities, including, but not limited to, a school.

[0924] "Multiple cameras" refer to imaging devices that are installed at different locations and serve to collect video data from multiple viewpoints simultaneously.

[0925] "Means for collecting video data in real time" refers to a method and device for instantly importing video captured by a camera into a system such as a server.

[0926] "Means for preprocessing and preparing for analysis" refers to methods and devices for reducing noise in acquired video data, correcting resolution, etc., to make the data suitable for analysis.

[0927] "Means for analyzing preprocessed video data and assessing subject behavior, facial expressions, voice, and positional relationships" refers to a method and apparatus that uses preprocessed data to identify and assess a subject's behavioral patterns, emotional expressions, vocal content, and spatial relationships.

[0928] "Means for detecting abnormal behavior or abnormal sounds and sending alerts to relevant parties" refers to a method and device for detecting unusual behavior or sounds from analyzed data and notifying relevant parties of that information.

[0929] An "emotion engine" is software or algorithms that analyze video and audio data to identify a subject's emotional state, such as joy, anger, sadness, or fear.

[0930] The "means for analyzing emotions" refers to a method and apparatus for analyzing emotions from specific data using an emotion engine.

[0931] "Means for detecting sudden disasters and issuing evacuation instructions" refers to methods and devices for instantly recognizing emergencies such as earthquakes and fires and issuing appropriate evacuation instructions based on that information.

[0932] "Means for monitoring evacuation situations in real time and confirming safety" refers to a method and device for monitoring the movements of people during evacuation in real time and confirming whether the safety of everyone is ensured.

[0933] "Means for notifying parents and related organizations of the completion of evacuation" refers to methods and devices for confirming that evacuation has been completed safely and for quickly and accurately communicating the results to parents and related organizations.

[0934] The present invention provides a system for ensuring safety in educational institutions or other facilities and for responding quickly and appropriately to abnormal behavior or emergencies. This system is constructed using the hardware and software described below.

[0935] The system involves installing multiple cameras in designated locations and collecting video data in real time. For example, in educational institutions, cameras are placed in classrooms, hallways, gymnasiums, and schoolyards, while in office buildings, cameras are installed at the entrance and in shared spaces on each floor.

[0936] The server receives the collected video data and performs preprocessing, which includes noise reduction and image resolution correction. The preprocessed video data is then analyzed to evaluate the subject's behavior, facial expression, voice, and positional relationship using behavior and voice recognition algorithms.

[0937] The server uses an emotion engine to analyze the subject's emotions from video and audio data. The emotion engine identifies emotions such as sadness, anger, and fear, improving the accuracy of detecting abnormal behavior. If abnormal behavior, audio, or emotion is detected, an alert is sent to the relevant parties. The alert is immediately displayed on a notification terminal, allowing the relevant parties to respond quickly.

[0938] In addition, the server constantly monitors data from earthquake sensors and fire alarms to detect sudden disasters. If a disaster is detected, the server automatically issues an emergency alert and sends evacuation instructions to all facilities. During an evacuation, camera footage is analyzed, the evacuation situation is monitored in real time, and safety is confirmed. Once everyone has evacuated, the server notifies parents and relevant organizations that the evacuation is complete.

[0939] As a concrete example, consider the case where someone shouts at the entrance of an office building. The server analyzes the camera footage and audio to detect anger. If it determines that the behavior is abnormal, an alert is immediately sent to the relevant parties. The alert includes the location of the abnormality and how to respond, allowing the relevant parties to quickly rush to the scene and take appropriate action.

[0940] An example prompt is, "We would like to develop a security system for an office building. Please design a system that analyzes camera footage, performs emotion recognition, and sends an alert if there is any abnormal behavior."

[0941] In this way, abnormal behavior and abnormal emotions can be detected in real time from multiple camera footage, enabling a quick and appropriate response. It can also be used to quickly issue evacuation instructions, monitor evacuation situations, and confirm safety in the event of a sudden disaster.

[0942] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0943] Step 1:

[0944] The server collects video data in real time from multiple cameras installed in educational institutions or other facilities.

[0945] Input: Video feed from camera

[0946] Output: Raw video data

[0947] Specific operation: Continuously collects video data from multiple locations where cameras are installed and sends it to a server.

[0948] Step 2:

[0949] The server pre-processes the collected video data.

[0950] Input: Raw video data

[0951] Output: Pre-processed video data

[0952] Specific operation: Performs noise reduction and image resolution correction to make the image suitable for analysis. For example, improves the quality of the video data using OpenCV.

[0953] Step 3:

[0954] The server analyzes the pre-processed video data and evaluates the subject's behavior, facial expression, voice, and position.

[0955] Input: Preprocessed video data

[0956] Output: Evaluation results on the subject's behavior, facial expression, voice, and positional relationship

[0957] Specific operation: Using behavioral and voice recognition algorithms, the system analyzes video and audio to identify the target's behavioral patterns, facial expressions, vocalizations, etc.

[0958] Step 4:

[0959] The server uses an emotion engine to analyze the emotion of the target from the video data and audio data.

[0960] Input: Preprocessed video and audio data

[0961] Output: Analysis results regarding the subject's emotions

[0962] Specific operation: For example, the EmotionRecognizer library is used to analyze facial expressions and voice tones in video to identify emotions. The emotion engine identifies emotions such as sadness, anger, and fear.

[0963] Step 5:

[0964] If the server detects abnormal behavior, abnormal voice, or abnormal emotions, it will send an alert to relevant parties.

[0965] Input: Analysis results of behavior, voice, and emotions

[0966] Output: Alert message

[0967] Specific operation: If abnormal behavior or abnormal emotions are detected, an alert message will be generated and sent to the device or smartphone, for example, via email or push notification.

[0968] Step 6:

[0969] The server detects sudden disasters and issues evacuation instructions.

[0970] Input: Data from disaster sensors and fire alarms

[0971] Output: Evacuation instruction message

[0972] Specific operation: Data from earthquake sensors and fire alarms is constantly monitored, and if a disaster is detected, an emergency alert is automatically issued and evacuation instructions are sent to all facilities.

[0973] Step 7:

[0974] The server monitors the evacuation situation in real time and confirms safety.

[0975] Input: Camera footage during evacuation

[0976] Output: Monitoring results of evacuation situation

[0977] Specific operation: Analyzes camera footage and monitors in real time whether people are evacuating properly. If there is a problem, it sends another warning.

[0978] Step 8:

[0979] After the evacuation is completed, the server notifies the parents and related organizations of the completion of the evacuation.

[0980] Input: Confirmation data of evacuation completion

[0981] Output: Evacuation completion notification

[0982] Specific operations: After confirming that everyone has evacuated, an evacuation completion notification will be sent to parents and relevant organizations via email or message.

[0983] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0984] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0985] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0986] [Fourth embodiment]

[0987] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0988] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0989] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0990] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0991] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0992] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0993] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0994] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0995] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0996] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0997] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0998] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0999] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1000] The present invention is a system for ensuring safety in educational institutions such as elementary and junior high schools, and for realizing a prompt and appropriate response when a problem occurs. A specific embodiment of this system is described below.

[1001] 1. Video data collection and preprocessing

[1002] The server collects video data in real time from multiple cameras installed within the educational institution. This video data is acquired from various areas such as classrooms, hallways, gymnasiums, and schoolyards. The server checks the quality of the collected video data and performs pre-processing such as noise reduction and image resolution correction.

[1003] 2. Video data analysis and problem detection

[1004] The server then inputs the preprocessed video data into behavioral and speech recognition algorithms for analysis. This allows the system to evaluate students' behavioral patterns and detect abnormal behavior or sounds. For example, if two students are making inappropriate contact in a hallway, the AI ​​will analyze this and recognize it as a sign of "bullying."

[1005] 3. Send real-time alerts to educators

[1006] If the server detects abnormal behavior or abnormal audio, it immediately generates an alert and sends it to the relevant educator's device. The device then displays the received alert on its screen and notifies the user via audio and visual means. The user (educator) can check the notification and rush to the scene to deal with the problematic behavior immediately.

[1007] 4. Disaster response

[1008] The server reads data from external devices such as earthquake sensors and fire alarms to detect sudden disasters. When a disaster occurs, the server immediately issues an emergency alert to the entire school and sends evacuation instructions to terminals in each area. For example, a terminal in a classroom might display a voice message saying, "Earthquake has occurred. Please evacuate," along with evacuation routes.

[1009] 5. Monitoring evacuation situations and sharing information

[1010] The server analyzes camera footage from each classroom and school in real time to monitor the evacuation status of students. It confirms that students have begun evacuation and issues another warning if any students are still unable to escape. Once safety is confirmed, the server automatically sends a notification that the evacuation is complete to parents and relevant organizations. The user (educator) receives the evacuation report and provides additional evacuation instructions or student care as necessary.

[1011] Specific examples

[1012] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[1013] As described above, the present invention can ensure safety in educational institutions such as elementary and junior high schools, and realize efficient and prompt responses.

[1014] The processing flow will be explained below.

[1015] Step 1:

[1016] The server collects video data in real time from multiple cameras installed throughout the school. Each camera is installed in a different area, such as a classroom, hallway, gymnasium, or schoolyard, and continuously monitors the behavior and situation of students.

[1017] Step 2:

[1018] The server checks the quality of the acquired video data and performs noise reduction and image resolution correction, resulting in clean data suitable for analysis. This pre-processing step is important to ensure data consistency and accuracy.

[1019] Step 3:

[1020] The preprocessed video data is input into an AI model integrated in the server. The AI ​​model evaluates students' behavior, facial expressions, and voice using behavioral and voice recognition algorithms to detect abnormal student behavior (e.g., bullying) and abnormal voice (e.g., shouting).

[1021] Step 4:

[1022] If any abnormal behavior or sound is detected, the server immediately generates an alert, which includes details of the problem and its location, and is sent to the device of the relevant educator.

[1023] Step 5:

[1024] The device will notify the educator of the received alert visually and audibly. The alert will include specific details of the abnormal behavior or abnormal sound, as well as recommended actions to take. The user (educator) will check the notification and immediately rush to the scene to address the problem.

[1025] Step 6:

[1026] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[1027] Step 7:

[1028] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[1029] Step 8:

[1030] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[1031] Step 9:

[1032] As a follow-up after the evacuation, the server records all data for later analysis and reporting, allowing for improvements to be identified and future disaster response capabilities to be enhanced.

[1033] Through these steps, the system improves safety within educational institutions and enables quick and effective responses to emergencies such as bullying and disasters.

[1034] Example 1

[1035] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1036] Conventional safety management systems in educational institutions have difficulty responding immediately to abnormal student behavior or sudden disasters. Detecting abnormal behavior, issuing evacuation instructions, and monitoring evacuation situations are all done manually, requiring rapid and accurate responses. This has led to problems with insufficient efforts to ensure student safety.

[1037] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1038] In this invention, the server includes means for collecting video data in real time from multiple visual sensors installed in the educational institution, means for quality checking the video data, performing noise reduction and image resolution correction, means for analyzing the preprocessed video data using behavior recognition algorithms and voice recognition algorithms to evaluate behavioral patterns, means for detecting abnormal behavior and abnormal voices and sending alerts to relevant educators in real time, means for reading data from earthquake sensors and fire alarms to detect sudden disasters and issue evacuation instructions, means for analyzing camera images in each area in real time, monitoring the evacuation status of students, and confirming the completion of evacuation, and means for notifying parents and relevant organizations of the completion of evacuation. This enables quick and accurate response to abnormal student behavior and sudden disasters.

[1039] "Educational institutions" are facilities that provide various types of education, such as elementary and junior high schools, high schools, vocational schools, and universities.

[1040] A "visual sensor" is a device for capturing video and images, and typically operates as a camera.

[1041] "Video data" refers to image and video information acquired by a visual sensor and stored in digital format.

[1042] "Quality check" refers to checking the clarity and resolution of collected video data, as well as the presence or absence of blur and noise, and then processing the data to maintain appropriate quality.

[1043] "Noise reduction" is the process of reducing or removing unwanted noise contained in video data.

[1044] "Image resolution correction" is the process of adjusting the resolution of video data to an appropriate level.

[1045] An "action recognition algorithm" is an algorithm that analyzes the movements and actions of people in video data and identifies and classifies them.

[1046] A "voice recognition algorithm" is an algorithm that analyzes voice data and identifies and classifies the content of speech and the type of sound.

[1047] An "alert" is a warning message sent to relevant educators when abnormal behavior or abnormal voice is detected.

[1048] An "earthquake sensor" is a device that detects earthquake vibrations and shaking, and is a sensor that responds quickly.

[1049] A "fire alarm" is a device that detects the occurrence of a fire and sounds an alarm.

[1050] An "evacuation order" is a message that instructs people to evacuate to a safe place when a sudden disaster occurs.

[1051] "Monitoring" is the process of monitoring student behavior and evacuation status in real time and managing them appropriately.

[1052] "Evacuation completed" refers to the state in which all students have properly evacuated to a safe location.

[1053] "Related agencies" refers to public agencies and support organizations related to school operations and student safety.

[1054] MODE FOR CARRYING OUT THE INVENTION

[1055] The present invention is a safety management system for educational institutions, and more specifically, a system that can detect abnormal behavior of students and respond to sudden disasters in real time.

[1056] 1. Video data collection and preprocessing

[1057] The server collects video data in real time from multiple visual sensors installed within the educational institution. Specific hardware used is network cameras. For example, cameras placed in each classroom, hallway, gymnasium, schoolyard, etc. send video data to the server. The server checks the quality of this data, and performs noise reduction and image resolution correction using a software library called OpenCV.

[1058] 2. Video data analysis and problem detection

[1059] The server then analyzes the preprocessed video data using behavioral and speech recognition algorithms. For this analysis, behavioral recognition algorithms using Google's TensorFlow and speech recognition algorithms using Amazon's Alexa Voice Service are used. For example, if two students are making inappropriate contact in a hallway, this is recognized as "bullying" and detected as an anomaly.

[1060] 3. Send real-time alerts to educators

[1061] If the server detects abnormal behavior or abnormal audio, it immediately generates an alert and sends it to the device of the relevant educator. The alert is displayed on the screen and notified through audio and visual means. For example, an alert stating "Inappropriate contact has occurred in classroom A" is displayed on the educator's device. The user (educator) can confirm this and quickly deal with the problematic behavior by rushing to the scene.

[1062] 4. Disaster response

[1063] The server reads data from earthquake sensors and fire alarms to detect sudden disasters. For example, if the earthquake sensor detects an earthquake, the server will send an emergency alert to the entire school, and a voice message saying "Please evacuate" and evacuation routes will be displayed on terminals in each area.

[1064] 5. Monitoring evacuation situations and sharing information

[1065] The server analyzes camera footage from each classroom and school in real time to monitor the evacuation situation. For example, it checks the cameras to see if students have begun evacuation, and issues another warning if any students are still unable to escape. Once safety is confirmed, the server automatically sends a notification that the evacuation is complete to parents and relevant organizations. The user (educator) receives the evacuation report and issues additional evacuation instructions or provides student care as necessary.

[1066] Specific examples

[1067] For example, if an earthquake occurs in a classroom, the server immediately detects it with the ShakeAlert sensor and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors camera footage in real time and confirms that all students have safely evacuated. Once evacuation is complete, parents and relevant organizations are notified and the specific situation is promptly shared.

[1068] Prompt Sentence Examples

[1069] Here are some example prompts for a generative AI model:

[1070] "Please explain in detail the system you use to detect abnormal behavior and sounds within your school. Please also provide details on the algorithms, hardware used, and notification methods."

[1071] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1072] Step 1:

[1073] The server collects real-time video data sent by multiple visual sensors (network cameras) installed in educational institutions. Specifically, video data from each camera is input to the server. The server receives this video data as a stream. The input of this step is the video data from the cameras, and the output is the raw video data stored on the server.

[1074] Step 2:

[1075] The server performs a quality check on the received video data. Specifically, it uses the OpenCV library to evaluate the clarity, resolution, and presence or absence of blur and noise in the video data. During this process, the server analyzes each frame and extracts frames that do not meet the quality criteria. The input to this step is raw video data, and the output is video data with quality information added.

[1076] Step 3:

[1077] The server then performs noise reduction and image resolution correction on the quality-checked video data. Specifically, it uses OpenCV's fastNlMeansDenoisingColored function to remove noise and the resize function to adjust the resolution appropriately. The input to this step is the video data with quality information added, and the output is the preprocessed video data.

[1078] Step 4:

[1079] The server inputs the preprocessed video data into behavior recognition and speech recognition algorithms for analysis. Google's TensorFlow is used for behavior recognition, and Amazon's Alexa Voice Service is used for speech recognition. Specifically, the server analyzes the movements and postures of people in the video and the content of the audio data to detect abnormal behavior or audio. The input for this step is the preprocessed video data, and the output is the analysis results.

[1080] Step 5:

[1081] If the server detects abnormal behavior or abnormal audio based on the results of behavior and audio recognition, it immediately generates an alert. Specifically, it creates an alert message that includes details of the location, time, and circumstances of the abnormal behavior or audio. The input to this step is the analysis results, and the output is an alert message.

[1082] Step 6:

[1083] The server sends the generated alert to the device of the relevant educator. The device displays the received alert message on the screen and notifies the user by voice and visual means. As a specific example, an alert stating "Inappropriate contact has occurred in classroom A. Please rush to the scene" is displayed on the device. The input of this step is the alert message, and the output is a notification to the device.

[1084] Step 7:

[1085] The server reads data from earthquake sensors and fire alarms in real time to detect sudden disasters. Specifically, it analyzes the data sent from the sensors and immediately detects the occurrence of earthquakes or fires. The input to this step is data from the sensors, and the output is the disaster detection results.

[1086] Step 8:

[1087] When the server detects a disaster, it issues an emergency alert to the entire school and sends evacuation instructions to terminals in each area. Specifically, it displays a voice message such as "Earthquake has occurred. Please evacuate" along with evacuation routes. The input to this step is the disaster detection result, and the output is an emergency alert and evacuation instructions.

[1088] Step 9:

[1089] The server analyzes the camera footage from each classroom and school in real time and monitors the evacuation status of students. Specifically, it confirms that students have begun evacuation and issues another warning if any students are unable to escape in time. The input for this step is the real-time camera footage, and the output is the analysis result of the evacuation status.

[1090] Step 10:

[1091] Once the safe evacuation of all students is confirmed, the server sends a notification of evacuation completion to parents and relevant organizations. Specifically, it automatically sends a message such as, "The evacuation of all students has been completed. Please rest assured." The input for this step is the analysis result of the evacuation situation, and the output is a notification of evacuation completion.

[1092] (Application example 1)

[1093] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1094] In the past, educational institutions and industrial facilities lacked systems for detecting abnormal behavior or abnormal sounds in real time and responding quickly. Furthermore, evacuation instructions and management of evacuation situations during sudden disasters were also inefficient. Furthermore, while industrial facilities require ensuring safety and responding quickly to abnormal situations, existing technologies were unable to adequately resolve these issues. This resulted in a significant loss of safety and efficiency in educational institutions and industrial facilities.

[1095] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1096] In this invention, the server includes means for collecting video data in real time from multiple camera devices installed in educational institutions or industrial facilities, means for preprocessing the video data and preparing for analysis, means for analyzing the preprocessed video data and evaluating the behavior, facial expression, voice, and positional relationship of a target, means for detecting abnormal behavior or abnormal voice and sending an alert to relevant parties, means for detecting a sudden disaster and issuing evacuation instructions, means for monitoring the evacuation situation in real time and confirming safety, means for notifying relevant parties and organizations of the completion of evacuation, means for evaluating the operating status of machines and generating an alert if an abnormality is detected, and means for ensuring safety at industrial facilities and realizing efficient and rapid response to abnormalities, thereby enabling rapid and accurate response to abnormal behavior or disasters in educational institutions and industrial facilities.

[1097] A "camera" is a device for collecting video data in real time.

[1098] A "server" is a computer system that processes, stores, and analyzes data.

[1099] "Video data" refers to the video signal and its video information collected from the imaging device.

[1100] "Preprocessing" refers to the process of removing noise and correcting the resolution of collected video data to prepare it for analysis.

[1101] "Analysis" refers to the evaluation of processed video data to analyze subject behavior, facial expressions, voice, and positional relationships.

[1102] "Evaluation" refers to analyzing data and judging, recognizing, or evaluating it based on specific criteria.

[1103] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and affects safety and efficiency.

[1104] "Abnormal sounds" refer to sounds that deviate from normal sound patterns and that affect safety and efficiency.

[1105] An "alert" is a signal that warns an individual about detected abnormal behavior or sound.

[1106] "Disaster" refers to sudden natural phenomena or accidents such as earthquakes and fires.

[1107] An "evacuation order" is an instruction to guide people to a safe place in the event of a disaster.

[1108] "Monitoring" means observing the situation in real time and observing changes or abnormalities.

[1109] A "motion recognition algorithm" is a mathematical and statistical method for analyzing and recognizing the behavior and movements of a subject.

[1110] A "speech recognition algorithm" is a mathematical and statistical method for analyzing and recognizing a target's speech.

[1111] An "environmental sensor" is a sensor device used to detect environmental changes such as earthquakes and fires.

[1112] The present invention provides a system for ensuring safety in educational institutions and industrial facilities and for responding to abnormalities efficiently and quickly. Specific embodiments of this system are described below.

[1113] The server collects video data in real time from camera devices installed in educational or industrial facilities. This collected video data undergoes quality checks, noise reduction, image resolution correction, and other pre-processing. This pre-processing removes noise and blurred areas from the collected video data, making it ready for accurate analysis.

[1114] After preprocessing, the video data is analyzed to evaluate the target's behavior, voice, and positional relationship. The server uses motion and voice recognition algorithms to detect abnormal behavior or voice. If an abnormality is detected, an alert is immediately generated and sent to the relevant device, enabling a prompt response on-site.

[1115] In addition, the server reads data from environmental sensors such as earthquake sensors and fire alarms to detect the occurrence of sudden disasters. In the event of a disaster, the server immediately issues evacuation instructions and displays evacuation routes and instructions on terminals in each area.

[1116] After the evacuation has begun, the server analyzes the camera footage and monitors the evacuation situation in real time. If any individuals are unable to escape in time, it issues another warning, and once safety is confirmed, it notifies relevant parties and organizations that the evacuation has been completed.

[1117] In industrial facilities, the server also monitors the operation status of machines and generates alerts if an abnormality is detected, thereby preventing breakdowns and problems before they occur.

[1118] For example, if machine A in a factory makes an abnormal operating noise, the server will detect it and send a notification to the manager's terminal saying, "Machine A is making an abnormal noise." At this time, the manager can immediately rush to the site and take action.

[1119] The system also utilizes a generative AI model, allowing users to input prompts to detect abnormal behavior and speech, such as:

[1120] An AI algorithm that detects abnormal behavior and sounds in real time from camera footage within a factory and generates alerts. For example, it recognizes inappropriate contact between workers or abnormal sounds and immediately sends a notification to the manager's device.

[1121] This will significantly improve the safety of educational institutions and industrial facilities and enable efficient and rapid response.

[1122] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1123] Step 1:

[1124] The server collects video data in real time from multiple camera devices installed in educational or industrial facilities. It receives the video data sent by the camera devices (input) and stores it in a database (output). This collection process involves continuously capturing the video stream from each camera and adding necessary metadata (timestamp, camera location, etc.).

[1125] Step 2:

[1126] The server preprocesses the collected video data, specifically by reducing noise and correcting image resolution (input: video data, output: preprocessed video data). This preprocessing step uses filtering techniques and edge detection algorithms to reduce noise and improve image clarity.

[1127] Step 3:

[1128] The server analyzes the preprocessed video data and evaluates the subject's behavior, facial expressions, voice, and positional relationship (input: preprocessed video data, output: analysis results). Specifically, it uses motion recognition and voice recognition algorithms to detect abnormalities in behavior and voice. This involves analyzing the subject's behavioral patterns and identifying abnormalities.

[1129] Step 4:

[1130] If the server detects abnormal behavior or abnormal audio, it generates an alert and sends it to the device of the relevant person (input: analysis result, output: alert notification). Specifically, when an abnormality is detected, it creates a notification message and sends it to the device. This message includes the type of abnormality, the location where it was detected, and a timestamp.

[1131] Step 5:

[1132] The server reads data from earthquake sensors, fire alarm systems, etc. to detect sudden disasters (input: sensor data, output: disaster alerts). It periodically monitors data from the sensors and generates a disaster alert if an abnormal value is detected.

[1133] Step 6:

[1134] In the event of a disaster, the server sends evacuation instructions to terminals in each area (input: disaster alert, output: evacuation instructions). Specifically, it displays a message containing safe evacuation routes and instructions on the terminals and issues instructions via voice message.

[1135] Step 7:

[1136] After evacuation has begun, the server analyzes the camera footage and monitors the evacuation status in real time (input: camera footage, output: evacuation status report). The server detects moving objects in the footage and evaluates the progress of the evacuation.

[1137] Step 8:

[1138] Once safety is confirmed, the server notifies the relevant parties and organizations that the evacuation has been completed (input: evacuation status report, output: evacuation completion notification). Specifically, after confirming that everyone is safe, the server automatically sends notifications to the relevant parties.

[1139] Step 9:

[1140] The server monitors the operation status of machines in industrial facilities and generates an alert if an abnormality is detected (input: machine operation data, output: machine abnormality alert). It uses a behavior recognition algorithm to identify abnormal machine behavior and immediately notify relevant parties.

[1141] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1142] The present invention is a system for ensuring safety in educational institutions such as elementary and junior high schools, enabling quick and accurate responses when problems occur, and is also combined with an emotion engine that recognizes the user's emotions. A specific embodiment of this system is shown below.

[1143] 1. Video data collection and preprocessing

[1144] The server collects video data in real time from multiple cameras installed around the school. This video data is collected from various areas, such as classrooms, hallways, gymnasiums, and schoolyards. The server checks the quality of the collected video data and performs pre-processing such as noise reduction and image resolution correction.

[1145] 2. Analysis of video data

[1146] The server then inputs the preprocessed video data into behavioral and speech recognition algorithms for analysis. This evaluates students' behavioral patterns, facial expressions, and voice to detect abnormal behavior or voice. For example, if two students are making inappropriate contact in a hallway, the AI ​​will analyze this and recognize it as a sign of "bullying."

[1147] 3. Emotion Recognition by Emotion Engine

[1148] The server uses an emotion engine to analyze students' emotions from video and audio data. The emotion engine identifies emotions such as sadness, anger, and fear, improving the accuracy of detecting abnormal behavior. For example, if a student appears sad in the classroom, this could be a sign of a potential problem, allowing for early intervention.

[1149] 4. Send real-time alerts to educators

[1150] If the server detects abnormal behavior, abnormal voice, or abnormal emotion based on the emotion engine, it immediately generates an alert, which includes detailed information about the problem and its location, and is sent to the device of the relevant educator.

[1151] The device will notify the educator of the received alert visually and audibly. The alert content will include specific details of abnormal behavior, abnormal voice, or abnormal emotion, as well as recommended actions to take. The user (educator) can check the notification and immediately rush to the scene to address the problem.

[1152] 5. Disaster response

[1153] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[1154] 6. Evacuation situation monitoring and information sharing

[1155] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[1156] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[1157] Specific examples

[1158] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[1159] This system, combined with an emotion engine, will further improve safety within educational institutions and enable quick and effective responses to emergencies such as bullying and disasters.

[1160] The processing flow will be explained below.

[1161] Step 1:

[1162] The server collects video data in real time from multiple cameras installed throughout the school. Each camera is installed in a different area, such as a classroom, hallway, gymnasium, or schoolyard, and continuously monitors the behavior and situation of students.

[1163] Step 2:

[1164] The server preprocesses the acquired video data, specifically by performing noise reduction and image resolution correction to obtain clean data suitable for analysis. This preprocessing step is important to ensure data consistency and accuracy.

[1165] Step 3:

[1166] The preprocessed video data is input into an AI model integrated on the server. The AI ​​model evaluates students' behavior, facial expressions, and voice using behavior and voice recognition algorithms to detect abnormal student behavior (e.g., bullying) and abnormal voice (e.g., shouting).

[1167] Step 4:

[1168] The server analyzes students' emotions from the video and audio data using an emotion engine, which identifies emotions such as sadness, anger, and fear, thereby improving the accuracy of detecting abnormal behavior.

[1169] Step 5:

[1170] If abnormal behavior, voice, or emotion is detected, the server immediately generates an alert, which includes details of the problem and its location, and is sent to the device of the relevant educator.

[1171] Step 6:

[1172] The device will notify the educator of the received alert visually and audibly. The alert content will include specific details of abnormal behavior, abnormal voice, or abnormal emotion, as well as recommended actions to take. The user (educator) can check the notification and immediately rush to the scene to address the problem.

[1173] Step 7:

[1174] The server constantly monitors data from earthquake sensors and fire alarms. If a sudden disaster (e.g., earthquake or fire) is detected, the server automatically issues an emergency alert and sends evacuation instructions to the entire school. Devices in each classroom and area display audio and visual messages such as "Earthquake has occurred. Please evacuate," urging students and educators to evacuate.

[1175] Step 8:

[1176] The server monitors the evacuation situation in real time, analyzing camera footage to check whether students are evacuating smoothly and whether any students are being delayed. If the evacuation route is blocked, it issues another warning.

[1177] Step 9:

[1178] Once the evacuation is deemed complete, the server sends a notification to parents and relevant organizations. This notification includes details of the evacuation and safety confirmation. The user (educator) can use this information to provide additional instructions and provide care for students.

[1179] Step 10:

[1180] As a follow-up after the evacuation, the server records all data for later analysis and reporting, allowing for improvements to be identified and future disaster response capabilities to be enhanced.

[1181] Example 2

[1182] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1183] Conventional safety management systems in educational institutions lack the functionality to detect abnormal student behavior or voice activity in real time and respond quickly. It is also difficult to issue evacuation instructions and monitor evacuation situations quickly during disasters, making it difficult to ensure student safety. Furthermore, they lack the technology to properly analyze student emotions and detect potential problems early. As a result, responses to issues such as bullying and violence are delayed, potentially leading to a decline in the quality of the educational environment.

[1184] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting video data in real time from multiple camera devices installed in the educational institution, means for preprocessing the video data to perform quality checks, noise reduction, and image resolution correction, means for analyzing the preprocessed video data and evaluating students' behavioral patterns, facial expressions, voice, and positional relationships, means for detecting abnormal behavior and voice and sending alerts to relevant educators, means for analyzing students' emotions (sadness, anger, fear, etc.) using an emotion recognition engine to improve the accuracy of detecting abnormal behavior, means for detecting sudden disasters and issuing evacuation instructions, means for monitoring the evacuation situation in real time and confirming safety, and means for notifying parents and relevant organizations of the completion of evacuation. This comprehensively ensures the safety of students and enables early detection and rapid response to problems.

[1185] "Capture equipment" refers to equipment such as cameras that are installed within educational institutions and are used to collect video data in real time.

[1186] "Video data" refers to digital video information that shows the students and facilities collected by a camera.

[1187] "Preprocessing" refers to the process of performing quality checks, noise reduction, image resolution correction, etc. on collected video data.

[1188] A "behavioral pattern" is a sequence of specific student actions or behaviors obtained from the analyzed video data.

[1189] "Facial expressions" refer to the emotions and reactions that can be read from the students' facial expressions.

[1190] "Speech" refers to acoustic information such as human voices and environmental sounds that are analyzed using a speech recognition algorithm.

[1191] "Positional relationship" is information that indicates the spatial arrangement and relationships of students and objects.

[1192] "Abnormal behavior" refers to behavior that deviates from normal patterns of behavior and requires vigilance or action.

[1193] "Abnormal sounds" refer to sounds that are different from normal sounds, such as cries or calls for help, that require vigilance or a response.

[1194] An "alert" is a warning message sent to relevant educators when abnormal behavior or sound is detected.

[1195] An "emotion recognition engine" is a software technology that analyzes video and audio data to identify emotions such as sadness, anger, and fear.

[1196] "Sudden disasters" refer to natural disasters and accidents, such as earthquakes and fires, that occur unexpectedly and require a rapid response.

[1197] An "evacuation order" is an instruction to students and educators to move to a safe place in the event of a sudden disaster.

[1198] "Evacuation situation" refers to the progress of evacuation actions by students and educators after an evacuation order is issued.

[1199] "Safety check" is the act of making sure that students and educators are safe when evacuation is complete.

[1200] "Guardian" refers to the person registered as the parent or guardian of a student.

[1201] "Affiliated institutions" refers to external organizations or facilities that have a relationship with an educational institution (e.g., fire department, police department, etc.).

[1202] This invention is a comprehensive surveillance system that improves safety in educational institutions and enables rapid and appropriate response when problems occur. The system collects video data in real time, preprocesses and analyzes it, and detects abnormal behavior or abnormal sounds and notifies educators. Furthermore, it uses an emotion recognition engine to analyze student emotions and improve the accuracy of detecting abnormal behavior. Furthermore, it issues evacuation instructions in the event of a sudden disaster, monitors evacuation conditions, and confirms safety.

[1203] Hardware and Software Configuration

[1204] The system uses the following hardware and software:

[1205] Hardware: High-resolution IP cameras, earthquake sensors, fire alarms

[1206] Software: OpenCV (image processing library), TensorFlow (machine learning library), EmotionAPI (emotion recognition API)

[1207] Server Roles

[1208] The server collects video data in real time from multiple high-resolution IP cameras installed on campus. The collected video data undergoes preprocessing, such as noise reduction and image resolution correction. The preprocessed video data is then analyzed using behavior recognition algorithms (using OpenCV and TensorFlow) to evaluate students' behavioral patterns, facial expressions, and voice. In addition, an emotion recognition engine (EmotionAPI) is used to analyze students' emotions from the video and audio data.

[1209] When abnormal behavior, abnormal voice, or abnormal emotion is detected by the emotion recognition engine, an alert is immediately generated. The server sends this alert to the device of the relevant educator, enabling a prompt response.

[1210] Device Role

[1211] The device will notify the educator of the received alert visually and audibly. The alert will include details of the problem and the location where it occurred. For example, it could say, "Student A is sad in the classroom. Details: Student A was captured on video crying alone. Action is required." After the educator confirms this, they can immediately rush to the scene and take action.

[1212] User Roles

[1213] The user (educator) checks the alert from the device and immediately rushes to the scene to deal with the problem. Also, in the event of a sudden disaster, the user receives evacuation instructions from the server and instructs students to evacuate. The server constantly monitors data from earthquake sensors and fire alarms, and immediately issues evacuation instructions when it detects a disaster. Devices in each classroom and area display audio messages and visual displays such as "Earthquake has occurred. Please evacuate," encouraging students and educators to take evacuation action.

[1214] Specific examples

[1215] For example, if an earthquake occurs in a classroom, the server immediately detects it and sends an emergency alert to the entire school. Devices in each classroom display evacuation instructions and guide students with voice messages. The server monitors the progress of the evacuation in real time and confirms that all students have evacuated safely. Once the evacuation is complete, it notifies parents and relevant organizations and quickly shares the specific situation.

[1216] Prompt Sentence Examples

[1217] The prompt sentence to be input to the generative AI model is shown below.

[1218] "You are building a safety management system for an elementary school. Explain how you can collect video data from cameras in real time, use behavior recognition algorithms and an emotion engine to detect abnormal behaviors and emotions, and send alerts to educators."

[1219] This prompt allows the generative AI model to suggest relevant technical approaches and detailed processing methods, thus comprehensively ensuring safety within educational institutions and enabling early detection and rapid response to problems.

[1220] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1221] Step 1:

[1222] The server collects video data in real time from multiple camera devices installed on campus. The server receives the collected video data as input, checks the video quality, and performs pre-processing such as noise reduction and image resolution correction. The output of this pre-processing is improved video data. Specifically, the server continuously acquires the video stream from the cameras, applies noise reduction filters, and executes correction algorithms to improve image quality.

[1223] Step 2:

[1224] The server inputs the preprocessed video data and analyzes it using behavioral and speech recognition algorithms. Specifically, it evaluates students' behavioral patterns, facial expressions, and audio data from the video data and detects abnormal behavior and audio based on these. The output of this analysis is the detection results of abnormal behavior and audio. For example, the behavioral recognition algorithm identifies whether a student is performing a specific action in the video, and the speech recognition algorithm detects audio events such as screaming or calling for help.

[1225] Step 3:

[1226] The server then inputs the analyzed video and audio data using an emotion recognition engine to identify the student's emotions. The output of this emotion recognition is an emotion label, such as sadness, anger, or fear. At this stage, the student's facial expressions and voice tone captured on video are analyzed and mapped to specific emotions. For example, EmotionAPI analyzes a student's facial expression and determines whether it corresponds to "sadness."

[1227] Step 4:

[1228] The server generates an alert when it detects abnormal behavior, abnormal voice, or abnormal emotion based on the analysis results. This alert includes detailed information about the problem and the location where it occurred. The generated alert is sent to the educator's device. The input for this transmission is the analysis result, and the output is notification information that is displayed on the relevant educator's device. Specifically, the server classifies the analysis result, composes an appropriate alert message, and sends it to the educator in real time.

[1229] Step 5:

[1230] The device will notify the educator of the received alert visually and audibly. The input here is the alert information sent from the server, and the output is an alert message displayed on the teacher's device. For example, a message such as "Student A is crying in the classroom. Details: Student A has been captured on video crying alone. Action required" will be displayed.

[1231] Step 6:

[1232] The server constantly monitors data from earthquake sensors and fire alarms, and if it detects a disaster, it automatically issues an emergency alert and sends evacuation instructions to the entire school. The input here is data from the earthquake sensors and fire alarms, and the output is an alert message that sends evacuation instructions. For example, the moment an earthquake is detected, an audio and visual message such as "Earthquake has occurred. Please evacuate" is displayed on terminals in each classroom and area.

[1233] Step 7:

[1234] The server analyzes camera footage in real time and monitors the evacuation status from each area. It checks for any obstructions in the evacuation routes and automatically issues a second warning. The input is video data from the school cameras, and the output is real-time information on the evacuation status. If a obstruction is detected, an alert message ordering a second evacuation is generated and sent to each device.

[1235] Step 8:

[1236] After the evacuation is complete, the server notifies parents and related organizations of the completion of the evacuation. The input here is the detection result of the evacuation completion, and the output is a notification message sent to parents and related organizations. Specifically, once safety confirmation is complete, a message such as "Evacuation Complete Notification: All students and educators have evacuated safely" is sent to parents and related organizations.

[1237] In this way, the entire system can comprehensively ensure student safety and enable early detection and rapid response to problems.

[1238] (Application example 2)

[1239] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1240] The present invention is required to ensure safety in places where many people gather, such as educational institutions and office buildings, and to respond quickly to abnormal behavior and emergencies. However, current systems are unable to adequately detect abnormal behavior and emotions in real time, making it difficult to respond quickly and accurately. Furthermore, even in the event of a sudden disaster, the system is unable to monitor evacuation situations in real time and confirm overall safety, which may result in delayed response.

[1241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1242] In this invention, the server includes: means for collecting video data in real time from multiple cameras installed in educational institutions or other facilities; means for preprocessing the video data and preparing for analysis; means for analyzing the preprocessed video data and evaluating the behavior, facial expressions, voice, and positional relationship of subjects; means for detecting abnormal behavior or abnormal voice and sending alerts to relevant parties; means for analyzing the emotions of subjects from the video data and voice data using an emotion engine and detecting abnormal emotions; means for detecting sudden disasters and issuing evacuation instructions; means for monitoring the evacuation situation in real time and confirming safety; and means for notifying parents and relevant organizations of the completion of evacuation. This enables the detection of abnormal behavior or abnormal emotions in real time from multiple camera footage, enabling rapid and accurate response. Furthermore, it is possible to quickly issue evacuation instructions, monitor the evacuation situation, and confirm safety even in the event of a sudden disaster.

[1243] An "educational institution or other facility" is a place where people gather to carry out educational, business, or other activities, including, but not limited to, a school.

[1244] "Multiple cameras" refer to imaging devices that are installed at different locations and serve to collect video data from multiple viewpoints simultaneously.

[1245] "Means for collecting video data in real time" refers to a method and device for instantly importing video captured by a camera into a system such as a server.

[1246] "Means for preprocessing and preparing for analysis" refers to methods and devices for reducing noise in acquired video data, correcting resolution, etc., to make the data suitable for analysis.

[1247] "Means for analyzing preprocessed video data and assessing subject behavior, facial expressions, voice, and positional relationships" refers to a method and apparatus that uses preprocessed data to identify and assess a subject's behavioral patterns, emotional expressions, vocal content, and spatial relationships.

[1248] "Means for detecting abnormal behavior or abnormal sounds and sending alerts to relevant parties" refers to a method and device for detecting unusual behavior or sounds from analyzed data and notifying relevant parties of that information.

[1249] An "emotion engine" is software or algorithms that analyze video and audio data to identify a subject's emotional state, such as joy, anger, sadness, or fear.

[1250] The "means for analyzing emotions" refers to a method and apparatus for analyzing emotions from specific data using an emotion engine.

[1251] "Means for detecting sudden disasters and issuing evacuation instructions" refers to methods and devices for instantly recognizing emergencies such as earthquakes and fires and issuing appropriate evacuation instructions based on that information.

[1252] "Means for monitoring evacuation situations in real time and confirming safety" refers to a method and device for monitoring the movements of people during evacuation in real time and confirming whether the safety of everyone is ensured.

[1253] "Means for notifying parents and related organizations of the completion of evacuation" refers to methods and devices for confirming that evacuation has been completed safely and for quickly and accurately communicating the results to parents and related organizations.

[1254] The present invention provides a system for ensuring safety in educational institutions or other facilities and for responding quickly and appropriately to abnormal behavior or emergencies. This system is constructed using the hardware and software described below.

[1255] The system involves installing multiple cameras in designated locations and collecting video data in real time. For example, in educational institutions, cameras are placed in classrooms, hallways, gymnasiums, and schoolyards, while in office buildings, cameras are installed at the entrance and in shared spaces on each floor.

[1256] The server receives the collected video data and performs preprocessing, which includes noise reduction and image resolution correction. The preprocessed video data is then analyzed to evaluate the subject's behavior, facial expression, voice, and positional relationship using behavior and voice recognition algorithms.

[1257] The server uses an emotion engine to analyze the subject's emotions from video and audio data. The emotion engine identifies emotions such as sadness, anger, and fear, improving the accuracy of detecting abnormal behavior. If abnormal behavior, audio, or emotion is detected, an alert is sent to the relevant parties. The alert is immediately displayed on a notification terminal, allowing the relevant parties to respond quickly.

[1258] In addition, the server constantly monitors data from earthquake sensors and fire alarms to detect sudden disasters. If a disaster is detected, the server automatically issues an emergency alert and sends evacuation instructions to all facilities. During an evacuation, camera footage is analyzed, the evacuation situation is monitored in real time, and safety is confirmed. Once everyone has evacuated, the server notifies parents and relevant organizations that the evacuation is complete.

[1259] As a concrete example, consider the case where someone shouts at the entrance of an office building. The server analyzes the camera footage and audio to detect anger. If it determines that the behavior is abnormal, an alert is immediately sent to the relevant parties. The alert includes the location of the abnormality and how to respond, allowing the relevant parties to quickly rush to the scene and take appropriate action.

[1260] An example prompt is, "We would like to develop a security system for an office building. Please design a system that analyzes camera footage, performs emotion recognition, and sends an alert if there is any abnormal behavior."

[1261] In this way, abnormal behavior and abnormal emotions can be detected in real time from multiple camera footage, enabling a quick and appropriate response. It can also be used to quickly issue evacuation instructions, monitor evacuation situations, and confirm safety in the event of a sudden disaster.

[1262] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1263] Step 1:

[1264] The server collects video data in real time from multiple cameras installed in educational institutions or other facilities.

[1265] Input: Video feed from camera

[1266] Output: Raw video data

[1267] Specific operation: Continuously collects video data from multiple locations where cameras are installed and sends it to a server.

[1268] Step 2:

[1269] The server pre-processes the collected video data.

[1270] Input: Raw video data

[1271] Output: Pre-processed video data

[1272] Specific operation: Performs noise reduction and image resolution correction to make the image suitable for analysis. For example, improves the quality of the video data using OpenCV.

[1273] Step 3:

[1274] The server analyzes the pre-processed video data and evaluates the subject's behavior, facial expression, voice, and position.

[1275] Input: Preprocessed video data

[1276] Output: Evaluation results on the subject's behavior, facial expression, voice, and positional relationship

[1277] Specific operation: Using behavioral and voice recognition algorithms, the system analyzes video and audio to identify the target's behavioral patterns, facial expressions, vocalizations, etc.

[1278] Step 4:

[1279] The server uses an emotion engine to analyze the emotion of the target from the video data and audio data.

[1280] Input: Preprocessed video and audio data

[1281] Output: Analysis results regarding the subject's emotions

[1282] Specific operation: For example, the EmotionRecognizer library is used to analyze facial expressions and voice tones in video to identify emotions. The emotion engine identifies emotions such as sadness, anger, and fear.

[1283] Step 5:

[1284] If the server detects abnormal behavior, abnormal voice, or abnormal emotions, it will send an alert to relevant parties.

[1285] Input: Analysis results of behavior, voice, and emotions

[1286] Output: Alert message

[1287] Specific operation: If abnormal behavior or abnormal emotions are detected, an alert message will be generated and sent to the device or smartphone, for example, via email or push notification.

[1288] Step 6:

[1289] The server detects sudden disasters and issues evacuation instructions.

[1290] Input: Data from disaster sensors and fire alarms

[1291] Output: Evacuation instruction message

[1292] Specific operation: Data from earthquake sensors and fire alarms is constantly monitored, and if a disaster is detected, an emergency alert is automatically issued and evacuation instructions are sent to all facilities.

[1293] Step 7:

[1294] The server monitors the evacuation situation in real time and confirms safety.

[1295] Input: Camera footage during evacuation

[1296] Output: Monitoring results of evacuation situation

[1297] Specific operation: Analyzes camera footage and monitors in real time whether people are evacuating properly. If there is a problem, it sends another warning.

[1298] Step 8:

[1299] After the evacuation is completed, the server notifies the parents and related organizations of the completion of the evacuation.

[1300] Input: Confirmation data of evacuation completion

[1301] Output: Evacuation completion notification

[1302] Specific operations: After confirming that everyone has evacuated, an evacuation completion notification will be sent to parents and relevant organizations via email or message.

[1303] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1304] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1305] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1306] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1307] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1308] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1309] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1310] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1311] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1312] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1313] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1314] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1315] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1316] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1317] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1318] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1319] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1320] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1321] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1322] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1323] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1324] The following is further disclosed regarding the above embodiment.

[1325] (Claim 1)

[1326] A means for collecting video data in real time from multiple cameras installed in educational institutions;

[1327] means for preprocessing the video data and preparing it for analysis;

[1328] A means for analyzing the preprocessed video data to evaluate student behavior, facial expressions, voice, and positional relationships;

[1329] A means of detecting abnormal behavior or abnormal voice and sending an alert to the relevant educator;

[1330] A means of detecting sudden disasters and issuing evacuation instructions,

[1331] A means of monitoring evacuation situations in real time and confirming safety,

[1332] A means of notifying parents and relevant agencies that evacuation has been completed;

[1333] A system including:

[1334] (Claim 2)

[1335] 10. The system of claim 1, comprising means for using behavior recognition algorithms and speech recognition algorithms to assess abnormal student behavior.

[1336] (Claim 3)

[1337] 10. The system of claim 1, further comprising means for reading data from earthquake sensors and fire alarms to provide for sudden disaster detection.

[1338] "Example 1"

[1339] (Claim 1)

[1340] a means for collecting video data in real time from a plurality of visual sensors installed in an educational institution;

[1341] means for checking the quality of the video data and performing noise reduction and image resolution correction;

[1342] means for analyzing the preprocessed video data using an activity recognition algorithm and a voice recognition algorithm to evaluate activity patterns;

[1343] A means of detecting abnormal behavior or abnormal voice and sending real-time alerts to relevant educators;

[1344] A means of reading data from earthquake sensors and fire alarms to detect sudden disasters and issue evacuation instructions,

[1345] Real-time analysis of camera footage from each area will be carried out to monitor the evacuation status of students and confirm completion of evacuation.

[1346] A means of notifying parents and relevant agencies that evacuation has been completed;

[1347] A system including:

[1348] (Claim 2)

[1349] 10. The system of claim 1, comprising means for assessing abnormal student behavior using behavior recognition algorithms and speech recognition algorithms.

[1350] (Claim 3)

[1351] 10. The system of claim 1, comprising means for reading data from earthquake sensors and fire alarm devices.

[1352] "Application Example 1"

[1353] (Claim 1)

[1354] means for collecting video data in real time from a plurality of image capture devices installed in an educational or industrial facility;

[1355] means for preprocessing the video data and preparing it for analysis;

[1356] means for analyzing the preprocessed video data to assess the subject's behavior, facial expression, voice, and position;

[1357] A means of detecting abnormal behavior or abnormal sounds and sending alerts to relevant parties;

[1358] A means of detecting sudden disasters and issuing evacuation instructions,

[1359] A means of monitoring evacuation situations in real time and confirming safety,

[1360] A means of notifying relevant parties and organizations of the completion of evacuation;

[1361] a means for evaluating the machine's operating status and generating an alert if an anomaly is detected;

[1362] A means to ensure safety in industrial facilities and realize efficient and rapid response to abnormalities;

[1363] A system including:

[1364] (Claim 2)

[1365] 10. The system of claim 1, further comprising means for using motion recognition algorithms and speech recognition algorithms to assess abnormal behavior of a subject or abnormal operation of a machine.

[1366] (Claim 3)

[1367] 10. The system of claim 1, further comprising means for reading data from earthquake sensors, fire alarms, and other environmental sensors to provide for sudden disaster detection.

[1368] "Example 2: Combining Emotion Engines"

[1369] (Claim 1)

[1370] a means for collecting video data in real time from a plurality of image capture devices installed in an educational institution;

[1371] means for pre-processing the video data to perform quality checks, noise reduction, and image resolution correction;

[1372] A means for analyzing the preprocessed video data and evaluating students' behavioral patterns, facial expressions, voice, and positional relationships;

[1373] A means of detecting abnormal behavior or abnormal voice and sending an alert to the relevant educator;

[1374] A means for analyzing students' emotions (sadness, anger, fear, etc.) using an emotion recognition engine to improve the accuracy of detecting abnormal behavior;

[1375] A means of detecting sudden disasters and issuing evacuation instructions,

[1376] A means of monitoring evacuation situations in real time and confirming safety,

[1377] A means of notifying parents and relevant agencies that evacuation has been completed;

[1378] A system including:

[1379] (Claim 2)

[1380] 10. The system of claim 1, comprising means for using behavior recognition algorithms and speech recognition algorithms to evaluate abnormal behavior and abnormal speech of students.

[1381] (Claim 3)

[1382] 10. The system of claim 1, comprising means for using an emotion recognition engine to analyze student emotions and improve accuracy of detecting anomalous behavior.

[1383] (Claim 4)

[1384] 10. The system of claim 1, further comprising means for reading data from earthquake sensors and fire alarms to provide for sudden disaster detection.

[1385] "Application example 2 when combining emotion engines"

[1386] (Claim 1)

[1387] a means for collecting video data in real time from multiple cameras installed in an educational institution or other facility;

[1388] means for preprocessing the video data and preparing it for analysis;

[1389] means for analyzing the preprocessed video data to assess the subject's behavior, facial expression, voice, and position;

[1390] A means of detecting abnormal behavior or abnormal sounds and sending alerts to relevant parties;

[1391] a means for analyzing the emotions of a subject from video data and audio data using an emotion engine and detecting abnormal emotions;

[1392] A means of detecting sudden disasters and issuing evacuation instructions,

[1393] A means of monitoring evacuation situations in real time and confirming safety,

[1394] A means of notifying parents and relevant agencies that evacuation has been completed;

[1395] A system including:

[1396] (Claim 2)

[1397] 10. The system of claim 1, comprising means for using behavior recognition algorithms and voice recognition algorithms to evaluate abnormal behavior and abnormal voice.

[1398] (Claim 3)

[1399] 10. The system of claim 1, further comprising means for reading data from sensors and alarms to perform sudden disaster detection. [Explanation of symbols]

[1400] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting video data in real time from multiple cameras installed in educational institutions; means for preprocessing the video data and preparing it for analysis; A means for analyzing the preprocessed video data to evaluate student behavior, facial expressions, voice, and positional relationships; A means of detecting abnormal behavior or abnormal voice and sending an alert to the relevant educator; A means of detecting sudden disasters and issuing evacuation instructions, A means of monitoring evacuation situations in real time and confirming safety, A means of notifying parents and relevant agencies that evacuation has been completed; A system including:

2. The system of claim 1 , further comprising means for using behavior recognition algorithms and speech recognition algorithms to assess abnormal student behavior.

3. 10. The system of claim 1, further comprising means for reading data from earthquake sensors and fire alarms to detect sudden disasters.

Images