system

Abstract

We provide the system. [Solution] A device for collecting audio and image data from a high-security area or a home-work environment, A central processing unit that uses a generative model to analyze collected audio data and detect dangerous keywords, Image analysis device means for analyzing collected image data and determining whether prohibited items are present, An evaluation device means for assessing potential information leaks and notifying a specific management department of the warning, A correction device means for masking parts of the screen and audio data after a warning, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a modern business environment, high-level information security is required. However, the risk of information leakage is increasing in various workplaces such as high-security areas and telecommuting environments. Under such circumstances, there is a need for a technology to effectively monitor risks using voice and image data and prevent the possibility of information leakage to the outside. In addition, there is a lack of reliable systems for evaluating potential information leakage in real time and responding promptly.

Means for Solving the Problems

[0005] This invention provides a system that utilizes a device to collect audio and image data from a high-security area or teleworking environment, and analyzes this data to detect potential information leaks. Specifically, a central processing unit using a generative model detects dangerous keywords in audio data, and an image analysis device determines prohibited items in image data. Furthermore, an evaluation device assesses the risk of information leaks and, if necessary, notifies the management department of a warning, enabling a rapid and effective response. After a warning is issued, this system further enhances information security by including a modification device to mask parts of the screen and audio data.

[0006] A "high-security area" refers to an environment or space containing information that requires special security protection, where access and the bringing in of devices are strictly restricted.

[0007] A "work-from-home environment" refers to a work environment where employees perform their duties at home and access company data and systems remotely.

[0008] "Audio and image data" refers to audio information acquired by a microphone and video information acquired by a camera, and these are the subjects of information analysis.

[0009] A "generative model" refers to artificial intelligence technology that learns from large amounts of data and recognizes new data patterns, and is particularly used for the automatic generation and analysis of speech and text.

[0010] A "central processing unit" refers to a computer system used for analyzing audio and image data, where the operation of generative models and the processing of analysis results are carried out centrally.

[0011] An "image analysis device" refers to a computer system that analyzes acquired image data to recognize specific objects or actions.

[0012] An "evaluation device" refers to a system that assesses the potential risk of information leakage based on data analysis results and determines the degree of risk.

[0013] A "correction device" refers to a device used to mask or modify screen or audio data when a risk of information leakage is detected.

[0014] "Dangerous keywords" refer to terms or phrases that are deemed to potentially lead to information leaks or security problems.

[0015] "Prohibited items" refer to devices or items whose bringing into or use in certain areas or situations is restricted for security reasons. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] Shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiments for Carrying out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] One embodiment of this invention is configured as an advanced monitoring system for preventing information leaks in high-security areas and home-work environments. This system consists of three main components: a server, a terminal, and a user, each working together to enhance information security.

[0038] Server role:

[0039] The server functions as the central hub of the system, analyzing audio and image data collected from terminals. Using generative models, the server converts audio data into text and detects designated dangerous keywords or phrases. For image data, an image analysis device performs object recognition to determine if prohibited items are present. For example, if a prohibited smartphone is visible in the video within a secure area, the server will identify its presence.

[0040] Furthermore, the server uses an evaluation device to assess the potential risk of information leakage based on the analysis results, and if a risk is detected, it sends an alert to the management department. This notification is provided via email or dashboard alert, enabling a quick response.

[0041] Terminal role:

[0042] The terminal operates on the user's side and performs information collection and masking as needed. The terminal collects data in real time through the microphone and camera and sends it to the server. Furthermore, if the server detects a risk, it will mask specific screen content or audio output to conceal confidential information. For example, if sensitive information is displayed on the screen, it will be blurred to prevent unnecessary disclosure of information to third parties.

[0043] User roles:

[0044] Users perform their normal work operations, but if they receive a security notification, they will stop or correct information sharing based on the displayed instructions. Users can also receive guidance to improve their security awareness, contributing to the protection of information in their daily work.

[0045] In this way, it is possible to build a system that minimizes the risk of information leakage through the interaction of servers, terminals, and users. As a specific example, one company is using this system to monitor audio during meetings and take measures to prevent confidential project information from being leaked to the outside. In this manner, the present invention dramatically improves information security in corporate activities.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] The device collects audio and image data in real time in high-security areas or work-from-home environments. It uses a microphone and camera to capture this data, converts it to an appropriate format, and prepares it for transmission to a server.

[0049] Step 2:

[0050] The terminal encrypts the collected data and sends it to the server. To prevent interception and tampering of communications, the SSL / TLS protocol is used for secure data transfer.

[0051] Step 3:

[0052] The server analyzes the audio data received from the terminal using a generative model. It utilizes automatic speech recognition (ASR) technology to generate text from the audio data and detect dangerous keywords and phrases.

[0053] Step 4:

[0054] The server analyzes the image data using an image analysis algorithm. Here, a machine learning model is used to check for the presence of prohibited items (e.g., smartphones), and if detected, the information is recorded.

[0055] Step 5:

[0056] The server evaluates the results of the analysis of audio and image data to identify potential data breach risks. If a risk is detected, it sends an alert to the security department to quickly notify the relevant personnel.

[0057] Step 6:

[0058] The terminal receives instructions from the server and, if necessary, masks some of the information on the screen and some of the audio. This process physically hides sensitive data from the user's display and output audio to prevent unintended information leaks.

[0059] Step 7:

[0060] When users receive a security notification, they will manually correct information or cease certain actions. Based on the notified measures, they will strive to mitigate the risk of information leakage by promptly taking action such as postponing or canceling information sharing.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] Information leaks pose a significant risk to companies and organizations, and mitigating this risk is crucial, especially in highly confidential areas and remote work environments. Traditional methods suffer from insufficient monitoring and analysis to prevent information leaks, and struggle to immediately detect specific languages ​​or items. There is a need for systems that can improve this situation and significantly reduce the risk of information leaks.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes a device means for collecting information from a highly confidential area or residential work environment, a central control device means that analyzes the collected audio information and uses a generation model to detect dangerous words, and an automatic analysis device means that analyzes the collected image information and determines the presence of prohibited items. This makes it possible to efficiently detect specific audio and visual hazard elements and take immediate countermeasures to reduce the risk of information leakage.

[0066] 1. A "highly confidential area" refers to an environment with a high risk of information leakage, and is a place where particularly protected information or activities are conducted.

[0067] 2. "Residential work environment" refers to an environment in which an individual performs work from their home or other location, and is a place where special monitoring is required for information protection.

[0068] 3. "Information-collecting device" refers to a device or system for collecting data such as voice and images in real time and transmitting it to the next processing stage.

[0069] 4. A "generative model" is a machine learning model used to analyze collected audio data and detect specified dangerous keywords or phrases.

[0070] 5. A "central control unit" is a central processing unit that oversees and manages the entire system and performs data analysis and evaluation.

[0071] 6. An "automated analysis device" is a device that analyzes collected image data using machine learning technology to detect specific items or scenarios.

[0072] 7. An "evaluation tool" is a mechanism for assessing the risk of information leakage and notifying the management department of warnings or corrective actions based on that assessment.

[0073] 8. A "regulation device" is a device used to control or conceal parts of the screen display or sound in order to prevent information leakage.

[0074] This invention is an advanced monitoring system aimed at preventing information leaks in highly confidential areas and remote work environments. The system primarily consists of three elements: a server, a terminal, and a user, each working together to enhance information security.

[0075] The server acts as the core of the system, analyzing audio and image information collected from terminals. The server uses a generative AI model to convert audio information into text, monitoring the information by detecting specific potentially dangerous words. Furthermore, it uses an automated analysis device to perform object recognition on image data to check for prohibited items. For example, one prohibited item might be the use of a smartphone within a specific area. Based on the analysis results, the server assesses the potential risk of information leakage and sends warnings to the management department as needed.

[0076] The terminal collects information in the user's environment and performs masking if necessary. Audio and image information is collected in real time through the terminal's microphone and camera. Based on instructions from the server, the terminal blurs specific content displayed on the monitor to prevent the leakage of confidential information.

[0077] Users receive security notifications while performing their normal duties. Following warnings from the server, users can stop or correct information sharing. Furthermore, users can enhance their awareness of information protection by receiving guidance to improve their security awareness.

[0078] As a concrete example, one company uses this system to monitor audio during meetings and prevent the leakage of information related to highly confidential projects. An example of a prompt message is the instruction, "Translate audio in the security area into text and detect the specified dangerous phrase." In this way, the present invention contributes to strengthening information security.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The device collects audio and image data and sends it to the server. Specifically, the device's microphone and camera record audio and video in real time and send that data to the server. In this process, audio and image data are provided as input and transferred to the server in their original format.

[0082] Step 2:

[0083] The server converts the received audio data into text using a generation AI model. The system receives audio data as input and generates text data through automatic speech recognition. The converted text is then used in the next analysis step.

[0084] Step 3:

[0085] The server analyzes the text data to detect whether it contains dangerous keywords or phrases. Using a generative AI model, it identifies risky phrases by evaluating specified language patterns. The output of this step is information on whether danger was detected.

[0086] Step 4:

[0087] The server processes image data using an automated analysis system to perform object recognition. It analyzes the received images to assess the likelihood of specific items or prohibited objects being present. The output provides information on whether hazardous materials were detected.

[0088] Step 5:

[0089] The server assesses the risk of data leakage based on the analysis results of audio and image data and sends a warning to the management department. If a risk is detected, the server sends an alert to the administrator in the form of an email or dashboard notification.

[0090] Step 6:

[0091] The terminal receives a warning instruction from the server and masks the screen display and audio output. It receives warning information provided by the server as input and performs controls such as mosaic processing to prevent the leakage of confidential information. The output of this step is a masked display or output.

[0092] Step 7:

[0093] Users receive security notifications and take necessary actions. The system receives notifications from terminals and servers as input, and takes necessary measures such as stopping or correcting information sharing. The output is a work environment with improvements or corrective actions implemented.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In high-security areas and remote work environments, a lack of effective monitoring and control measures to prevent information leaks is a challenge. In particular, there is a need to analyze voice and image data in real time to quickly detect and address potential risks, but conventional systems are unable to adequately achieve this.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes a device means for collecting voice and image data from a high-security area or a work-from-home environment; a central processing unit means that analyzes the collected voice data and uses a generative model to detect dangerous keywords; and an evaluation device means that evaluates potential information leaks and notifies a specific management department of the warning. This enables real-time information leak prevention and rapid risk management response.

[0099] A "high-security area" is a region where the risk of information leakage is high, and therefore strict security management is required.

[0100] A "work-from-home environment" refers to a work environment where the home is used as the base of operations when performing work.

[0101] A "device for collecting audio and image data" is a device that uses a microphone and a camera to acquire audio and images.

[0102] A "central processing unit using a generative model" is a device equipped with generative AI technology to analyze audio data and detect specified dangerous keywords.

[0103] An "image analysis device" is a device that uses machine learning algorithms to analyze image data and determine the presence or absence of an object.

[0104] An "evaluation device" is a device that assesses potential information leakage risks and issues warnings as necessary.

[0105] A "correction device" is a device that, after a warning, masks parts of the screen and audio data to block confidential information.

[0106] A "blocking device" is a device that has the function of blocking specific information in order to prevent unauthorized access to that information.

[0107] This system is designed to prevent information leaks in high-security areas or remote work environments. It consists of three main components: servers, terminals, and users.

[0108] The server functions as the central hub for analyzing audio and image data. Audio data is collected from the terminal via a microphone and transferred to the server. There, it is converted to text using automatic speech recognition technology such as Google® Speech-to-Text API. A generative AI model is used to detect specified dangerous keywords from this text data. Image data is transmitted from the terminal via a camera and analyzed using image processing libraries such as OpenCV. Machine learning algorithms are used to determine, for example, the presence or absence of prohibited items.

[0109] Based on these analysis results, the server assesses the potential risk of data breaches and generates an alert via the assessment device. This alert is then sent to the management department via email or a dashboard alert.

[0110] If a warning is issued, the terminal controls the display and output of data, and a correction device masks certain information. This includes processes to prevent the leakage of confidential information, such as muting audio and blurring text and images on the screen.

[0111] Users will perform their normal duties, but when they receive a warning, they will restrict or modify data disclosure based on the instructions given. Furthermore, users will receive guidance to raise their awareness of information security, thereby strengthening information protection in their daily work.

[0112] As a concrete example, if this system is implemented in a certain corporate environment and a project code name is mentioned during a meeting, the server will immediately detect the risk and alert the participants, thereby preventing information leakage to external parties.

[0113] Examples of prompt statements include the following:

[0114] "We analyzed audio data from corporate meetings and, if the keyword 'project code' was found, we issued a warning notification based on that information. Because there is a risk of information leakage if this project code is disclosed externally, we urge participants who receive the notification to handle it with caution."

[0115] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0116] Step 1:

[0117] The device collects audio and image data from the environment in real time. This collected data is acquired via a microphone and camera and prepared for transmission to a server. The input is raw audio and image data, and the output is transmission data transferred to the server.

[0118] Step 2:

[0119] The server converts received audio data from speech to text using the Google Speech-to-Text API. It then utilizes a generative AI model to analyze the text data and detect specified dangerous keywords. The input is audio data, and the output is text data along with the detected keywords. The analysis lists specific words and phrases, and a risk assessment is performed based on this list.

[0120] Step 3:

[0121] The server analyzes image data using OpenCV and performs object recognition using machine learning algorithms. It determines whether specific prohibited items are present in the image. The input to this process is image data, and the output is the object recognition result. If a prohibited item is identified through analysis, the warning process is immediately initiated.

[0122] Step 4:

[0123] The server uses an evaluation device to assess the risk of information leakage based on the analysis results. If a risk is detected, it creates a notification to send an alert to the management department. The input is the analyzed keywords and item data, and the output is an alert notification. The notification is sent via email or a dashboard, enabling a quick response.

[0124] Step 5:

[0125] The terminal controls the screen or audio and masks certain information according to instructions received from the server. The input is the control instructions from the server, and the output is the display of masked data. This process ensures that measures are taken to prevent sensitive information from being leaked to third parties.

[0126] Step 6:

[0127] When a user receives a warning notification from their device, they take the necessary action. The input is the warning notification from the device, and the output is the user's response to the risk. The user will stop publishing data or perform necessary corrective actions according to this notification.

[0128] Each step works closely together to minimize the risk of data breaches and help build a highly secure environment.

[0129] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0130] One embodiment of this invention is a monitoring system aimed at preventing information leaks in high-security areas and teleworking environments, which achieves even more advanced security management by combining it with an emotion engine. This system consists of three main components: a server, a terminal, and a user.

[0131] Server role:

[0132] The server functions as the core of the entire system, analyzing audio and image data transmitted from terminals. The server uses generative models to analyze audio data and identify dangerous keywords and phrases. Furthermore, it uses image analysis equipment to check for the presence of prohibited items in image data.

[0133] The newly integrated emotion engine works to recognize the user's emotional state from voice data. In this process, the server analyzes changes in voice tone and emphasis to identify emotions such as joy, anger, and sadness. Simultaneously, it analyzes the user's facial expressions from image data to provide information that complements the emotional state. These emotional changes are considered as potential security risks.

[0134] Terminal role:

[0135] The device uses sensors, cameras, and microphones to collect data in real time and transmit it to a server. This data provides material for analysis by an emotion engine. Furthermore, based on instructions from the server, the device masks screen displays and audio output to prevent information leakage.

[0136] User roles:

[0137] Users receive feedback from the system while performing their normal tasks. In particular, if the system issues a notification as a result of sentiment analysis, users will follow the instructions and take the necessary actions. For example, if a suspicious sentiment pattern is detected, users may be asked to review their work and revise their behavior.

[0138] For example, if a user working from home is having a conversation about confidential documents and the system detects signs of anxiety in their voice, it will use this information to notify the security department and prompt them to take action. In this way, by referencing emotional states, it is possible to complementarily manage the risk of information leaks that are difficult to detect with conventional monitoring systems.

[0139] As described above, the present invention utilizes an emotion engine to realize a system that provides advanced analysis of information leakage risks and proactive countermeasures.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The device collects the user's voice and image data in real time. Here, it uses a microphone to acquire voice data and a camera to record facial expressions. This data is immediately prepared for transmission to the server.

[0143] Step 2:

[0144] The terminal encrypts the collected data to prevent eavesdropping and tampering, and sends it to the server using a secure protocol. The data arrives at the server instantly and awaits analysis.

[0145] Step 3:

[0146] The server analyzes the received audio data using a generative model. Here, it converts the data into text using automatic speech recognition and compares it against pre-configured dangerous keywords and phrases.

[0147] Step 4:

[0148] The server simultaneously analyzes the image data to check for prohibited items within the image. This process utilizes machine learning algorithms for object recognition and records the results.

[0149] Step 5:

[0150] The server activates an emotion engine to detect the user's emotional state from their voice. It analyzes changes in tone and volume to determine the user's emotional state. This information is added to the log as emotion analysis results.

[0151] Step 6:

[0152] The server further uses image data to analyze changes in the user's facial expressions and obtain supplementary information about their emotional state. It checks the movement of facial muscles and uses this information to support the results of the voice analysis.

[0153] Step 7:

[0154] The server integrates voice, image, and sentiment analysis data to assess potential data breach risks. If a risk is detected, it sends an alert to the security department to prompt a swift response.

[0155] Step 8:

[0156] The terminal receives masking instructions from the server and appropriately masks the display and audio output as needed. This protects confidential information from the user and their surroundings.

[0157] Step 9:

[0158] Users review security notifications and, based on the alert content, take necessary actions such as stopping or correcting information sharing. By following instructions and responding promptly, users can minimize the risk of information leakage.

[0159] (Example 2)

[0160] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0161] In high-security environments and remote work settings, preventing the leakage of user voice and image information is a major challenge. Conventional monitoring systems have struggled to accurately detect the risk of information leakage and identify potential risks arising from emotional shifts. Therefore, more advanced and consistent information leakage prevention measures are required.

[0162] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0163] In this invention, the server includes a central processing means that uses a generative model to analyze voice information and detect dangerous phrases, an image analysis means that analyzes image information and determines the presence of prohibited items, and a means that uses an emotion recognition engine to evaluate the user's emotional state from the voice and image information. This makes it possible to more accurately identify the risk of information leakage and to provide warnings and countermeasures that take into account the user's emotional changes.

[0164] A "high-security domain" refers to a physical or virtual environment where information security is of particular importance and strict access restrictions and protective measures are required.

[0165] A "remote work environment" refers to a network-based work environment that allows employees to perform their duties from home or other locations without having to go to the office.

[0166] "Audio information" refers to data, including human conversations and other sounds, that are collected and analyzed by machines.

[0167] "Image information" refers to visual data, including still images or videos, acquired by cameras or sensors.

[0168] A "generative model" is an algorithm that generates new data based on audio or text data and detects specific patterns or features.

[0169] A "central processing unit" is a computing device that serves as the central hub of an entire system, responsible for data aggregation, analysis, and the issuance of instructions.

[0170] "Image analysis equipment" refers to technologies and devices used to analyze image information and detect specific features or patterns.

[0171] An "emotion recognition engine" refers to algorithms and technologies that analyze audio and image information to identify and evaluate a person's emotional state.

[0172] A "warning" is a notification that informs administrators and users of potential risks detected by the system and prompts them to take action.

[0173] A "correction device" refers to a device or function that edits or partially deletes collected audio and image information as needed to prevent information leakage.

[0174] In terms of embodiments for carrying out the invention, this system consists of a monitoring device aimed at preventing information leakage in high-security areas and remote work environments. This system includes three main components: a server, a terminal, and a user.

[0175] The server functions as the core of the entire system, responsible for analyzing audio and image information. Specifically, the server uses a generative AI model to analyze audio information and detect dangerous keywords and phrases. This process utilizes automated speech recognition technology. The server also uses an image analysis device to analyze image information and determine if prohibited objects are present. In this process, machine learning algorithms are used for object recognition.

[0176] Furthermore, the server incorporates an emotion recognition engine that identifies the user's emotional state from audio and image information. The server analyzes the tone and emphasis of the voice, identifies facial expressions from image information, and evaluates changes in emotion. This analysis of emotional state is considered a potential factor in the risk of information leakage.

[0177] The device uses sensors, cameras, and microphones to collect data in real time and transmit it to the server. This collected data then serves as material for analysis on the server. Furthermore, the device prevents information leakage by following instructions from the server, such as masking the display screen or modifying audio output as needed.

[0178] Users receive feedback from the system while performing their normal tasks. In particular, if a warning is issued based on the results of sentiment analysis, users will follow the instructions and take the necessary actions. For example, if a suspicious emotional pattern is detected, users may be asked to review their work and revise their behavior.

[0179] For example, if a user working from home is having a conversation about confidential information and anxiety is detected in their voice, the server will notify the security department. In this way, by evaluating emotional states, it is possible to complementarily manage the risk of information leaks that are difficult to detect with conventional monitoring systems.

[0180] An example of a prompt for the generating AI model is: "Analyze voice data from someone working from home and suggest a response scenario if feelings of anxiety are detected."

[0181] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0182] Step 1:

[0183] The device collects audio and image data from its environment in real time. Audio data is input from the microphone, and image data from the camera. This collected data is transmitted to a server using a secure communication method. During this process, the device's sensors capture all audio and video from its surroundings.

[0184] Step 2:

[0185] The server analyzes the received audio data using a generating AI model. The input audio data includes parts of human conversation and other sounds. The server converts the audio to text using automatic speech recognition technology and extracts dangerous keywords and phrases. This process also analyzes changes in tone and emphasis in the speech to identify the user's emotional state. The output is the analysis results in text format and an emotional assessment.

[0186] Step 3:

[0187] The server analyzes the received image data using an image analysis device. The input image data includes people and objects in the environment. Machine learning algorithms are used to recognize faces and analyze facial expressions from the images. Object recognition functionality is also used to determine if there are any prohibited items. The output provides evaluation results regarding objects and facial expressions.

[0188] Step 4:

[0189] The server integrates the analysis results of audio and image data to evaluate the user's emotional state. This evaluation may flag emotional changes as a potential risk of information leakage. The input is evaluation information from audio and images, and the output is a determination of whether a warning is necessary.

[0190] Step 5:

[0191] The server generates warnings if necessary and creates prompts to notify the management department. It also instructs the terminal to take administrative actions, such as masking the screen display or correcting audio, where applicable. The output shows the administrative actions to be taken and the content of the notification.

[0192] Step 6:

[0193] The user receives warnings and instructions as feedback from the server. The user reviews the task details and takes the necessary actions based on the instructions. This process ensures that appropriate measures are taken against the risk of information leakage. The output is improved user behavior and strengthened security management.

[0194] (Application Example 2)

[0195] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0196] In today's highly secure areas and remote work environments, information leaks are a major concern. When handling information using voice and image data, conventional systems struggle to fully detect potential risks, and effective management of information leaks, particularly those related to emotional shifts, is essential.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0198] In this invention, the server includes means for analyzing audio and image information collected using a communication device, means for detecting dangerous words from audio information using a central processing unit with a generation format model, means for determining prohibited items using a visual analysis device, evaluation device means for evaluating the user's emotional state and warning of potential risks, and modification device means for changing the display and audio information after the warning. This makes it possible to comprehensively and effectively manage the potential risk of information leakage.

[0199] A "communication device" is a means of collecting voice and image information from a high-security environment or a remote work environment.

[0200] A "generative model" is an algorithm used to analyze audio information and detect potentially dangerous words or phrases.

[0201] The "Central Processing Unit" is the core processing unit used to analyze the collected audio information.

[0202] A "visual analysis device" is a device that analyzes image information to determine whether or not prohibited items are present.

[0203] An "evaluation device" is a device that analyzes a user's emotional state and warns the user if there is a potential risk of information leakage.

[0204] A "correction device" is a means of modifying some of the display and audio information after a warning has been issued.

[0205] The embodiment for carrying out this invention is a monitoring system mainly composed of three elements: a server, a terminal, and a user.

[0206] The server performs advanced data processing using communication equipment to analyze audio and image information. The server has a central processing unit equipped with a generative model that analyzes audio information and detects potentially dangerous words. Furthermore, a visual analysis device analyzes image information to determine the presence of prohibited items. The server uses an evaluation device to assess the user's emotional state and identify and warn of potential risks of information leakage.

[0207] The terminal transmits collected audio and image information to the server. The terminal is equipped with a microphone and camera suitable for the environment, enabling it to collect data in real time and transmit it to the server. If a warning is issued, the terminal uses a correction device to modify the display and some of the audio.

[0208] Users receive real-time feedback from this system as they perform their daily tasks. For example, when handling confidential data while working from home, if a sudden change in voice tone is detected, the system performs sentiment analysis and warns of a potential risk of information leakage. This warning allows users to review their work and take necessary measures.

[0209] For example, if a user is having an important meeting while working from home, and the system detects feelings of anxiety or impatience from the audio data, the system will immediately issue a warning: "We have detected an emotional shift that requires attention. Please review the details and take necessary actions." This is achieved through the analysis of audio and image data using a generative AI model.

[0210] An example of a prompt for a generative AI model would be: "Analyze user emotions from audio and image data and identify information leakage risks. Imagine a system that immediately notifies the user if a risk is detected."

[0211] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0212] Step 1:

[0213] The device collects audio and image data from the environment. It uses its microphone and camera as input to acquire data in real time and sends it to the server. The output is audio and image data for analysis by the server. Specifically, the device continuously captures data at regular intervals.

[0214] Step 2:

[0215] The server analyzes received audio data using a generation AI model. The input is audio data sent from a terminal, and data processing is performed to detect dangerous words and emotional tones. The output is an evaluation of dangerous words or inappropriate emotional states. Specifically, the server performs language processing using a speech recognition algorithm and evaluates tone changes with an emotion analysis engine.

[0216] Step 3:

[0217] The server analyzes the received image data using a visual analysis device. Using image data transmitted from the terminal as input, it performs data calculations to recognize prohibited items. The output is a judgment result regarding the presence or absence of the item. Specifically, the server uses a machine learning algorithm to analyze elements within the image through object recognition.

[0218] Step 4:

[0219] The server performs a comprehensive evaluation based on the results of audio and image analysis, and if a potential data leakage risk is detected, it issues a warning to the user. The input uses the analysis results from audio and images to identify risks. The output is a warning message to the user. Specifically, the server provides feedback on the evaluation results and generates notifications in real time.

[0220] Step 5:

[0221] The user receives a warning from the server, reviews the work content, and takes necessary actions. The input is the warning message received from the server. The output is the action taken to avoid risk. Specifically, the user reviews work procedures based on the notification content and performs safety checks.

[0222] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0223] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0224] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0225] [Second Embodiment]

[0226] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0227] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0228] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0229] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0230] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0232] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0233] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0234] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0235] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0236] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0237] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0238] One embodiment of this invention is configured as an advanced monitoring system for preventing information leaks in high-security areas and home-work environments. This system consists of three main components: a server, a terminal, and a user, each working together to enhance information security.

[0239] Server role:

[0240] The server functions as the central hub of the system, analyzing audio and image data collected from terminals. Using generative models, the server converts audio data into text and detects designated dangerous keywords or phrases. For image data, an image analysis device performs object recognition to determine if prohibited items are present. For example, if a prohibited smartphone is visible in the video within a secure area, the server will identify its presence.

[0241] Furthermore, the server uses an evaluation device to assess the potential risk of information leakage based on the analysis results, and if a risk is detected, it sends an alert to the management department. This notification is provided via email or dashboard alert, enabling a quick response.

[0242] Terminal role:

[0243] The terminal operates on the user's side and performs information collection and masking as needed. The terminal collects data in real time through the microphone and camera and sends it to the server. Furthermore, if the server detects a risk, it will mask specific screen content or audio output to conceal confidential information. For example, if sensitive information is displayed on the screen, it will be blurred to prevent unnecessary disclosure of information to third parties.

[0244] User roles:

[0245] Users perform their normal work operations, but if they receive a security notification, they will stop or correct information sharing based on the displayed instructions. Users can also receive guidance to improve their security awareness, contributing to the protection of information in their daily work.

[0246] In this way, it is possible to build a system that minimizes the risk of information leakage through the interaction of servers, terminals, and users. As a specific example, one company is using this system to monitor audio during meetings and take measures to prevent confidential project information from being leaked to the outside. In this manner, the present invention dramatically improves information security in corporate activities.

[0247] The following describes the processing flow.

[0248] Step 1:

[0249] The device collects audio and image data in real time in high-security areas or work-from-home environments. It uses a microphone and camera to capture this data, converts it to an appropriate format, and prepares it for transmission to a server.

[0250] Step 2:

[0251] The terminal encrypts the collected data and sends it to the server. To prevent interception and tampering of communications, the SSL / TLS protocol is used for secure data transfer.

[0252] Step 3:

[0253] The server analyzes the audio data received from the terminal using a generative model. It utilizes automatic speech recognition (ASR) technology to generate text from the audio data and detect dangerous keywords and phrases.

[0254] Step 4:

[0255] The server analyzes the image data using an image analysis algorithm. Here, a machine learning model is used to check for the presence of prohibited items (e.g., smartphones), and if detected, the information is recorded.

[0256] Step 5:

[0257] The server evaluates the results of the analysis of audio and image data to identify potential data breach risks. If a risk is detected, it sends an alert to the security department to quickly notify the relevant personnel.

[0258] Step 6:

[0259] The terminal receives instructions from the server and, if necessary, masks some of the information on the screen and some of the audio. This process physically hides sensitive data from the user's display and output audio to prevent unintended information leaks.

[0260] Step 7:

[0261] When users receive a security notification, they will manually correct information or cease certain actions. Based on the notified measures, they will strive to mitigate the risk of information leakage by promptly taking action such as postponing or canceling information sharing.

[0262] (Example 1)

[0263] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0264] Information leaks pose a significant risk to companies and organizations, and mitigating this risk is crucial, especially in highly confidential areas and remote work environments. Traditional methods suffer from insufficient monitoring and analysis to prevent information leaks, and struggle to immediately detect specific languages ​​or items. There is a need for systems that can improve this situation and significantly reduce the risk of information leaks.

[0265] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0266] In this invention, the server includes a device means for collecting information from a highly confidential area or residential work environment, a central control device means that analyzes the collected audio information and uses a generation model to detect dangerous words, and an automatic analysis device means that analyzes the collected image information and determines the presence of prohibited items. This makes it possible to efficiently detect specific audio and visual hazard elements and take immediate countermeasures to reduce the risk of information leakage.

[0267] 1. A "highly confidential area" refers to an environment with a high risk of information leakage, and is a place where particularly protected information or activities are conducted.

[0268] 2. "Residential work environment" refers to an environment in which an individual performs work from their home or other location, and is a place where special monitoring is required for information protection.

[0269] 3. "Information-collecting device" refers to a device or system for collecting data such as voice and images in real time and transmitting it to the next processing stage.

[0270] 4. A "generative model" is a machine learning model used to analyze collected audio data and detect specified dangerous keywords or phrases.

[0271] 5. A "central control unit" is a central processing unit that oversees and manages the entire system and performs data analysis and evaluation.

[0272] 6. An "automated analysis device" is a device that analyzes collected image data using machine learning technology to detect specific items or scenarios.

[0273] 7. An "evaluation tool" is a mechanism for assessing the risk of information leakage and notifying the management department of warnings or corrective actions based on that assessment.

[0274] 8. A "regulation device" is a device used to control or conceal parts of the screen display or sound in order to prevent information leakage.

[0275] This invention is an advanced monitoring system aimed at preventing information leaks in highly confidential areas and remote work environments. The system primarily consists of three elements: a server, a terminal, and a user, each working together to enhance information security.

[0276] The server acts as the core of the system, analyzing audio and image information collected from terminals. The server uses a generative AI model to convert audio information into text, monitoring the information by detecting specific potentially dangerous words. Furthermore, it uses an automated analysis device to perform object recognition on image data to check for prohibited items. For example, one prohibited item might be the use of a smartphone within a specific area. Based on the analysis results, the server assesses the potential risk of information leakage and sends warnings to the management department as needed.

[0277] The terminal collects information in the user's environment and performs masking if necessary. Audio and image information is collected in real time through the terminal's microphone and camera. Based on instructions from the server, the terminal blurs specific content displayed on the monitor to prevent the leakage of confidential information.

[0278] Users receive security notifications while performing their normal duties. Following warnings from the server, users can stop or correct information sharing. Furthermore, users can enhance their awareness of information protection by receiving guidance to improve their security awareness.

[0279] As a concrete example, one company uses this system to monitor audio during meetings and prevent the leakage of information related to highly confidential projects. An example of a prompt message is the instruction, "Translate audio in the security area into text and detect the specified dangerous phrase." In this way, the present invention contributes to strengthening information security.

[0280] The flow of the specific process in Example 1 will be described with reference to FIG. 11.

[0281] Step 1:

[0282] The terminal collects voice and image data and transmits it to the server. Specifically, the microphone and camera of the terminal record audio and video in real time and transmit the data to the server. In this process, voice and image data are provided as input and transferred to the server in the same format.

[0283] Step 2:

[0284] The server converts the received voice data into text using a generative AI model. By receiving the voice data as input and performing automatic speech recognition, text data is generated. The converted text is used in the next analysis step.

[0285] Step 3:

[0286] The server analyzes the text data to detect whether it contains dangerous keywords or phrases. By using a generative AI model to evaluate the specified language patterns, phrases with risks are identified. The output of this step is information on whether a danger has been detected.

[0287] Step 4:

[0288] The server processes the image data with an automatic analysis device to perform object recognition. By analyzing the received image, the possibility of the presence of specific articles or prohibited objects is evaluated. As output, information on whether dangerous articles have been detected is obtained.

[0289] Step 5:

[0290] The server evaluates the risk of information leakage based on the analysis results of the voice and image data and sends a warning to the management department. If a risk is detected, the server outputs an alert to the administrator in the form of an email or dashboard notification.

[0291] Step 6:

[0292] The terminal receives a warning instruction from the server and masks the screen display and audio output. It receives warning information provided by the server as input and performs controls such as mosaic processing to prevent the leakage of confidential information. The output of this step is a masked display or output.

[0293] Step 7:

[0294] Users receive security notifications and take necessary actions. The system receives notifications from terminals and servers as input, and takes necessary measures such as stopping or correcting information sharing. The output is a work environment with improvements or corrective actions implemented.

[0295] (Application Example 1)

[0296] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0297] In high-security areas and remote work environments, a lack of effective monitoring and control measures to prevent information leaks is a challenge. In particular, there is a need to analyze voice and image data in real time to quickly detect and address potential risks, but conventional systems are unable to adequately achieve this.

[0298] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0299] In this invention, the server includes device means for collecting voice and image data from a high-security area or a telecommuting environment, central processing unit means using a generative model to analyze the collected voice data and detect dangerous keywords, and evaluation device means for evaluating potential information leakage and notifying a warning to a specific management department. Thereby, it becomes possible to suppress information leakage in real time and respond quickly to risk management.

[0300] The "high-security area" is an area where strict security management is required because of the high risk of information leakage.

[0301] The "telecommuting environment" is a working environment used with the home as a base when performing business.

[0302] The "device for collecting voice and image data" is a device for acquiring voice and images using a microphone and a camera.

[0303] The "central processing unit using a generative model" is a device equipped with generative AI technology for analyzing voice data and detecting specified dangerous keywords.

[0304] The "image analysis device" is a device that uses a machine learning algorithm to analyze image data and determine the presence or absence of an object.

[0305] The "evaluation device" is a device for evaluating potential information leakage risks and issuing warnings as necessary.

[0306] The "modification device" is a device for masking a part of screen and voice data to block confidential information after a warning.

[0307] The "blocking device" is a device having a function of blocking specific information to prevent unauthorized information access.

[0308] This system is designed to prevent information leaks in high-security areas or remote work environments. It consists of three main components: servers, terminals, and users.

[0309] The server functions as a central hub for analyzing audio and image data. Audio data is collected from the device via a microphone and transferred to the server. There, it is converted to text using automatic speech recognition technology such as the Google Speech-to-Text API. A generative AI model is used to detect specified dangerous keywords from this text data. Image data is transmitted from the device via a camera and analyzed using image processing libraries such as OpenCV. Machine learning algorithms are used to determine, for example, the presence or absence of prohibited items.

[0310] Based on these analysis results, the server assesses the potential risk of data breaches and generates an alert via the assessment device. This alert is then sent to the management department via email or a dashboard alert.

[0311] If a warning is issued, the terminal controls the display and output of data, and a correction device masks certain information. This includes processes to prevent the leakage of confidential information, such as muting audio and blurring text and images on the screen.

[0312] Users will perform their normal duties, but when they receive a warning, they will restrict or modify data disclosure based on the instructions given. Furthermore, users will receive guidance to raise their awareness of information security, thereby strengthening information protection in their daily work.

[0313] As a concrete example, if this system is implemented in a certain corporate environment and a project code name is mentioned during a meeting, the server will immediately detect the risk and alert the participants, thereby preventing information leakage to external parties.

[0314] Examples of prompt statements include the following:

[0315] "We analyzed audio data from corporate meetings and, if the keyword 'project code' was found, we issued a warning notification based on that information. Because there is a risk of information leakage if this project code is disclosed externally, we urge participants who receive the notification to handle it with caution."

[0316] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0317] Step 1:

[0318] The device collects audio and image data from the environment in real time. This collected data is acquired via a microphone and camera and prepared for transmission to a server. The input is raw audio and image data, and the output is transmission data transferred to the server.

[0319] Step 2:

[0320] The server converts received audio data from speech to text using the Google Speech-to-Text API. It then utilizes a generative AI model to analyze the text data and detect specified dangerous keywords. The input is audio data, and the output is text data along with the detected keywords. The analysis lists specific words and phrases, and a risk assessment is performed based on this list.

[0321] Step 3:

[0322] The server analyzes image data using OpenCV and performs object recognition using machine learning algorithms. It determines whether specific prohibited items are present in the image. The input to this process is image data, and the output is the object recognition result. If a prohibited item is identified through analysis, the warning process is immediately initiated.

[0323] Step 4:

[0324] The server uses an evaluation device to assess the risk of information leakage based on the analysis results. If a risk is detected, it creates a notification to send an alert to the management department. The input is the analyzed keywords and item data, and the output is an alert notification. The notification is sent via email or a dashboard, enabling a quick response.

[0325] Step 5:

[0326] The terminal controls the screen or audio and masks certain information according to instructions received from the server. The input is the control instructions from the server, and the output is the display of masked data. This process ensures that measures are taken to prevent sensitive information from being leaked to third parties.

[0327] Step 6:

[0328] When a user receives a warning notification from their device, they take the necessary action. The input is the warning notification from the device, and the output is the user's response to the risk. The user will stop publishing data or perform necessary corrective actions according to this notification.

[0329] Each step works closely together to minimize the risk of data breaches and help build a highly secure environment.

[0330] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0331] One embodiment of this invention is a monitoring system aimed at preventing information leaks in high-security areas and teleworking environments, which achieves even more advanced security management by combining it with an emotion engine. This system consists of three main components: a server, a terminal, and a user.

[0332] Server role:

[0333] The server functions as the core of the entire system, analyzing audio and image data transmitted from terminals. The server uses generative models to analyze audio data and identify dangerous keywords and phrases. Furthermore, it uses image analysis equipment to check for the presence of prohibited items in image data.

[0334] The newly integrated emotion engine works to recognize the user's emotional state from voice data. In this process, the server analyzes changes in voice tone and emphasis to identify emotions such as joy, anger, and sadness. Simultaneously, it analyzes the user's facial expressions from image data to provide information that complements the emotional state. These emotional changes are considered as potential security risks.

[0335] Terminal role:

[0336] The device uses sensors, cameras, and microphones to collect data in real time and transmit it to a server. This data provides material for analysis by an emotion engine. Furthermore, based on instructions from the server, the device masks screen displays and audio output to prevent information leakage.

[0337] User roles:

[0338] Users receive feedback from the system while performing their normal tasks. In particular, if the system issues a notification as a result of sentiment analysis, users will follow the instructions and take the necessary actions. For example, if a suspicious sentiment pattern is detected, users may be asked to review their work and revise their behavior.

[0339] For example, if a user working from home is having a conversation about confidential documents and the system detects signs of anxiety in their voice, it will use this information to notify the security department and prompt them to take action. In this way, by referencing emotional states, it is possible to complementarily manage the risk of information leaks that are difficult to detect with conventional monitoring systems.

[0340] As described above, the present invention utilizes an emotion engine to realize a system that provides advanced analysis of information leakage risks and proactive countermeasures.

[0341] The following describes the processing flow.

[0342] Step 1:

[0343] The device collects the user's voice and image data in real time. Here, it uses a microphone to acquire voice data and a camera to record facial expressions. This data is immediately prepared for transmission to the server.

[0344] Step 2:

[0345] The terminal encrypts the collected data to prevent eavesdropping and tampering, and sends it to the server using a secure protocol. The data arrives at the server instantly and awaits analysis.

[0346] Step 3:

[0347] The server analyzes the received audio data using a generative model. Here, it converts the data into text using automatic speech recognition and compares it against pre-configured dangerous keywords and phrases.

[0348] Step 4:

[0349] The server simultaneously analyzes the image data to check for prohibited items within the image. This process utilizes machine learning algorithms for object recognition and records the results.

[0350] Step 5:

[0351] The server activates an emotion engine to detect the user's emotional state from their voice. It analyzes changes in tone and volume to determine the user's emotional state. This information is added to the log as emotion analysis results.

[0352] Step 6:

[0353] The server further uses image data to analyze changes in the user's facial expressions and obtain supplementary information about their emotional state. It checks the movement of facial muscles and uses this information to support the results of the voice analysis.

[0354] Step 7:

[0355] The server integrates voice, image, and sentiment analysis data to assess potential data breach risks. If a risk is detected, it sends an alert to the security department to prompt a swift response.

[0356] Step 8:

[0357] The terminal receives masking instructions from the server and appropriately masks the display and audio output as needed. This protects confidential information from the user and their surroundings.

[0358] Step 9:

[0359] Users review security notifications and, based on the alert content, take necessary actions such as stopping or correcting information sharing. By following instructions and responding promptly, users can minimize the risk of information leakage.

[0360] (Example 2)

[0361] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0362] In high-security environments and remote work settings, preventing the leakage of user voice and image information is a major challenge. Conventional monitoring systems have struggled to accurately detect the risk of information leakage and identify potential risks arising from emotional shifts. Therefore, more advanced and consistent information leakage prevention measures are required.

[0363] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0364] In this invention, the server includes a central processing means that uses a generative model to analyze voice information and detect dangerous phrases, an image analysis means that analyzes image information and determines the presence of prohibited items, and a means that uses an emotion recognition engine to evaluate the user's emotional state from the voice and image information. This makes it possible to more accurately identify the risk of information leakage and to provide warnings and countermeasures that take into account the user's emotional changes.

[0365] A "high-security domain" refers to a physical or virtual environment where information security is of particular importance and strict access restrictions and protective measures are required.

[0366] A "remote work environment" refers to a network-based work environment that allows employees to perform their duties from home or other locations without having to go to the office.

[0367] "Audio information" refers to data, including human conversations and other sounds, that are collected and analyzed by machines.

[0368] "Image information" refers to visual data, including still images or videos, acquired by cameras or sensors.

[0369] A "generative model" is an algorithm that generates new data based on audio or text data and detects specific patterns or features.

[0370] A "central processing unit" is a computing device that serves as the central hub of an entire system, responsible for data aggregation, analysis, and the issuance of instructions.

[0371] "Image analysis equipment" refers to technologies and devices used to analyze image information and detect specific features or patterns.

[0372] An "emotion recognition engine" refers to algorithms and technologies that analyze audio and image information to identify and evaluate a person's emotional state.

[0373] A "warning" is a notification that informs administrators and users of potential risks detected by the system and prompts them to take action.

[0374] A "correction device" refers to a device or function that edits or partially deletes collected audio and image information as needed to prevent information leakage.

[0375] In terms of embodiments for carrying out the invention, this system consists of a monitoring device aimed at preventing information leakage in high-security areas and remote work environments. This system includes three main components: a server, a terminal, and a user.

[0376] The server functions as the core of the entire system, responsible for analyzing audio and image information. Specifically, the server uses a generative AI model to analyze audio information and detect dangerous keywords and phrases. This process utilizes automated speech recognition technology. The server also uses an image analysis device to analyze image information and determine if prohibited objects are present. In this process, machine learning algorithms are used for object recognition.

[0377] Furthermore, the server incorporates an emotion recognition engine that identifies the user's emotional state from audio and image information. The server analyzes the tone and emphasis of the voice, identifies facial expressions from image information, and evaluates changes in emotion. This analysis of emotional state is considered a potential factor in the risk of information leakage.

[0378] The device uses sensors, cameras, and microphones to collect data in real time and transmit it to the server. This collected data then serves as material for analysis on the server. Furthermore, the device prevents information leakage by following instructions from the server, such as masking the display screen or modifying audio output as needed.

[0379] Users receive feedback from the system while performing their normal tasks. In particular, if a warning is issued based on the results of sentiment analysis, users will follow the instructions and take the necessary actions. For example, if a suspicious emotional pattern is detected, users may be asked to review their work and revise their behavior.

[0380] For example, if a user working from home is having a conversation about confidential information and anxiety is detected in their voice, the server will notify the security department. In this way, by evaluating emotional states, it is possible to complementarily manage the risk of information leaks that are difficult to detect with conventional monitoring systems.

[0381] An example of a prompt for the generating AI model is: "Analyze voice data from someone working from home and suggest a response scenario if feelings of anxiety are detected."

[0382] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0383] Step 1:

[0384] The device collects audio and image data from its environment in real time. Audio data is input from the microphone, and image data from the camera. This collected data is transmitted to a server using a secure communication method. During this process, the device's sensors capture all audio and video from its surroundings.

[0385] Step 2:

[0386] The server analyzes the received audio data using a generating AI model. The input audio data includes parts of human conversation and other sounds. The server converts the audio to text using automatic speech recognition technology and extracts dangerous keywords and phrases. This process also analyzes changes in tone and emphasis in the speech to identify the user's emotional state. The output is the analysis results in text format and an emotional assessment.

[0387] Step 3:

[0388] The server analyzes the received image data using an image analysis device. The input image data includes people and objects in the environment. Machine learning algorithms are used to recognize faces and analyze facial expressions from the images. Object recognition functionality is also used to determine if there are any prohibited items. The output provides evaluation results regarding objects and facial expressions.

[0389] Step 4:

[0390] The server integrates the analysis results of audio and image data to evaluate the user's emotional state. This evaluation may flag emotional changes as a potential risk of information leakage. The input is evaluation information from audio and images, and the output is a determination of whether a warning is necessary.

[0391] Step 5:

[0392] The server generates warnings if necessary and creates prompts to notify the management department. It also instructs the terminal to take administrative actions, such as masking the screen display or correcting audio, where applicable. The output shows the administrative actions to be taken and the content of the notification.

[0393] Step 6:

[0394] The user receives warnings and instructions as feedback from the server. The user reviews the task details and takes the necessary actions based on the instructions. This process ensures that appropriate measures are taken against the risk of information leakage. The output is improved user behavior and strengthened security management.

[0395] (Application Example 2)

[0396] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0397] In today's highly secure areas and remote work environments, information leaks are a major concern. When handling information using voice and image data, conventional systems struggle to fully detect potential risks, and effective management of information leaks, particularly those related to emotional shifts, is essential.

[0398] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0399] In this invention, the server includes means for analyzing audio and image information collected using a communication device, means for detecting dangerous words from audio information using a central processing unit with a generation format model, means for determining prohibited items using a visual analysis device, evaluation device means for evaluating the user's emotional state and warning of potential risks, and modification device means for changing the display and audio information after the warning. This makes it possible to comprehensively and effectively manage the potential risk of information leakage.

[0400] A "communication device" is a means of collecting voice and image information from a high-security environment or a remote work environment.

[0401] A "generative model" is an algorithm used to analyze audio information and detect potentially dangerous words or phrases.

[0402] The "Central Processing Unit" is the core processing unit used to analyze the collected audio information.

[0403] A "visual analysis device" is a device that analyzes image information to determine whether or not prohibited items are present.

[0404] An "evaluation device" is a device that analyzes a user's emotional state and warns the user if there is a potential risk of information leakage.

[0405] A "correction device" is a means of modifying some of the display and audio information after a warning has been issued.

[0406] The embodiment for carrying out this invention is a monitoring system mainly composed of three elements: a server, a terminal, and a user.

[0407] The server performs advanced data processing using communication equipment to analyze audio and image information. The server has a central processing unit equipped with a generative model that analyzes audio information and detects potentially dangerous words. Furthermore, a visual analysis device analyzes image information to determine the presence of prohibited items. The server uses an evaluation device to assess the user's emotional state and identify and warn of potential risks of information leakage.

[0408] The terminal transmits collected audio and image information to the server. The terminal is equipped with a microphone and camera suitable for the environment, enabling it to collect data in real time and transmit it to the server. If a warning is issued, the terminal uses a correction device to modify the display and some of the audio.

[0409] Users receive real-time feedback from this system as they perform their daily tasks. For example, when handling confidential data while working from home, if a sudden change in voice tone is detected, the system performs sentiment analysis and warns of a potential risk of information leakage. This warning allows users to review their work and take necessary measures.

[0410] For example, if a user is having an important meeting while working from home, and the system detects feelings of anxiety or impatience from the audio data, the system will immediately issue a warning: "We have detected an emotional shift that requires attention. Please review the details and take necessary actions." This is achieved through the analysis of audio and image data using a generative AI model.

[0411] An example of a prompt for a generative AI model would be: "Analyze user emotions from audio and image data and identify information leakage risks. Imagine a system that immediately notifies the user if a risk is detected."

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The device collects audio and image data from the environment. It uses its microphone and camera as input to acquire data in real time and sends it to the server. The output is audio and image data for analysis by the server. Specifically, the device continuously captures data at regular intervals.

[0415] Step 2:

[0416] The server analyzes received audio data using a generation AI model. The input is audio data sent from a terminal, and data processing is performed to detect dangerous words and emotional tones. The output is an evaluation of dangerous words or inappropriate emotional states. Specifically, the server performs language processing using a speech recognition algorithm and evaluates tone changes with an emotion analysis engine.

[0417] Step 3:

[0418] The server analyzes the received image data using a visual analysis device. Using image data transmitted from the terminal as input, it performs data calculations to recognize prohibited items. The output is a judgment result regarding the presence or absence of the item. Specifically, the server uses a machine learning algorithm to analyze elements within the image through object recognition.

[0419] Step 4:

[0420] The server performs a comprehensive evaluation based on the results of audio and image analysis, and if a potential data leakage risk is detected, it issues a warning to the user. The input uses the analysis results from audio and images to identify risks. The output is a warning message to the user. Specifically, the server provides feedback on the evaluation results and generates notifications in real time.

[0421] Step 5:

[0422] The user receives a warning from the server, reviews the work content, and takes necessary actions. The input is the warning message received from the server. The output is the action taken to avoid risk. Specifically, the user reviews work procedures based on the notification content and performs safety checks.

[0423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0424] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0425] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0426] [Third Embodiment]

[0427] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0428] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0429] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0430] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0431] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0432] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0433] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0434] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0435] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0436] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0437] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0438] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0439] One embodiment of this invention is configured as an advanced monitoring system for preventing information leaks in high-security areas and home-work environments. This system consists of three main components: a server, a terminal, and a user, each working together to enhance information security.

[0440] Server role:

[0441] The server functions as the central hub of the system, analyzing audio and image data collected from terminals. Using generative models, the server converts audio data into text and detects designated dangerous keywords or phrases. For image data, an image analysis device performs object recognition to determine if prohibited items are present. For example, if a prohibited smartphone is visible in the video within a secure area, the server will identify its presence.

[0442] Furthermore, the server uses an evaluation device to assess the potential risk of information leakage based on the analysis results, and if a risk is detected, it sends an alert to the management department. This notification is provided via email or dashboard alert, enabling a quick response.

[0443] Terminal role:

[0444] The terminal operates on the user's side and performs information collection and masking as needed. The terminal collects data in real time through the microphone and camera and sends it to the server. Furthermore, if the server detects a risk, it will mask specific screen content or audio output to conceal confidential information. For example, if sensitive information is displayed on the screen, it will be blurred to prevent unnecessary disclosure of information to third parties.

[0445] User roles:

[0446] Users perform their normal work operations, but if they receive a security notification, they will stop or correct information sharing based on the displayed instructions. Users can also receive guidance to improve their security awareness, contributing to the protection of information in their daily work.

[0447] In this way, it is possible to build a system that minimizes the risk of information leakage through the interaction of servers, terminals, and users. As a specific example, one company is using this system to monitor audio during meetings and take measures to prevent confidential project information from being leaked to the outside. In this manner, the present invention dramatically improves information security in corporate activities.

[0448] The following describes the processing flow.

[0449] Step 1:

[0450] The device collects audio and image data in real time in high-security areas or work-from-home environments. It uses a microphone and camera to capture this data, converts it to an appropriate format, and prepares it for transmission to a server.

[0451] Step 2:

[0452] The terminal encrypts the collected data and sends it to the server. To prevent interception and tampering of communications, the SSL / TLS protocol is used for secure data transfer.

[0453] Step 3:

[0454] The server analyzes the audio data received from the terminal using a generative model. It utilizes automatic speech recognition (ASR) technology to generate text from the audio data and detect dangerous keywords and phrases.

[0455] Step 4:

[0456] The server analyzes the image data using an image analysis algorithm. Here, a machine learning model is used to check for the presence of prohibited items (e.g., smartphones), and if detected, the information is recorded.

[0457] Step 5:

[0458] The server evaluates the results of the analysis of audio and image data to identify potential data breach risks. If a risk is detected, it sends an alert to the security department to quickly notify the relevant personnel.

[0459] Step 6:

[0460] The terminal receives instructions from the server and, if necessary, masks some of the information on the screen and some of the audio. This process physically hides sensitive data from the user's display and output audio to prevent unintended information leaks.

[0461] Step 7:

[0462] When users receive a security notification, they will manually correct information or cease certain actions. Based on the notified measures, they will strive to mitigate the risk of information leakage by promptly taking action such as postponing or canceling information sharing.

[0463] (Example 1)

[0464] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0465] Information leaks pose a significant risk to companies and organizations, and mitigating this risk is crucial, especially in highly confidential areas and remote work environments. Traditional methods suffer from insufficient monitoring and analysis to prevent information leaks, and struggle to immediately detect specific languages ​​or items. There is a need for systems that can improve this situation and significantly reduce the risk of information leaks.

[0466] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0467] In this invention, the server includes a device means for collecting information from a highly confidential area or residential work environment, a central control device means that analyzes the collected audio information and uses a generation model to detect dangerous words, and an automatic analysis device means that analyzes the collected image information and determines the presence of prohibited items. This makes it possible to efficiently detect specific audio and visual hazard elements and take immediate countermeasures to reduce the risk of information leakage.

[0468] 1. A "highly confidential area" refers to an environment with a high risk of information leakage, and is a place where particularly protected information or activities are conducted.

[0469] 2. "Residential work environment" refers to an environment in which an individual performs work from their home or other location, and is a place where special monitoring is required for information protection.

[0470] 3. "Information-collecting device" refers to a device or system for collecting data such as voice and images in real time and transmitting it to the next processing stage.

[0471] 4. A "generative model" is a machine learning model used to analyze collected audio data and detect specified dangerous keywords or phrases.

[0472] 5. A "central control unit" is a central processing unit that oversees and manages the entire system and performs data analysis and evaluation.

[0473] 6. An "automated analysis device" is a device that analyzes collected image data using machine learning technology to detect specific items or scenarios.

[0474] 7. An "evaluation tool" is a mechanism for assessing the risk of information leakage and notifying the management department of warnings or corrective actions based on that assessment.

[0475] 8. A "regulation device" is a device used to control or conceal parts of the screen display or sound in order to prevent information leakage.

[0476] This invention is an advanced monitoring system aimed at preventing information leaks in highly confidential areas and remote work environments. The system primarily consists of three elements: a server, a terminal, and a user, each working together to enhance information security.

[0477] The server acts as the core of the system, analyzing audio and image information collected from terminals. The server uses a generative AI model to convert audio information into text, monitoring the information by detecting specific potentially dangerous words. Furthermore, it uses an automated analysis device to perform object recognition on image data to check for prohibited items. For example, one prohibited item might be the use of a smartphone within a specific area. Based on the analysis results, the server assesses the potential risk of information leakage and sends warnings to the management department as needed.

[0478] The terminal collects information in the user's environment and performs masking if necessary. Audio and image information is collected in real time through the terminal's microphone and camera. Based on instructions from the server, the terminal blurs specific content displayed on the monitor to prevent the leakage of confidential information.

[0479] Users receive security notifications while performing their normal duties. Following warnings from the server, users can stop or correct information sharing. Furthermore, users can enhance their awareness of information protection by receiving guidance to improve their security awareness.

[0480] As a concrete example, one company uses this system to monitor audio during meetings and prevent the leakage of information related to highly confidential projects. An example of a prompt message is the instruction, "Translate audio in the security area into text and detect the specified dangerous phrase." In this way, the present invention contributes to strengthening information security.

[0481] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0482] Step 1:

[0483] The device collects audio and image data and sends it to the server. Specifically, the device's microphone and camera record audio and video in real time and send that data to the server. In this process, audio and image data are provided as input and transferred to the server in their original format.

[0484] Step 2:

[0485] The server converts the received audio data into text using a generation AI model. The system receives audio data as input and generates text data through automatic speech recognition. The converted text is then used in the next analysis step.

[0486] Step 3:

[0487] The server analyzes the text data to detect whether it contains dangerous keywords or phrases. Using a generative AI model, it identifies risky phrases by evaluating specified language patterns. The output of this step is information on whether danger was detected.

[0488] Step 4:

[0489] The server processes image data using an automated analysis system to perform object recognition. It analyzes the received images to assess the likelihood of specific items or prohibited objects being present. The output provides information on whether hazardous materials were detected.

[0490] Step 5:

[0491] The server assesses the risk of data leakage based on the analysis results of audio and image data and sends a warning to the management department. If a risk is detected, the server sends an alert to the administrator in the form of an email or dashboard notification.

[0492] Step 6:

[0493] The terminal receives a warning instruction from the server and masks the screen display and audio output. It receives warning information provided by the server as input and performs controls such as mosaic processing to prevent the leakage of confidential information. The output of this step is a masked display or output.

[0494] Step 7:

[0495] Users receive security notifications and take necessary actions. The system receives notifications from terminals and servers as input, and takes necessary measures such as stopping or correcting information sharing. The output is a work environment with improvements or corrective actions implemented.

[0496] (Application Example 1)

[0497] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0498] In high-security areas and remote work environments, a lack of effective monitoring and control measures to prevent information leaks is a challenge. In particular, there is a need to analyze voice and image data in real time to quickly detect and address potential risks, but conventional systems are unable to adequately achieve this.

[0499] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0500] In this invention, the server includes a device means for collecting voice and image data from a high-security area or a work-from-home environment; a central processing unit means that analyzes the collected voice data and uses a generative model to detect dangerous keywords; and an evaluation device means that evaluates potential information leaks and notifies a specific management department of the warning. This enables real-time information leak prevention and rapid risk management response.

[0501] A "high-security area" is a region where the risk of information leakage is high, and therefore strict security management is required.

[0502] A "work-from-home environment" refers to a work environment where the home is used as the base of operations when performing work.

[0503] A "device for collecting audio and image data" is a device that uses a microphone and a camera to acquire audio and images.

[0504] A "central processing unit using a generative model" is a device equipped with generative AI technology to analyze audio data and detect specified dangerous keywords.

[0505] An "image analysis device" is a device that uses machine learning algorithms to analyze image data and determine the presence or absence of an object.

[0506] An "evaluation device" is a device that assesses potential information leakage risks and issues warnings as necessary.

[0507] A "correction device" is a device that, after a warning, masks parts of the screen and audio data to block confidential information.

[0508] A "blocking device" is a device that has the function of blocking specific information in order to prevent unauthorized access to that information.

[0509] This system is designed to prevent information leaks in high-security areas or remote work environments. It consists of three main components: servers, terminals, and users.

[0510] The server functions as a central hub for analyzing audio and image data. Audio data is collected from the device via a microphone and transferred to the server. There, it is converted to text using automatic speech recognition technology such as the Google Speech-to-Text API. A generative AI model is used to detect specified dangerous keywords from this text data. Image data is transmitted from the device via a camera and analyzed using image processing libraries such as OpenCV. Machine learning algorithms are used to determine, for example, the presence or absence of prohibited items.

[0511] Based on these analysis results, the server assesses the potential risk of data breaches and generates an alert via the assessment device. This alert is then sent to the management department via email or a dashboard alert.

[0512] If a warning is issued, the terminal controls the display and output of data, and a correction device masks certain information. This includes processes to prevent the leakage of confidential information, such as muting audio and blurring text and images on the screen.

[0513] Users will perform their normal duties, but when they receive a warning, they will restrict or modify data disclosure based on the instructions given. Furthermore, users will receive guidance to raise their awareness of information security, thereby strengthening information protection in their daily work.

[0514] As a concrete example, if this system is implemented in a certain corporate environment and a project code name is mentioned during a meeting, the server will immediately detect the risk and alert the participants, thereby preventing information leakage to external parties.

[0515] Examples of prompt statements include the following:

[0516] "We analyzed audio data from corporate meetings and, if the keyword 'project code' was found, we issued a warning notification based on that information. Because there is a risk of information leakage if this project code is disclosed externally, we urge participants who receive the notification to handle it with caution."

[0517] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0518] Step 1:

[0519] The device collects audio and image data from the environment in real time. This collected data is acquired via a microphone and camera and prepared for transmission to a server. The input is raw audio and image data, and the output is transmission data transferred to the server.

[0520] Step 2:

[0521] The server converts received audio data from speech to text using the Google Speech-to-Text API. It then utilizes a generative AI model to analyze the text data and detect specified dangerous keywords. The input is audio data, and the output is text data along with the detected keywords. The analysis lists specific words and phrases, and a risk assessment is performed based on this list.

[0522] Step 3:

[0523] The server analyzes image data using OpenCV and performs object recognition using machine learning algorithms. It determines whether specific prohibited items are present in the image. The input to this process is image data, and the output is the object recognition result. If a prohibited item is identified through analysis, the warning process is immediately initiated.

[0524] Step 4:

[0525] The server uses an evaluation device to assess the risk of information leakage based on the analysis results. If a risk is detected, it creates a notification to send an alert to the management department. The input is the analyzed keywords and item data, and the output is an alert notification. The notification is sent via email or a dashboard, enabling a quick response.

[0526] Step 5:

[0527] The terminal controls the screen or audio and masks certain information according to instructions received from the server. The input is the control instructions from the server, and the output is the display of masked data. This process ensures that measures are taken to prevent sensitive information from being leaked to third parties.

[0528] Step 6:

[0529] When a user receives a warning notification from their device, they take the necessary action. The input is the warning notification from the device, and the output is the user's response to the risk. The user will stop publishing data or perform necessary corrective actions according to this notification.

[0530] Each step works closely together to minimize the risk of data breaches and help build a highly secure environment.

[0531] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0532] One embodiment of this invention is a monitoring system aimed at preventing information leaks in high-security areas and teleworking environments, which achieves even more advanced security management by combining it with an emotion engine. This system consists of three main components: a server, a terminal, and a user.

[0533] Server role:

[0534] The server functions as the core of the entire system, analyzing audio and image data transmitted from terminals. The server uses generative models to analyze audio data and identify dangerous keywords and phrases. Furthermore, it uses image analysis equipment to check for the presence of prohibited items in image data.

[0535] The newly integrated emotion engine works to recognize the user's emotional state from voice data. In this process, the server analyzes changes in voice tone and emphasis to identify emotions such as joy, anger, and sadness. Simultaneously, it analyzes the user's facial expressions from image data to provide information that complements the emotional state. These emotional changes are considered as potential security risks.

[0536] Terminal role:

[0537] The device uses sensors, cameras, and microphones to collect data in real time and transmit it to a server. This data provides material for analysis by an emotion engine. Furthermore, based on instructions from the server, the device masks screen displays and audio output to prevent information leakage.

[0538] User roles:

[0539] Users receive feedback from the system while performing their normal tasks. In particular, if the system issues a notification as a result of sentiment analysis, users will follow the instructions and take the necessary actions. For example, if a suspicious sentiment pattern is detected, users may be asked to review their work and revise their behavior.

[0540] For example, if a user working from home is having a conversation about confidential documents and the system detects signs of anxiety in their voice, it will use this information to notify the security department and prompt them to take action. In this way, by referencing emotional states, it is possible to complementarily manage the risk of information leaks that are difficult to detect with conventional monitoring systems.

[0541] As described above, the present invention utilizes an emotion engine to realize a system that provides advanced analysis of information leakage risks and proactive countermeasures.

[0542] The following describes the processing flow.

[0543] Step 1:

[0544] The device collects the user's voice and image data in real time. Here, it uses a microphone to acquire voice data and a camera to record facial expressions. This data is immediately prepared for transmission to the server.

[0545] Step 2:

[0546] The terminal encrypts the collected data to prevent eavesdropping and tampering, and sends it to the server using a secure protocol. The data arrives at the server instantly and awaits analysis.

[0547] Step 3:

[0548] The server analyzes the received audio data using a generative model. Here, it converts the data into text using automatic speech recognition and compares it against pre-configured dangerous keywords and phrases.

[0549] Step 4:

[0550] The server simultaneously analyzes the image data to check for prohibited items within the image. This process utilizes machine learning algorithms for object recognition and records the results.

[0551] Step 5:

[0552] The server activates an emotion engine to detect the user's emotional state from their voice. It analyzes changes in tone and volume to determine the user's emotional state. This information is added to the log as emotion analysis results.

[0553] Step 6:

[0554] The server further uses image data to analyze changes in the user's facial expressions and obtain supplementary information about their emotional state. It checks the movement of facial muscles and uses this information to support the results of the voice analysis.

[0555] Step 7:

[0556] The server integrates voice, image, and sentiment analysis data to assess potential data breach risks. If a risk is detected, it sends an alert to the security department to prompt a swift response.

[0557] Step 8:

[0558] The terminal receives masking instructions from the server and appropriately masks the display and audio output as needed. This protects confidential information from the user and their surroundings.

[0559] Step 9:

[0560] Users review security notifications and, based on the alert content, take necessary actions such as stopping or correcting information sharing. By following instructions and responding promptly, users can minimize the risk of information leakage.

[0561] (Example 2)

[0562] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0563] In high-security environments and remote work settings, preventing the leakage of user voice and image information is a major challenge. Conventional monitoring systems have struggled to accurately detect the risk of information leakage and identify potential risks arising from emotional shifts. Therefore, more advanced and consistent information leakage prevention measures are required.

[0564] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0565] In this invention, the server includes a central processing means that uses a generative model to analyze voice information and detect dangerous phrases, an image analysis means that analyzes image information and determines the presence of prohibited items, and a means that uses an emotion recognition engine to evaluate the user's emotional state from the voice and image information. This makes it possible to more accurately identify the risk of information leakage and to provide warnings and countermeasures that take into account the user's emotional changes.

[0566] A "high-security domain" refers to a physical or virtual environment where information security is of particular importance and strict access restrictions and protective measures are required.

[0567] A "remote work environment" refers to a network-based work environment that allows employees to perform their duties from home or other locations without having to go to the office.

[0568] "Audio information" refers to data, including human conversations and other sounds, that are collected and analyzed by machines.

[0569] "Image information" refers to visual data, including still images or videos, acquired by cameras or sensors.

[0570] A "generative model" is an algorithm that generates new data based on audio or text data and detects specific patterns or features.

[0571] A "central processing unit" is a computing device that serves as the central hub of an entire system, responsible for data aggregation, analysis, and the issuance of instructions.

[0572] "Image analysis equipment" refers to technologies and devices used to analyze image information and detect specific features or patterns.

[0573] An "emotion recognition engine" refers to algorithms and technologies that analyze audio and image information to identify and evaluate a person's emotional state.

[0574] A "warning" is a notification that informs administrators and users of potential risks detected by the system and prompts them to take action.

[0575] A "correction device" refers to a device or function that edits or partially deletes collected audio and image information as needed to prevent information leakage.

[0576] In terms of embodiments for carrying out the invention, this system consists of a monitoring device aimed at preventing information leakage in high-security areas and remote work environments. This system includes three main components: a server, a terminal, and a user.

[0577] The server functions as the core of the entire system, responsible for analyzing audio and image information. Specifically, the server uses a generative AI model to analyze audio information and detect dangerous keywords and phrases. This process utilizes automated speech recognition technology. The server also uses an image analysis device to analyze image information and determine if prohibited objects are present. In this process, machine learning algorithms are used for object recognition.

[0578] Furthermore, the server incorporates an emotion recognition engine that identifies the user's emotional state from audio and image information. The server analyzes the tone and emphasis of the voice, identifies facial expressions from image information, and evaluates changes in emotion. This analysis of emotional state is considered a potential factor in the risk of information leakage.

[0579] The device uses sensors, cameras, and microphones to collect data in real time and transmit it to the server. This collected data then serves as material for analysis on the server. Furthermore, the device prevents information leakage by following instructions from the server, such as masking the display screen or modifying audio output as needed.

[0580] Users receive feedback from the system while performing their normal tasks. In particular, if a warning is issued based on the results of sentiment analysis, users will follow the instructions and take the necessary actions. For example, if a suspicious emotional pattern is detected, users may be asked to review their work and revise their behavior.

[0581] For example, if a user working from home is having a conversation about confidential information and anxiety is detected in their voice, the server will notify the security department. In this way, by evaluating emotional states, it is possible to complementarily manage the risk of information leaks that are difficult to detect with conventional monitoring systems.

[0582] An example of a prompt for the generating AI model is: "Analyze voice data from someone working from home and suggest a response scenario if feelings of anxiety are detected."

[0583] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0584] Step 1:

[0585] The device collects audio and image data from its environment in real time. Audio data is input from the microphone, and image data from the camera. This collected data is transmitted to a server using a secure communication method. During this process, the device's sensors capture all audio and video from its surroundings.

[0586] Step 2:

[0587] The server analyzes the received audio data using a generating AI model. The input audio data includes parts of human conversation and other sounds. The server converts the audio to text using automatic speech recognition technology and extracts dangerous keywords and phrases. This process also analyzes changes in tone and emphasis in the speech to identify the user's emotional state. The output is the analysis results in text format and an emotional assessment.

[0588] Step 3:

[0589] The server analyzes the received image data using an image analysis device. The input image data includes people and objects in the environment. Machine learning algorithms are used to recognize faces and analyze facial expressions from the images. Object recognition functionality is also used to determine if there are any prohibited items. The output provides evaluation results regarding objects and facial expressions.

[0590] Step 4:

[0591] The server integrates the analysis results of audio and image data to evaluate the user's emotional state. This evaluation may flag emotional changes as a potential risk of information leakage. The input is evaluation information from audio and images, and the output is a determination of whether a warning is necessary.

[0592] Step 5:

[0593] The server generates warnings if necessary and creates prompts to notify the management department. It also instructs the terminal to take administrative actions, such as masking the screen display or correcting audio, where applicable. The output shows the administrative actions to be taken and the content of the notification.

[0594] Step 6:

[0595] The user receives warnings and instructions as feedback from the server. The user reviews the task details and takes the necessary actions based on the instructions. This process ensures that appropriate measures are taken against the risk of information leakage. The output is improved user behavior and strengthened security management.

[0596] (Application Example 2)

[0597] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0598] In today's highly secure areas and remote work environments, information leaks are a major concern. When handling information using voice and image data, conventional systems struggle to fully detect potential risks, and effective management of information leaks, particularly those related to emotional shifts, is essential.

[0599] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0600] In this invention, the server includes means for analyzing audio and image information collected using a communication device, means for detecting dangerous words from audio information using a central processing unit with a generation format model, means for determining prohibited items using a visual analysis device, evaluation device means for evaluating the user's emotional state and warning of potential risks, and modification device means for changing the display and audio information after the warning. This makes it possible to comprehensively and effectively manage the potential risk of information leakage.

[0601] A "communication device" is a means of collecting voice and image information from a high-security environment or a remote work environment.

[0602] A "generative model" is an algorithm used to analyze audio information and detect potentially dangerous words or phrases.

[0603] The "Central Processing Unit" is the core processing unit used to analyze the collected audio information.

[0604] A "visual analysis device" is a device that analyzes image information to determine whether or not prohibited items are present.

[0605] An "evaluation device" is a device that analyzes a user's emotional state and warns the user if there is a potential risk of information leakage.

[0606] A "correction device" is a means of modifying some of the display and audio information after a warning has been issued.

[0607] The embodiment for carrying out this invention is a monitoring system mainly composed of three elements: a server, a terminal, and a user.

[0608] The server performs advanced data processing using communication equipment to analyze audio and image information. The server has a central processing unit equipped with a generative model that analyzes audio information and detects potentially dangerous words. Furthermore, a visual analysis device analyzes image information to determine the presence of prohibited items. The server uses an evaluation device to assess the user's emotional state and identify and warn of potential risks of information leakage.

[0609] The terminal transmits collected audio and image information to the server. The terminal is equipped with a microphone and camera suitable for the environment, enabling it to collect data in real time and transmit it to the server. If a warning is issued, the terminal uses a correction device to modify the display and some of the audio.

[0610] Users receive real-time feedback from this system as they perform their daily tasks. For example, when handling confidential data while working from home, if a sudden change in voice tone is detected, the system performs sentiment analysis and warns of a potential risk of information leakage. This warning allows users to review their work and take necessary measures.

[0611] For example, if a user is having an important meeting while working from home, and the system detects feelings of anxiety or impatience from the audio data, the system will immediately issue a warning: "We have detected an emotional shift that requires attention. Please review the details and take necessary actions." This is achieved through the analysis of audio and image data using a generative AI model.

[0612] An example of a prompt for a generative AI model would be: "Analyze user emotions from audio and image data and identify information leakage risks. Imagine a system that immediately notifies the user if a risk is detected."

[0613] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0614] Step 1:

[0615] The device collects audio and image data from the environment. It uses its microphone and camera as input to acquire data in real time and sends it to the server. The output is audio and image data for analysis by the server. Specifically, the device continuously captures data at regular intervals.

[0616] Step 2:

[0617] The server analyzes received audio data using a generation AI model. The input is audio data sent from a terminal, and data processing is performed to detect dangerous words and emotional tones. The output is an evaluation of dangerous words or inappropriate emotional states. Specifically, the server performs language processing using a speech recognition algorithm and evaluates tone changes with an emotion analysis engine.

[0618] Step 3:

[0619] The server analyzes the received image data using a visual analysis device. Using image data transmitted from the terminal as input, it performs data calculations to recognize prohibited items. The output is a judgment result regarding the presence or absence of the item. Specifically, the server uses a machine learning algorithm to analyze elements within the image through object recognition.

[0620] Step 4:

[0621] The server performs a comprehensive evaluation based on the results of audio and image analysis, and if a potential data leakage risk is detected, it issues a warning to the user. The input uses the analysis results from audio and images to identify risks. The output is a warning message to the user. Specifically, the server provides feedback on the evaluation results and generates notifications in real time.

[0622] Step 5:

[0623] The user receives a warning from the server, reviews the work content, and takes necessary actions. The input is the warning message received from the server. The output is the action taken to avoid risk. Specifically, the user reviews work procedures based on the notification content and performs safety checks.

[0624] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0625] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0626] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0627] [Fourth Embodiment]

[0628] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0629] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0630] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0631] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0632] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0633] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0634] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0635] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0636] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0637] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0638] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0639] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0640] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0641] One embodiment of this invention is configured as an advanced monitoring system for preventing information leaks in high-security areas and home-work environments. This system consists of three main components: a server, a terminal, and a user, each working together to enhance information security.

[0642] Server role:

[0643] The server functions as the central hub of the system, analyzing audio and image data collected from terminals. Using generative models, the server converts audio data into text and detects designated dangerous keywords or phrases. For image data, an image analysis device performs object recognition to determine if prohibited items are present. For example, if a prohibited smartphone is visible in the video within a secure area, the server will identify its presence.

[0644] Furthermore, the server uses an evaluation device to assess the potential risk of information leakage based on the analysis results, and if a risk is detected, it sends an alert to the management department. This notification is provided via email or dashboard alert, enabling a quick response.

[0645] Terminal role:

[0646] The terminal operates on the user's side and performs information collection and masking as needed. The terminal collects data in real time through the microphone and camera and sends it to the server. Furthermore, if the server detects a risk, it will mask specific screen content or audio output to conceal confidential information. For example, if sensitive information is displayed on the screen, it will be blurred to prevent unnecessary disclosure of information to third parties.

[0647] User roles:

[0648] Users perform their normal work operations, but if they receive a security notification, they will stop or correct information sharing based on the displayed instructions. Users can also receive guidance to improve their security awareness, contributing to the protection of information in their daily work.

[0649] In this way, it is possible to build a system that minimizes the risk of information leakage through the interaction of servers, terminals, and users. As a specific example, one company is using this system to monitor audio during meetings and take measures to prevent confidential project information from being leaked to the outside. In this manner, the present invention dramatically improves information security in corporate activities.

[0650] The following describes the processing flow.

[0651] Step 1:

[0652] The device collects audio and image data in real time in high-security areas or work-from-home environments. It uses a microphone and camera to capture this data, converts it to an appropriate format, and prepares it for transmission to a server.

[0653] Step 2:

[0654] The terminal encrypts the collected data and sends it to the server. To prevent interception and tampering of communications, the SSL / TLS protocol is used for secure data transfer.

[0655] Step 3:

[0656] The server analyzes the audio data received from the terminal using a generative model. It utilizes automatic speech recognition (ASR) technology to generate text from the audio data and detect dangerous keywords and phrases.

[0657] Step 4:

[0658] The server analyzes the image data using an image analysis algorithm. Here, a machine learning model is used to check for the presence of prohibited items (e.g., smartphones), and if detected, the information is recorded.

[0659] Step 5:

[0660] The server evaluates the results of the analysis of audio and image data to identify potential data breach risks. If a risk is detected, it sends an alert to the security department to quickly notify the relevant personnel.

[0661] Step 6:

[0662] The terminal receives instructions from the server and, if necessary, masks some of the information on the screen and some of the audio. This process physically hides sensitive data from the user's display and output audio to prevent unintended information leaks.

[0663] Step 7:

[0664] When users receive a security notification, they will manually correct information or cease certain actions. Based on the notified measures, they will strive to mitigate the risk of information leakage by promptly taking action such as postponing or canceling information sharing.

[0665] (Example 1)

[0666] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0667] Information leaks pose a significant risk to companies and organizations, and mitigating this risk is crucial, especially in highly confidential areas and remote work environments. Traditional methods suffer from insufficient monitoring and analysis to prevent information leaks, and struggle to immediately detect specific languages ​​or items. There is a need for systems that can improve this situation and significantly reduce the risk of information leaks.

[0668] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0669] In this invention, the server includes a device means for collecting information from a highly confidential area or residential work environment, a central control device means that analyzes the collected audio information and uses a generation model to detect dangerous words, and an automatic analysis device means that analyzes the collected image information and determines the presence of prohibited items. This makes it possible to efficiently detect specific audio and visual hazard elements and take immediate countermeasures to reduce the risk of information leakage.

[0670] 1. A "highly confidential area" refers to an environment with a high risk of information leakage, and is a place where particularly protected information or activities are conducted.

[0671] 2. "Residential work environment" refers to an environment in which an individual performs work from their home or other location, and is a place where special monitoring is required for information protection.

[0672] 3. "Information-collecting device" refers to a device or system for collecting data such as voice and images in real time and transmitting it to the next processing stage.

[0673] 4. A "generative model" is a machine learning model used to analyze collected audio data and detect specified dangerous keywords or phrases.

[0674] 5. A "central control unit" is a central processing unit that oversees and manages the entire system and performs data analysis and evaluation.

[0675] 6. An "automated analysis device" is a device that analyzes collected image data using machine learning technology to detect specific items or scenarios.

[0676] 7. An "evaluation tool" is a mechanism for assessing the risk of information leakage and notifying the management department of warnings or corrective actions based on that assessment.

[0677] 8. A "regulation device" is a device used to control or conceal parts of the screen display or sound in order to prevent information leakage.

[0678] This invention is an advanced monitoring system aimed at preventing information leaks in highly confidential areas and remote work environments. The system primarily consists of three elements: a server, a terminal, and a user, each working together to enhance information security.

[0679] The server acts as the core of the system, analyzing audio and image information collected from terminals. The server uses a generative AI model to convert audio information into text, monitoring the information by detecting specific potentially dangerous words. Furthermore, it uses an automated analysis device to perform object recognition on image data to check for prohibited items. For example, one prohibited item might be the use of a smartphone within a specific area. Based on the analysis results, the server assesses the potential risk of information leakage and sends warnings to the management department as needed.

[0680] The terminal collects information in the user's environment and performs masking if necessary. Audio and image information is collected in real time through the terminal's microphone and camera. Based on instructions from the server, the terminal blurs specific content displayed on the monitor to prevent the leakage of confidential information.

[0681] Users receive security notifications while performing their normal duties. Following warnings from the server, users can stop or correct information sharing. Furthermore, users can enhance their awareness of information protection by receiving guidance to improve their security awareness.

[0682] As a concrete example, one company uses this system to monitor audio during meetings and prevent the leakage of information related to highly confidential projects. An example of a prompt message is the instruction, "Translate audio in the security area into text and detect the specified dangerous phrase." In this way, the present invention contributes to strengthening information security.

[0683] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0684] Step 1:

[0685] The device collects audio and image data and sends it to the server. Specifically, the device's microphone and camera record audio and video in real time and send that data to the server. In this process, audio and image data are provided as input and transferred to the server in their original format.

[0686] Step 2:

[0687] The server converts the received audio data into text using a generation AI model. The system receives audio data as input and generates text data through automatic speech recognition. The converted text is then used in the next analysis step.

[0688] Step 3:

[0689] The server analyzes the text data to detect whether it contains dangerous keywords or phrases. Using a generative AI model, it identifies risky phrases by evaluating specified language patterns. The output of this step is information on whether danger was detected.

[0690] Step 4:

[0691] The server processes image data using an automated analysis system to perform object recognition. It analyzes the received images to assess the likelihood of specific items or prohibited objects being present. The output provides information on whether hazardous materials were detected.

[0692] Step 5:

[0693] The server assesses the risk of data leakage based on the analysis results of audio and image data and sends a warning to the management department. If a risk is detected, the server sends an alert to the administrator in the form of an email or dashboard notification.

[0694] Step 6:

[0695] The terminal receives a warning instruction from the server and masks the screen display and audio output. It receives warning information provided by the server as input and performs controls such as mosaic processing to prevent the leakage of confidential information. The output of this step is a masked display or output.

[0696] Step 7:

[0697] Users receive security notifications and take necessary actions. The system receives notifications from terminals and servers as input, and takes necessary measures such as stopping or correcting information sharing. The output is a work environment with improvements or corrective actions implemented.

[0698] (Application Example 1)

[0699] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0700] In high-security areas and remote work environments, a lack of effective monitoring and control measures to prevent information leaks is a challenge. In particular, there is a need to analyze voice and image data in real time to quickly detect and address potential risks, but conventional systems are unable to adequately achieve this.

[0701] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0702] In this invention, the server includes a device means for collecting voice and image data from a high-security area or a work-from-home environment; a central processing unit means that analyzes the collected voice data and uses a generative model to detect dangerous keywords; and an evaluation device means that evaluates potential information leaks and notifies a specific management department of the warning. This enables real-time information leak prevention and rapid risk management response.

[0703] A "high-security area" is a region where the risk of information leakage is high, and therefore strict security management is required.

[0704] A "work-from-home environment" refers to a work environment where the home is used as the base of operations when performing work.

[0705] A "device for collecting audio and image data" is a device that uses a microphone and a camera to acquire audio and images.

[0706] A "central processing unit using a generative model" is a device equipped with generative AI technology to analyze audio data and detect specified dangerous keywords.

[0707] An "image analysis device" is a device that uses machine learning algorithms to analyze image data and determine the presence or absence of an object.

[0708] An "evaluation device" is a device that assesses potential information leakage risks and issues warnings as necessary.

[0709] A "correction device" is a device that, after a warning, masks parts of the screen and audio data to block confidential information.

[0710] A "blocking device" is a device that has the function of blocking specific information in order to prevent unauthorized access to that information.

[0711] This system is designed to prevent information leaks in high-security areas or remote work environments. It consists of three main components: servers, terminals, and users.

[0712] The server functions as a central hub for analyzing audio and image data. Audio data is collected from the device via a microphone and transferred to the server. There, it is converted to text using automatic speech recognition technology such as the Google Speech-to-Text API. A generative AI model is used to detect specified dangerous keywords from this text data. Image data is transmitted from the device via a camera and analyzed using image processing libraries such as OpenCV. Machine learning algorithms are used to determine, for example, the presence or absence of prohibited items.

[0713] Based on these analysis results, the server assesses the potential risk of data breaches and generates an alert via the assessment device. This alert is then sent to the management department via email or a dashboard alert.

[0714] If a warning is issued, the terminal controls the display and output of data, and a correction device masks certain information. This includes processes to prevent the leakage of confidential information, such as muting audio and blurring text and images on the screen.

[0715] Users will perform their normal duties, but when they receive a warning, they will restrict or modify data disclosure based on the instructions given. Furthermore, users will receive guidance to raise their awareness of information security, thereby strengthening information protection in their daily work.

[0716] As a concrete example, if this system is implemented in a certain corporate environment and a project code name is mentioned during a meeting, the server will immediately detect the risk and alert the participants, thereby preventing information leakage to external parties.

[0717] Examples of prompt statements include the following:

[0718] "We analyzed audio data from corporate meetings and, if the keyword 'project code' was found, we issued a warning notification based on that information. Because there is a risk of information leakage if this project code is disclosed externally, we urge participants who receive the notification to handle it with caution."

[0719] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0720] Step 1:

[0721] The device collects audio and image data from the environment in real time. This collected data is acquired via a microphone and camera and prepared for transmission to a server. The input is raw audio and image data, and the output is transmission data transferred to the server.

[0722] Step 2:

[0723] The server converts received audio data from speech to text using the Google Speech-to-Text API. It then utilizes a generative AI model to analyze the text data and detect specified dangerous keywords. The input is audio data, and the output is text data along with the detected keywords. The analysis lists specific words and phrases, and a risk assessment is performed based on this list.

[0724] Step 3:

[0725] The server analyzes image data using OpenCV and performs object recognition using machine learning algorithms. It determines whether specific prohibited items are present in the image. The input to this process is image data, and the output is the object recognition result. If a prohibited item is identified through analysis, the warning process is immediately initiated.

[0726] Step 4:

[0727] The server uses an evaluation device to assess the risk of information leakage based on the analysis results. If a risk is detected, it creates a notification to send an alert to the management department. The input is the analyzed keywords and item data, and the output is an alert notification. The notification is sent via email or a dashboard, enabling a quick response.

[0728] Step 5:

[0729] The terminal controls the screen or audio and masks certain information according to instructions received from the server. The input is the control instructions from the server, and the output is the display of masked data. This process ensures that measures are taken to prevent sensitive information from being leaked to third parties.

[0730] Step 6:

[0731] When a user receives a warning notification from their device, they take the necessary action. The input is the warning notification from the device, and the output is the user's response to the risk. The user will stop publishing data or perform necessary corrective actions according to this notification.

[0732] Each step works closely together to minimize the risk of data breaches and help build a highly secure environment.

[0733] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0734] One embodiment of this invention is a monitoring system aimed at preventing information leaks in high-security areas and teleworking environments, which achieves even more advanced security management by combining it with an emotion engine. This system consists of three main components: a server, a terminal, and a user.

[0735] Server role:

[0736] The server functions as the core of the entire system, analyzing audio and image data transmitted from terminals. The server uses generative models to analyze audio data and identify dangerous keywords and phrases. Furthermore, it uses image analysis equipment to check for the presence of prohibited items in image data.

[0737] The newly integrated emotion engine works to recognize the user's emotional state from voice data. In this process, the server analyzes changes in voice tone and emphasis to identify emotions such as joy, anger, and sadness. Simultaneously, it analyzes the user's facial expressions from image data to provide information that complements the emotional state. These emotional changes are considered as potential security risks.

[0738] Terminal role:

[0739] The device uses sensors, cameras, and microphones to collect data in real time and transmit it to a server. This data provides material for analysis by an emotion engine. Furthermore, based on instructions from the server, the device masks screen displays and audio output to prevent information leakage.

[0740] User roles:

[0741] Users receive feedback from the system while performing their normal tasks. In particular, if the system issues a notification as a result of sentiment analysis, users will follow the instructions and take the necessary actions. For example, if a suspicious sentiment pattern is detected, users may be asked to review their work and revise their behavior.

[0742] For example, if a user working from home is having a conversation about confidential documents and the system detects signs of anxiety in their voice, it will use this information to notify the security department and prompt them to take action. In this way, by referencing emotional states, it is possible to complementarily manage the risk of information leaks that are difficult to detect with conventional monitoring systems.

[0743] As described above, the present invention utilizes an emotion engine to realize a system that provides advanced analysis of information leakage risks and proactive countermeasures.

[0744] The following describes the processing flow.

[0745] Step 1:

[0746] The device collects the user's voice and image data in real time. Here, it uses a microphone to acquire voice data and a camera to record facial expressions. This data is immediately prepared for transmission to the server.

[0747] Step 2:

[0748] The terminal encrypts the collected data to prevent eavesdropping and tampering, and sends it to the server using a secure protocol. The data arrives at the server instantly and awaits analysis.

[0749] Step 3:

[0750] The server analyzes the received audio data using a generative model. Here, it converts the data into text using automatic speech recognition and compares it against pre-configured dangerous keywords and phrases.

[0751] Step 4:

[0752] The server simultaneously analyzes the image data to check for prohibited items within the image. This process utilizes machine learning algorithms for object recognition and records the results.

[0753] Step 5:

[0754] The server activates an emotion engine to detect the user's emotional state from their voice. It analyzes changes in tone and volume to determine the user's emotional state. This information is added to the log as emotion analysis results.

[0755] Step 6:

[0756] The server further uses image data to analyze changes in the user's facial expressions and obtain supplementary information about their emotional state. It checks the movement of facial muscles and uses this information to support the results of the voice analysis.

[0757] Step 7:

[0758] The server integrates voice, image, and sentiment analysis data to assess potential data breach risks. If a risk is detected, it sends an alert to the security department to prompt a swift response.

[0759] Step 8:

[0760] The terminal receives masking instructions from the server and appropriately masks the display and audio output as needed. This protects confidential information from the user and their surroundings.

[0761] Step 9:

[0762] Users review security notifications and, based on the alert content, take necessary actions such as stopping or correcting information sharing. By following instructions and responding promptly, users can minimize the risk of information leakage.

[0763] (Example 2)

[0764] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0765] In high-security environments and remote work settings, preventing the leakage of user voice and image information is a major challenge. Conventional monitoring systems have struggled to accurately detect the risk of information leakage and identify potential risks arising from emotional shifts. Therefore, more advanced and consistent information leakage prevention measures are required.

[0766] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0767] In this invention, the server includes a central processing means that uses a generative model to analyze voice information and detect dangerous phrases, an image analysis means that analyzes image information and determines the presence of prohibited items, and a means that uses an emotion recognition engine to evaluate the user's emotional state from the voice and image information. This makes it possible to more accurately identify the risk of information leakage and to provide warnings and countermeasures that take into account the user's emotional changes.

[0768] A "high-security domain" refers to a physical or virtual environment where information security is of particular importance and strict access restrictions and protective measures are required.

[0769] A "remote work environment" refers to a network-based work environment that allows employees to perform their duties from home or other locations without having to go to the office.

[0770] "Audio information" refers to data, including human conversations and other sounds, that are collected and analyzed by machines.

[0771] "Image information" refers to visual data, including still images or videos, acquired by cameras or sensors.

[0772] A "generative model" is an algorithm that generates new data based on audio or text data and detects specific patterns or features.

[0773] A "central processing unit" is a computing device that serves as the central hub of an entire system, responsible for data aggregation, analysis, and the issuance of instructions.

[0774] "Image analysis equipment" refers to technologies and devices used to analyze image information and detect specific features or patterns.

[0775] An "emotion recognition engine" refers to algorithms and technologies that analyze audio and image information to identify and evaluate a person's emotional state.

[0776] A "warning" is a notification that informs administrators and users of potential risks detected by the system and prompts them to take action.

[0777] A "correction device" refers to a device or function that edits or partially deletes collected audio and image information as needed to prevent information leakage.

[0778] In terms of embodiments for carrying out the invention, this system consists of a monitoring device aimed at preventing information leakage in high-security areas and remote work environments. This system includes three main components: a server, a terminal, and a user.

[0779] The server functions as the core of the entire system, responsible for analyzing audio and image information. Specifically, the server uses a generative AI model to analyze audio information and detect dangerous keywords and phrases. This process utilizes automated speech recognition technology. The server also uses an image analysis device to analyze image information and determine if prohibited objects are present. In this process, machine learning algorithms are used for object recognition.

[0780] Furthermore, the server incorporates an emotion recognition engine that identifies the user's emotional state from audio and image information. The server analyzes the tone and emphasis of the voice, identifies facial expressions from image information, and evaluates changes in emotion. This analysis of emotional state is considered a potential factor in the risk of information leakage.

[0781] The device uses sensors, cameras, and microphones to collect data in real time and transmit it to the server. This collected data then serves as material for analysis on the server. Furthermore, the device prevents information leakage by following instructions from the server, such as masking the display screen or modifying audio output as needed.

[0782] Users receive feedback from the system while performing their normal tasks. In particular, if a warning is issued based on the results of sentiment analysis, users will follow the instructions and take the necessary actions. For example, if a suspicious emotional pattern is detected, users may be asked to review their work and revise their behavior.

[0783] For example, if a user working from home is having a conversation about confidential information and anxiety is detected in their voice, the server will notify the security department. In this way, by evaluating emotional states, it is possible to complementarily manage the risk of information leaks that are difficult to detect with conventional monitoring systems.

[0784] An example of a prompt for the generating AI model is: "Analyze voice data from someone working from home and suggest a response scenario if feelings of anxiety are detected."

[0785] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0786] Step 1:

[0787] The device collects audio and image data from its environment in real time. Audio data is input from the microphone, and image data from the camera. This collected data is transmitted to a server using a secure communication method. During this process, the device's sensors capture all audio and video from its surroundings.

[0788] Step 2:

[0789] The server analyzes the received audio data using a generating AI model. The input audio data includes parts of human conversation and other sounds. The server converts the audio to text using automatic speech recognition technology and extracts dangerous keywords and phrases. This process also analyzes changes in tone and emphasis in the speech to identify the user's emotional state. The output is the analysis results in text format and an emotional assessment.

[0790] Step 3:

[0791] The server analyzes the received image data using an image analysis device. The input image data includes people and objects in the environment. Machine learning algorithms are used to recognize faces and analyze facial expressions from the images. Object recognition functionality is also used to determine if there are any prohibited items. The output provides evaluation results regarding objects and facial expressions.

[0792] Step 4:

[0793] The server integrates the analysis results of audio and image data to evaluate the user's emotional state. This evaluation may flag emotional changes as a potential risk of information leakage. The input is evaluation information from audio and images, and the output is a determination of whether a warning is necessary.

[0794] Step 5:

[0795] The server generates warnings if necessary and creates prompts to notify the management department. It also instructs the terminal to take administrative actions, such as masking the screen display or correcting audio, where applicable. The output shows the administrative actions to be taken and the content of the notification.

[0796] Step 6:

[0797] The user receives warnings and instructions as feedback from the server. The user reviews the task details and takes the necessary actions based on the instructions. This process ensures that appropriate measures are taken against the risk of information leakage. The output is improved user behavior and strengthened security management.

[0798] (Application Example 2)

[0799] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0800] In today's highly secure areas and remote work environments, information leaks are a major concern. When handling information using voice and image data, conventional systems struggle to fully detect potential risks, and effective management of information leaks, particularly those related to emotional shifts, is essential.

[0801] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0802] In this invention, the server includes means for analyzing audio and image information collected using a communication device, means for detecting dangerous words from audio information using a central processing unit with a generation format model, means for determining prohibited items using a visual analysis device, evaluation device means for evaluating the user's emotional state and warning of potential risks, and modification device means for changing the display and audio information after the warning. This makes it possible to comprehensively and effectively manage the potential risk of information leakage.

[0803] A "communication device" is a means of collecting voice and image information from a high-security environment or a remote work environment.

[0804] A "generative model" is an algorithm used to analyze audio information and detect potentially dangerous words or phrases.

[0805] The "Central Processing Unit" is the core processing unit used to analyze the collected audio information.

[0806] A "visual analysis device" is a device that analyzes image information to determine whether or not prohibited items are present.

[0807] An "evaluation device" is a device that analyzes a user's emotional state and warns the user if there is a potential risk of information leakage.

[0808] A "correction device" is a means of modifying some of the display and audio information after a warning has been issued.

[0809] The embodiment for carrying out this invention is a monitoring system mainly composed of three elements: a server, a terminal, and a user.

[0810] The server performs advanced data processing using communication equipment to analyze audio and image information. The server has a central processing unit equipped with a generative model that analyzes audio information and detects potentially dangerous words. Furthermore, a visual analysis device analyzes image information to determine the presence of prohibited items. The server uses an evaluation device to assess the user's emotional state and identify and warn of potential risks of information leakage.

[0811] The terminal transmits collected audio and image information to the server. The terminal is equipped with a microphone and camera suitable for the environment, enabling it to collect data in real time and transmit it to the server. If a warning is issued, the terminal uses a correction device to modify the display and some of the audio.

[0812] Users receive real-time feedback from this system as they perform their daily tasks. For example, when handling confidential data while working from home, if a sudden change in voice tone is detected, the system performs sentiment analysis and warns of a potential risk of information leakage. This warning allows users to review their work and take necessary measures.

[0813] For example, if a user is having an important meeting while working from home, and the system detects feelings of anxiety or impatience from the audio data, the system will immediately issue a warning: "We have detected an emotional shift that requires attention. Please review the details and take necessary actions." This is achieved through the analysis of audio and image data using a generative AI model.

[0814] An example of a prompt for a generative AI model would be: "Analyze user emotions from audio and image data and identify information leakage risks. Imagine a system that immediately notifies the user if a risk is detected."

[0815] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0816] Step 1:

[0817] The device collects audio and image data from the environment. It uses its microphone and camera as input to acquire data in real time and sends it to the server. The output is audio and image data for analysis by the server. Specifically, the device continuously captures data at regular intervals.

[0818] Step 2:

[0819] The server analyzes received audio data using a generation AI model. The input is audio data sent from a terminal, and data processing is performed to detect dangerous words and emotional tones. The output is an evaluation of dangerous words or inappropriate emotional states. Specifically, the server performs language processing using a speech recognition algorithm and evaluates tone changes with an emotion analysis engine.

[0820] Step 3:

[0821] The server analyzes the received image data using a visual analysis device. Using image data transmitted from the terminal as input, it performs data calculations to recognize prohibited items. The output is a judgment result regarding the presence or absence of the item. Specifically, the server uses a machine learning algorithm to analyze elements within the image through object recognition.

[0822] Step 4:

[0823] The server performs a comprehensive evaluation based on the results of audio and image analysis, and if a potential data leakage risk is detected, it issues a warning to the user. The input uses the analysis results from audio and images to identify risks. The output is a warning message to the user. Specifically, the server provides feedback on the evaluation results and generates notifications in real time.

[0824] Step 5:

[0825] The user receives a warning from the server, reviews the work content, and takes necessary actions. The input is the warning message received from the server. The output is the action taken to avoid risk. Specifically, the user reviews work procedures based on the notification content and performs safety checks.

[0826] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0827] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0828] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0829] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0830] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0831] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0832] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0833] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0834] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0835] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0836] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0837] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0838] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0839] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0840] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0841] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0842] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0843] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0844] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0845] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0846] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0847] The following is further disclosed regarding the embodiments described above.

[0848] (Claim 1)

[0849] A device for collecting audio and image data from a high-security area or a home-work environment,

[0850] A central processing unit that uses a generative model to analyze collected audio data and detect dangerous keywords,

[0851] Image analysis device means for analyzing collected image data and determining whether prohibited items are present,

[0852] An evaluation device means for assessing potential information leaks and notifying a specific management department of the warning,

[0853] A correction device means for masking parts of the screen and audio data after a warning,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, wherein in the analysis of audio data, a generative model utilizes automatic speech recognition technology to detect dangerous phrases.

[0857] (Claim 3)

[0858] The system according to claim 1, which uses a machine learning algorithm to perform object recognition in the analysis of image data and determines the presence of prohibited objects.

[0859] "Example 1"

[0860] (Claim 1)

[0861] Device and means for collecting information from a highly confidential area or residential / work environment,

[0862] A central control device that uses a generative model to analyze collected audio information and detect dangerous words,

[0863] An automated analysis device means that analyzes collected image information and determines the presence of prohibited items,

[0864] An assessment mechanism that evaluates the potential risk of information leakage and sends warnings to specific management departments,

[0865] After a warning, an adjustment device means for concealing some of the visual and auditory information,

[0866] A system that includes this.

[0867] (Claim 2)

[0868] The system according to claim 1, wherein a generative model uses speech recognition technology to detect dangerous language structures in the analysis of speech information.

[0869] (Claim 3)

[0870] The system according to claim 1, which uses machine learning techniques to perform object recognition in the analysis of image information and determines the presence of prohibited objects.

[0871] "Application Example 1"

[0872] (Claim 1)

[0873] A device for collecting audio and image data from a high-security area or a home-work environment,

[0874] A central processing unit that uses a generative model to analyze collected audio data and detect dangerous keywords,

[0875] Image analysis device means for analyzing collected image data and determining whether prohibited items are present,

[0876] An evaluation device means for assessing potential information leaks and notifying a specific management department of the warning,

[0877] A correction device means for masking parts of the screen and audio data after a warning,

[0878] A blocking device means for blocking specified information on a smart device and preventing third-party access to that information,

[0879] A system that includes this.

[0880] (Claim 2)

[0881] The system according to claim 1, wherein in the analysis of audio data, a generative model utilizes automatic speech recognition technology to detect dangerous phrases.

[0882] (Claim 3)

[0883] The system according to claim 1, which uses a machine learning algorithm to perform object recognition in the analysis of image data and determines the presence of prohibited objects.

[0884] "Example 2 of combining an emotion engine"

[0885] (Claim 1)

[0886] A device means for collecting audio and image information from a high-security area or remote work environment,

[0887] A central processing unit means that analyzes collected audio information and uses a generative model to detect dangerous words,

[0888] An image analysis device means that analyzes collected image information and determines whether a prohibited object is present,

[0889] A device means that uses an emotion recognition engine based on voice and image information to evaluate the user's emotional state and assess the potential risk of information leakage,

[0890] Based on the evaluation results, a correction device means notifies a specific management body of the warning and modifies part of the display and audio information as necessary.

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, in analyzing speech information, a generative model utilizes automatic speech recognition technology to detect dangerous phrases and analyzes the tone and emphasis of the voice to recognize the emotional state.

[0894] (Claim 3)

[0895] The system according to claim 1, which, in analyzing image information, uses a machine learning algorithm to recognize a person's facial expression and supplement their emotional state, and also performs object recognition to determine the presence of prohibited objects.

[0896] "Application example 2 when combining with an emotional engine"

[0897] (Claim 1)

[0898] A device means for collecting voice and image information from a highly secure environment or remote work environment using a communication device,

[0899] A central computing device means that analyzes collected audio information and uses a generation format model to detect potentially dangerous words,

[0900] A visual analysis device means for analyzing collected image information and determining whether prohibited items are present,

[0901] An evaluation device means for analyzing a user's emotional state and issuing a warning to the user if there is a potential risk of information leakage,

[0902] After the warning, a modification device means for changing part of the display device and audio information,

[0903] A system that includes this.

[0904] (Claim 2)

[0905] The system according to claim 1, wherein in the analysis of speech information, the generation format model uses speech recognition technology to detect dangerous words.

[0906] (Claim 3)

[0907] The system according to claim 1, which uses a machine learning algorithm to perform object recognition in the analysis of image information and determines the presence of prohibited items. [Explanation of symbols]

[0908] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A device for collecting audio and image data from a high-security area or a home-work environment, A central processing unit that uses a generative model to analyze collected audio data and detect dangerous keywords, Image analysis device means for analyzing collected image data and determining whether prohibited items are present, An evaluation device means for assessing potential information leaks and notifying a specific management department of the warning, A correction device means for masking parts of the screen and audio data after a warning, A system that includes this.

2. The system according to claim 1, wherein in the analysis of audio data, the generative model uses automatic speech recognition technology to detect dangerous phrases.

3. The system according to claim 1, which uses a machine learning algorithm to perform object recognition in the analysis of image data and determines the presence of prohibited objects.

Images