System

Abstract

A system is provided.SOLUTION: A system comprising: means for reading a near-miss incident database; means for analyzing a received image using a large-scale language model; means for generating a predicted accident and a preventive measure thereof based on an analysis result; and means for outputting the generated accident prediction result and preventive measure.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many accidents occur in the home, some of which can result in serious injury or even death. To prevent these accidents, it is important to understand the risk factors surrounding us and take appropriate measures. However, it is difficult to anticipate all risks, and it is unrealistic for individuals to understand all the hidden risks lurking in the home. Therefore, the purpose of this invention is to provide an automatic near-miss prevention system that uses a large-scale language model (LLM) and a near-miss database to predict potential accidents that may occur in the home and propose preventive measures. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means. A system is provided that includes a means for reading a near-miss database, a means for analyzing received images using a large-scale language model (LLM), a means for generating predicted accidents and preventive measures based on the analysis results, and a means for outputting the generated accident prediction results and preventive measures. In this system, the near-miss database includes cases of accidents that have occurred within the home, and the large-scale language model (LLM) is trained based on these accident cases. The system also includes a means for capturing images of the home using a camera and sending the images to a server. This means makes it possible to grasp the risk of accidents within the home in advance and take appropriate measures.

[0006] A "near miss database" is a database that collects past near miss cases and is used as a knowledge base that is useful for preventing accidents and problems.

[0007] A "large-scale language model (LLM)" is an advanced machine learning model that learns from text data and performs natural language processing, giving computers the ability to understand and generate language.

[0008] "Image analysis" is a technology that processes image data acquired by a camera and extracts significant information and patterns from it.

[0009] "Prediction results" refer to future events and risks estimated based on analysis, and are information that indicates the possibility of an accident occurring.

[0010] "Preventive measures" are specific actions or methods taken to avoid predicted risks.

[0011] A "server" is a computer system that provides various services and data to clients (terminals).

[0012] A "terminal" is a computer device that is directly operated by a user, and includes input / output devices such as a camera and a display.

[0013] A "camera" is a device for taking images or videos, and is an equipment that acquires still images or videos as digital data.

[0014] "Image data" is a digital representation of visual information captured by a camera.

[0015] "Analysis means" refers to a function or system that processes input data and extracts the desired information.

[0016] "Output means" refers to a device or mechanism for displaying information obtained as a result of calculation processing to the user. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to an automatic near-miss prevention system that analyzes images of a home, predicts accidents, and proposes preventive measures. This system combines a near-miss database with a large-scale language model (LLM) to predict potential accidents that may occur in the home and propose preventive measures. An embodiment of the present invention will be described in detail below.

[0039] System Overview

[0040] This system consists of the following main components:

[0041] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0042] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0043] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[0044] 4. Server: It is the main computing unit for performing image analysis and generating accident prediction and prevention measures.

[0045] 5. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results.

[0046] Program processing

[0047] 1. System initialization

[0048] The server loads the near-miss database and large-scale language model (LLM) into memory and completes preparation.

[0049] The terminal activates the camera and switches to image capture mode.

[0050] 2. Image Acquisition

[0051] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the camera.

[0052] The terminal transmits the captured image to the server.

[0053] 3. Image Analysis

[0054] The server preprocesses the received image data and inputs it into a large-scale language model (LLM). Preprocessing includes image denoising and resolution adjustment.

[0055] LLM analyzes risk factors within an image and generates a prediction result.

[0056] 4. Accident prediction and prevention measures

[0057] The server generates specific accident predictions and prevention measures based on the analysis results from the LLM.

[0058] For example, the prediction result is "there is a high risk of fire because there are flammable objects near the stove," and the preventive measure is "keep flammable objects a certain distance away from the stove."

[0059] 5. Displaying the results

[0060] The server transmits the generated prediction results and preventive measures to the terminal.

[0061] The terminal displays the received results on the screen for the user.

[0062] Specific examples

[0063] Examples of areas around the kitchen stove

[0064] 1. System initialization

[0065] The server loads the near miss database and LLM model and is ready to go.

[0066] The terminal activates the camera and switches to image capture mode.

[0067] 2. Image Acquisition

[0068] The user selects the area around the kitchen stove and takes an image with the camera.

[0069] The terminal transmits the captured image of the kitchen to the server.

[0070] 3. Image Analysis

[0071] The server preprocesses the received images and then inputs them into the LLM model.

[0072] The LLM model detects the presence of flammable materials near a stove and predicts the risk of fire.

[0073] 4. Accident prediction and prevention measures

[0074] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove."

[0075] 5. Displaying the results

[0076] The server transmits the generated prediction results and preventive measures to the terminal.

[0077] The terminal displays the received results to the user.

[0078] This allows the user to know in advance the risks that may be present in the home and take appropriate measures. This is a specific example of an embodiment of the present invention.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] The server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization, which prepares the system for analysis and prediction.

[0082] Step 2:

[0083] The device starts the camera and prepares the user to capture images. It displays a message that the camera has been successfully started and prompts the user to perform an operation.

[0084] Step 3:

[0085] The user selects the room or specific location (such as the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[0086] Step 4:

[0087] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[0088] Step 5:

[0089] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[0090] Step 6:

[0091] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[0092] Step 7:

[0093] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[0094] Step 8:

[0095] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model. For example, the server may predict that "there is a high risk of fire due to flammable materials near the stove" and generate preventive measures such as "keep flammable materials at a certain distance away from the stove."

[0096] Step 9:

[0097] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[0098] Step 10:

[0099] The terminal displays the received accident prediction results and prevention measures to the user in a format that makes it easy for the user to check the results.

[0100] Step 11:

[0101] The user checks the prediction results and preventive measures displayed on the device screen and implements the proposed preventive measures, such as moving flammable materials away from the stove.

[0102] Through these steps, the system of the present invention can detect potential risks in the home in advance and provide users with specific preventative measures, thereby improving safety in the home and preventing accidents from occurring.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] A wide variety of accidents can occur in the home, and preventing them requires quick and accurate judgment. Especially in households with elderly people or children, even minor near-misses can lead to serious accidents. Therefore, there is a need for a system that can analyze potential home risks in real time and quickly present predictions and preventative measures. However, current technology does not adequately predict accidents or present preventative measures. Furthermore, there are issues with the system, such as complex operation for end users and low accuracy.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for reading the near-miss database, means for analyzing received images using a large-scale language model (LLM), means for generating predicted accidents and preventive measures based on the analysis results, means for receiving image data from an image acquisition device and performing preprocessing, means for inputting the preprocessed image data into the large-scale language model, and means for notifying the user of the prediction results and preventive measures. This makes it possible to analyze risks that may occur in the home in real time with high accuracy and quickly predict accidents and present preventive measures.

[0108] The "near miss database" is a database that collects and records past accident and risk cases that have occurred within the home.

[0109] A "large-scale language model (LLM)" is a model based on natural language processing technology that is trained from large amounts of text data and is used for image analysis and text generation.

[0110] "Image capture device" refers to a hardware device, such as a camera or sensor, used to capture images in the home.

[0111] "Preprocessing" refers to a series of processes that involve performing noise removal, resolution adjustment, etc. on image data to prepare it in a format suitable for analysis.

[0112] "Prediction results" refers to risk assessments and information indicating the possibility of an accident occurring obtained as a result of image analysis.

[0113] "Preventive measures" refers to instructions and advice to propose specific countermeasures and actions based on predicted accident risks.

[0114] "User notification means" refers to the interface or protocol for informing users of the analysis results and preventive measures.

[0115] The present invention relates to an automatic near-miss prevention system that analyzes images of a home, predicts accidents, and proposes preventive measures. This system combines a near-miss database with a large-scale language model (LLM) to predict potential accidents that may occur in the home and propose preventive measures. An embodiment of the present invention will be described in detail below.

[0116] System Overview

[0117] This system consists of the following main components:

[0118] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0119] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0120] 3. Image capture device: A camera for capturing images in the home and its operation interface.

[0121] 4. Server: This is the main computational unit for performing image analysis and generating accident predictions and prevention measures. The server is responsible for reading the near-miss incident database, preprocessing the received image data, and inputting it into the large-scale language model (LLM).

[0122] 5. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results.

[0123] The main processing of this system is as follows.

[0124] System initialization

[0125] The server reads the initial configuration file, loads the near-miss database, and deploys a large-scale language model (e.g., GPT-4) in memory to prepare for analysis. Meanwhile, the device starts the camera and prepares to switch to image acquisition mode.

[0126] Image acquisition and transmission

[0127] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the device's camera. The captured image is then sent from the device to the server using a secure protocol (for example, HTTPS).

[0128] Image analysis

[0129] The server performs preprocessing on the received image data, including noise removal and resolution adjustment. The preprocessed image data is then input into a large-scale language model (LLM), which analyzes risk factors and generates prediction results. For example, it can detect the presence of flammable materials near a stove and predict a high risk of fire.

[0130] Accident prediction and prevention measures

[0131] The server generates specific accident predictions and prevention measures as text based on the output of the LLM. For example, in response to a prediction that "there is a high risk of fire because there are flammable objects near the stove," it generates a prevention measure such as "keep flammable objects a certain distance away from the stove."

[0132] Displaying the results

[0133] The server sends the generated prediction results and preventive measures to the terminal, which displays them on the user interface. The user can check the displayed results and take necessary measures.

[0134] Specific examples

[0135] Analysis example around a kitchen stove

[0136] After initializing the system, the user takes a picture of the area around the kitchen stove with a camera, and the device sends the image data to the server. The server preprocesses the image data and analyzes risk elements in the image using a large-scale language model (LLM). As a result, it predicts that there are flammable objects near the stove and determines that there is a fire risk. The server then generates a preventive measure, such as "keep flammable objects a certain distance away from the stove," and sends it to the device. The device displays this to the user, who can then take appropriate measures.

[0137] Prompt Sentence Examples

[0138] "We would like you to analyze images from within a home and propose ways to predict and prevent accidents. Please use the near-miss database and LLM model below to provide examples and explain.

[0139] 1. Near miss database: Examples of accidents that may occur in the home (e.g., falls, fires, electric shocks, etc.).

[0140] 2. LLM model: Analyzes risk factors within the home, predicts accidents, and proposes preventive measures.

[0141] Examples:

[0142] After analyzing images of the area around the kitchen stove, it was determined that there was a high risk of fire.

[0143] As a preventative measure, they suggest "keeping flammable materials away from the stove."

[0144] Under the above conditions, please generate the image analysis results and countermeasures for your home.

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Step 1: Initialize the system

[0147] The server reads the initial configuration file and loads the near-miss database and large-scale language model (LLM) into memory. During this process, a database containing past near-miss cases is loaded into the server. The large-scale language model (e.g., GPT-4) is initialized based on these databases. The input is the initial configuration file, and the output is the database and model loaded into memory.

[0148] Step 2: Launch the camera

[0149] The device sends a camera start command and switches to image capture mode. The user can confirm that the camera is ready through the device interface. The input is the camera start command, and the output is the camera in image capture mode.

[0150] Step 3: Capture an image

[0151] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the device's camera. Image data is captured by pressing the capture button. The input is the area selected by the user's operation, and the output is the captured image.

[0152] Step 4: Sending image data

[0153] The terminal compresses the captured image data and sends it to the server using a secure protocol (e.g., HTTPS). The input is the captured image data, and the output is the compressed data sent to the server.

[0154] Step 5: Preprocessing the image data

[0155] The server performs preprocessing on the received image data, such as noise reduction and resolution adjustment. The preprocessed image data is input into a large-scale language model (LLM). The input is the received image data, and the output is the preprocessed image data.

[0156] Step 6: Image analysis

[0157] The server inputs the preprocessed image data into a large-scale language model (LLM) to analyze risk factors. The model performs a risk assessment based on the input image and generates a prediction result. For example, it detects the presence of flammable materials near a stove and predicts a high risk of fire. The input is the preprocessed image data, and the output is a risk assessment and prediction result.

[0158] Step 7: Accident prediction and prevention measures

[0159] The server generates specific accident predictions and preventive measures based on the output of the LLM. For example, in response to a prediction result that "there is a high risk of fire because flammable objects are near the stove," the server generates a preventive measure as a text message, such as "keep flammable objects a certain distance away from the stove." The input is the analysis result, and the output is a text message containing the preventive measures.

[0160] Step 8: Sending the results

[0161] The server encodes the generated prediction results and prevention measures and sends them to the terminal. The input is the generated text data, and the output is the encoded data sent to the terminal.

[0162] Step 9: View the results

[0163] The terminal decodes the received data and displays it on the user interface. The user can check the displayed results and take necessary measures. The input is the received encoded data, and the output is the predicted results and preventative measures displayed to the user.

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] Unforeseen safety risks may occur inside an autonomous vehicle due to the placement of passengers and luggage. While conventional vehicle safety systems have made some progress in addressing risks in the external environment, they are insufficient in addressing risks inside the vehicle. Therefore, there is a need for a system that can monitor the situation inside the vehicle in real time, predict potential risks, and quickly propose preventative measures.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes means for reading the near-miss database, means for capturing received images of the inside of the vehicle with a camera, means for preprocessing the captured images and inputting them into a large-scale language model (LLM), means for generating accident predictions and preventive measures based on the analysis results, and means for outputting the generated prediction results and preventive measures. This makes it possible to predict safety risks inside the vehicle in real time and quickly present appropriate preventive measures.

[0169] The "near miss database" is a database that collects and manages various accident and risk cases that have occurred in the past.

[0170] A "large-scale language model (LLM)" is an artificial intelligence model that learns from large amounts of text data and has advanced natural language processing capabilities.

[0171] A "received image" or "captured image" refers to image data captured using a device such as a camera.

[0172] "Preprocessing" is the process of removing noise from data, adjusting resolution, and preparing the data in an appropriate format before using it for image analysis.

[0173] "Analysis results" refers to information generated as a result of analyzing image data using a large-scale language model (LLM).

[0174] "Accident prediction" refers to estimating accidents and risks that may occur in the future based on the results of analysis.

[0175] "Preventive measures" are specific methods and means for preventing predicted risks and accidents.

[0176] "Inside the vehicle" refers to the area inside the cabin of an autonomous vehicle, i.e., the space where passengers sit.

[0177] A "camera" is a photographing device for capturing visual information and recording it as image data.

[0178] A "server" is the main computing unit that processes the entire system and manages and analyzes data.

[0179] "Means" are the functions or methods used by the system to achieve a particular purpose.

[0180] A specific system for implementing this invention is configured as follows: The server reads the near-miss incident database and uses a large-scale language model (LLM) to analyze the received images of the vehicle interior. A camera is used to capture images of the vehicle interior in real time, which are preprocessed and then input into the LLM. Accident predictions and corresponding preventive measures are generated based on the analysis results, and this information is output to a terminal.

[0181] Program processing

[0182] System initialization

[0183] The server loads the near-miss database and large-scale language model (LLM) into memory, preparing the system for quick access to the information needed for analysis. The device also activates the camera and switches it to image acquisition mode.

[0184] Image acquisition

[0185] The user selects a specific area in the vehicle (e.g., the back seat) and takes an image with the camera. The captured image is then sent to the server via the terminal.

[0186] Image preprocessing

[0187] The server preprocesses the received image data, including removing image noise and adjusting the resolution, especially by resizing and normalizing the image so that the LLM can efficiently analyze the input data.

[0188] Image analysis

[0189] The pre-processed image data is input into a large-scale language model (LLM) to analyze risk factors. The LLM determines whether a particular situation inside the vehicle is dangerous based on information learned from a near-miss database.

[0190] Accident prediction and prevention measures

[0191] The server uses the analysis results from the LLM to predict accidents and generate preventative measures. For example, if it determines that there is a risk of luggage being thrown off in the back seat during sudden braking, it will suggest a preventative measure such as "moving the luggage to the trunk."

[0192] Displaying the results

[0193] The generated prediction results and preventative measures are sent to the device and presented to the user. Providing information in real time via a display or voice notification encourages prompt action.

[0194] Hardware and software used

[0195] Hardware:

[0196] Camera (installed inside the vehicle)

[0197] Server (for analysis and data management)

[0198] Terminal (for user interface)

[0199] software:

[0200] Near miss database

[0201] Large-scale Language Models (LLMs)

[0202] Image Preprocessing Algorithms

[0203] Specific examples

[0204] Example: Rear seat luggage management

[0205] For example, if luggage is left in the back seat, the system will focus on this luggage and identify the risk of harm to passengers in the event of sudden braking, etc. It will then suggest specific preventative measures to the user, such as "Please move the luggage to the trunk."

[0206] Prompt Sentence Examples

[0207] "Identify potential hazards inside the vehicle in this image and suggest preventative measures to notify the occupants."

[0208] This will significantly improve safety inside self-driving vehicles and increase passenger confidence.

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Step 1: System initialization

[0211] The server loads the near-miss database and large-scale language model (LLM) into memory. This makes the data and models necessary for analysis immediately available. The terminal also starts the camera and switches it to image acquisition mode. This prepares the camera to capture images. The inputs are the near-miss database, LLM, and camera start commands, and the output is the system's overall ready state.

[0212] Step 2: Image acquisition

[0213] The user selects a specific area in the vehicle (e.g., the back seat) and takes an image with the camera. At this time, the device acquires the captured image and sends it to the server. The input is the area in the vehicle selected by the user and the image acquired through the camera, and the output is the image data sent to the server.

[0214] Step 3: Image preprocessing

[0215] The server preprocesses the received image data. First, it removes noise from the image and adjusts the resolution. Then it resizes and normalizes the image to an appropriate size so that the LLM can analyze it efficiently. The input is the image data sent from the terminal, and the output is the preprocessed image data.

[0216] Step 4: Image analysis

[0217] The server inputs the preprocessed image data into a large-scale language model (LLM). The LLM uses information learned from a near-miss database to identify and analyze risk elements in the image. The input is the preprocessed image data, and the output is the result of the risk elements analyzed by the LLM.

[0218] Step 5: Accident prediction and prevention measures

[0219] The server generates accident predictions and preventive measures based on the analysis results obtained from the LLM. For example, if it is determined that there is a risk that luggage placed in the back seat will fly off during sudden braking, it generates the prediction result along with the preventive measure of "moving the luggage to the trunk." The input is the analysis result from the LLM, and the output is the specific prediction result and preventive measures.

[0220] Step 6: View the results

[0221] The server sends the generated prediction results and preventive measures to the device, which then presents them to the user. At this time, information can be provided in real time through a display or voice notification. The input is the prediction results and preventive measures sent from the server, and the output is the information presented to the user.

[0222] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0223] The present invention relates to an automatic near-miss prevention system that analyzes images of the home, predicts accidents, and suggests preventive measures, and also to a system that combines an emotion engine that recognizes the user's emotions. This system not only predicts potential accidents that may occur in the home in advance and suggests preventive measures using a near-miss database and a large-scale language model (LLM), but also improves the user experience by suggesting appropriate measures taking the user's emotional state into consideration. An embodiment of the present invention will be described in detail below.

[0224] System Overview

[0225] This system consists of the following main components:

[0226] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0227] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0228] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[0229] 4. Emotion engine: This engine analyzes emotions from the user's voice and facial expressions, and adjusts the predictions and preventive measures presented by the system based on the analysis results.

[0230] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[0231] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[0232] Program processing

[0233] 1. System initialization

[0234] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes it. The emotion engine is also started at the same time. This completes the preparation for the system to analyze and make predictions.

[0235] The device starts the camera and switches it to image capture mode. It displays a message that the camera was successfully started and prompts the user to perform an operation.

[0236] 2. Image Acquisition

[0237] The user selects the room or specific location (for example, the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[0238] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[0239] 3. Sending images

[0240] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[0241] 4. Image Analysis

[0242] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[0243] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[0244] 5. Emotion analysis

[0245] The server inputs the user's voice input and the user's facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[0246] 6. Accident prediction and prevention measures

[0247] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model and the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventive measure such as "keep flammable materials at a certain distance away from the stove." Based on the results of the emotion engine, if the user is nervous, the server provides concise, explicit instructions, and if the user is relaxed, it provides detailed explanations.

[0248] 7. Displaying the results

[0249] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[0250] The device displays the received accident prediction results and prevention measures to the user in a format that is easy for the user to check. The display format and content are adjusted based on the analysis results of the emotion engine.

[0251] Specific examples

[0252] Examples of areas around the kitchen stove

[0253] 1. System initialization

[0254] The server loads the near-miss database and the LLM model, starts the emotion engine, and is ready to go.

[0255] The terminal activates the camera and switches to image capture mode.

[0256] 2. Image Acquisition

[0257] The user selects the area around the kitchen stove and takes an image with the camera.

[0258] The terminal transmits the captured image of the kitchen to the server.

[0259] 3. Sending images

[0260] The terminal transmits image data to the server, and the server confirms receipt of the data.

[0261] 4. Image Analysis

[0262] The server preprocesses the received images and then inputs them into the LLM model.

[0263] The LLM model detects the presence of flammable materials near a stove and predicts the risk of fire.

[0264] 5. Emotion analysis

[0265] The server inputs the user's voice input and facial expression data into the emotion engine, which then analyzes that the user is nervous.

[0266] 6. Accident prediction and prevention measures

[0267] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[0268] 7. Displaying the results

[0269] The server transmits the generated prediction results and preventive measures to the terminal.

[0270] The terminal displays the received results to the user, displaying them as brief instructions.

[0271] This allows users to understand potential risks in their homes in advance and take appropriate measures. In particular, taking the user's emotional state into consideration enables more effective and prompt responses. This is a specific example of an embodiment of the present invention.

[0272] The processing flow will be explained below.

[0273] Step 1:

[0274] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes it. At the same time, it initializes the emotion engine and prepares for user emotion analysis. This completes the preparation of the entire system for analysis and prediction.

[0275] Step 2:

[0276] The device starts the camera and switches it to image capture mode. It displays a message that the camera was successfully started and prompts the user to perform an operation.

[0277] Step 3:

[0278] The user selects the room or specific location (such as the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[0279] Step 4:

[0280] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[0281] Step 5:

[0282] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[0283] Step 6:

[0284] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[0285] Step 7:

[0286] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[0287] Step 8:

[0288] The server inputs the user's voice input and the user's facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[0289] Step 9:

[0290] The server generates specific accident predictions and preventive measures based on the analysis results from the LLM and the emotion engine. For example, it predicts that there is a high risk of fire because of flammable materials near a stove, and generates a preventive measure to keep flammable materials at a certain distance away from the stove. Based on the results of the emotion engine, it provides concise and clear instructions if the user is nervous, or provides detailed explanations if the user is relaxed.

[0291] Step 10:

[0292] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[0293] Step 11:

[0294] The device displays the received accident prediction results and prevention measures to the user in a format that is easy for the user to check. The display format and content are adjusted based on the analysis results of the emotion engine.

[0295] Step 12:

[0296] Users can check the prediction results and preventive measures displayed on their device screen and implement the suggested preventive measures (e.g., moving flammable materials away from the stove), thereby effectively reducing the risk of accidents in the home.

[0297] Through these steps, the system of the present invention can detect potential risks in the home in advance and provide specific preventive measures to the user. In particular, by taking the user's emotional state into consideration, more effective and prompt responses are possible. This improves safety in the home and prevents accidents from occurring.

[0298] Example 2

[0299] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] Conventional home accident prevention systems focus on predicting risks and suggesting preventive measures, but rarely take into account the user's emotional state. As a result, if the user is nervous or distracted, the proposed preventive measures cannot be implemented properly, and the risk of an accident still remains.

[0301] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for reading a near-miss database, means for analyzing received images using a large-scale language model (LLM), means for using an emotion engine that analyzes the user's emotional state, means for generating predicted accidents and preventive measures based on the analysis results and emotion analysis results, and means for outputting the generated accident prediction results and preventive measures to a terminal. This makes it possible to predict the risk of domestic accidents in advance and present appropriate preventive measures according to the user's emotional state.

[0302] The "near miss database" is a database that collects and records accident cases that occur within the home, and is an information source used for training large-scale language models (LLMs).

[0303] A "large-scale language model (LLM)" is a model learned from a near-miss database and is an artificial intelligence system for analyzing received images.

[0304] The "emotion engine" is a system that analyzes the user's voice input and facial expression data to determine the user's emotional state.

[0305] The "analysis results" are risk assessments and prediction information generated by a large-scale language model (LLM) based on the image data received.

[0306] The "emotion analysis result" is evaluation information of the emotional state extracted by the emotion engine from the user's voice and facial expression.

[0307] The "accident prediction result" is information on the risk of accidents within the home predicted based on the analysis results.

[0308] "Preventive measures" are specific countermeasures and guidelines for action proposed based on the results of accident predictions.

[0309] A "terminal" is a device used by a user, which includes a camera and a display and provides an interface for displaying instructions and information from the system.

[0310] A "camera" is a photographing device that takes pictures of the home and transmits them to the system.

[0311] The "server" is the main computing unit that performs various processes such as image analysis, emotion analysis, accident prediction, and prevention measures generation.

[0312] This invention relates to an automatic near-miss prevention system that analyzes images from within the home, predicts accidents, and suggests preventive measures, and also to a system that combines an emotion engine that recognizes the user's emotions. This system not only predicts potential accidents that may occur within the home and suggests preventive measures using a near-miss database and a large-scale language model (LLM), but also improves the user experience by suggesting appropriate measures taking into account the user's emotional state.

[0313] This system consists of the following main components:

[0314] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0315] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0316] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[0317] 4. Emotion engine: This engine analyzes emotions from the user's voice and facial expressions, and adjusts the predictions and preventive measures presented by the system based on the analysis results.

[0318] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[0319] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[0320] Program processing

[0321] During system initialization, the server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization. At the same time, the emotion engine is also started, completing preparations for the system for analysis and prediction. The device then starts the camera, switches to image acquisition mode, and prompts the user for operation.

[0322] To acquire images, the user selects the room or specific location (such as the area around the kitchen stove) they want to analyze and takes an image with the camera. The device converts the captured image data into data packets and prepares them to be sent to the server.

[0323] In image transmission, the device sends image data packets to the server, which then confirms receipt of the data. The received data undergoes preprocessing, such as noise removal and resolution adjustment, at the server. The preprocessed data is then input into a large-scale language model (LLM) to analyze risk factors.

[0324] In emotion analysis, the server inputs the user's voice and facial expressions captured by the camera into an emotion engine to analyze the user's emotional state. The analysis results are reflected in accident prediction and the generation of preventive measures.

[0325] To predict accidents and generate preventive measures, the server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM and the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventive measure such as "keep flammable materials at a certain distance away from the stove." Based on the results of the emotion engine, the server provides concise, explicit instructions if the user is nervous, and detailed explanations if the user is relaxed.

[0326] The generated prediction results and preventive measures are sent to the device and displayed to the user. The device adjusts the display format and content based on the analysis results of the emotion engine, and presents the results in a format that is easy for the user to check.

[0327] Specific examples

[0328] Examples of areas around the kitchen stove

[0329] 1. System initialization

[0330] The server loads the near-miss database and the LLM model and starts the emotion engine.

[0331] The terminal activates the camera and switches to image capture mode.

[0332] 2. Image Acquisition

[0333] The user selects the area around the kitchen stove and takes an image with the camera.

[0334] The terminal transmits the captured image data of the kitchen to the server.

[0335] 3. Sending images

[0336] The terminal transmits image data to the server, and the server confirms receipt of the data.

[0337] 4. Image Analysis

[0338] The server pre-processes the received images and then inputs them into the LLM.

[0339] LLM detects the presence of flammable materials near the stove and predicts the risk of fire.

[0340] 5. Emotion analysis

[0341] The server inputs the user's voice input and facial expression data into the emotion engine, which then analyzes that the user is nervous.

[0342] 6. Accident prediction and prevention measures

[0343] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[0344] 7. Displaying the results

[0345] The server transmits the generated prediction results and preventive measures to the terminal.

[0346] The terminal presents the received results to the user, displaying them as simple instructions.

[0347] Prompt Sentence Examples

[0348] Example: "Analyze images of the kitchen stove area to determine if there is a risk of fire and suggest preventative measures. Also, provide brief instructions if the user is nervous, or detailed instructions if the user is relaxed."

[0349] This allows users to identify potential risks in the home in advance and take appropriate measures. In particular, taking the user's emotional state into consideration enables more effective responses.

[0350] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0351] Step 1: System initialization

[0352] The server loads the near-miss database and the large-scale language model (LLM) into memory and initializes the system. The server then starts the emotion engine and prepares the system for analyzing the user's voice input and facial expression data.

[0353] Input: Near miss database, LLM model

[0354] Output: System initialized, emotion engine activated

[0355] Specific operations: Loads near-miss data and LLM into the server's internal data memory, executes the emotion engine script, and records in the system log that the user is ready to start using the system.

[0356] Step 2: Image acquisition

[0357] The device starts up the camera and switches to image capture mode. The user selects the room or location they want to analyze and takes an image with the camera.

[0358] Input: Camera

[0359] Output: Captured image data

[0360] Specific operation: The user operates the device's camera app and presses the image capture button. The camera takes an image of the specified location and saves the image data in the device's internal memory.

[0361] Step 3: Send image

[0362] The device converts the captured image data into data packets and sends them to the server, which then confirms receipt.

[0363] Input: Photographed image data

[0364] Output: Image data packet, acknowledgement

[0365] Specific operation: Image data is converted into packet format through the terminal's internal network interface and sent to the server. The server verifies the received data and returns a receipt confirmation response to the terminal.

[0366] Step 4: Image analysis

[0367] The server preprocesses the received image data, which includes noise removal, resolution adjustment, and image cropping if necessary. The preprocessed image data is then input into a large-scale language model (LLM) to analyze risk factors.

[0368] Input: Received image data

[0369] Output: Analysis results

[0370] Specific operation: After using image processing software to remove noise, optimize the image resolution, and crop only the necessary parts, the preprocessed image is input into LLM, which analyzes the risk factors in the image and stores the analysis results in the server's memory.

[0371] Step 5: Sentiment Analysis

[0372] The server receives the user's voice input and facial expression data captured by a camera, and analyzes them using an emotion engine to determine the user's emotional state.

[0373] Input: Voice data, facial expression data

[0374] Output: Emotion analysis results

[0375] Specific operation: The server feeds the voice data acquired from the user and facial expression data captured by the camera to the emotion engine, which performs emotion analysis based on the tone, pitch, and extension characteristics of the voice and the movement of facial muscles. The obtained emotion analysis results are stored on the server.

[0376] Step 6: Accident prediction and prevention measures

[0377] The server generates accident predictions and preventative measures based on the image analysis results from the LLM and the emotion analysis results from the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventative measure such as "keep flammable materials at a certain distance away from the stove." The instructions are adjusted according to the user's emotional state.

[0378] Input: Analysis results, emotion analysis results

[0379] Output: Accident prediction results, prevention measures

[0380] Specific operation: The processing unit inside the server integrates the results of the LLM and the emotion engine, evaluates the risk level, and then generates appropriate prevention measures. It outputs short and clear instructions to users who are nervous, and more detailed explanations to users who are relaxed.

[0381] Step 7: View the results

[0382] The server sends the generated accident prediction results and preventive measures to the terminal, which then displays them to the user. The display format and content are adjusted based on the user's emotional state.

[0383] Input: Accident prediction results, prevention measures

[0384] Output: What the user sees

[0385] Specific operation: The server converts the generated results into packet format and sends them to the terminal. The terminal interprets the received data and displays it to the user in an appropriate format. For example, the control panel may display a message saying, "There are flammable materials near the stove. There is a high risk of fire. Please move the flammable materials away."

[0386] This allows the system to take into account the user's emotional state, detect hidden risks in the home early, and provide appropriate preventive measures, thereby effectively preventing accidents in the home.

[0387] (Application example 2)

[0388] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0389] It is important not only to predict and prevent potential accidents in the home, but also to present appropriate countermeasures that take the user's emotional state into account. However, conventional systems have had the problem of insufficient safety measures that take the user's emotional state into account, resulting in a lack of improvement in the user experience. The present invention aims to solve these problems and provide more effective accident prediction and prevention measures.

[0390] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for reading a near-miss incident database, means for analyzing received images using a large-scale language model (LLM), means for generating predicted accidents and preventive measures based on the analysis results, means for outputting the generated accident prediction results and preventive measures, means for analyzing the user's emotions, and means for adjusting the prediction results and measures based on the emotion analysis results. This makes it possible to provide appropriate accident predictions and preventive measures that take into account the user's emotional state.

[0391] The "near miss database" is a database that collects near miss cases that have occurred in the past, and records various accident cases that occurred in the home or on mobile vehicles.

[0392] A "large-scale language model (LLM)" is an advanced machine learning model that learns from massive amounts of text and image data, and is used to analyze received images and predict risks.

[0393] "Means for generating predicted accidents and preventive measures based on the analysis results" refers to a means by which the server predicts potential risks based on the image analysis results and derives specific preventive measures to avoid those risks.

[0394] The "means for outputting the generated accident prediction results and preventive measures" refers to a means for displaying or notifying the user of the predicted accident risk and preventive measures on a device such as a terminal or display used by the user.

[0395] The "means for analyzing user emotions" refers to a software routine or model for analyzing user emotions from voice input or camera footage and obtaining the results.

[0396] "Means for adjusting prediction results and countermeasures based on emotion analysis results" refers to means for optimizing the presentation method and content of accident prediction results and preventive measures, taking into account the user's emotional state.

[0397] MODE FOR CARRYING OUT THE INVENTION

[0398] System Overview

[0399] The system of the present invention consists of the following main components:

[0400] 1. Near-miss database: This database collects past near-miss incidents, recording various accidents occurring within the home and on mobile devices.

[0401] 2. Large-scale language model (LLM): A model trained from a near-miss database that analyzes received images and predicts risks.

[0402] 3. Image acquisition means: A camera and its operating interface for acquiring images of the home or moving objects.

[0403] 4. Emotion Engine: This engine analyzes the user's emotions from their voice and facial expressions. The results of the emotion analysis are used to adjust the predictions and preventative measures presented by the system.

[0404] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[0405] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[0406] Program processing

[0407] The system of the present invention performs processing in the following steps.

[0408] System initialization

[0409] The server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization. The emotion engine is also started at the same time. This completes preparation for the system for analysis and prediction. The device starts the camera and switches to image acquisition mode. A message is displayed indicating that the camera was successfully started, prompting the user to perform an operation.

[0410] Image acquisition

[0411] The user selects the room or specific location they want to analyze (such as the living room or the area around the stove) and takes an image with the camera. The device converts the captured image data into data packets and prepares them to be sent to the server.

[0412] Image analysis

[0413] The server preprocesses the received image data. This includes denoising the image, adjusting the resolution, and cropping the image if necessary. The server then inputs the preprocessed image data into a large-scale language model (LLM). The LLM analyzes the risk elements in the image and generates a prediction result.

[0414] Emotion analysis

[0415] The server inputs the user's voice input and facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[0416] Accident prediction and prevention measures

[0417] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model and the emotion engine. For example, predictions such as "risk of falls in the living room" and "high risk of fire due to flammable objects near the stove" may be provided, along with preventive measures such as "rearranging furniture" and "keeping flammable objects a certain distance away from the stove." If the user is nervous, the server provides concise, explicit instructions, while if they are relaxed, it provides detailed explanations.

[0418] Displaying the results

[0419] The server sends the generated prediction results and preventive measures to the terminal. The terminal confirms that they have been received correctly. The terminal then displays the received accident prediction results and preventive measures to the user. The results are displayed in a format that is easy for the user to check, and the display format and content are adjusted based on the analysis results of the emotion engine.

[0420] Specific examples

[0421] Examples of areas around the kitchen stove

[0422] 1. System initialization

[0423] The server loads the near-miss database and LLM model, starts the emotion engine, and is ready to go.

[0424] The device will start the camera and switch to image capture mode.

[0425] 2. Image Acquisition

[0426] The user selects the area around the kitchen stove and takes a picture with the camera.

[0427] The device sends the captured image of the kitchen to the server.

[0428] 3. Image Analysis

[0429] The server preprocesses the received images and inputs them into the LLM model.

[0430] The LLM model detects the presence of flammable materials near the stove and predicts the risk of fire.

[0431] 4. Emotion analysis

[0432] The server inputs the user's voice input and facial expression data into an emotion engine and analyzes whether the user is nervous.

[0433] 5. Accident prediction and prevention measures

[0434] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[0435] 6. Displaying the results

[0436] The server transmits the generated prediction results and preventive measures to the terminal.

[0437] The terminal displays the received results to the user, displaying them as brief instructions.

[0438] Example prompts for generative AI models

[0439] "Analyze live video to detect potential risks, recognize the user's emotional state, and suggest appropriate countermeasures based on that."

[0440] The above is a detailed description of the embodiment of the present invention. This allows users to understand potential risks in their homes in advance and take appropriate measures. In particular, it is possible to provide appropriate accident predictions and prevention measures that take into account the emotional state of the user.

[0441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0442] Step 1:

[0443] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes the system. The emotion engine is also started at the same time. This completes preparation for the system to perform analysis and prediction. The input is the near-miss database, LLM, and the emotion engine program code, and the output is the initialized system state.

[0444] Step 2:

[0445] The device starts the camera and switches to image acquisition mode. It displays to the user that the camera was successfully started and prompts them to operate. The input is the camera start command, and the output is that the camera is ready to acquire images.

[0446] Step 3:

[0447] The user selects the room or specific area (e.g., living room, stove area) they want to analyze and takes an image with the camera. The input is the user's selection and the captured image, and the output is the image data captured by the camera.

[0448] Step 4:

[0449] The device converts the captured image data into data packets and prepares them for transmission to the server. The input is the captured image data, and the output is the data packets for transmission.

[0450] Step 5:

[0451] The terminal sends image data packets to the server. The server receives the data and prepares it for data analysis. The input is the image data packets sent from the terminal, and the output is the received image data ready for preprocessing.

[0452] Step 6:

[0453] The server pre-processes the received image data. Pre-processing includes image noise removal, resolution adjustment, and image cropping if necessary. The input is the received image data and the output is the pre-processed image data.

[0454] Step 7:

[0455] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes the risk elements in the image and generates prediction results. The input is the preprocessed image data, and the output is the risk elements and their prediction results.

[0456] Step 8:

[0457] The server receives the user's voice input and facial expressions captured by a camera, and inputs them into the emotion engine. The emotion engine analyzes the user's emotional state and obtains the results. The input is voice data and facial expression data, and the output is the analyzed emotional state.

[0458] Step 9:

[0459] The server generates specific accident predictions and prevention measures based on the analysis results from the LLM model and the emotion engine. The inputs are risk prediction results and the user's emotional state, and the output is optimized accident predictions and prevention measures.

[0460] Step 10:

[0461] The server sends the generated prediction results and prevention measures to the terminal. The input is the optimized accident prediction and prevention measures, and the output is the transmitted data.

[0462] Step 11:

[0463] The device displays the received accident prediction results and prevention measures to the user. The input is the prediction results and prevention measures sent from the server, and the output is the content displayed to the user. The display format and content are adjusted based on the analysis results of the emotion engine, so that the user can easily check the results.

[0464] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0465] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0466] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0467] [Second embodiment]

[0468] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0469] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0470] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0471] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0472] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0473] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0474] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0475] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0478] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0479] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0480] The present invention relates to an automatic near-miss prevention system that analyzes images of a home, predicts accidents, and proposes preventive measures. This system combines a near-miss database with a large-scale language model (LLM) to predict potential accidents that may occur in the home and propose preventive measures. An embodiment of the present invention will be described in detail below.

[0481] System Overview

[0482] This system consists of the following main components:

[0483] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0484] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0485] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[0486] 4. Server: It is the main computing unit for performing image analysis and generating accident prediction and prevention measures.

[0487] 5. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results.

[0488] Program processing

[0489] 1. System initialization

[0490] The server loads the near-miss database and large-scale language model (LLM) into memory and completes preparation.

[0491] The terminal activates the camera and switches to image capture mode.

[0492] 2. Image Acquisition

[0493] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the camera.

[0494] The terminal transmits the captured image to the server.

[0495] 3. Image Analysis

[0496] The server preprocesses the received image data and inputs it into a large-scale language model (LLM). Preprocessing includes image denoising and resolution adjustment.

[0497] LLM analyzes risk factors within an image and generates a prediction result.

[0498] 4. Accident prediction and prevention measures

[0499] The server generates specific accident predictions and prevention measures based on the analysis results from the LLM.

[0500] For example, the prediction result is "there is a high risk of fire because there are flammable objects near the stove," and the preventive measure is "keep flammable objects a certain distance away from the stove."

[0501] 5. Displaying the results

[0502] The server transmits the generated prediction results and preventive measures to the terminal.

[0503] The terminal displays the received results on the screen for the user.

[0504] Specific examples

[0505] Examples of areas around the kitchen stove

[0506] 1. System initialization

[0507] The server loads the near miss database and LLM model and is ready to go.

[0508] The terminal activates the camera and switches to image capture mode.

[0509] 2. Image Acquisition

[0510] The user selects the area around the kitchen stove and takes an image with the camera.

[0511] The terminal transmits the captured image of the kitchen to the server.

[0512] 3. Image Analysis

[0513] The server preprocesses the received images and then inputs them into the LLM model.

[0514] The LLM model detects the presence of flammable materials near a stove and predicts the risk of fire.

[0515] 4. Accident prediction and prevention measures

[0516] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove."

[0517] 5. Displaying the results

[0518] The server transmits the generated prediction results and preventive measures to the terminal.

[0519] The terminal displays the received results to the user.

[0520] This allows the user to know in advance the risks that may be present in the home and take appropriate measures. This is a specific example of an embodiment of the present invention.

[0521] The processing flow will be explained below.

[0522] Step 1:

[0523] The server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization, which prepares the system for analysis and prediction.

[0524] Step 2:

[0525] The device starts the camera and prepares the user to capture images. It displays a message that the camera has been successfully started and prompts the user to perform an operation.

[0526] Step 3:

[0527] The user selects the room or specific location (such as the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[0528] Step 4:

[0529] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[0530] Step 5:

[0531] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[0532] Step 6:

[0533] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[0534] Step 7:

[0535] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[0536] Step 8:

[0537] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model. For example, the server may predict that "there is a high risk of fire due to flammable materials near the stove" and generate preventive measures such as "keep flammable materials at a certain distance away from the stove."

[0538] Step 9:

[0539] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[0540] Step 10:

[0541] The terminal displays the received accident prediction results and prevention measures to the user in a format that makes it easy for the user to check the results.

[0542] Step 11:

[0543] The user checks the prediction results and preventive measures displayed on the device screen and implements the proposed preventive measures, such as moving flammable materials away from the stove.

[0544] Through these steps, the system of the present invention can detect potential risks in the home in advance and provide users with specific preventative measures, thereby improving safety in the home and preventing accidents from occurring.

[0545] Example 1

[0546] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0547] A wide variety of accidents can occur in the home, and preventing them requires quick and accurate judgment. Especially in households with elderly people or children, even minor near-misses can lead to serious accidents. Therefore, there is a need for a system that can analyze potential home risks in real time and quickly present predictions and preventative measures. However, current technology does not adequately predict accidents or present preventative measures. Furthermore, there are issues with the system, such as complex operation for end users and low accuracy.

[0548] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0549] In this invention, the server includes means for reading the near-miss database, means for analyzing received images using a large-scale language model (LLM), means for generating predicted accidents and preventive measures based on the analysis results, means for receiving image data from an image acquisition device and performing preprocessing, means for inputting the preprocessed image data into the large-scale language model, and means for notifying the user of the prediction results and preventive measures. This makes it possible to analyze risks that may occur in the home in real time with high accuracy and quickly predict accidents and present preventive measures.

[0550] The "near miss database" is a database that collects and records past accident and risk cases that have occurred within the home.

[0551] A "large-scale language model (LLM)" is a model based on natural language processing technology that is trained from large amounts of text data and is used for image analysis and text generation.

[0552] "Image capture device" refers to a hardware device, such as a camera or sensor, used to capture images in the home.

[0553] "Preprocessing" refers to a series of processes that involve performing noise removal, resolution adjustment, etc. on image data to prepare it in a format suitable for analysis.

[0554] "Prediction results" refers to risk assessments and information indicating the possibility of an accident occurring obtained as a result of image analysis.

[0555] "Preventive measures" refers to instructions and advice to propose specific countermeasures and actions based on predicted accident risks.

[0556] "User notification means" refers to the interface or protocol for informing users of the analysis results and preventive measures.

[0557] The present invention relates to an automatic near-miss prevention system that analyzes images of a home, predicts accidents, and proposes preventive measures. This system combines a near-miss database with a large-scale language model (LLM) to predict potential accidents that may occur in the home and propose preventive measures. An embodiment of the present invention will be described in detail below.

[0558] System Overview

[0559] This system consists of the following main components:

[0560] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0561] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0562] 3. Image capture device: A camera for capturing images in the home and its operation interface.

[0563] 4. Server: This is the main computational unit for performing image analysis and generating accident predictions and prevention measures. The server is responsible for reading the near-miss incident database, preprocessing the received image data, and inputting it into the large-scale language model (LLM).

[0564] 5. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results.

[0565] The main processing of this system is as follows.

[0566] System initialization

[0567] The server reads the initial configuration file, loads the near-miss database, and deploys a large-scale language model (e.g., GPT-4) in memory to prepare for analysis. Meanwhile, the device starts the camera and prepares to switch to image acquisition mode.

[0568] Image acquisition and transmission

[0569] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the device's camera. The captured image is then sent from the device to the server using a secure protocol (for example, HTTPS).

[0570] Image analysis

[0571] The server performs preprocessing on the received image data, including noise removal and resolution adjustment. The preprocessed image data is then input into a large-scale language model (LLM), which analyzes risk factors and generates prediction results. For example, it can detect the presence of flammable materials near a stove and predict a high risk of fire.

[0572] Accident prediction and prevention measures

[0573] The server generates specific accident predictions and prevention measures as text based on the output of the LLM. For example, in response to a prediction that "there is a high risk of fire because there are flammable objects near the stove," it generates a prevention measure such as "keep flammable objects a certain distance away from the stove."

[0574] Displaying the results

[0575] The server sends the generated prediction results and preventive measures to the terminal, which displays them on the user interface. The user can check the displayed results and take necessary measures.

[0576] Specific examples

[0577] Analysis example around a kitchen stove

[0578] After initializing the system, the user takes a picture of the area around the kitchen stove with a camera, and the device sends the image data to the server. The server preprocesses the image data and analyzes risk elements in the image using a large-scale language model (LLM). As a result, it predicts that there are flammable objects near the stove and determines that there is a fire risk. The server then generates a preventive measure, such as "keep flammable objects a certain distance away from the stove," and sends it to the device. The device displays this to the user, who can then take appropriate measures.

[0579] Prompt Sentence Examples

[0580] "We would like you to analyze images from within a home and propose ways to predict and prevent accidents. Please use the near-miss database and LLM model below to provide examples and explain.

[0581] 1. Near miss database: Examples of accidents that may occur in the home (e.g., falls, fires, electric shocks, etc.).

[0582] 2. LLM model: Analyzes risk factors within the home, predicts accidents, and proposes preventive measures.

[0583] Examples:

[0584] After analyzing images of the area around the kitchen stove, it was determined that there was a high risk of fire.

[0585] As a preventative measure, they suggest "keeping flammable materials away from the stove."

[0586] Under the above conditions, please generate the image analysis results and countermeasures for your home.

[0587] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0588] Step 1: Initialize the system

[0589] The server reads the initial configuration file and loads the near-miss database and large-scale language model (LLM) into memory. During this process, a database containing past near-miss cases is loaded into the server. The large-scale language model (e.g., GPT-4) is initialized based on these databases. The input is the initial configuration file, and the output is the database and model loaded into memory.

[0590] Step 2: Launch the camera

[0591] The device sends a camera start command and switches to image capture mode. The user can confirm that the camera is ready through the device interface. The input is the camera start command, and the output is the camera in image capture mode.

[0592] Step 3: Capture an image

[0593] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the device's camera. Image data is captured by pressing the capture button. The input is the area selected by the user's operation, and the output is the captured image.

[0594] Step 4: Sending image data

[0595] The terminal compresses the captured image data and sends it to the server using a secure protocol (e.g., HTTPS). The input is the captured image data, and the output is the compressed data sent to the server.

[0596] Step 5: Preprocessing the image data

[0597] The server performs preprocessing on the received image data, such as noise reduction and resolution adjustment. The preprocessed image data is input into a large-scale language model (LLM). The input is the received image data, and the output is the preprocessed image data.

[0598] Step 6: Image analysis

[0599] The server inputs the preprocessed image data into a large-scale language model (LLM) to analyze risk factors. The model performs a risk assessment based on the input image and generates a prediction result. For example, it detects the presence of flammable materials near a stove and predicts a high risk of fire. The input is the preprocessed image data, and the output is a risk assessment and prediction result.

[0600] Step 7: Accident prediction and prevention measures

[0601] The server generates specific accident predictions and preventive measures based on the output of the LLM. For example, in response to a prediction result that "there is a high risk of fire because flammable objects are near the stove," the server generates a preventive measure as a text message, such as "keep flammable objects a certain distance away from the stove." The input is the analysis result, and the output is a text message containing the preventive measures.

[0602] Step 8: Sending the results

[0603] The server encodes the generated prediction results and prevention measures and sends them to the terminal. The input is the generated text data, and the output is the encoded data sent to the terminal.

[0604] Step 9: View the results

[0605] The terminal decodes the received data and displays it on the user interface. The user can check the displayed results and take necessary measures. The input is the received encoded data, and the output is the predicted results and preventative measures displayed to the user.

[0606] (Application example 1)

[0607] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0608] Unforeseen safety risks may occur inside an autonomous vehicle due to the placement of passengers and luggage. While conventional vehicle safety systems have made some progress in addressing risks in the external environment, they are insufficient in addressing risks inside the vehicle. Therefore, there is a need for a system that can monitor the situation inside the vehicle in real time, predict potential risks, and quickly propose preventative measures.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0610] In this invention, the server includes means for reading the near-miss database, means for capturing received images of the inside of the vehicle with a camera, means for preprocessing the captured images and inputting them into a large-scale language model (LLM), means for generating accident predictions and preventive measures based on the analysis results, and means for outputting the generated prediction results and preventive measures. This makes it possible to predict safety risks inside the vehicle in real time and quickly present appropriate preventive measures.

[0611] The "near miss database" is a database that collects and manages various accident and risk cases that have occurred in the past.

[0612] A "large-scale language model (LLM)" is an artificial intelligence model that learns from large amounts of text data and has advanced natural language processing capabilities.

[0613] A "received image" or "captured image" refers to image data captured using a device such as a camera.

[0614] "Preprocessing" is the process of removing noise from data, adjusting resolution, and preparing the data in an appropriate format before using it for image analysis.

[0615] "Analysis results" refers to information generated as a result of analyzing image data using a large-scale language model (LLM).

[0616] "Accident prediction" refers to estimating accidents and risks that may occur in the future based on the results of analysis.

[0617] "Preventive measures" are specific methods and means for preventing predicted risks and accidents.

[0618] "Inside the vehicle" refers to the area inside the cabin of an autonomous vehicle, i.e., the space where passengers sit.

[0619] A "camera" is a photographing device for capturing visual information and recording it as image data.

[0620] A "server" is the main computing unit that processes the entire system and manages and analyzes data.

[0621] "Means" are the functions or methods used by the system to achieve a particular purpose.

[0622] A specific system for implementing this invention is configured as follows: The server reads the near-miss incident database and uses a large-scale language model (LLM) to analyze the received images of the vehicle interior. A camera is used to capture images of the vehicle interior in real time, which are preprocessed and then input into the LLM. Accident predictions and corresponding preventive measures are generated based on the analysis results, and this information is output to a terminal.

[0623] Program processing

[0624] System initialization

[0625] The server loads the near-miss database and large-scale language model (LLM) into memory, preparing the system for quick access to the information needed for analysis. The device also activates the camera and switches it to image acquisition mode.

[0626] Image acquisition

[0627] The user selects a specific area in the vehicle (e.g., the back seat) and takes an image with the camera. The captured image is then sent to the server via the terminal.

[0628] Image preprocessing

[0629] The server preprocesses the received image data, including removing image noise and adjusting the resolution, especially by resizing and normalizing the image so that the LLM can efficiently analyze the input data.

[0630] Image analysis

[0631] The pre-processed image data is input into a large-scale language model (LLM) to analyze risk factors. The LLM determines whether a particular situation inside the vehicle is dangerous based on information learned from a near-miss database.

[0632] Accident prediction and prevention measures

[0633] The server uses the analysis results from the LLM to predict accidents and generate preventative measures. For example, if it determines that there is a risk of luggage being thrown off in the back seat during sudden braking, it will suggest a preventative measure such as "moving the luggage to the trunk."

[0634] Displaying the results

[0635] The generated prediction results and preventative measures are sent to the device and presented to the user. Providing information in real time via a display or voice notification encourages prompt action.

[0636] Hardware and software used

[0637] Hardware:

[0638] Camera (installed inside the vehicle)

[0639] Server (for analysis and data management)

[0640] Terminal (for user interface)

[0641] software:

[0642] Near miss database

[0643] Large-scale Language Models (LLMs)

[0644] Image Preprocessing Algorithms

[0645] Specific examples

[0646] Example: Rear seat luggage management

[0647] For example, if luggage is left in the back seat, the system will focus on this luggage and identify the risk of harm to passengers in the event of sudden braking, etc. It will then suggest specific preventative measures to the user, such as "Please move the luggage to the trunk."

[0648] Prompt Sentence Examples

[0649] "Identify potential hazards inside the vehicle in this image and suggest preventative measures to notify the occupants."

[0650] This will significantly improve safety inside self-driving vehicles and increase passenger confidence.

[0651] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0652] Step 1: System initialization

[0653] The server loads the near-miss database and large-scale language model (LLM) into memory. This makes the data and models necessary for analysis immediately available. The terminal also starts the camera and switches it to image acquisition mode. This prepares the camera to capture images. The inputs are the near-miss database, LLM, and camera start commands, and the output is the system's overall ready state.

[0654] Step 2: Image acquisition

[0655] The user selects a specific area in the vehicle (e.g., the back seat) and takes an image with the camera. At this time, the device acquires the captured image and sends it to the server. The input is the area in the vehicle selected by the user and the image acquired through the camera, and the output is the image data sent to the server.

[0656] Step 3: Image preprocessing

[0657] The server preprocesses the received image data. First, it removes noise from the image and adjusts the resolution. Then it resizes and normalizes the image to an appropriate size so that the LLM can analyze it efficiently. The input is the image data sent from the terminal, and the output is the preprocessed image data.

[0658] Step 4: Image analysis

[0659] The server inputs the preprocessed image data into a large-scale language model (LLM). The LLM uses information learned from a near-miss database to identify and analyze risk elements in the image. The input is the preprocessed image data, and the output is the result of the risk elements analyzed by the LLM.

[0660] Step 5: Accident prediction and prevention measures

[0661] The server generates accident predictions and preventive measures based on the analysis results obtained from the LLM. For example, if it is determined that there is a risk that luggage placed in the back seat will fly off during sudden braking, it generates the prediction result along with the preventive measure of "moving the luggage to the trunk." The input is the analysis result from the LLM, and the output is the specific prediction result and preventive measures.

[0662] Step 6: View the results

[0663] The server sends the generated prediction results and preventive measures to the device, which then presents them to the user. At this time, information can be provided in real time through a display or voice notification. The input is the prediction results and preventive measures sent from the server, and the output is the information presented to the user.

[0664] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0665] The present invention relates to an automatic near-miss prevention system that analyzes images of the home, predicts accidents, and suggests preventive measures, and also to a system that combines an emotion engine that recognizes the user's emotions. This system not only predicts potential accidents that may occur in the home in advance and suggests preventive measures using a near-miss database and a large-scale language model (LLM), but also improves the user experience by suggesting appropriate measures taking the user's emotional state into consideration. An embodiment of the present invention will be described in detail below.

[0666] System Overview

[0667] This system consists of the following main components:

[0668] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0669] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0670] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[0671] 4. Emotion engine: This engine analyzes emotions from the user's voice and facial expressions, and adjusts the predictions and preventive measures presented by the system based on the analysis results.

[0672] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[0673] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[0674] Program processing

[0675] 1. System initialization

[0676] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes it. The emotion engine is also started at the same time. This completes the preparation for the system to analyze and make predictions.

[0677] The device starts the camera and switches it to image capture mode. It displays a message that the camera was successfully started and prompts the user to perform an operation.

[0678] 2. Image Acquisition

[0679] The user selects the room or specific location (for example, the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[0680] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[0681] 3. Sending images

[0682] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[0683] 4. Image Analysis

[0684] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[0685] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[0686] 5. Emotion analysis

[0687] The server inputs the user's voice input and the user's facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[0688] 6. Accident prediction and prevention measures

[0689] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model and the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventive measure such as "keep flammable materials at a certain distance away from the stove." Based on the results of the emotion engine, if the user is nervous, the server provides concise, explicit instructions, and if the user is relaxed, it provides detailed explanations.

[0690] 7. Displaying the results

[0691] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[0692] The device displays the received accident prediction results and prevention measures to the user in a format that is easy for the user to check. The display format and content are adjusted based on the analysis results of the emotion engine.

[0693] Specific examples

[0694] Examples of areas around the kitchen stove

[0695] 1. System initialization

[0696] The server loads the near-miss database and the LLM model, starts the emotion engine, and is ready to go.

[0697] The terminal activates the camera and switches to image capture mode.

[0698] 2. Image Acquisition

[0699] The user selects the area around the kitchen stove and takes an image with the camera.

[0700] The terminal transmits the captured image of the kitchen to the server.

[0701] 3. Sending images

[0702] The terminal transmits image data to the server, and the server confirms receipt of the data.

[0703] 4. Image Analysis

[0704] The server preprocesses the received images and then inputs them into the LLM model.

[0705] The LLM model detects the presence of flammable materials near a stove and predicts the risk of fire.

[0706] 5. Emotion analysis

[0707] The server inputs the user's voice input and facial expression data into the emotion engine, which then analyzes that the user is nervous.

[0708] 6. Accident prediction and prevention measures

[0709] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[0710] 7. Displaying the results

[0711] The server transmits the generated prediction results and preventive measures to the terminal.

[0712] The terminal displays the received results to the user, displaying them as brief instructions.

[0713] This allows users to understand potential risks in their homes in advance and take appropriate measures. In particular, taking the user's emotional state into consideration enables more effective and prompt responses. This is a specific example of an embodiment of the present invention.

[0714] The processing flow will be explained below.

[0715] Step 1:

[0716] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes it. At the same time, it initializes the emotion engine and prepares for user emotion analysis. This completes the preparation of the entire system for analysis and prediction.

[0717] Step 2:

[0718] The device starts the camera and switches it to image capture mode. It displays a message that the camera was successfully started and prompts the user to perform an operation.

[0719] Step 3:

[0720] The user selects the room or specific location (such as the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[0721] Step 4:

[0722] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[0723] Step 5:

[0724] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[0725] Step 6:

[0726] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[0727] Step 7:

[0728] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[0729] Step 8:

[0730] The server inputs the user's voice input and the user's facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[0731] Step 9:

[0732] The server generates specific accident predictions and preventive measures based on the analysis results from the LLM and the emotion engine. For example, it predicts that there is a high risk of fire because of flammable materials near a stove, and generates a preventive measure to keep flammable materials at a certain distance away from the stove. Based on the results of the emotion engine, it provides concise and clear instructions if the user is nervous, or provides detailed explanations if the user is relaxed.

[0733] Step 10:

[0734] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[0735] Step 11:

[0736] The device displays the received accident prediction results and prevention measures to the user in a format that is easy for the user to check. The display format and content are adjusted based on the analysis results of the emotion engine.

[0737] Step 12:

[0738] Users can check the prediction results and preventive measures displayed on their device screen and implement the suggested preventive measures (e.g., moving flammable materials away from the stove), thereby effectively reducing the risk of accidents in the home.

[0739] Through these steps, the system of the present invention can detect potential risks in the home in advance and provide specific preventive measures to the user. In particular, by taking the user's emotional state into consideration, more effective and prompt responses are possible. This improves safety in the home and prevents accidents from occurring.

[0740] Example 2

[0741] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] Conventional home accident prevention systems focus on predicting risks and suggesting preventive measures, but rarely take into account the user's emotional state. As a result, if the user is nervous or distracted, the proposed preventive measures cannot be implemented properly, and the risk of an accident still remains.

[0743] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for reading a near-miss database, means for analyzing received images using a large-scale language model (LLM), means for using an emotion engine that analyzes the user's emotional state, means for generating predicted accidents and preventive measures based on the analysis results and emotion analysis results, and means for outputting the generated accident prediction results and preventive measures to a terminal. This makes it possible to predict the risk of domestic accidents in advance and present appropriate preventive measures according to the user's emotional state.

[0744] The "near miss database" is a database that collects and records accident cases that occur within the home, and is an information source used for training large-scale language models (LLMs).

[0745] A "large-scale language model (LLM)" is a model learned from a near-miss database and is an artificial intelligence system for analyzing received images.

[0746] The "emotion engine" is a system that analyzes the user's voice input and facial expression data to determine the user's emotional state.

[0747] The "analysis results" are risk assessments and prediction information generated by a large-scale language model (LLM) based on the image data received.

[0748] The "emotion analysis result" is evaluation information of the emotional state extracted by the emotion engine from the user's voice and facial expression.

[0749] The "accident prediction result" is information on the risk of accidents within the home predicted based on the analysis results.

[0750] "Preventive measures" are specific countermeasures and guidelines for action proposed based on the results of accident predictions.

[0751] A "terminal" is a device used by a user, which includes a camera and a display and provides an interface for displaying instructions and information from the system.

[0752] A "camera" is a photographing device that takes pictures of the home and transmits them to the system.

[0753] The "server" is the main computing unit that performs various processes such as image analysis, emotion analysis, accident prediction, and prevention measures generation.

[0754] This invention relates to an automatic near-miss prevention system that analyzes images from within the home, predicts accidents, and suggests preventive measures, and also to a system that combines an emotion engine that recognizes the user's emotions. This system not only predicts potential accidents that may occur within the home and suggests preventive measures using a near-miss database and a large-scale language model (LLM), but also improves the user experience by suggesting appropriate measures taking into account the user's emotional state.

[0755] This system consists of the following main components:

[0756] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0757] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0758] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[0759] 4. Emotion engine: This engine analyzes emotions from the user's voice and facial expressions, and adjusts the predictions and preventive measures presented by the system based on the analysis results.

[0760] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[0761] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[0762] Program processing

[0763] During system initialization, the server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization. At the same time, the emotion engine is also started, completing preparations for the system for analysis and prediction. The device then starts the camera, switches to image acquisition mode, and prompts the user for operation.

[0764] To acquire images, the user selects the room or specific location (such as the area around the kitchen stove) they want to analyze and takes an image with the camera. The device converts the captured image data into data packets and prepares them to be sent to the server.

[0765] In image transmission, the device sends image data packets to the server, which then confirms receipt of the data. The received data undergoes preprocessing, such as noise removal and resolution adjustment, at the server. The preprocessed data is then input into a large-scale language model (LLM) to analyze risk factors.

[0766] In emotion analysis, the server inputs the user's voice and facial expressions captured by the camera into an emotion engine to analyze the user's emotional state. The analysis results are reflected in accident prediction and the generation of preventive measures.

[0767] To predict accidents and generate preventive measures, the server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM and the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventive measure such as "keep flammable materials at a certain distance away from the stove." Based on the results of the emotion engine, the server provides concise, explicit instructions if the user is nervous, and detailed explanations if the user is relaxed.

[0768] The generated prediction results and preventive measures are sent to the device and displayed to the user. The device adjusts the display format and content based on the analysis results of the emotion engine, and presents the results in a format that is easy for the user to check.

[0769] Specific examples

[0770] Examples of areas around the kitchen stove

[0771] 1. System initialization

[0772] The server loads the near-miss database and the LLM model and starts the emotion engine.

[0773] The terminal activates the camera and switches to image capture mode.

[0774] 2. Image Acquisition

[0775] The user selects the area around the kitchen stove and takes an image with the camera.

[0776] The terminal transmits the captured image data of the kitchen to the server.

[0777] 3. Sending images

[0778] The terminal transmits image data to the server, and the server confirms receipt of the data.

[0779] 4. Image Analysis

[0780] The server pre-processes the received images and then inputs them into the LLM.

[0781] LLM detects the presence of flammable materials near the stove and predicts the risk of fire.

[0782] 5. Emotion analysis

[0783] The server inputs the user's voice input and facial expression data into the emotion engine, which then analyzes that the user is nervous.

[0784] 6. Accident prediction and prevention measures

[0785] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[0786] 7. Displaying the results

[0787] The server transmits the generated prediction results and preventive measures to the terminal.

[0788] The terminal presents the received results to the user, displaying them as simple instructions.

[0789] Prompt Sentence Examples

[0790] Example: "Analyze images of the kitchen stove area to determine if there is a risk of fire and suggest preventative measures. Also, provide brief instructions if the user is nervous, or detailed instructions if the user is relaxed."

[0791] This allows users to identify potential risks in the home in advance and take appropriate measures. In particular, taking the user's emotional state into consideration enables more effective responses.

[0792] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0793] Step 1: System initialization

[0794] The server loads the near-miss database and the large-scale language model (LLM) into memory and initializes the system. The server then starts the emotion engine and prepares the system for analyzing the user's voice input and facial expression data.

[0795] Input: Near miss database, LLM model

[0796] Output: System initialized, emotion engine activated

[0797] Specific operations: Loads near-miss data and LLM into the server's internal data memory, executes the emotion engine script, and records in the system log that the user is ready to start using the system.

[0798] Step 2: Image acquisition

[0799] The device starts up the camera and switches to image capture mode. The user selects the room or location they want to analyze and takes an image with the camera.

[0800] Input: Camera

[0801] Output: Captured image data

[0802] Specific operation: The user operates the device's camera app and presses the image capture button. The camera takes an image of the specified location and saves the image data in the device's internal memory.

[0803] Step 3: Send image

[0804] The device converts the captured image data into data packets and sends them to the server, which then confirms receipt.

[0805] Input: Photographed image data

[0806] Output: Image data packet, acknowledgement

[0807] Specific operation: Image data is converted into packet format through the terminal's internal network interface and sent to the server. The server verifies the received data and returns a receipt confirmation response to the terminal.

[0808] Step 4: Image analysis

[0809] The server preprocesses the received image data, which includes noise removal, resolution adjustment, and image cropping if necessary. The preprocessed image data is then input into a large-scale language model (LLM) to analyze risk factors.

[0810] Input: Received image data

[0811] Output: Analysis results

[0812] Specific operation: After using image processing software to remove noise, optimize the image resolution, and crop only the necessary parts, the preprocessed image is input into LLM, which analyzes the risk factors in the image and stores the analysis results in the server's memory.

[0813] Step 5: Sentiment Analysis

[0814] The server receives the user's voice input and facial expression data captured by a camera, and analyzes them using an emotion engine to determine the user's emotional state.

[0815] Input: Voice data, facial expression data

[0816] Output: Emotion analysis results

[0817] Specific operation: The server feeds the voice data acquired from the user and facial expression data captured by the camera to the emotion engine, which performs emotion analysis based on the tone, pitch, and extension characteristics of the voice and the movement of facial muscles. The obtained emotion analysis results are stored on the server.

[0818] Step 6: Accident prediction and prevention measures

[0819] The server generates accident predictions and preventative measures based on the image analysis results from the LLM and the emotion analysis results from the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventative measure such as "keep flammable materials at a certain distance away from the stove." The instructions are adjusted according to the user's emotional state.

[0820] Input: Analysis results, emotion analysis results

[0821] Output: Accident prediction results, prevention measures

[0822] Specific operation: The processing unit inside the server integrates the results of the LLM and the emotion engine, evaluates the risk level, and then generates appropriate prevention measures. It outputs short and clear instructions to users who are nervous, and more detailed explanations to users who are relaxed.

[0823] Step 7: View the results

[0824] The server sends the generated accident prediction results and preventive measures to the terminal, which then displays them to the user. The display format and content are adjusted based on the user's emotional state.

[0825] Input: Accident prediction results, prevention measures

[0826] Output: What the user sees

[0827] Specific operation: The server converts the generated results into packet format and sends them to the terminal. The terminal interprets the received data and displays it to the user in an appropriate format. For example, the control panel may display a message saying, "There are flammable materials near the stove. There is a high risk of fire. Please move the flammable materials away."

[0828] This allows the system to take into account the user's emotional state, detect hidden risks in the home early, and provide appropriate preventive measures, thereby effectively preventing accidents in the home.

[0829] (Application example 2)

[0830] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0831] It is important not only to predict and prevent potential accidents in the home, but also to present appropriate countermeasures that take the user's emotional state into account. However, conventional systems have had the problem of insufficient safety measures that take the user's emotional state into account, resulting in a lack of improvement in the user experience. The present invention aims to solve these problems and provide more effective accident prediction and prevention measures.

[0832] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for reading a near-miss incident database, means for analyzing received images using a large-scale language model (LLM), means for generating predicted accidents and preventive measures based on the analysis results, means for outputting the generated accident prediction results and preventive measures, means for analyzing the user's emotions, and means for adjusting the prediction results and measures based on the emotion analysis results. This makes it possible to provide appropriate accident predictions and preventive measures that take into account the user's emotional state.

[0833] The "near miss database" is a database that collects near miss cases that have occurred in the past, and records various accident cases that occurred in the home or on mobile vehicles.

[0834] A "large-scale language model (LLM)" is an advanced machine learning model that learns from massive amounts of text and image data, and is used to analyze received images and predict risks.

[0835] "Means for generating predicted accidents and preventive measures based on the analysis results" refers to a means by which the server predicts potential risks based on the image analysis results and derives specific preventive measures to avoid those risks.

[0836] The "means for outputting the generated accident prediction results and preventive measures" refers to a means for displaying or notifying the user of the predicted accident risk and preventive measures on a device such as a terminal or display used by the user.

[0837] The "means for analyzing user emotions" refers to a software routine or model for analyzing user emotions from voice input or camera footage and obtaining the results.

[0838] "Means for adjusting prediction results and countermeasures based on emotion analysis results" refers to means for optimizing the presentation method and content of accident prediction results and preventive measures, taking into account the user's emotional state.

[0839] MODE FOR CARRYING OUT THE INVENTION

[0840] System Overview

[0841] The system of the present invention consists of the following main components:

[0842] 1. Near-miss database: This database collects past near-miss incidents, recording various accidents occurring within the home and on mobile devices.

[0843] 2. Large-scale language model (LLM): A model trained from a near-miss database that analyzes received images and predicts risks.

[0844] 3. Image acquisition means: A camera and its operating interface for acquiring images of the home or moving objects.

[0845] 4. Emotion Engine: This engine analyzes the user's emotions from their voice and facial expressions. The results of the emotion analysis are used to adjust the predictions and preventative measures presented by the system.

[0846] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[0847] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[0848] Program processing

[0849] The system of the present invention performs processing in the following steps.

[0850] System initialization

[0851] The server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization. The emotion engine is also started at the same time. This completes preparation for the system for analysis and prediction. The device starts the camera and switches to image acquisition mode. A message is displayed indicating that the camera was successfully started, prompting the user to perform an operation.

[0852] Image acquisition

[0853] The user selects the room or specific location they want to analyze (such as the living room or the area around the stove) and takes an image with the camera. The device converts the captured image data into data packets and prepares them to be sent to the server.

[0854] Image analysis

[0855] The server preprocesses the received image data. This includes denoising the image, adjusting the resolution, and cropping the image if necessary. The server then inputs the preprocessed image data into a large-scale language model (LLM). The LLM analyzes the risk elements in the image and generates a prediction result.

[0856] Emotion analysis

[0857] The server inputs the user's voice input and facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[0858] Accident prediction and prevention measures

[0859] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model and the emotion engine. For example, predictions such as "risk of falls in the living room" and "high risk of fire due to flammable objects near the stove" may be provided, along with preventive measures such as "rearranging furniture" and "keeping flammable objects a certain distance away from the stove." If the user is nervous, the server provides concise, explicit instructions, while if they are relaxed, it provides detailed explanations.

[0860] Displaying the results

[0861] The server sends the generated prediction results and preventive measures to the terminal. The terminal confirms that they have been received correctly. The terminal then displays the received accident prediction results and preventive measures to the user. The results are displayed in a format that is easy for the user to check, and the display format and content are adjusted based on the analysis results of the emotion engine.

[0862] Specific examples

[0863] Examples of areas around the kitchen stove

[0864] 1. System initialization

[0865] The server loads the near-miss database and LLM model, starts the emotion engine, and is ready to go.

[0866] The device will start the camera and switch to image capture mode.

[0867] 2. Image Acquisition

[0868] The user selects the area around the kitchen stove and takes a picture with the camera.

[0869] The device sends the captured image of the kitchen to the server.

[0870] 3. Image Analysis

[0871] The server preprocesses the received images and inputs them into the LLM model.

[0872] The LLM model detects the presence of flammable materials near the stove and predicts the risk of fire.

[0873] 4. Emotion analysis

[0874] The server inputs the user's voice input and facial expression data into an emotion engine and analyzes whether the user is nervous.

[0875] 5. Accident prediction and prevention measures

[0876] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[0877] 6. Displaying the results

[0878] The server transmits the generated prediction results and preventive measures to the terminal.

[0879] The terminal displays the received results to the user, displaying them as brief instructions.

[0880] Example prompts for generative AI models

[0881] "Analyze live video to detect potential risks, recognize the user's emotional state, and suggest appropriate countermeasures based on that."

[0882] The above is a detailed description of the embodiment of the present invention. This allows users to understand potential risks in their homes in advance and take appropriate measures. In particular, it is possible to provide appropriate accident predictions and prevention measures that take into account the emotional state of the user.

[0883] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0884] Step 1:

[0885] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes the system. The emotion engine is also started at the same time. This completes preparation for the system to perform analysis and prediction. The input is the near-miss database, LLM, and the emotion engine program code, and the output is the initialized system state.

[0886] Step 2:

[0887] The device starts the camera and switches to image acquisition mode. It displays to the user that the camera was successfully started and prompts them to operate. The input is the camera start command, and the output is that the camera is ready to acquire images.

[0888] Step 3:

[0889] The user selects the room or specific area (e.g., living room, stove area) they want to analyze and takes an image with the camera. The input is the user's selection and the captured image, and the output is the image data captured by the camera.

[0890] Step 4:

[0891] The device converts the captured image data into data packets and prepares them for transmission to the server. The input is the captured image data, and the output is the data packets for transmission.

[0892] Step 5:

[0893] The terminal sends image data packets to the server. The server receives the data and prepares it for data analysis. The input is the image data packets sent from the terminal, and the output is the received image data ready for preprocessing.

[0894] Step 6:

[0895] The server pre-processes the received image data. Pre-processing includes image noise removal, resolution adjustment, and image cropping if necessary. The input is the received image data and the output is the pre-processed image data.

[0896] Step 7:

[0897] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes the risk elements in the image and generates prediction results. The input is the preprocessed image data, and the output is the risk elements and their prediction results.

[0898] Step 8:

[0899] The server receives the user's voice input and facial expressions captured by a camera, and inputs them into the emotion engine. The emotion engine analyzes the user's emotional state and obtains the results. The input is voice data and facial expression data, and the output is the analyzed emotional state.

[0900] Step 9:

[0901] The server generates specific accident predictions and prevention measures based on the analysis results from the LLM model and the emotion engine. The inputs are risk prediction results and the user's emotional state, and the output is optimized accident predictions and prevention measures.

[0902] Step 10:

[0903] The server sends the generated prediction results and prevention measures to the terminal. The input is the optimized accident prediction and prevention measures, and the output is the transmitted data.

[0904] Step 11:

[0905] The device displays the received accident prediction results and prevention measures to the user. The input is the prediction results and prevention measures sent from the server, and the output is the content displayed to the user. The display format and content are adjusted based on the analysis results of the emotion engine, so that the user can easily check the results.

[0906] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0907] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0908] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0909] [Third embodiment]

[0910] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0911] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0912] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0913] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0914] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0915] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0916] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0917] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0918] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0919] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0920] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0921] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0922] The present invention relates to an automatic near-miss prevention system that analyzes images of a home, predicts accidents, and proposes preventive measures. This system combines a near-miss database with a large-scale language model (LLM) to predict potential accidents that may occur in the home and propose preventive measures. An embodiment of the present invention will be described in detail below.

[0923] System Overview

[0924] This system consists of the following main components:

[0925] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[0926] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[0927] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[0928] 4. Server: It is the main computing unit for performing image analysis and generating accident prediction and prevention measures.

[0929] 5. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results.

[0930] Program processing

[0931] 1. System initialization

[0932] The server loads the near-miss database and large-scale language model (LLM) into memory and completes preparation.

[0933] The terminal activates the camera and switches to image capture mode.

[0934] 2. Image Acquisition

[0935] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the camera.

[0936] The terminal transmits the captured image to the server.

[0937] 3. Image Analysis

[0938] The server preprocesses the received image data and inputs it into a large-scale language model (LLM). Preprocessing includes image denoising and resolution adjustment.

[0939] LLM analyzes risk factors within an image and generates a prediction result.

[0940] 4. Accident prediction and prevention measures

[0941] The server generates specific accident predictions and prevention measures based on the analysis results from the LLM.

[0942] For example, the prediction result is "there is a high risk of fire because there are flammable objects near the stove," and the preventive measure is "keep flammable objects a certain distance away from the stove."

[0943] 5. Displaying the results

[0944] The server transmits the generated prediction results and preventive measures to the terminal.

[0945] The terminal displays the received results on the screen for the user.

[0946] Specific examples

[0947] Examples of areas around the kitchen stove

[0948] 1. System initialization

[0949] The server loads the near miss database and LLM model and is ready to go.

[0950] The terminal activates the camera and switches to image capture mode.

[0951] 2. Image Acquisition

[0952] The user selects the area around the kitchen stove and takes an image with the camera.

[0953] The terminal transmits the captured image of the kitchen to the server.

[0954] 3. Image Analysis

[0955] The server preprocesses the received images and then inputs them into the LLM model.

[0956] The LLM model detects the presence of flammable materials near a stove and predicts the risk of fire.

[0957] 4. Accident prediction and prevention measures

[0958] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove."

[0959] 5. Displaying the results

[0960] The server transmits the generated prediction results and preventive measures to the terminal.

[0961] The terminal displays the received results to the user.

[0962] This allows the user to know in advance the risks that may be present in the home and take appropriate measures. This is a specific example of an embodiment of the present invention.

[0963] The processing flow will be explained below.

[0964] Step 1:

[0965] The server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization, which prepares the system for analysis and prediction.

[0966] Step 2:

[0967] The device starts the camera and prepares the user to capture images. It displays a message that the camera has been successfully started and prompts the user to perform an operation.

[0968] Step 3:

[0969] The user selects the room or specific location (such as the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[0970] Step 4:

[0971] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[0972] Step 5:

[0973] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[0974] Step 6:

[0975] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[0976] Step 7:

[0977] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[0978] Step 8:

[0979] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model. For example, the server may predict that "there is a high risk of fire due to flammable materials near the stove" and generate preventive measures such as "keep flammable materials at a certain distance away from the stove."

[0980] Step 9:

[0981] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[0982] Step 10:

[0983] The terminal displays the received accident prediction results and prevention measures to the user in a format that makes it easy for the user to check the results.

[0984] Step 11:

[0985] The user checks the prediction results and preventive measures displayed on the device screen and implements the proposed preventive measures, such as moving flammable materials away from the stove.

[0986] Through these steps, the system of the present invention can detect potential risks in the home in advance and provide users with specific preventative measures, thereby improving safety in the home and preventing accidents from occurring.

[0987] Example 1

[0988] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0989] A wide variety of accidents can occur in the home, and preventing them requires quick and accurate judgment. Especially in households with elderly people or children, even minor near-misses can lead to serious accidents. Therefore, there is a need for a system that can analyze potential home risks in real time and quickly present predictions and preventative measures. However, current technology does not adequately predict accidents or present preventative measures. Furthermore, there are issues with the system, such as complex operation for end users and low accuracy.

[0990] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0991] In this invention, the server includes means for reading the near-miss database, means for analyzing received images using a large-scale language model (LLM), means for generating predicted accidents and preventive measures based on the analysis results, means for receiving image data from an image acquisition device and performing preprocessing, means for inputting the preprocessed image data into the large-scale language model, and means for notifying the user of the prediction results and preventive measures. This makes it possible to analyze risks that may occur in the home in real time with high accuracy and quickly predict accidents and present preventive measures.

[0992] The "near miss database" is a database that collects and records past accident and risk cases that have occurred within the home.

[0993] A "large-scale language model (LLM)" is a model based on natural language processing technology that is trained from large amounts of text data and is used for image analysis and text generation.

[0994] "Image capture device" refers to a hardware device, such as a camera or sensor, used to capture images in the home.

[0995] "Preprocessing" refers to a series of processes that involve performing noise removal, resolution adjustment, etc. on image data to prepare it in a format suitable for analysis.

[0996] "Prediction results" refers to risk assessments and information indicating the possibility of an accident occurring obtained as a result of image analysis.

[0997] "Preventive measures" refers to instructions and advice to propose specific countermeasures and actions based on predicted accident risks.

[0998] "User notification means" refers to the interface or protocol for informing users of the analysis results and preventive measures.

[0999] The present invention relates to an automatic near-miss prevention system that analyzes images of a home, predicts accidents, and proposes preventive measures. This system combines a near-miss database with a large-scale language model (LLM) to predict potential accidents that may occur in the home and propose preventive measures. An embodiment of the present invention will be described in detail below.

[1000] System Overview

[1001] This system consists of the following main components:

[1002] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[1003] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[1004] 3. Image capture device: A camera for capturing images in the home and its operation interface.

[1005] 4. Server: This is the main computational unit for performing image analysis and generating accident predictions and prevention measures. The server is responsible for reading the near-miss incident database, preprocessing the received image data, and inputting it into the large-scale language model (LLM).

[1006] 5. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results.

[1007] The main processing of this system is as follows.

[1008] System initialization

[1009] The server reads the initial configuration file, loads the near-miss database, and deploys a large-scale language model (e.g., GPT-4) in memory to prepare for analysis. Meanwhile, the device starts the camera and prepares to switch to image acquisition mode.

[1010] Image acquisition and transmission

[1011] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the device's camera. The captured image is then sent from the device to the server using a secure protocol (for example, HTTPS).

[1012] Image analysis

[1013] The server performs preprocessing on the received image data, including noise removal and resolution adjustment. The preprocessed image data is then input into a large-scale language model (LLM), which analyzes risk factors and generates prediction results. For example, it can detect the presence of flammable materials near a stove and predict a high risk of fire.

[1014] Accident prediction and prevention measures

[1015] The server generates specific accident predictions and prevention measures as text based on the output of the LLM. For example, in response to a prediction that "there is a high risk of fire because there are flammable objects near the stove," it generates a prevention measure such as "keep flammable objects a certain distance away from the stove."

[1016] Displaying the results

[1017] The server sends the generated prediction results and preventive measures to the terminal, which displays them on the user interface. The user can check the displayed results and take necessary measures.

[1018] Specific examples

[1019] Analysis example around a kitchen stove

[1020] After initializing the system, the user takes a picture of the area around the kitchen stove with a camera, and the device sends the image data to the server. The server preprocesses the image data and analyzes risk elements in the image using a large-scale language model (LLM). As a result, it predicts that there are flammable objects near the stove and determines that there is a fire risk. The server then generates a preventive measure, such as "keep flammable objects a certain distance away from the stove," and sends it to the device. The device displays this to the user, who can then take appropriate measures.

[1021] Prompt Sentence Examples

[1022] "We would like you to analyze images from within a home and propose ways to predict and prevent accidents. Please use the near-miss database and LLM model below to provide examples and explain.

[1023] 1. Near miss database: Examples of accidents that may occur in the home (e.g., falls, fires, electric shocks, etc.).

[1024] 2. LLM model: Analyzes risk factors within the home, predicts accidents, and proposes preventive measures.

[1025] Examples:

[1026] After analyzing images of the area around the kitchen stove, it was determined that there was a high risk of fire.

[1027] As a preventative measure, they suggest "keeping flammable materials away from the stove."

[1028] Under the above conditions, please generate the image analysis results and countermeasures for your home.

[1029] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1030] Step 1: Initialize the system

[1031] The server reads the initial configuration file and loads the near-miss database and large-scale language model (LLM) into memory. During this process, a database containing past near-miss cases is loaded into the server. The large-scale language model (e.g., GPT-4) is initialized based on these databases. The input is the initial configuration file, and the output is the database and model loaded into memory.

[1032] Step 2: Launch the camera

[1033] The device sends a camera start command and switches to image capture mode. The user can confirm that the camera is ready through the device interface. The input is the camera start command, and the output is the camera in image capture mode.

[1034] Step 3: Capture an image

[1035] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the device's camera. Image data is captured by pressing the capture button. The input is the area selected by the user's operation, and the output is the captured image.

[1036] Step 4: Sending image data

[1037] The terminal compresses the captured image data and sends it to the server using a secure protocol (e.g., HTTPS). The input is the captured image data, and the output is the compressed data sent to the server.

[1038] Step 5: Preprocessing the image data

[1039] The server performs preprocessing on the received image data, such as noise reduction and resolution adjustment. The preprocessed image data is input into a large-scale language model (LLM). The input is the received image data, and the output is the preprocessed image data.

[1040] Step 6: Image analysis

[1041] The server inputs the preprocessed image data into a large-scale language model (LLM) to analyze risk factors. The model performs a risk assessment based on the input image and generates a prediction result. For example, it detects the presence of flammable materials near a stove and predicts a high risk of fire. The input is the preprocessed image data, and the output is a risk assessment and prediction result.

[1042] Step 7: Accident prediction and prevention measures

[1043] The server generates specific accident predictions and preventive measures based on the output of the LLM. For example, in response to a prediction result that "there is a high risk of fire because flammable objects are near the stove," the server generates a preventive measure as a text message, such as "keep flammable objects a certain distance away from the stove." The input is the analysis result, and the output is a text message containing the preventive measures.

[1044] Step 8: Sending the results

[1045] The server encodes the generated prediction results and prevention measures and sends them to the terminal. The input is the generated text data, and the output is the encoded data sent to the terminal.

[1046] Step 9: View the results

[1047] The terminal decodes the received data and displays it on the user interface. The user can check the displayed results and take necessary measures. The input is the received encoded data, and the output is the predicted results and preventative measures displayed to the user.

[1048] (Application example 1)

[1049] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] Unforeseen safety risks may occur inside an autonomous vehicle due to the placement of passengers and luggage. While conventional vehicle safety systems have made some progress in addressing risks in the external environment, they are insufficient in addressing risks inside the vehicle. Therefore, there is a need for a system that can monitor the situation inside the vehicle in real time, predict potential risks, and quickly propose preventative measures.

[1051] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1052] In this invention, the server includes means for reading the near-miss database, means for capturing received images of the inside of the vehicle with a camera, means for preprocessing the captured images and inputting them into a large-scale language model (LLM), means for generating accident predictions and preventive measures based on the analysis results, and means for outputting the generated prediction results and preventive measures. This makes it possible to predict safety risks inside the vehicle in real time and quickly present appropriate preventive measures.

[1053] The "near miss database" is a database that collects and manages various accident and risk cases that have occurred in the past.

[1054] A "large-scale language model (LLM)" is an artificial intelligence model that learns from large amounts of text data and has advanced natural language processing capabilities.

[1055] A "received image" or "captured image" refers to image data captured using a device such as a camera.

[1056] "Preprocessing" is the process of removing noise from data, adjusting resolution, and preparing the data in an appropriate format before using it for image analysis.

[1057] "Analysis results" refers to information generated as a result of analyzing image data using a large-scale language model (LLM).

[1058] "Accident prediction" refers to estimating accidents and risks that may occur in the future based on the results of analysis.

[1059] "Preventive measures" are specific methods and means for preventing predicted risks and accidents.

[1060] "Inside the vehicle" refers to the area inside the cabin of an autonomous vehicle, i.e., the space where passengers sit.

[1061] A "camera" is a photographing device for capturing visual information and recording it as image data.

[1062] A "server" is the main computing unit that processes the entire system and manages and analyzes data.

[1063] "Means" are the functions or methods used by the system to achieve a particular purpose.

[1064] A specific system for implementing this invention is configured as follows: The server reads the near-miss incident database and uses a large-scale language model (LLM) to analyze the received images of the vehicle interior. A camera is used to capture images of the vehicle interior in real time, which are preprocessed and then input into the LLM. Accident predictions and corresponding preventive measures are generated based on the analysis results, and this information is output to a terminal.

[1065] Program processing

[1066] System initialization

[1067] The server loads the near-miss database and large-scale language model (LLM) into memory, preparing the system for quick access to the information needed for analysis. The device also activates the camera and switches it to image acquisition mode.

[1068] Image acquisition

[1069] The user selects a specific area in the vehicle (e.g., the back seat) and takes an image with the camera. The captured image is then sent to the server via the terminal.

[1070] Image preprocessing

[1071] The server preprocesses the received image data, including removing image noise and adjusting the resolution, especially by resizing and normalizing the image so that the LLM can efficiently analyze the input data.

[1072] Image analysis

[1073] The pre-processed image data is input into a large-scale language model (LLM) to analyze risk factors. The LLM determines whether a particular situation inside the vehicle is dangerous based on information learned from a near-miss database.

[1074] Accident prediction and prevention measures

[1075] The server uses the analysis results from the LLM to predict accidents and generate preventative measures. For example, if it determines that there is a risk of luggage being thrown off in the back seat during sudden braking, it will suggest a preventative measure such as "moving the luggage to the trunk."

[1076] Displaying the results

[1077] The generated prediction results and preventative measures are sent to the device and presented to the user. Providing information in real time via a display or voice notification encourages prompt action.

[1078] Hardware and software used

[1079] Hardware:

[1080] Camera (installed inside the vehicle)

[1081] Server (for analysis and data management)

[1082] Terminal (for user interface)

[1083] software:

[1084] Near miss database

[1085] Large-scale Language Models (LLMs)

[1086] Image Preprocessing Algorithms

[1087] Specific examples

[1088] Example: Rear seat luggage management

[1089] For example, if luggage is left in the back seat, the system will focus on this luggage and identify the risk of harm to passengers in the event of sudden braking, etc. It will then suggest specific preventative measures to the user, such as "Please move the luggage to the trunk."

[1090] Prompt Sentence Examples

[1091] "Identify potential hazards inside the vehicle in this image and suggest preventative measures to notify the occupants."

[1092] This will significantly improve safety inside self-driving vehicles and increase passenger confidence.

[1093] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1094] Step 1: System initialization

[1095] The server loads the near-miss database and large-scale language model (LLM) into memory. This makes the data and models necessary for analysis immediately available. The terminal also starts the camera and switches it to image acquisition mode. This prepares the camera to capture images. The inputs are the near-miss database, LLM, and camera start commands, and the output is the system's overall ready state.

[1096] Step 2: Image acquisition

[1097] The user selects a specific area in the vehicle (e.g., the back seat) and takes an image with the camera. At this time, the device acquires the captured image and sends it to the server. The input is the area in the vehicle selected by the user and the image acquired through the camera, and the output is the image data sent to the server.

[1098] Step 3: Image preprocessing

[1099] The server preprocesses the received image data. First, it removes noise from the image and adjusts the resolution. Then it resizes and normalizes the image to an appropriate size so that the LLM can analyze it efficiently. The input is the image data sent from the terminal, and the output is the preprocessed image data.

[1100] Step 4: Image analysis

[1101] The server inputs the preprocessed image data into a large-scale language model (LLM). The LLM uses information learned from a near-miss database to identify and analyze risk elements in the image. The input is the preprocessed image data, and the output is the result of the risk elements analyzed by the LLM.

[1102] Step 5: Accident prediction and prevention measures

[1103] The server generates accident predictions and preventive measures based on the analysis results obtained from the LLM. For example, if it is determined that there is a risk that luggage placed in the back seat will fly off during sudden braking, it generates the prediction result along with the preventive measure of "moving the luggage to the trunk." The input is the analysis result from the LLM, and the output is the specific prediction result and preventive measures.

[1104] Step 6: View the results

[1105] The server sends the generated prediction results and preventive measures to the device, which then presents them to the user. At this time, information can be provided in real time through a display or voice notification. The input is the prediction results and preventive measures sent from the server, and the output is the information presented to the user.

[1106] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1107] The present invention relates to an automatic near-miss prevention system that analyzes images of the home, predicts accidents, and suggests preventive measures, and also to a system that combines an emotion engine that recognizes the user's emotions. This system not only predicts potential accidents that may occur in the home in advance and suggests preventive measures using a near-miss database and a large-scale language model (LLM), but also improves the user experience by suggesting appropriate measures taking the user's emotional state into consideration. An embodiment of the present invention will be described in detail below.

[1108] System Overview

[1109] This system consists of the following main components:

[1110] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[1111] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[1112] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[1113] 4. Emotion engine: This engine analyzes emotions from the user's voice and facial expressions, and adjusts the predictions and preventive measures presented by the system based on the analysis results.

[1114] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[1115] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[1116] Program processing

[1117] 1. System initialization

[1118] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes it. The emotion engine is also started at the same time. This completes the preparation for the system to analyze and make predictions.

[1119] The device starts the camera and switches it to image capture mode. It displays a message that the camera was successfully started and prompts the user to perform an operation.

[1120] 2. Image Acquisition

[1121] The user selects the room or specific location (for example, the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[1122] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[1123] 3. Sending images

[1124] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[1125] 4. Image Analysis

[1126] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[1127] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[1128] 5. Emotion analysis

[1129] The server inputs the user's voice input and the user's facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[1130] 6. Accident prediction and prevention measures

[1131] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model and the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventive measure such as "keep flammable materials at a certain distance away from the stove." Based on the results of the emotion engine, if the user is nervous, the server provides concise, explicit instructions, and if the user is relaxed, it provides detailed explanations.

[1132] 7. Displaying the results

[1133] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[1134] The device displays the received accident prediction results and prevention measures to the user in a format that is easy for the user to check. The display format and content are adjusted based on the analysis results of the emotion engine.

[1135] Specific examples

[1136] Examples of areas around the kitchen stove

[1137] 1. System initialization

[1138] The server loads the near-miss database and the LLM model, starts the emotion engine, and is ready to go.

[1139] The terminal activates the camera and switches to image capture mode.

[1140] 2. Image Acquisition

[1141] The user selects the area around the kitchen stove and takes an image with the camera.

[1142] The terminal transmits the captured image of the kitchen to the server.

[1143] 3. Sending images

[1144] The terminal transmits image data to the server, and the server confirms receipt of the data.

[1145] 4. Image Analysis

[1146] The server preprocesses the received images and then inputs them into the LLM model.

[1147] The LLM model detects the presence of flammable materials near a stove and predicts the risk of fire.

[1148] 5. Emotion analysis

[1149] The server inputs the user's voice input and facial expression data into the emotion engine, which then analyzes that the user is nervous.

[1150] 6. Accident prediction and prevention measures

[1151] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[1152] 7. Displaying the results

[1153] The server transmits the generated prediction results and preventive measures to the terminal.

[1154] The terminal displays the received results to the user, displaying them as brief instructions.

[1155] This allows users to understand potential risks in their homes in advance and take appropriate measures. In particular, taking the user's emotional state into consideration enables more effective and prompt responses. This is a specific example of an embodiment of the present invention.

[1156] The processing flow will be explained below.

[1157] Step 1:

[1158] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes it. At the same time, it initializes the emotion engine and prepares for user emotion analysis. This completes the preparation of the entire system for analysis and prediction.

[1159] Step 2:

[1160] The device starts the camera and switches it to image capture mode. It displays a message that the camera was successfully started and prompts the user to perform an operation.

[1161] Step 3:

[1162] The user selects the room or specific location (such as the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[1163] Step 4:

[1164] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[1165] Step 5:

[1166] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[1167] Step 6:

[1168] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[1169] Step 7:

[1170] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[1171] Step 8:

[1172] The server inputs the user's voice input and the user's facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[1173] Step 9:

[1174] The server generates specific accident predictions and preventive measures based on the analysis results from the LLM and the emotion engine. For example, it predicts that there is a high risk of fire because of flammable materials near a stove, and generates a preventive measure to keep flammable materials at a certain distance away from the stove. Based on the results of the emotion engine, it provides concise and clear instructions if the user is nervous, or provides detailed explanations if the user is relaxed.

[1175] Step 10:

[1176] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[1177] Step 11:

[1178] The device displays the received accident prediction results and prevention measures to the user in a format that is easy for the user to check. The display format and content are adjusted based on the analysis results of the emotion engine.

[1179] Step 12:

[1180] Users can check the prediction results and preventive measures displayed on their device screen and implement the suggested preventive measures (e.g., moving flammable materials away from the stove), thereby effectively reducing the risk of accidents in the home.

[1181] Through these steps, the system of the present invention can detect potential risks in the home in advance and provide specific preventive measures to the user. In particular, by taking the user's emotional state into consideration, more effective and prompt responses are possible. This improves safety in the home and prevents accidents from occurring.

[1182] Example 2

[1183] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1184] Conventional home accident prevention systems focus on predicting risks and suggesting preventive measures, but rarely take into account the user's emotional state. As a result, if the user is nervous or distracted, the proposed preventive measures cannot be implemented properly, and the risk of an accident still remains.

[1185] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for reading a near-miss database, means for analyzing received images using a large-scale language model (LLM), means for using an emotion engine that analyzes the user's emotional state, means for generating predicted accidents and preventive measures based on the analysis results and emotion analysis results, and means for outputting the generated accident prediction results and preventive measures to a terminal. This makes it possible to predict the risk of domestic accidents in advance and present appropriate preventive measures according to the user's emotional state.

[1186] The "near miss database" is a database that collects and records accident cases that occur within the home, and is an information source used for training large-scale language models (LLMs).

[1187] A "large-scale language model (LLM)" is a model learned from a near-miss database and is an artificial intelligence system for analyzing received images.

[1188] The "emotion engine" is a system that analyzes the user's voice input and facial expression data to determine the user's emotional state.

[1189] The "analysis results" are risk assessments and prediction information generated by a large-scale language model (LLM) based on the image data received.

[1190] The "emotion analysis result" is evaluation information of the emotional state extracted by the emotion engine from the user's voice and facial expression.

[1191] The "accident prediction result" is information on the risk of accidents within the home predicted based on the analysis results.

[1192] "Preventive measures" are specific countermeasures and guidelines for action proposed based on the results of accident predictions.

[1193] A "terminal" is a device used by a user, which includes a camera and a display and provides an interface for displaying instructions and information from the system.

[1194] A "camera" is a photographing device that takes pictures of the home and transmits them to the system.

[1195] The "server" is the main computing unit that performs various processes such as image analysis, emotion analysis, accident prediction, and prevention measures generation.

[1196] This invention relates to an automatic near-miss prevention system that analyzes images from within the home, predicts accidents, and suggests preventive measures, and also to a system that combines an emotion engine that recognizes the user's emotions. This system not only predicts potential accidents that may occur within the home and suggests preventive measures using a near-miss database and a large-scale language model (LLM), but also improves the user experience by suggesting appropriate measures taking into account the user's emotional state.

[1197] This system consists of the following main components:

[1198] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[1199] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[1200] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[1201] 4. Emotion engine: This engine analyzes emotions from the user's voice and facial expressions, and adjusts the predictions and preventive measures presented by the system based on the analysis results.

[1202] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[1203] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[1204] Program processing

[1205] During system initialization, the server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization. At the same time, the emotion engine is also started, completing preparations for the system for analysis and prediction. The device then starts the camera, switches to image acquisition mode, and prompts the user for operation.

[1206] To acquire images, the user selects the room or specific location (such as the area around the kitchen stove) they want to analyze and takes an image with the camera. The device converts the captured image data into data packets and prepares them to be sent to the server.

[1207] In image transmission, the device sends image data packets to the server, which then confirms receipt of the data. The received data undergoes preprocessing, such as noise removal and resolution adjustment, at the server. The preprocessed data is then input into a large-scale language model (LLM) to analyze risk factors.

[1208] In emotion analysis, the server inputs the user's voice and facial expressions captured by the camera into an emotion engine to analyze the user's emotional state. The analysis results are reflected in accident prediction and the generation of preventive measures.

[1209] To predict accidents and generate preventive measures, the server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM and the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventive measure such as "keep flammable materials at a certain distance away from the stove." Based on the results of the emotion engine, the server provides concise, explicit instructions if the user is nervous, and detailed explanations if the user is relaxed.

[1210] The generated prediction results and preventive measures are sent to the device and displayed to the user. The device adjusts the display format and content based on the analysis results of the emotion engine, and presents the results in a format that is easy for the user to check.

[1211] Specific examples

[1212] Examples of areas around the kitchen stove

[1213] 1. System initialization

[1214] The server loads the near-miss database and the LLM model and starts the emotion engine.

[1215] The terminal activates the camera and switches to image capture mode.

[1216] 2. Image Acquisition

[1217] The user selects the area around the kitchen stove and takes an image with the camera.

[1218] The terminal transmits the captured image data of the kitchen to the server.

[1219] 3. Sending images

[1220] The terminal transmits image data to the server, and the server confirms receipt of the data.

[1221] 4. Image Analysis

[1222] The server pre-processes the received images and then inputs them into the LLM.

[1223] LLM detects the presence of flammable materials near the stove and predicts the risk of fire.

[1224] 5. Emotion analysis

[1225] The server inputs the user's voice input and facial expression data into the emotion engine, which then analyzes that the user is nervous.

[1226] 6. Accident prediction and prevention measures

[1227] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[1228] 7. Displaying the results

[1229] The server transmits the generated prediction results and preventive measures to the terminal.

[1230] The terminal presents the received results to the user, displaying them as simple instructions.

[1231] Prompt Sentence Examples

[1232] Example: "Analyze images of the kitchen stove area to determine if there is a risk of fire and suggest preventative measures. Also, provide brief instructions if the user is nervous, or detailed instructions if the user is relaxed."

[1233] This allows users to identify potential risks in the home in advance and take appropriate measures. In particular, taking the user's emotional state into consideration enables more effective responses.

[1234] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1235] Step 1: System initialization

[1236] The server loads the near-miss database and the large-scale language model (LLM) into memory and initializes the system. The server then starts the emotion engine and prepares the system for analyzing the user's voice input and facial expression data.

[1237] Input: Near miss database, LLM model

[1238] Output: System initialized, emotion engine activated

[1239] Specific operations: Loads near-miss data and LLM into the server's internal data memory, executes the emotion engine script, and records in the system log that the user is ready to start using the system.

[1240] Step 2: Image acquisition

[1241] The device starts up the camera and switches to image capture mode. The user selects the room or location they want to analyze and takes an image with the camera.

[1242] Input: Camera

[1243] Output: Captured image data

[1244] Specific operation: The user operates the device's camera app and presses the image capture button. The camera takes an image of the specified location and saves the image data in the device's internal memory.

[1245] Step 3: Send image

[1246] The device converts the captured image data into data packets and sends them to the server, which then confirms receipt.

[1247] Input: Photographed image data

[1248] Output: Image data packet, acknowledgement

[1249] Specific operation: Image data is converted into packet format through the terminal's internal network interface and sent to the server. The server verifies the received data and returns a receipt confirmation response to the terminal.

[1250] Step 4: Image analysis

[1251] The server preprocesses the received image data, which includes noise removal, resolution adjustment, and image cropping if necessary. The preprocessed image data is then input into a large-scale language model (LLM) to analyze risk factors.

[1252] Input: Received image data

[1253] Output: Analysis results

[1254] Specific operation: After using image processing software to remove noise, optimize the image resolution, and crop only the necessary parts, the preprocessed image is input into LLM, which analyzes the risk factors in the image and stores the analysis results in the server's memory.

[1255] Step 5: Sentiment Analysis

[1256] The server receives the user's voice input and facial expression data captured by a camera, and analyzes them using an emotion engine to determine the user's emotional state.

[1257] Input: Voice data, facial expression data

[1258] Output: Emotion analysis results

[1259] Specific operation: The server feeds the voice data acquired from the user and facial expression data captured by the camera to the emotion engine, which performs emotion analysis based on the tone, pitch, and extension characteristics of the voice and the movement of facial muscles. The obtained emotion analysis results are stored on the server.

[1260] Step 6: Accident prediction and prevention measures

[1261] The server generates accident predictions and preventative measures based on the image analysis results from the LLM and the emotion analysis results from the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventative measure such as "keep flammable materials at a certain distance away from the stove." The instructions are adjusted according to the user's emotional state.

[1262] Input: Analysis results, emotion analysis results

[1263] Output: Accident prediction results, prevention measures

[1264] Specific operation: The processing unit inside the server integrates the results of the LLM and the emotion engine, evaluates the risk level, and then generates appropriate prevention measures. It outputs short and clear instructions to users who are nervous, and more detailed explanations to users who are relaxed.

[1265] Step 7: View the results

[1266] The server sends the generated accident prediction results and preventive measures to the terminal, which then displays them to the user. The display format and content are adjusted based on the user's emotional state.

[1267] Input: Accident prediction results, prevention measures

[1268] Output: What the user sees

[1269] Specific operation: The server converts the generated results into packet format and sends them to the terminal. The terminal interprets the received data and displays it to the user in an appropriate format. For example, the control panel may display a message saying, "There are flammable materials near the stove. There is a high risk of fire. Please move the flammable materials away."

[1270] This allows the system to take into account the user's emotional state, detect hidden risks in the home early, and provide appropriate preventive measures, thereby effectively preventing accidents in the home.

[1271] (Application example 2)

[1272] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1273] It is important not only to predict and prevent potential accidents in the home, but also to present appropriate countermeasures that take the user's emotional state into account. However, conventional systems have had the problem of insufficient safety measures that take the user's emotional state into account, resulting in a lack of improvement in the user experience. The present invention aims to solve these problems and provide more effective accident prediction and prevention measures.

[1274] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for reading a near-miss incident database, means for analyzing received images using a large-scale language model (LLM), means for generating predicted accidents and preventive measures based on the analysis results, means for outputting the generated accident prediction results and preventive measures, means for analyzing the user's emotions, and means for adjusting the prediction results and measures based on the emotion analysis results. This makes it possible to provide appropriate accident predictions and preventive measures that take into account the user's emotional state.

[1275] The "near miss database" is a database that collects near miss cases that have occurred in the past, and records various accident cases that occurred in the home or on mobile vehicles.

[1276] A "large-scale language model (LLM)" is an advanced machine learning model that learns from massive amounts of text and image data, and is used to analyze received images and predict risks.

[1277] "Means for generating predicted accidents and preventive measures based on the analysis results" refers to a means by which the server predicts potential risks based on the image analysis results and derives specific preventive measures to avoid those risks.

[1278] The "means for outputting the generated accident prediction results and preventive measures" refers to a means for displaying or notifying the user of the predicted accident risk and preventive measures on a device such as a terminal or display used by the user.

[1279] The "means for analyzing user emotions" refers to a software routine or model for analyzing user emotions from voice input or camera footage and obtaining the results.

[1280] "Means for adjusting prediction results and countermeasures based on emotion analysis results" refers to means for optimizing the presentation method and content of accident prediction results and preventive measures, taking into account the user's emotional state.

[1281] MODE FOR CARRYING OUT THE INVENTION

[1282] System Overview

[1283] The system of the present invention consists of the following main components:

[1284] 1. Near-miss database: This database collects past near-miss incidents, recording various accidents occurring within the home and on mobile devices.

[1285] 2. Large-scale language model (LLM): A model trained from a near-miss database that analyzes received images and predicts risks.

[1286] 3. Image acquisition means: A camera and its operating interface for acquiring images of the home or moving objects.

[1287] 4. Emotion Engine: This engine analyzes the user's emotions from their voice and facial expressions. The results of the emotion analysis are used to adjust the predictions and preventative measures presented by the system.

[1288] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[1289] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[1290] Program processing

[1291] The system of the present invention performs processing in the following steps.

[1292] System initialization

[1293] The server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization. The emotion engine is also started at the same time. This completes preparation for the system for analysis and prediction. The device starts the camera and switches to image acquisition mode. A message is displayed indicating that the camera was successfully started, prompting the user to perform an operation.

[1294] Image acquisition

[1295] The user selects the room or specific location they want to analyze (such as the living room or the area around the stove) and takes an image with the camera. The device converts the captured image data into data packets and prepares them to be sent to the server.

[1296] Image analysis

[1297] The server preprocesses the received image data. This includes denoising the image, adjusting the resolution, and cropping the image if necessary. The server then inputs the preprocessed image data into a large-scale language model (LLM). The LLM analyzes the risk elements in the image and generates a prediction result.

[1298] Emotion analysis

[1299] The server inputs the user's voice input and facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[1300] Accident prediction and prevention measures

[1301] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model and the emotion engine. For example, predictions such as "risk of falls in the living room" and "high risk of fire due to flammable objects near the stove" may be provided, along with preventive measures such as "rearranging furniture" and "keeping flammable objects a certain distance away from the stove." If the user is nervous, the server provides concise, explicit instructions, while if they are relaxed, it provides detailed explanations.

[1302] Displaying the results

[1303] The server sends the generated prediction results and preventive measures to the terminal. The terminal confirms that they have been received correctly. The terminal then displays the received accident prediction results and preventive measures to the user. The results are displayed in a format that is easy for the user to check, and the display format and content are adjusted based on the analysis results of the emotion engine.

[1304] Specific examples

[1305] Examples of areas around the kitchen stove

[1306] 1. System initialization

[1307] The server loads the near-miss database and LLM model, starts the emotion engine, and is ready to go.

[1308] The device will start the camera and switch to image capture mode.

[1309] 2. Image Acquisition

[1310] The user selects the area around the kitchen stove and takes a picture with the camera.

[1311] The device sends the captured image of the kitchen to the server.

[1312] 3. Image Analysis

[1313] The server preprocesses the received images and inputs them into the LLM model.

[1314] The LLM model detects the presence of flammable materials near the stove and predicts the risk of fire.

[1315] 4. Emotion analysis

[1316] The server inputs the user's voice input and facial expression data into an emotion engine and analyzes whether the user is nervous.

[1317] 5. Accident prediction and prevention measures

[1318] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[1319] 6. Displaying the results

[1320] The server transmits the generated prediction results and preventive measures to the terminal.

[1321] The terminal displays the received results to the user, displaying them as brief instructions.

[1322] Example prompts for generative AI models

[1323] "Analyze live video to detect potential risks, recognize the user's emotional state, and suggest appropriate countermeasures based on that."

[1324] The above is a detailed description of the embodiment of the present invention. This allows users to understand potential risks in their homes in advance and take appropriate measures. In particular, it is possible to provide appropriate accident predictions and prevention measures that take into account the emotional state of the user.

[1325] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1326] Step 1:

[1327] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes the system. The emotion engine is also started at the same time. This completes preparation for the system to perform analysis and prediction. The input is the near-miss database, LLM, and the emotion engine program code, and the output is the initialized system state.

[1328] Step 2:

[1329] The device starts the camera and switches to image acquisition mode. It displays to the user that the camera was successfully started and prompts them to operate. The input is the camera start command, and the output is that the camera is ready to acquire images.

[1330] Step 3:

[1331] The user selects the room or specific area (e.g., living room, stove area) they want to analyze and takes an image with the camera. The input is the user's selection and the captured image, and the output is the image data captured by the camera.

[1332] Step 4:

[1333] The device converts the captured image data into data packets and prepares them for transmission to the server. The input is the captured image data, and the output is the data packets for transmission.

[1334] Step 5:

[1335] The terminal sends image data packets to the server. The server receives the data and prepares it for data analysis. The input is the image data packets sent from the terminal, and the output is the received image data ready for preprocessing.

[1336] Step 6:

[1337] The server pre-processes the received image data. Pre-processing includes image noise removal, resolution adjustment, and image cropping if necessary. The input is the received image data and the output is the pre-processed image data.

[1338] Step 7:

[1339] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes the risk elements in the image and generates prediction results. The input is the preprocessed image data, and the output is the risk elements and their prediction results.

[1340] Step 8:

[1341] The server receives the user's voice input and facial expressions captured by a camera, and inputs them into the emotion engine. The emotion engine analyzes the user's emotional state and obtains the results. The input is voice data and facial expression data, and the output is the analyzed emotional state.

[1342] Step 9:

[1343] The server generates specific accident predictions and prevention measures based on the analysis results from the LLM model and the emotion engine. The inputs are risk prediction results and the user's emotional state, and the output is optimized accident predictions and prevention measures.

[1344] Step 10:

[1345] The server sends the generated prediction results and prevention measures to the terminal. The input is the optimized accident prediction and prevention measures, and the output is the transmitted data.

[1346] Step 11:

[1347] The device displays the received accident prediction results and prevention measures to the user. The input is the prediction results and prevention measures sent from the server, and the output is the content displayed to the user. The display format and content are adjusted based on the analysis results of the emotion engine, so that the user can easily check the results.

[1348] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1349] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1350] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1351] [Fourth embodiment]

[1352] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1353] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1354] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1355] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1356] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1357] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1358] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1359] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1360] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1361] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1362] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1363] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1364] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1365] The present invention relates to an automatic near-miss prevention system that analyzes images of a home, predicts accidents, and proposes preventive measures. This system combines a near-miss database with a large-scale language model (LLM) to predict potential accidents that may occur in the home and propose preventive measures. An embodiment of the present invention will be described in detail below.

[1366] System Overview

[1367] This system consists of the following main components:

[1368] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[1369] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[1370] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[1371] 4. Server: It is the main computing unit for performing image analysis and generating accident prediction and prevention measures.

[1372] 5. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results.

[1373] Program processing

[1374] 1. System initialization

[1375] The server loads the near-miss database and large-scale language model (LLM) into memory and completes preparation.

[1376] The terminal activates the camera and switches to image capture mode.

[1377] 2. Image Acquisition

[1378] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the camera.

[1379] The terminal transmits the captured image to the server.

[1380] 3. Image Analysis

[1381] The server preprocesses the received image data and inputs it into a large-scale language model (LLM). Preprocessing includes image denoising and resolution adjustment.

[1382] LLM analyzes risk factors within an image and generates a prediction result.

[1383] 4. Accident prediction and prevention measures

[1384] The server generates specific accident predictions and prevention measures based on the analysis results from the LLM.

[1385] For example, the prediction result is "there is a high risk of fire because there are flammable objects near the stove," and the preventive measure is "keep flammable objects a certain distance away from the stove."

[1386] 5. Displaying the results

[1387] The server transmits the generated prediction results and preventive measures to the terminal.

[1388] The terminal displays the received results on the screen for the user.

[1389] Specific examples

[1390] Examples of areas around the kitchen stove

[1391] 1. System initialization

[1392] The server loads the near miss database and LLM model and is ready to go.

[1393] The terminal activates the camera and switches to image capture mode.

[1394] 2. Image Acquisition

[1395] The user selects the area around the kitchen stove and takes an image with the camera.

[1396] The terminal transmits the captured image of the kitchen to the server.

[1397] 3. Image Analysis

[1398] The server preprocesses the received images and then inputs them into the LLM model.

[1399] The LLM model detects the presence of flammable materials near a stove and predicts the risk of fire.

[1400] 4. Accident prediction and prevention measures

[1401] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove."

[1402] 5. Displaying the results

[1403] The server transmits the generated prediction results and preventive measures to the terminal.

[1404] The terminal displays the received results to the user.

[1405] This allows the user to know in advance the risks that may be present in the home and take appropriate measures. This is a specific example of an embodiment of the present invention.

[1406] The processing flow will be explained below.

[1407] Step 1:

[1408] The server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization, which prepares the system for analysis and prediction.

[1409] Step 2:

[1410] The device starts the camera and prepares the user to capture images. It displays a message that the camera has been successfully started and prompts the user to perform an operation.

[1411] Step 3:

[1412] The user selects the room or specific location (such as the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[1413] Step 4:

[1414] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[1415] Step 5:

[1416] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[1417] Step 6:

[1418] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[1419] Step 7:

[1420] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[1421] Step 8:

[1422] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model. For example, the server may predict that "there is a high risk of fire due to flammable materials near the stove" and generate preventive measures such as "keep flammable materials at a certain distance away from the stove."

[1423] Step 9:

[1424] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[1425] Step 10:

[1426] The terminal displays the received accident prediction results and prevention measures to the user in a format that makes it easy for the user to check the results.

[1427] Step 11:

[1428] The user checks the prediction results and preventive measures displayed on the device screen and implements the proposed preventive measures, such as moving flammable materials away from the stove.

[1429] Through these steps, the system of the present invention can detect potential risks in the home in advance and provide users with specific preventative measures, thereby improving safety in the home and preventing accidents from occurring.

[1430] Example 1

[1431] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] A wide variety of accidents can occur in the home, and preventing them requires quick and accurate judgment. Especially in households with elderly people or children, even minor near-misses can lead to serious accidents. Therefore, there is a need for a system that can analyze potential home risks in real time and quickly present predictions and preventative measures. However, current technology does not adequately predict accidents or present preventative measures. Furthermore, there are issues with the system, such as complex operation for end users and low accuracy.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1434] In this invention, the server includes means for reading the near-miss database, means for analyzing received images using a large-scale language model (LLM), means for generating predicted accidents and preventive measures based on the analysis results, means for receiving image data from an image acquisition device and performing preprocessing, means for inputting the preprocessed image data into the large-scale language model, and means for notifying the user of the prediction results and preventive measures. This makes it possible to analyze risks that may occur in the home in real time with high accuracy and quickly predict accidents and present preventive measures.

[1435] The "near miss database" is a database that collects and records past accident and risk cases that have occurred within the home.

[1436] A "large-scale language model (LLM)" is a model based on natural language processing technology that is trained from large amounts of text data and is used for image analysis and text generation.

[1437] "Image capture device" refers to a hardware device, such as a camera or sensor, used to capture images in the home.

[1438] "Preprocessing" refers to a series of processes that involve performing noise removal, resolution adjustment, etc. on image data to prepare it in a format suitable for analysis.

[1439] "Prediction results" refers to risk assessments and information indicating the possibility of an accident occurring obtained as a result of image analysis.

[1440] "Preventive measures" refers to instructions and advice to propose specific countermeasures and actions based on predicted accident risks.

[1441] "User notification means" refers to the interface or protocol for informing users of the analysis results and preventive measures.

[1442] The present invention relates to an automatic near-miss prevention system that analyzes images of a home, predicts accidents, and proposes preventive measures. This system combines a near-miss database with a large-scale language model (LLM) to predict potential accidents that may occur in the home and propose preventive measures. An embodiment of the present invention will be described in detail below.

[1443] System Overview

[1444] This system consists of the following main components:

[1445] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[1446] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[1447] 3. Image capture device: A camera for capturing images in the home and its operation interface.

[1448] 4. Server: This is the main computational unit for performing image analysis and generating accident predictions and prevention measures. The server is responsible for reading the near-miss incident database, preprocessing the received image data, and inputting it into the large-scale language model (LLM).

[1449] 5. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results.

[1450] The main processing of this system is as follows.

[1451] System initialization

[1452] The server reads the initial configuration file, loads the near-miss database, and deploys a large-scale language model (e.g., GPT-4) in memory to prepare for analysis. Meanwhile, the device starts the camera and prepares to switch to image acquisition mode.

[1453] Image acquisition and transmission

[1454] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the device's camera. The captured image is then sent from the device to the server using a secure protocol (for example, HTTPS).

[1455] Image analysis

[1456] The server performs preprocessing on the received image data, including noise removal and resolution adjustment. The preprocessed image data is then input into a large-scale language model (LLM), which analyzes risk factors and generates prediction results. For example, it can detect the presence of flammable materials near a stove and predict a high risk of fire.

[1457] Accident prediction and prevention measures

[1458] The server generates specific accident predictions and prevention measures as text based on the output of the LLM. For example, in response to a prediction that "there is a high risk of fire because there are flammable objects near the stove," it generates a prevention measure such as "keep flammable objects a certain distance away from the stove."

[1459] Displaying the results

[1460] The server sends the generated prediction results and preventive measures to the terminal, which displays them on the user interface. The user can check the displayed results and take necessary measures.

[1461] Specific examples

[1462] Analysis example around a kitchen stove

[1463] After initializing the system, the user takes a picture of the area around the kitchen stove with a camera, and the device sends the image data to the server. The server preprocesses the image data and analyzes risk elements in the image using a large-scale language model (LLM). As a result, it predicts that there are flammable objects near the stove and determines that there is a fire risk. The server then generates a preventive measure, such as "keep flammable objects a certain distance away from the stove," and sends it to the device. The device displays this to the user, who can then take appropriate measures.

[1464] Prompt Sentence Examples

[1465] "We would like you to analyze images from within a home and propose ways to predict and prevent accidents. Please use the near-miss database and LLM model below to provide examples and explain.

[1466] 1. Near miss database: Examples of accidents that may occur in the home (e.g., falls, fires, electric shocks, etc.).

[1467] 2. LLM model: Analyzes risk factors within the home, predicts accidents, and proposes preventive measures.

[1468] Examples:

[1469] After analyzing images of the area around the kitchen stove, it was determined that there was a high risk of fire.

[1470] As a preventative measure, they suggest "keeping flammable materials away from the stove."

[1471] Under the above conditions, please generate the image analysis results and countermeasures for your home.

[1472] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1473] Step 1: Initialize the system

[1474] The server reads the initial configuration file and loads the near-miss database and large-scale language model (LLM) into memory. During this process, a database containing past near-miss cases is loaded into the server. The large-scale language model (e.g., GPT-4) is initialized based on these databases. The input is the initial configuration file, and the output is the database and model loaded into memory.

[1475] Step 2: Launch the camera

[1476] The device sends a camera start command and switches to image capture mode. The user can confirm that the camera is ready through the device interface. The input is the camera start command, and the output is the camera in image capture mode.

[1477] Step 3: Capture an image

[1478] The user selects the area they want to analyze (for example, the area around the kitchen stove) and takes an image with the device's camera. Image data is captured by pressing the capture button. The input is the area selected by the user's operation, and the output is the captured image.

[1479] Step 4: Sending image data

[1480] The terminal compresses the captured image data and sends it to the server using a secure protocol (e.g., HTTPS). The input is the captured image data, and the output is the compressed data sent to the server.

[1481] Step 5: Preprocessing the image data

[1482] The server performs preprocessing on the received image data, such as noise reduction and resolution adjustment. The preprocessed image data is input into a large-scale language model (LLM). The input is the received image data, and the output is the preprocessed image data.

[1483] Step 6: Image analysis

[1484] The server inputs the preprocessed image data into a large-scale language model (LLM) to analyze risk factors. The model performs a risk assessment based on the input image and generates a prediction result. For example, it detects the presence of flammable materials near a stove and predicts a high risk of fire. The input is the preprocessed image data, and the output is a risk assessment and prediction result.

[1485] Step 7: Accident prediction and prevention measures

[1486] The server generates specific accident predictions and preventive measures based on the output of the LLM. For example, in response to a prediction result that "there is a high risk of fire because flammable objects are near the stove," the server generates a preventive measure as a text message, such as "keep flammable objects a certain distance away from the stove." The input is the analysis result, and the output is a text message containing the preventive measures.

[1487] Step 8: Sending the results

[1488] The server encodes the generated prediction results and prevention measures and sends them to the terminal. The input is the generated text data, and the output is the encoded data sent to the terminal.

[1489] Step 9: View the results

[1490] The terminal decodes the received data and displays it on the user interface. The user can check the displayed results and take necessary measures. The input is the received encoded data, and the output is the predicted results and preventative measures displayed to the user.

[1491] (Application example 1)

[1492] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1493] Unforeseen safety risks may occur inside an autonomous vehicle due to the placement of passengers and luggage. While conventional vehicle safety systems have made some progress in addressing risks in the external environment, they are insufficient in addressing risks inside the vehicle. Therefore, there is a need for a system that can monitor the situation inside the vehicle in real time, predict potential risks, and quickly propose preventative measures.

[1494] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1495] In this invention, the server includes means for reading the near-miss database, means for capturing received images of the inside of the vehicle with a camera, means for preprocessing the captured images and inputting them into a large-scale language model (LLM), means for generating accident predictions and preventive measures based on the analysis results, and means for outputting the generated prediction results and preventive measures. This makes it possible to predict safety risks inside the vehicle in real time and quickly present appropriate preventive measures.

[1496] The "near miss database" is a database that collects and manages various accident and risk cases that have occurred in the past.

[1497] A "large-scale language model (LLM)" is an artificial intelligence model that learns from large amounts of text data and has advanced natural language processing capabilities.

[1498] A "received image" or "captured image" refers to image data captured using a device such as a camera.

[1499] "Preprocessing" is the process of removing noise from data, adjusting resolution, and preparing the data in an appropriate format before using it for image analysis.

[1500] "Analysis results" refers to information generated as a result of analyzing image data using a large-scale language model (LLM).

[1501] "Accident prediction" refers to estimating accidents and risks that may occur in the future based on the results of analysis.

[1502] "Preventive measures" are specific methods and means for preventing predicted risks and accidents.

[1503] "Inside the vehicle" refers to the area inside the cabin of an autonomous vehicle, i.e., the space where passengers sit.

[1504] A "camera" is a photographing device for capturing visual information and recording it as image data.

[1505] A "server" is the main computing unit that processes the entire system and manages and analyzes data.

[1506] "Means" are the functions or methods used by the system to achieve a particular purpose.

[1507] A specific system for implementing this invention is configured as follows: The server reads the near-miss incident database and uses a large-scale language model (LLM) to analyze the received images of the vehicle interior. A camera is used to capture images of the vehicle interior in real time, which are preprocessed and then input into the LLM. Accident predictions and corresponding preventive measures are generated based on the analysis results, and this information is output to a terminal.

[1508] Program processing

[1509] System initialization

[1510] The server loads the near-miss database and large-scale language model (LLM) into memory, preparing the system for quick access to the information needed for analysis. The device also activates the camera and switches it to image acquisition mode.

[1511] Image acquisition

[1512] The user selects a specific area in the vehicle (e.g., the back seat) and takes an image with the camera. The captured image is then sent to the server via the terminal.

[1513] Image preprocessing

[1514] The server preprocesses the received image data, including removing image noise and adjusting the resolution, especially by resizing and normalizing the image so that the LLM can efficiently analyze the input data.

[1515] Image analysis

[1516] The pre-processed image data is input into a large-scale language model (LLM) to analyze risk factors. The LLM determines whether a particular situation inside the vehicle is dangerous based on information learned from a near-miss database.

[1517] Accident prediction and prevention measures

[1518] The server uses the analysis results from the LLM to predict accidents and generate preventative measures. For example, if it determines that there is a risk of luggage being thrown off in the back seat during sudden braking, it will suggest a preventative measure such as "moving the luggage to the trunk."

[1519] Displaying the results

[1520] The generated prediction results and preventative measures are sent to the device and presented to the user. Providing information in real time via a display or voice notification encourages prompt action.

[1521] Hardware and software used

[1522] Hardware:

[1523] Camera (installed inside the vehicle)

[1524] Server (for analysis and data management)

[1525] Terminal (for user interface)

[1526] software:

[1527] Near miss database

[1528] Large-scale Language Models (LLMs)

[1529] Image Preprocessing Algorithms

[1530] Specific examples

[1531] Example: Rear seat luggage management

[1532] For example, if luggage is left in the back seat, the system will focus on this luggage and identify the risk of harm to passengers in the event of sudden braking, etc. It will then suggest specific preventative measures to the user, such as "Please move the luggage to the trunk."

[1533] Prompt Sentence Examples

[1534] "Identify potential hazards inside the vehicle in this image and suggest preventative measures to notify the occupants."

[1535] This will significantly improve safety inside self-driving vehicles and increase passenger confidence.

[1536] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1537] Step 1: System initialization

[1538] The server loads the near-miss database and large-scale language model (LLM) into memory. This makes the data and models necessary for analysis immediately available. The terminal also starts the camera and switches it to image acquisition mode. This prepares the camera to capture images. The inputs are the near-miss database, LLM, and camera start commands, and the output is the system's overall ready state.

[1539] Step 2: Image acquisition

[1540] The user selects a specific area in the vehicle (e.g., the back seat) and takes an image with the camera. At this time, the device acquires the captured image and sends it to the server. The input is the area in the vehicle selected by the user and the image acquired through the camera, and the output is the image data sent to the server.

[1541] Step 3: Image preprocessing

[1542] The server preprocesses the received image data. First, it removes noise from the image and adjusts the resolution. Then it resizes and normalizes the image to an appropriate size so that the LLM can analyze it efficiently. The input is the image data sent from the terminal, and the output is the preprocessed image data.

[1543] Step 4: Image analysis

[1544] The server inputs the preprocessed image data into a large-scale language model (LLM). The LLM uses information learned from a near-miss database to identify and analyze risk elements in the image. The input is the preprocessed image data, and the output is the result of the risk elements analyzed by the LLM.

[1545] Step 5: Accident prediction and prevention measures

[1546] The server generates accident predictions and preventive measures based on the analysis results obtained from the LLM. For example, if it is determined that there is a risk that luggage placed in the back seat will fly off during sudden braking, it generates the prediction result along with the preventive measure of "moving the luggage to the trunk." The input is the analysis result from the LLM, and the output is the specific prediction result and preventive measures.

[1547] Step 6: View the results

[1548] The server sends the generated prediction results and preventive measures to the device, which then presents them to the user. At this time, information can be provided in real time through a display or voice notification. The input is the prediction results and preventive measures sent from the server, and the output is the information presented to the user.

[1549] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1550] The present invention relates to an automatic near-miss prevention system that analyzes images of the home, predicts accidents, and suggests preventive measures, and also to a system that combines an emotion engine that recognizes the user's emotions. This system not only predicts potential accidents that may occur in the home in advance and suggests preventive measures using a near-miss database and a large-scale language model (LLM), but also improves the user experience by suggesting appropriate measures taking the user's emotional state into consideration. An embodiment of the present invention will be described in detail below.

[1551] System Overview

[1552] This system consists of the following main components:

[1553] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[1554] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[1555] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[1556] 4. Emotion engine: This engine analyzes emotions from the user's voice and facial expressions, and adjusts the predictions and preventive measures presented by the system based on the analysis results.

[1557] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[1558] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[1559] Program processing

[1560] 1. System initialization

[1561] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes it. The emotion engine is also started at the same time. This completes the preparation for the system to analyze and make predictions.

[1562] The device starts the camera and switches it to image capture mode. It displays a message that the camera was successfully started and prompts the user to perform an operation.

[1563] 2. Image Acquisition

[1564] The user selects the room or specific location (for example, the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[1565] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[1566] 3. Sending images

[1567] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[1568] 4. Image Analysis

[1569] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[1570] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[1571] 5. Emotion analysis

[1572] The server inputs the user's voice input and the user's facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[1573] 6. Accident prediction and prevention measures

[1574] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model and the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventive measure such as "keep flammable materials at a certain distance away from the stove." Based on the results of the emotion engine, if the user is nervous, the server provides concise, explicit instructions, and if the user is relaxed, it provides detailed explanations.

[1575] 7. Displaying the results

[1576] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[1577] The device displays the received accident prediction results and prevention measures to the user in a format that is easy for the user to check. The display format and content are adjusted based on the analysis results of the emotion engine.

[1578] Specific examples

[1579] Examples of areas around the kitchen stove

[1580] 1. System initialization

[1581] The server loads the near-miss database and the LLM model, starts the emotion engine, and is ready to go.

[1582] The terminal activates the camera and switches to image capture mode.

[1583] 2. Image Acquisition

[1584] The user selects the area around the kitchen stove and takes an image with the camera.

[1585] The terminal transmits the captured image of the kitchen to the server.

[1586] 3. Sending images

[1587] The terminal transmits image data to the server, and the server confirms receipt of the data.

[1588] 4. Image Analysis

[1589] The server preprocesses the received images and then inputs them into the LLM model.

[1590] The LLM model detects the presence of flammable materials near a stove and predicts the risk of fire.

[1591] 5. Emotion analysis

[1592] The server inputs the user's voice input and facial expression data into the emotion engine, which then analyzes that the user is nervous.

[1593] 6. Accident prediction and prevention measures

[1594] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[1595] 7. Displaying the results

[1596] The server transmits the generated prediction results and preventive measures to the terminal.

[1597] The terminal displays the received results to the user, displaying them as brief instructions.

[1598] This allows users to understand potential risks in their homes in advance and take appropriate measures. In particular, taking the user's emotional state into consideration enables more effective and prompt responses. This is a specific example of an embodiment of the present invention.

[1599] The processing flow will be explained below.

[1600] Step 1:

[1601] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes it. At the same time, it initializes the emotion engine and prepares for user emotion analysis. This completes the preparation of the entire system for analysis and prediction.

[1602] Step 2:

[1603] The device starts the camera and switches it to image capture mode. It displays a message that the camera was successfully started and prompts the user to perform an operation.

[1604] Step 3:

[1605] The user selects the room or specific location (such as the area around the kitchen stove) they want to analyze, takes a picture with the camera, and presses the capture button to acquire the image data.

[1606] Step 4:

[1607] The device converts the captured image data into data packets and prepares to send them to the server. The device checks the format and content of the data to be sent.

[1608] Step 5:

[1609] The terminal transmits image data packets to the server, and manages the data transmission status until the server confirms receipt of the data.

[1610] Step 6:

[1611] The server pre-processes the received image data, which includes denoising the image, adjusting the resolution, and cropping the image if necessary.

[1612] Step 7:

[1613] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes risk elements in the image and generates prediction results.

[1614] Step 8:

[1615] The server inputs the user's voice input and the user's facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[1616] Step 9:

[1617] The server generates specific accident predictions and preventive measures based on the analysis results from the LLM and the emotion engine. For example, it predicts that there is a high risk of fire because of flammable materials near a stove, and generates a preventive measure to keep flammable materials at a certain distance away from the stove. Based on the results of the emotion engine, it provides concise and clear instructions if the user is nervous, or provides detailed explanations if the user is relaxed.

[1618] Step 10:

[1619] The server sends the generated prediction results and prevention measures to the terminal, and confirms that the terminal has received them correctly.

[1620] Step 11:

[1621] The device displays the received accident prediction results and prevention measures to the user in a format that is easy for the user to check. The display format and content are adjusted based on the analysis results of the emotion engine.

[1622] Step 12:

[1623] Users can check the prediction results and preventive measures displayed on their device screen and implement the suggested preventive measures (e.g., moving flammable materials away from the stove), thereby effectively reducing the risk of accidents in the home.

[1624] Through these steps, the system of the present invention can detect potential risks in the home in advance and provide specific preventive measures to the user. In particular, by taking the user's emotional state into consideration, more effective and prompt responses are possible. This improves safety in the home and prevents accidents from occurring.

[1625] Example 2

[1626] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1627] Conventional home accident prevention systems focus on predicting risks and suggesting preventive measures, but rarely take into account the user's emotional state. As a result, if the user is nervous or distracted, the proposed preventive measures cannot be implemented properly, and the risk of an accident still remains.

[1628] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for reading a near-miss database, means for analyzing received images using a large-scale language model (LLM), means for using an emotion engine that analyzes the user's emotional state, means for generating predicted accidents and preventive measures based on the analysis results and emotion analysis results, and means for outputting the generated accident prediction results and preventive measures to a terminal. This makes it possible to predict the risk of domestic accidents in advance and present appropriate preventive measures according to the user's emotional state.

[1629] The "near miss database" is a database that collects and records accident cases that occur within the home, and is an information source used for training large-scale language models (LLMs).

[1630] A "large-scale language model (LLM)" is a model learned from a near-miss database and is an artificial intelligence system for analyzing received images.

[1631] The "emotion engine" is a system that analyzes the user's voice input and facial expression data to determine the user's emotional state.

[1632] The "analysis results" are risk assessments and prediction information generated by a large-scale language model (LLM) based on the image data received.

[1633] The "emotion analysis result" is evaluation information of the emotional state extracted by the emotion engine from the user's voice and facial expression.

[1634] The "accident prediction result" is information on the risk of accidents within the home predicted based on the analysis results.

[1635] "Preventive measures" are specific countermeasures and guidelines for action proposed based on the results of accident predictions.

[1636] A "terminal" is a device used by a user, which includes a camera and a display and provides an interface for displaying instructions and information from the system.

[1637] A "camera" is a photographing device that takes pictures of the home and transmits them to the system.

[1638] The "server" is the main computing unit that performs various processes such as image analysis, emotion analysis, accident prediction, and prevention measures generation.

[1639] This invention relates to an automatic near-miss prevention system that analyzes images from within the home, predicts accidents, and suggests preventive measures, and also to a system that combines an emotion engine that recognizes the user's emotions. This system not only predicts potential accidents that may occur within the home and suggests preventive measures using a near-miss database and a large-scale language model (LLM), but also improves the user experience by suggesting appropriate measures taking into account the user's emotional state.

[1640] This system consists of the following main components:

[1641] 1. Near-miss database: A database that collects past near-miss incidents. This database records various accidents that occurred within the home and is used for LLM learning.

[1642] 2. Large-scale language model (LLM): A model trained from a near-miss database and used for image analysis.

[1643] 3. Image acquisition means: A camera for acquiring images within the home and its operation interface.

[1644] 4. Emotion engine: This engine analyzes emotions from the user's voice and facial expressions, and adjusts the predictions and preventive measures presented by the system based on the analysis results.

[1645] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[1646] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[1647] Program processing

[1648] During system initialization, the server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization. At the same time, the emotion engine is also started, completing preparations for the system for analysis and prediction. The device then starts the camera, switches to image acquisition mode, and prompts the user for operation.

[1649] To acquire images, the user selects the room or specific location (such as the area around the kitchen stove) they want to analyze and takes an image with the camera. The device converts the captured image data into data packets and prepares them to be sent to the server.

[1650] In image transmission, the device sends image data packets to the server, which then confirms receipt of the data. The received data undergoes preprocessing, such as noise removal and resolution adjustment, at the server. The preprocessed data is then input into a large-scale language model (LLM) to analyze risk factors.

[1651] In emotion analysis, the server inputs the user's voice and facial expressions captured by the camera into an emotion engine to analyze the user's emotional state. The analysis results are reflected in accident prediction and the generation of preventive measures.

[1652] To predict accidents and generate preventive measures, the server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM and the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventive measure such as "keep flammable materials at a certain distance away from the stove." Based on the results of the emotion engine, the server provides concise, explicit instructions if the user is nervous, and detailed explanations if the user is relaxed.

[1653] The generated prediction results and preventive measures are sent to the device and displayed to the user. The device adjusts the display format and content based on the analysis results of the emotion engine, and presents the results in a format that is easy for the user to check.

[1654] Specific examples

[1655] Examples of areas around the kitchen stove

[1656] 1. System initialization

[1657] The server loads the near-miss database and the LLM model and starts the emotion engine.

[1658] The terminal activates the camera and switches to image capture mode.

[1659] 2. Image Acquisition

[1660] The user selects the area around the kitchen stove and takes an image with the camera.

[1661] The terminal transmits the captured image data of the kitchen to the server.

[1662] 3. Sending images

[1663] The terminal transmits image data to the server, and the server confirms receipt of the data.

[1664] 4. Image Analysis

[1665] The server pre-processes the received images and then inputs them into the LLM.

[1666] LLM detects the presence of flammable materials near the stove and predicts the risk of fire.

[1667] 5. Emotion analysis

[1668] The server inputs the user's voice input and facial expression data into the emotion engine, which then analyzes that the user is nervous.

[1669] 6. Accident prediction and prevention measures

[1670] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[1671] 7. Displaying the results

[1672] The server transmits the generated prediction results and preventive measures to the terminal.

[1673] The terminal presents the received results to the user, displaying them as simple instructions.

[1674] Prompt Sentence Examples

[1675] Example: "Analyze images of the kitchen stove area to determine if there is a risk of fire and suggest preventative measures. Also, provide brief instructions if the user is nervous, or detailed instructions if the user is relaxed."

[1676] This allows users to identify potential risks in the home in advance and take appropriate measures. In particular, taking the user's emotional state into consideration enables more effective responses.

[1677] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1678] Step 1: System initialization

[1679] The server loads the near-miss database and the large-scale language model (LLM) into memory and initializes the system. The server then starts the emotion engine and prepares the system for analyzing the user's voice input and facial expression data.

[1680] Input: Near miss database, LLM model

[1681] Output: System initialized, emotion engine activated

[1682] Specific operations: Loads near-miss data and LLM into the server's internal data memory, executes the emotion engine script, and records in the system log that the user is ready to start using the system.

[1683] Step 2: Image acquisition

[1684] The device starts up the camera and switches to image capture mode. The user selects the room or location they want to analyze and takes an image with the camera.

[1685] Input: Camera

[1686] Output: Captured image data

[1687] Specific operation: The user operates the device's camera app and presses the image capture button. The camera takes an image of the specified location and saves the image data in the device's internal memory.

[1688] Step 3: Send image

[1689] The device converts the captured image data into data packets and sends them to the server, which then confirms receipt.

[1690] Input: Photographed image data

[1691] Output: Image data packet, acknowledgement

[1692] Specific operation: Image data is converted into packet format through the terminal's internal network interface and sent to the server. The server verifies the received data and returns a receipt confirmation response to the terminal.

[1693] Step 4: Image analysis

[1694] The server preprocesses the received image data, which includes noise removal, resolution adjustment, and image cropping if necessary. The preprocessed image data is then input into a large-scale language model (LLM) to analyze risk factors.

[1695] Input: Received image data

[1696] Output: Analysis results

[1697] Specific operation: After using image processing software to remove noise, optimize the image resolution, and crop only the necessary parts, the preprocessed image is input into LLM, which analyzes the risk factors in the image and stores the analysis results in the server's memory.

[1698] Step 5: Sentiment Analysis

[1699] The server receives the user's voice input and facial expression data captured by a camera, and analyzes them using an emotion engine to determine the user's emotional state.

[1700] Input: Voice data, facial expression data

[1701] Output: Emotion analysis results

[1702] Specific operation: The server feeds the voice data acquired from the user and facial expression data captured by the camera to the emotion engine, which performs emotion analysis based on the tone, pitch, and extension characteristics of the voice and the movement of facial muscles. The obtained emotion analysis results are stored on the server.

[1703] Step 6: Accident prediction and prevention measures

[1704] The server generates accident predictions and preventative measures based on the image analysis results from the LLM and the emotion analysis results from the emotion engine. For example, a prediction result such as "there is a high risk of fire due to flammable materials near the stove" may be provided, along with a preventative measure such as "keep flammable materials at a certain distance away from the stove." The instructions are adjusted according to the user's emotional state.

[1705] Input: Analysis results, emotion analysis results

[1706] Output: Accident prediction results, prevention measures

[1707] Specific operation: The processing unit inside the server integrates the results of the LLM and the emotion engine, evaluates the risk level, and then generates appropriate prevention measures. It outputs short and clear instructions to users who are nervous, and more detailed explanations to users who are relaxed.

[1708] Step 7: View the results

[1709] The server sends the generated accident prediction results and preventive measures to the terminal, which then displays them to the user. The display format and content are adjusted based on the user's emotional state.

[1710] Input: Accident prediction results, prevention measures

[1711] Output: What the user sees

[1712] Specific operation: The server converts the generated results into packet format and sends them to the terminal. The terminal interprets the received data and displays it to the user in an appropriate format. For example, the control panel may display a message saying, "There are flammable materials near the stove. There is a high risk of fire. Please move the flammable materials away."

[1713] This allows the system to take into account the user's emotional state, detect hidden risks in the home early, and provide appropriate preventive measures, thereby effectively preventing accidents in the home.

[1714] (Application example 2)

[1715] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1716] It is important not only to predict and prevent potential accidents in the home, but also to present appropriate countermeasures that take the user's emotional state into account. However, conventional systems have had the problem of insufficient safety measures that take the user's emotional state into account, resulting in a lack of improvement in the user experience. The present invention aims to solve these problems and provide more effective accident prediction and prevention measures.

[1717] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for reading a near-miss incident database, means for analyzing received images using a large-scale language model (LLM), means for generating predicted accidents and preventive measures based on the analysis results, means for outputting the generated accident prediction results and preventive measures, means for analyzing the user's emotions, and means for adjusting the prediction results and measures based on the emotion analysis results. This makes it possible to provide appropriate accident predictions and preventive measures that take into account the user's emotional state.

[1718] The "near miss database" is a database that collects near miss cases that have occurred in the past, and records various accident cases that occurred in the home or on mobile vehicles.

[1719] A "large-scale language model (LLM)" is an advanced machine learning model that learns from massive amounts of text and image data, and is used to analyze received images and predict risks.

[1720] "Means for generating predicted accidents and preventive measures based on the analysis results" refers to a means by which the server predicts potential risks based on the image analysis results and derives specific preventive measures to avoid those risks.

[1721] The "means for outputting the generated accident prediction results and preventive measures" refers to a means for displaying or notifying the user of the predicted accident risk and preventive measures on a device such as a terminal or display used by the user.

[1722] The "means for analyzing user emotions" refers to a software routine or model for analyzing user emotions from voice input or camera footage and obtaining the results.

[1723] "Means for adjusting prediction results and countermeasures based on emotion analysis results" refers to means for optimizing the presentation method and content of accident prediction results and preventive measures, taking into account the user's emotional state.

[1724] MODE FOR CARRYING OUT THE INVENTION

[1725] System Overview

[1726] The system of the present invention consists of the following main components:

[1727] 1. Near-miss database: This database collects past near-miss incidents, recording various accidents occurring within the home and on mobile devices.

[1728] 2. Large-scale language model (LLM): A model trained from a near-miss database that analyzes received images and predicts risks.

[1729] 3. Image acquisition means: A camera and its operating interface for acquiring images of the home or moving objects.

[1730] 4. Emotion Engine: This engine analyzes the user's emotions from their voice and facial expressions. The results of the emotion analysis are used to adjust the predictions and preventative measures presented by the system.

[1731] 5. Server: It is the main computing unit for performing image analysis and sentiment analysis, and generating accident prediction and prevention measures.

[1732] 6. Terminal: A device operated by the user that acquires images and provides an interface for displaying analysis results and emotion analysis results.

[1733] Program processing

[1734] The system of the present invention performs processing in the following steps.

[1735] System initialization

[1736] The server loads the near-miss database and large-scale language model (LLM) into memory and performs initialization. The emotion engine is also started at the same time. This completes preparation for the system for analysis and prediction. The device starts the camera and switches to image acquisition mode. A message is displayed indicating that the camera was successfully started, prompting the user to perform an operation.

[1737] Image acquisition

[1738] The user selects the room or specific location they want to analyze (such as the living room or the area around the stove) and takes an image with the camera. The device converts the captured image data into data packets and prepares them to be sent to the server.

[1739] Image analysis

[1740] The server preprocesses the received image data. This includes denoising the image, adjusting the resolution, and cropping the image if necessary. The server then inputs the preprocessed image data into a large-scale language model (LLM). The LLM analyzes the risk elements in the image and generates a prediction result.

[1741] Emotion analysis

[1742] The server inputs the user's voice input and facial expressions captured by the camera into the emotion engine, which then analyzes the user's emotional state and obtains the results.

[1743] Accident prediction and prevention measures

[1744] The server generates specific accident predictions and preventive measures based on the analysis results obtained from the LLM model and the emotion engine. For example, predictions such as "risk of falls in the living room" and "high risk of fire due to flammable objects near the stove" may be provided, along with preventive measures such as "rearranging furniture" and "keeping flammable objects a certain distance away from the stove." If the user is nervous, the server provides concise, explicit instructions, while if they are relaxed, it provides detailed explanations.

[1745] Displaying the results

[1746] The server sends the generated prediction results and preventive measures to the terminal. The terminal confirms that they have been received correctly. The terminal then displays the received accident prediction results and preventive measures to the user. The results are displayed in a format that is easy for the user to check, and the display format and content are adjusted based on the analysis results of the emotion engine.

[1747] Specific examples

[1748] Examples of areas around the kitchen stove

[1749] 1. System initialization

[1750] The server loads the near-miss database and LLM model, starts the emotion engine, and is ready to go.

[1751] The device will start the camera and switch to image capture mode.

[1752] 2. Image Acquisition

[1753] The user selects the area around the kitchen stove and takes a picture with the camera.

[1754] The device sends the captured image of the kitchen to the server.

[1755] 3. Image Analysis

[1756] The server preprocesses the received images and inputs them into the LLM model.

[1757] The LLM model detects the presence of flammable materials near the stove and predicts the risk of fire.

[1758] 4. Emotion analysis

[1759] The server inputs the user's voice input and facial expression data into an emotion engine and analyzes whether the user is nervous.

[1760] 5. Accident prediction and prevention measures

[1761] The server generates a prediction result that "there is a high risk of fire because there are flammable objects near the stove" and a preventive measure that "keep flammable objects a certain distance away from the stove," and incorporates concise and explicit instructions taking into account the results of the emotion engine.

[1762] 6. Displaying the results

[1763] The server transmits the generated prediction results and preventive measures to the terminal.

[1764] The terminal displays the received results to the user, displaying them as brief instructions.

[1765] Example prompts for generative AI models

[1766] "Analyze live video to detect potential risks, recognize the user's emotional state, and suggest appropriate countermeasures based on that."

[1767] The above is a detailed description of the embodiment of the present invention. This allows users to understand potential risks in their homes in advance and take appropriate measures. In particular, it is possible to provide appropriate accident predictions and prevention measures that take into account the emotional state of the user.

[1768] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1769] Step 1:

[1770] The server loads the near-miss database and large-scale language model (LLM) into memory and initializes the system. The emotion engine is also started at the same time. This completes preparation for the system to perform analysis and prediction. The input is the near-miss database, LLM, and the emotion engine program code, and the output is the initialized system state.

[1771] Step 2:

[1772] The device starts the camera and switches to image acquisition mode. It displays to the user that the camera was successfully started and prompts them to operate. The input is the camera start command, and the output is that the camera is ready to acquire images.

[1773] Step 3:

[1774] The user selects the room or specific area (e.g., living room, stove area) they want to analyze and takes an image with the camera. The input is the user's selection and the captured image, and the output is the image data captured by the camera.

[1775] Step 4:

[1776] The device converts the captured image data into data packets and prepares them for transmission to the server. The input is the captured image data, and the output is the data packets for transmission.

[1777] Step 5:

[1778] The terminal sends image data packets to the server. The server receives the data and prepares it for data analysis. The input is the image data packets sent from the terminal, and the output is the received image data ready for preprocessing.

[1779] Step 6:

[1780] The server pre-processes the received image data. Pre-processing includes image noise removal, resolution adjustment, and image cropping if necessary. The input is the received image data and the output is the pre-processed image data.

[1781] Step 7:

[1782] The server inputs the preprocessed image data into a large-scale language model (LLM), which analyzes the risk elements in the image and generates prediction results. The input is the preprocessed image data, and the output is the risk elements and their prediction results.

[1783] Step 8:

[1784] The server receives the user's voice input and facial expressions captured by a camera, and inputs them into the emotion engine. The emotion engine analyzes the user's emotional state and obtains the results. The input is voice data and facial expression data, and the output is the analyzed emotional state.

[1785] Step 9:

[1786] The server generates specific accident predictions and prevention measures based on the analysis results from the LLM model and the emotion engine. The inputs are risk prediction results and the user's emotional state, and the output is optimized accident predictions and prevention measures.

[1787] Step 10:

[1788] The server sends the generated prediction results and prevention measures to the terminal. The input is the optimized accident prediction and prevention measures, and the output is the transmitted data.

[1789] Step 11:

[1790] The device displays the received accident prediction results and prevention measures to the user. The input is the prediction results and prevention measures sent from the server, and the output is the content displayed to the user. The display format and content are adjusted based on the analysis results of the emotion engine, so that the user can easily check the results.

[1791] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1792] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1793] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1794] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1795] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1796] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1797] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1798] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1799] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1800] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1801] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1802] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1803] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1804] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1805] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1806] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1807] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1808] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1809] Furthermore, the hardware structure of these various processors can be, more specific...

Claims

1. A means of reading the near miss database, means for analyzing received images using a large scale language model; A means for generating predicted accidents and preventive measures based on the analysis results; a means for outputting the generated accident prediction results and preventive measures; A system including:

2. The near-miss database contains cases of accidents that occurred within the home.

2. The system of claim 1, wherein the large-scale language model is trained based on the accident cases.

3. The camera captures images of the home, 2. The system of claim 1, further comprising means for transmitting the image to a server.

Images