Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises a camera with a human body induction sensor; the device is used for receiving data acquired by the camera and detecting the existence of people in the vehicle; the device is used for giving an alarm in a voice manner if a person is detected in the vehicle when the vehicle key is pulled out; means for transmitting the alarm information to an external communication device; and means for transmitting notifications to the plurality of registered terminals in accordance with the transmitted alarm information.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech. Summary of the Invention

[0003] This invention provides an information processing system, including a camera with a human body sensor capable of detecting and identifying the presence of occupants inside a vehicle. It is configured to automatically determine the presence of occupants when the vehicle key is removed and promptly issue an alarm via voice, simultaneously sending the alarm information to an external communication device and pushing notifications to multiple registered terminals based on this information. Simultaneously, a mobile device for monitoring the vehicle's interior temperature is included, capable of detecting abnormal temperatures and triggering an alarm process. Furthermore, the system also includes a dialogue device based on natural language processing technology, which proactively interacts with occupants via voice upon receiving an alarm, collecting information about the situation and providing self-rescue or assistance guidance. Relevant information is sent in real-time to external communication devices and management terminals, thereby effectively reducing the safety risks of children left in vehicles.

[0004] "Human body sensor" refers to a sensing device that can detect and sense the presence of a human body inside a vehicle, identifying human activity or body temperature signals through infrared, thermal, or other means.

[0005] A "camera" is a video capture device used to photograph and monitor the interior environment of a vehicle, capture image data, and communicate with a data processing device.

[0006] A "vehicle key" is a control device used to start or disconnect a vehicle's power supply. By detecting whether the key is inserted or removed, the vehicle's operating and departure status can be determined.

[0007] "Alarm information" refers to the prompt data generated by the system when it detects abnormal conditions inside the vehicle (such as occupants left behind or abnormal temperature), and is used to convey warning signals.

[0008] "External communication devices" refer to communication terminals outside the vehicle system that have the function of receiving and displaying alarm information, such as network-connected devices like smartphones and tablets.

[0009] "Multiple registered terminals" refers to multiple user mobile devices or communication terminals that have been pre-registered and associated with the system to receive alarm notifications.

[0010] "Mobile device" refers to a terminal that is easy to carry and can detect the ambient temperature of a vehicle and communicate with the system, such as a monitor or smart handheld device with a temperature sensor.

[0011] "Temperature data" refers to digital information that reflects the temperature status of the vehicle interior environment, collected through mobile devices or temperature sensors.

[0012] "Abnormal temperature" refers to high or low temperature conditions that exceed the safe and comfortable range of a vehicle and may endanger the safety of the occupants.

[0013] Natural Language Processing (NLP) technology refers to information processing methods and algorithms that enable human-computer voice or text interaction, and to understand, analyze, and generate natural language content.

[0014] "Dialogue device" refers to an electronic device that, based on natural language processing, can automatically interact with people inside a vehicle via voice or text, and thereby collect information or provide operational guidance.

[0015] "Situation information" refers to dynamic data collected through dialogue or sensors that reflects the current status, location, and needs of the people inside the vehicle. Attached Figure Description

[0016] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0017] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0018] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0019] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0020] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0021] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0022] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0023] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0024] Figure 9 This represents an emotion map that maps multiple emotions.

[0025] Figure 10 This represents an emotion map that maps multiple emotions.

[0026] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0027] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0028] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0029] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0030] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0031] First, let me explain the terminology used in the following instructions.

[0032] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0033] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0034] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0035] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0036] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0037] First Implementation Method

[0038] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0039] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0040] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0041] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0042] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0043] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0044] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0045] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0046] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0047] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0048] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0049] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0050] Example 1

[0051] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0052] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0053] In this invention, the server includes: a device for determining whether an object exists based on information from a detection device; a device for monitoring startup and operation status and issuing an alarm via sound when conditions are met; a device for sending alarm information to an external information processing device and pushing notifications to multiple terminals via a network; a device for calling a generative artificial intelligence model and automatically generating scenario dialogues and escape guidance based on prompts; and a device for recording and forwarding dialogue content and status information. This enables real-time detection of object presence and environmental anomalies, timely notification across multiple platforms, intelligent dialogue guidance, and emergency collaboration, improving the object's self-rescue capabilities and safety level in abnormal situations.

[0054] "Detection device" refers to a hardware unit or combination module that can acquire and output data on the existence status of objects or environmental parameters (such as temperature, humidity, etc.) within a space.

[0055] "Object" refers to the subject being monitored in a specific space, including but not limited to people, animals, or other targets requiring monitoring.

[0056] "Space" refers to a limited physical area that requires security monitoring and management, such as the interior of a vehicle or a room.

[0057] "Starting device" refers to a device that can indicate or control the operating status of spatial hardware, such as the ignition system of a vehicle engine or a power switch.

[0058] "Operating components" refer to components within the corresponding space used to control entry, exit, or operation, such as car keys, door locks, and handles.

[0059] "Alarm information" refers to notifications generated based on detection results or abnormal situations, used to remind relevant personnel to pay attention or take measures.

[0060] "External information processing device" refers to computing hardware that is connected to the system via a communication network and is capable of receiving and processing information such as alarms or dialogues, including servers and cloud platforms.

[0061] "Terminal devices" refer to various hardware devices that can receive notifications, display information, or allow users to interact, such as smartphones, tablets, and computers.

[0062] "Communication network" refers to wired or wireless networks that enable data exchange between devices in a system, including local area networks, mobile communication networks, and the Internet.

[0063] "Generative artificial intelligence models" refer to algorithmic models and systems that can generate natural language dialogues, suggestions, or operational instructions based on input content.

[0064] "Prompt statements" are instructional texts used to guide generative artificial intelligence models in generating specific content or dialogue.

[0065] "Scenario-based dialogue" refers to multi-round interactive communication content automatically generated by the system, which involves inquiring about the situation, providing reassurance, and offering emergency guidance tailored to specific scenarios.

[0066] "Status information" refers to real-time status data and related descriptive information of a space or object obtained through detection, dialogue, or other methods.

[0067] "Environmental information" refers to a set of parameters that reflect physical environmental data such as temperature and humidity within a space.

[0068] This invention relates to a system for the safety protection and intelligent rescue of objects (such as young children) in specific spaces (such as inside a vehicle). The system consists of a terminal, a server, and multiple user terminal devices working together, with each component responsible for different functional modules, forming a highly efficient safety linkage system.

[0069] The terminal can utilize embedded computing hardware (such as Raspberry Pi, embedded controllers), integrated human body sensors (such as infrared sensors), cameras, temperature sensors, microphones, and speakers. Installed in a vehicle or other space, the terminal continuously acquires data on the presence of objects and environmental parameters, and monitors the vehicle's startup status and the position of operational components such as the key. The terminal uses a Linux operating system, OpenCV (an image processing library), a local database, and timed data acquisition and uploading software modules. Temperature data is acquired and periodically stored by a digital temperature sensor (such as a DS18B20), while human body detection and vehicle status are monitored in real time by corresponding sensor modules.

[0070] When the terminal detects that the engine is off and the key has been removed, or that the ambient temperature exceeds a set threshold (e.g., 35 degrees Celsius), it will immediately issue a voice alarm through the speaker (e.g., "Someone is inside the vehicle, please check immediately!" "The interior temperature is too high, please open the door immediately!"). Simultaneously, the terminal uploads the detected alarm information to the server via HTTP or MQTT protocols using communication modules such as 4G or Wi-Fi. The uploaded data includes the terminal device ID, detection type, timestamp, and sensor results.

[0071] The server can be configured based on a general cloud platform (such as ECS cloud servers) or a local rack server, utilizing API services (such as Python Flask, Django, etc.) to receive and parse alarm data from multiple endpoints. The server stores the data in a database (such as MySQL) and monitors for new alarms that need to be pushed. The server integrates with third-party push service platforms (such as JPush, Firebase Push, etc.) to send notifications to all registered user terminal devices (such as mobile phones, tablets, etc.). For example, the relevant push content might be: "Alert: A person has been detected abandoned in the vehicle; please confirm as soon as possible."

[0072] Upon receiving an alert, the server automatically invokes a generative artificial intelligence model. This model could be ChatGPT, a large natural language processing API, or a dedicated chatbot platform. The server sends a pre-defined prompt:

[0073] “A child has been detected left in the vehicle. Please speak with the child to confirm the situation and provide guidance.”

[0074] "The temperature inside the car has exceeded the safe threshold. Please give a gentle warning to the children in the car and instruct them to take safety measures."

[0075] Generative AI models combine prompts and actual detection data to automatically generate targeted multi-turn dialogue content. The server then transmits the generated dialogue content back to the terminal in text and voice formats, where the terminal interacts with objects in the space through a speaker, inquiring about the situation and providing operational suggestions, such as "Can you find the unlock button on the car door? Can you press it?"

[0076] All AI-generated dialogue processes and interaction results are recorded in the database in real time by the server and synchronized to all relevant user terminal devices via push notifications, making it easy for parents or administrators to monitor the space situation in real time and intervene in a timely manner.

[0077] For example, at 9 AM one day, the terminal detects that the vehicle is turned off and the key has been removed, but there is still a movement signal inside the car. The terminal immediately issues a voice alarm and uploads the alarm information to the server. The server pushes a message to three parents: "Alarm: A person has been detected left in the vehicle." At the same time, the AI ​​model is activated, and a dialogue is generated based on the prompt: "Please check the situation of the child in the car and guide them to help themselves." The AI ​​first sends, "Hello, little friend, can you hear me?" If the child responds, it continues, "Please look around for a red button. This is the unlock button. Please try it." The server pushes this dialogue process to all parents' terminals in real time.

[0078] This invention provides a highly integrated, multi-layered security and intelligent interactive space safety system that can be widely used in vehicles, classrooms, enclosed spaces and other fields, and is particularly suitable for scenarios that require key safety protection for vulnerable groups.

[0079] use Figure 11 The processing procedure is explained.

[0080] Step 1:

[0081] The terminal periodically collects information on the existence of objects in the space and environmental data.

[0082] Inputs include: human body sensor signals, camera images, temperature sensor values, etc.

[0083] The terminal identifies and processes the sensor input data, judges human movement, features in the image, temperature, etc., outputs whether there is an object in the current space and the current environmental parameters, and stores the results in local storage.

[0084] Step 2:

[0085] The terminal monitors the status of the starting device and the operating components in real time.

[0086] The inputs include: the vehicle engine switch signal, key insertion / removal status, etc.

[0087] The terminal reads vehicle bus information, judges engine status and key insertion / removal, and outputs monitoring results on whether the engine is off and whether the key has been removed.

[0088] Step 3:

[0089] The terminal performs anomaly detection and alarm when the conditions are met.

[0090] The input is the result of step 1 and step 2.

[0091] The terminal determines whether someone has been left behind or if there is an abnormal temperature by comparing data with evidence of engine shutdown, key removal, or ambient temperature exceeding a threshold, and then makes logical judgments. If someone is found to have been left behind or the temperature is abnormal, an alarm is triggered, and a voice warning is played through the speaker.

[0092] Step 4:

[0093] The terminal reports the alarm information to the server.

[0094] The inputs are: alarm status, detection data, terminal number, timestamp, etc. generated in step 3.

[0095] The terminal encapsulates alarm data and uploads it to the server interface via 4G, Wi-Fi, or other means, outputting a report message containing detailed status information.

[0096] Step 5:

[0097] The server receives and parses the alarm information.

[0098] The input is: the alarm data packet sent by the terminal.

[0099] The server parses the message packet, extracts the object type, time, and reason for the exception, records the alarm in the database, determines whether to distribute the notification, and outputs the database record and the notification tasks to be pushed.

[0100] Step 6:

[0101] The server sends notifications to registered user terminals via push service.

[0102] The input consists of: the content of the alarm notifications to be pushed from the database, and a list of registered users.

[0103] The server retrieves all relevant user devices, calls the push API, assembles the notification text, and sends it, such as to the parent's mobile app, outputting feedback information indicating whether the push was successful or failed.

[0104] Step 7:

[0105] The server invokes a generative artificial intelligence model to generate emergency dialogue content.

[0106] The input includes: alarm event information, detected abnormal content, and preset prompt statements (such as "A child has been found left behind in the vehicle. Please confirm the situation with the child and guide them on how to proceed").

[0107] The server sends the input to the AI ​​model, which performs natural language processing and reasoning to generate targeted, multi-turn communication text. The output is the generated dialogue content.

[0108] Step 8:

[0109] The terminal receives AI dialogue content returned by the server and interacts with objects in the space.

[0110] The input is the dialogue text generated by the server.

[0111] The terminal uses a text-to-speech module to convert the content into speech, plays it through a speaker, and listens for the subject's voice feedback (such as a child's response). The output is a record of the interactive Q&A with the subject.

[0112] Step 9:

[0113] The server records the entire dialogue process and updates the user's terminal synchronously.

[0114] The input consists of: the dialogue content, interaction details, and new status information returned by the terminal.

[0115] The server organizes and archives all interaction data, analyzes the latest status, and pushes important information to the user's terminal in real time, outputting updated dialogue logs and user messages.

[0116] Step 10:

[0117] Users receive notifications and make operational decisions through their terminals.

[0118] The input includes: server-push alerts, conversation content, and the latest status information.

[0119] The user reads the notification content, combines it with the interaction information between AI and the object, and decides on actions such as calling for rescue, rushing to the scene, or providing remote guidance, outputting the response measures to be taken by the user.

[0120] Application Example 1

[0121] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0122] Existing safety management systems often suffer from problems such as delayed response, insufficiently specific alarm information, lack of effective interaction, and inability to dynamically respond based on actual conditions and personnel emotions. Especially in dangerous or special locations such as factories and vehicles, existing systems struggle to collect personnel status and environmental information in a timely and accurate manner and provide personalized instructions during emergencies. This can easily lead to untimely evacuation, low rescue efficiency, and an inability to fully protect personnel safety. Furthermore, it makes it difficult for managers to grasp the true situation on-site in a timely manner.

[0123] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0124] In this invention, the server includes: a device for receiving sensor data from a detection device and detecting the presence of people in a space; a device for outputting a sound signal when a predetermined trigger condition is met and people are present in the space; a device for receiving temperature data from a measuring device and determining temperature anomalies; a device for sending anomaly detection information or personnel presence detection information to a communication device; a device for sending notifications to multiple information terminals based on the above information; a device for generating prompt content using a generative artificial intelligence model; a device for engaging in dialogue with the user and collecting status information or response information based on the prompt content using natural language processing technology; a device for generating and outputting appropriate instruction content based on the collected information; and a device for saving all status information and response information to a storage device. Furthermore, it may also include a device for making sentiment inferences based on prompt content generated by a generative artificial intelligence model, collected status information, or response information, or a device for dynamically optimizing instruction content based on sentiment inference results and status information and notifying multiple terminals. This enables intelligent real-time monitoring and management of personnel and environmental status, timely automatic alarms, and personalized, dynamic, and context-appropriate safety dialogues and instructions with users. Furthermore, it allows for customized response plans based on users' emotional states, thereby significantly improving the automation and reliability of safety management and effectively protecting the lives and property of personnel.

[0125] "Detection device" refers to various sensing devices that can acquire information about the status of people or the environment in a space, including but not limited to human sensors, cameras, etc.

[0126] "Sensing data" refers to digital or analog data collected and output by detection devices to reflect the status of people, temperature, etc. in a space.

[0127] "Space" refers to the physical environment area that is being detected or monitored, such as factory areas, vehicle interiors, warehouses, etc.

[0128] "Measuring device" refers to a sensing device that can measure environmental parameters (such as temperature), including temperature sensors, humidity sensors, etc.

[0129] "Temperature data" refers to information that represents the ambient temperature value obtained through a measuring device.

[0130] "Communication equipment" refers to terminal equipment capable of receiving, sending, and exchanging data, including servers, smartphones, tablets, network repeaters, etc.

[0131] "Information terminal" refers to user-side devices that can receive and display notification information, including wearable devices, mobile terminals, desktop computers, etc.

[0132] "Storage device" refers to a storage device used to save and manage data or information generated by a system, including local storage, cloud storage, databases, etc.

[0133] "Generative AI models" refer to AI technology models that can automatically generate text, speech, dialogue, or other data forms based on input content, such as deep learning generative models and natural language generation models.

[0134] "Prompt content" refers to specific text, voice, or image data generated by a generative artificial intelligence model based on system detection results and on-site conditions, used for notification, instruction, or dialogue.

[0135] Natural Language Processing (NLP) refers to various algorithms and technologies that use computers to understand and generate human language (text or speech), including but not limited to word segmentation, semantic understanding, and automatic question answering.

[0136] "Status information" refers to various data information that reflects the current status, environmental conditions, and event progress of the monitored object.

[0137] "Response information" refers to the feedback data, such as text and voice, that users provide in response to system prompts.

[0138] "Instruction content" refers to the information generated by the system based on the current situation and user response, used for guidance, reminders, control, and other purposes.

[0139] "Emotional estimation" refers to the methods and processes used to process user input data and estimate their emotional state (such as anxiety, calmness, panic, etc.) using sentiment analysis algorithms or models.

[0140] To facilitate the implementation of this invention, a specific embodiment of the invention is described in detail below.

[0141] This invention relates to a security monitoring and dialogue system. This system utilizes detection devices (such as human-sensing cameras, temperature sensors, etc.), information terminals (such as wearable devices, mobile devices, etc.), servers, and generative artificial intelligence models to achieve functions such as real-time environmental monitoring, emergency warnings, interactive commands, and emotion recognition. The main hardware includes: embedded camera equipment (such as a camera integrated into a single-board computing platform), temperature sensors (such as general-purpose digital temperature sensors), information terminals (such as smart wearable devices, mobile communication terminals), a central server (optionally implemented on a cloud platform), and network communication components. The core software includes a database management system (such as a relational database), a natural language processing platform, generative artificial intelligence models (such as AI APIs for text generation or dialogue generation), push message services, and a sentiment analysis module.

[0142] In practice, endpoints (such as human-sensing cameras) periodically collect images of the area. A local computing unit performs basic human detection and sends data such as the presence of personnel, corresponding time, and endpoint ID to the server via a wireless network. Endpoints (such as temperature sensors) periodically collect ambient temperature data and upload it to the server in real-time as data packets. Upon receiving this data, the server uses a pre-defined safety threshold algorithm to determine if a person is in a high-risk environment or if the ambient temperature is abnormal. If the detection results indicate that a person is in a high-risk environment or that the ambient temperature is abnormal, the server automatically generates and outputs an alarm signal, pushing voice or text alerts to information terminals, on-site speakers, wearable devices, etc., via wired or wireless means. Simultaneously, the server invokes a generative artificial intelligence model to generate prompts for dialogue based on the alarm scenario and environmental data, and uses natural language processing technology to engage in two-way interaction with the user. Users can reply via voice or text; the system automatically collects and analyzes this information, and uses an emotion inference module to analyze the user's current state, further adjusting the instructions and interaction methods to achieve personalized security measures.

[0143] For example, if a person is detected in an area at the end of a factory production line and the ambient temperature reaches a dangerous threshold (e.g., 40 degrees Celsius), the system will push a voice or text alert via wearable devices or other information terminals: "Factory temperature abnormal, please evacuate this area immediately!" and immediately initiate a dialogue: "What is the situation in the area now? Do you need help?" If the user replies "I feel very hot and unwell," the system can provide a humanized response through sentiment analysis: "Please remain calm, rescue teams are on their way, please check if there are any other people nearby." All dialogue and status information are synchronously stored in the server database for subsequent management and analysis.

[0144] For example, to address the risk of people remaining inside vehicles, when the terminal (human-sensing camera) detects that the vehicle is turned off and there are people inside, it automatically sends a prompt such as "Person detected inside the vehicle, please confirm immediately!" through the information terminal, and engages in dialogue with the monitored person to continuously track their status.

[0145] This system has broad application prospects in various hazardous work sites and densely populated areas. It forms an intelligent closed loop of data flow, command flow, and emergency response flow between the server, terminals, and users, effectively improving safety management levels in various scenarios.

[0146] Commonly used prompt statements include:

[0147] "The factory temperature has reached a dangerous level. Please immediately implement the evacuation procedures and restore the current situation."

[0148] "Someone has been detected in the danger zone. Please describe the current situation."

[0149] "The temperature inside the vehicle exceeds the safe threshold. Please check and ensure safety as soon as possible."

[0150] "Are you safe right now? Do you need any assistance?"

[0151] "Please remain calm and wait for the rescue team to arrive. Do not rush into any action."

[0152] In summary, through the software and hardware integration of the various components and processes of this invention, intelligent and efficient fully automated operation of the entire process, including hazard identification, alarm notification, contextual dialogue, instruction push, and emotional care, can be achieved, ensuring user safety.

[0153] use Figure 12 The processing procedure is explained.

[0154] Step 1:

[0155] The terminal uses detection devices (such as human detection cameras and temperature sensors) to periodically collect image data and ambient temperature data within the monitored area. The input is real-time images and temperature information of the physical environment. The terminal performs image recognition to determine if there is human activity within the area, and generates a data packet by combining the recognition results with the temperature measurement values. The output is a data packet containing the human detection status and temperature values.

[0156] Step 2:

[0157] The terminal uses a network communication module (such as WiFi or cellular network) to upload the above data packets to the server. The input is the data packet generated by the terminal. The terminal packages and formats the data and sends it to the server's API interface via a specified communication protocol (such as HTTP, MQTT, etc.). The output is the data stream that has reached the server.

[0158] Step 3:

[0159] The server receives the data stream uploaded by the terminal and stores the personnel detection status and temperature data in the database system. The input is the received data stream containing personnel status and temperature values. After data verification, classification, and information integrity checks, the server performs persistent data storage. The output is the historical status records in the database.

[0160] Step 4:

[0161] The server periodically analyzes new data in the database to determine if there are any preset risk conditions (such as temperature exceeding a threshold or personnel detected in a dangerous area). The input is the latest personnel detection and temperature data from the database. The server calls a logical judgment program to perform threshold comparisons and condition checks on each data entry. The output is the judgment result for either an abnormal event or a normal state.

[0162] Step 5:

[0163] When the server detects a risk event, it automatically generates an alarm message and calls the speech synthesis module to generate corresponding voice content. Simultaneously, it pushes text and voice alarms to endpoints (such as information terminals, wearable devices, and on-site speakers) via the network. The input is the risk assessment result. The server converts the assessment information into multimedia alarm content and broadcasts it to all target terminals via a push service. The output is that the terminal device is in an alarm state.

[0164] Step 6:

[0165] Based on risk events, the server invokes a generative artificial intelligence model to generate prompts relevant to the situation (such as natural language questions for dialogue). The input is risk scenario data (such as anomaly types, environmental parameters, etc.). The server then inputs this data into the AI ​​model to generate prompts specific to the scenario. The output is personalized dialogue prompt text.

[0166] Step 7:

[0167] The server uses natural language processing technology to generate prompts and initiate a dialogue with the user via a messaging terminal. Input includes the prompt text and the user's voice or text response. The server performs language understanding and key intent recognition on the user's feedback, and generates subsequent communication information based on the dialogue content. Output includes responses, suggestions, or new instructions to the user.

[0168] Step 8:

[0169] The server uses a sentiment analysis module to analyze user responses and determine the user's current psychological state. The input is the user's natural language feedback. The server uses a sentiment analysis algorithm to analyze the text and infer whether the user is calm, nervous, or panicked. The output is the user's emotion type and sentiment label.

[0170] Step 9:

[0171] The server dynamically adjusts and optimizes alarm content, dialogue prompts, and instructions based on the user's emotional state, and pushes the final instruction or notification to all relevant information terminals. Input includes the user's emotional type, the scene situation, and dialogue history. The server integrates this information and selects the most appropriate content to push. Output is a dynamically optimized instruction, notification text, or voice, which is broadcast to the designated terminal.

[0172] Step 10:

[0173] After receiving the notification, the user takes appropriate safety measures according to the instructions through an information terminal (such as a wearable device or mobile phone). The input consists of the alarms, instructions, and suggestions received by the terminal. Based on the system feedback, the user takes actions such as evacuation, calling for help, and reporting the current status. The output is the user's actual actions, and the user can continue to report the current status through the terminal, forming a closed-loop interaction.

[0174] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0175] Example 2

[0176] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0177] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0178] In this invention, the server includes an information processing device for receiving data from a human perception information acquisition device and a temperature detection information device; a notification integration device for executing audio alarms and remote notifications based on judgment results; a communication control device for enabling data linkage between various devices; an artificial intelligence processing device for dialogue support through a generative artificial intelligence model for natural language dialogue, situation collection, and emotion analysis; and an information sharing device for pushing collected intelligence and emotional states to guardians. This enables high-precision presence detection and environmental monitoring of children or vulnerable individuals in the vehicle, automatically pushing alarms to multiple terminals in real time, and combining artificial intelligence for situation assessment and humane reassurance, providing guardians with real-time and complete information and emergency response guidance, thereby significantly improving the level of in-vehicle safety management.

[0179] "Human perception information acquisition device" refers to sensors and related acquisition modules installed in a vehicle to detect the presence or movement of people, including but not limited to infrared sensors, cameras and other devices.

[0180] "Information processing device" refers to a software or hardware processing unit that can receive and analyze data from various sensing and environmental information acquisition devices, and make status judgments and generate alarms and other responses.

[0181] "Vehicle operating device" refers to the related sensing and control modules used to detect and acquire operational information such as vehicle power status, engine status, and key insertion / removal.

[0182] "Notification device" refers to a device used to send audio warnings, voice prompts, and other sound and light reminder signals to the inside or outside of the vehicle.

[0183] "Communication device" refers to a communication module that enables the exchange and transmission of alarm information, data and instructions between the vehicle and an external network or server.

[0184] "Telematics processing device" refers to a server or cloud processing system that receives vehicle status data and alarm information through a communication network outside the vehicle, and then performs subsequent analysis, storage and push.

[0185] "Information terminal" refers to various user devices that can receive alarm notifications and interact with information, such as smartphones, tablets, and wearable devices.

[0186] "Notification integration device" refers to a system or module that integrates and manages alarm and environmental information and pushes this information to multiple information terminals in a unified manner.

[0187] "Communication control device" refers to the software and hardware modules that control and coordinate data transmission and communication flows between various devices in a system.

[0188] "Information management device" refers to a data management module that stores, manages, and retrieves all alarm logs, environmental information, personnel status data, etc. collected in the system.

[0189] "Environmental information acquisition device" refers to various sensor devices that can sense and detect environmental parameters (such as temperature and humidity) inside the vehicle.

[0190] "Temperature detection information acquisition device" refers to sensors and modules specifically designed for real-time detection and output of vehicle interior temperature data.

[0191] "Artificial intelligence processing device" refers to a data processing system that can realize natural language processing, information interaction, and reasoning and judgment based on artificial intelligence technologies such as machine learning and deep learning.

[0192] "Generative AI model" refers to an AI model system that uses generative pre-training technology to generate content, perform automatic dialogue, and reason about scenarios.

[0193] "Dialogue support device" refers to a device or module based on artificial intelligence model that can achieve multi-turn natural language interaction with people in the vehicle, obtain status, analyze emotions, and provide guidance and suggestions.

[0194] "Information sharing device" refers to a module used to integrate status information and alarm data collected and analyzed within the system, and then push or distribute them to a remote processing system or multiple user terminals.

[0195] To better understand the implementation of the system of the present invention, a detailed description is provided below, taking into account the specific hardware and software structure, data processing flow, and actual application scenarios.

[0196] This invention relates to an in-vehicle safety monitoring and intelligent dialogue system, which consists of a terminal, a server, and various information terminals used by the user. The system primarily achieves automatic detection and alarm push for occupants left behind in the vehicle or abnormal environmental conditions. It also interacts with occupants left behind in the vehicle through a generative artificial intelligence model, effectively enhancing in-vehicle safety.

[0197] Terminal section:

[0198] The terminal equipment includes human perception information acquisition devices (such as infrared sensors and cameras) and temperature detection information acquisition devices (such as temperature sensors) configured inside the vehicle. These hardware components can utilize general-purpose single-board computers (such as Raspberry Pi), microcontrollers (such as Arduino), and corresponding sensor acquisition modules. The terminal performs periodic data acquisition, processing, and status judgment through local programming, has local audio playback functionality (such as outputting alarms via speakers), and communicates with the server via Wi-Fi, 4G, or other methods.

[0199] Server components:

[0200] The server typically utilizes cloud servers or edge computing devices to achieve unified reception and processing of data from all endpoints. The server can run a Linux system and employ backend development technologies such as Python and Node.js, combined with a database to manage alarms, logs, and status information. The server integrates a communication management module, utilizing communication protocols such as HTTP / REST and MQTT for bidirectional data interaction with remote terminals. Simultaneously, the server is also equipped with an artificial intelligence processing unit, capable of calling generative AI models (such as a GPT-based dialogue engine) to automatically generate and push multi-terminal warning information based on received alarms, achieving natural language dialogue and sentiment analysis.

[0201] User section:

[0202] Users can interact with the system in real time via a companion app installed on smartphones, tablets, and other information terminals. Within the app, users can set temperature thresholds, receive real-time push notifications for alerts, and observe the dialogue and sentiment analysis results between the AI ​​and occupants. Users can then take emergency intervention measures based on the information provided by the system to ensure the safety of those inside the vehicle.

[0203] Specific examples:

[0204] Imagine a vehicle has been parked and the driver has left. The terminal camera, using a human detection module, detects that a child is still left inside. The terminal broadcasts an alarm through its speaker while simultaneously uploading data and images of the abnormal scene wirelessly to a server. Once the server determines it's an emergency, it automatically pushes a message, "Emergency! Someone has been left in the car!" to the parents' and guardians' mobile phones and other devices. At the same time, the server activates a generative artificial intelligence model to engage in dialogue with the child inside the car. For example, the AI ​​can gently say to the child, "Hello, little one, it's safe now. Can you hear me? Tell me where you are." If the AI ​​detects that the child is distressed or calling for help, it will promptly soothe the child and guide them on self-rescue or waiting for assistance, and push all dialogue and emotional reports to the guardian.

[0205] Example of a prompt statement:

[0206] When you speak to a child left in a car, prioritize calming them down, asking about their physical sensations and location, and instructing them on how to remain safe while waiting for rescue.

[0207] If you detect your child saying "I'm scared" or "Help," your first words of comfort should be to guide them to press the horn or find the door button.

[0208] Send all the children's answers and noise clues from the scene to the guardians, and advise them to go to the vehicle immediately.

[0209] As the child conversation assistant, you have detected a child forgotten in the vehicle. Please speak to the child in a gentle, reassuring, and easy-to-understand tone, asking about their status and location as much as possible, and providing safety guidance.

[0210] This system uses general-purpose hardware such as Raspberry Pi and Arduino, combined with generative artificial intelligence models and communication push technology, which greatly enhances the safety protection capabilities for children and vulnerable groups in vehicles, and can realize real-time detection, warning, intelligent dialogue and emergency response in emergency situations.

[0211] use Figure 13 The processing procedure is explained.

[0212] Step 1:

[0213] The terminal periodically collects information on the vehicle's interior environment and personnel. Input consists of raw data collected in real-time by hardware devices such as infrared sensors, cameras, and temperature sensors. The terminal, through local programming, acquires data such as the presence of people, their current location, and the real-time temperature inside the vehicle, performing data analysis and preliminary status determination on various signals. Output consists of structured personnel status data and environmental parameters.

[0214] Step 2:

[0215] The terminal determines the vehicle's status and safety conditions. Inputs include the vehicle's power status, key insertion / removal information, and the personnel and temperature data obtained in step 1. The terminal combines this information with signals from the vehicle's onboard system to determine whether the vehicle is off and locked, and accordingly decides whether to initiate the anomaly detection process. Outputs include the vehicle's current status and the result indicating whether the alarm triggering conditions are met.

[0216] Step 3:

[0217] The terminal determines whether a risk event exists and triggers an alarm. Inputs include vehicle status, occupant information, temperature readings, and user-defined safety thresholds. The terminal compares the actual values ​​with the preset thresholds. If it detects a risk such as someone remaining inside or an excessively high temperature, it issues a real-time voice alarm via speaker and simultaneously uploads the event data to the server. Outputs include a voice warning and an alarm data packet.

[0218] Step 4:

[0219] The terminal uploads detected alarm data to the server via the communication module. Input consists of identified alarm event information, including personnel status, environmental data, alarm type, and event time. The terminal formats and encrypts this data before sending it to the remote server via the communication network. Output is the completed remote data upload operation.

[0220] Step 5:

[0221] The server receives and parses alarm data. The input is the alarm data packet from the endpoint. The server unpacks, verifies, and analyzes the received data, stores the alarm information in the database, and prepares notification content. Data processing includes event type determination and filtering of associated user terminals. The output is an alarm message object that can be pushed and the stored alarm records.

[0222] Step 6:

[0223] The server uses a push notification service to send alarm notifications to registered users' information terminals. The input consists of the alarm message object generated in step 5 and the user information. The server calls the push interface to push the specific alarm content (e.g., "Child detected inside the vehicle" or "Abnormal temperature inside the vehicle") to the user's smartphone, tablet, or other devices in real time. The output is the alarm notification that pops up on the user's terminal.

[0224] Step 7:

[0225] The server automatically invokes a generative artificial intelligence model based on the alarm type to initiate intelligent dialogue. Input includes alarm details, personal identification information, and preset prompts. The server provides specific prompts to the AI ​​model, such as "You need to soothe and guide the child currently trapped in the car," and the AI ​​generates information for dialogue with the occupants. Output is dialogue text or voice content appropriate for the current situation.

[0226] Step 8:

[0227] The server transmits AI-generated dialogue to occupants via speakers or a display screen and collects their feedback. The input is an AI-generated greeting or instruction phrase, and the output is the occupants' voice and behavioral feedback signals. The server performs sentiment analysis and content recognition on the collected feedback information to further adjust subsequent dialogue.

[0228] Step 9:

[0229] The server pushes sentiment analysis, dialogue content, and on-site data to the parent's or guardian's terminal. The input consists of the feedback information and analysis results obtained in step 8. The server integrates the dialogue record, emotional state, and suggested actions, and delivers them to each guardian and family member via push message. The output is a detailed on-site report and guidance received by the family member's terminal.

[0230] Step 10:

[0231] Users take appropriate action based on alarms and system notifications. Inputs include alarm notifications, dialogue content, and operational suggestions. After viewing detailed information on their mobile phones or messaging devices, users assess the urgency of the situation and respond accordingly, such as notifying other family members or immediately heading to the vehicle's location. Outputs indicate whether the user has proactively provided assistance or taken further safety precautions.

[0232] Application Example 2

[0233] The process of a specific process in Application Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. In addition, the data processing device 12 is referred to as the "server", and the smart device 14 is referred to as the "terminal".

[0234] In recent years, with the popularization of autonomous vehicles and smart carriers, the risk of personal safety accidents caused by infants and children left in vehicles has been increasing continuously. Traditional vehicle safety systems cannot effectively and real-time detect the presence and emotional state of monitored objects inside the vehicle, lack comprehensive monitoring and multi-party notification mechanisms for abnormal environments (such as high temperature), and cannot provide timely interaction and guidance. Therefore, there is an urgent need for a comprehensive safety system that can automatically detect targets and environmental states inside the vehicle, give alarms in a timely manner, interact with monitored objects, provide dynamic instructions, and notify managers and relevant terminals in real time, so as to ensure the life safety of people inside the vehicle, especially children.

[0235] The specific process executed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is achieved by the following means.

[0236] In the present invention, the server includes a biometric detection image acquisition device, an environmental state detection terminal, an information processing device, an alarm unit, a communication unit, an artificial intelligence dialogue unit, a situation analysis and instruction generation unit, and a record sending unit for simultaneous multi-terminal notification. Thus, it is possible to achieve automatic detection of monitored objects in a vehicle or space, multi-factor joint discrimination such as high-temperature abnormality, multi-channel automatic alarm in case of abnormality, generate prompt statements through artificial intelligence to interact with monitored objects in real time, analyze the emotional state, and automatically generate specific response instructions. At the same time, all alarm and interaction information can be synchronously notified to multiple information terminals in real time, realizing real-time multi-party linkage response, and significantly improving the vehicle safety and emergency handling capabilities.

[0237] The "biometric detection function" refers to a technical function that can detect or identify whether there is a human or animal with physiological characteristics in a space.

[0238] The "image acquisition device" refers to a device used to capture, record, and output spatial image information.

[0239] The "information processing device" refers to an arithmetic unit with the capabilities of data reception, storage, analysis, and processing, which can perform logical judgment and feedback control on input information.

[0240] The "discrimination unit" refers to an information processing module that judges the existence or state change of a target object according to set conditions.

[0241] An "alarm unit" refers to an output module that sends warning information via sound, light, or other means when an abnormality or early warning condition is detected.

[0242] "Communication function" refers to the ability of a device to exchange data with external devices or networks via wired or wireless means.

[0243] "Transmission unit" refers to a functional component that transmits information from this system to external devices or systems.

[0244] A "transmission unit" refers to a module that distributes notifications or data to multiple terminals according to predetermined conditions.

[0245] "Information terminal device" refers to a portable or fixed terminal device that can receive and display information sent from a system for prompting or warning purposes.

[0246] "Response information" refers to the information returned to the system by the monitored object or related terminal after interaction or receiving instructions.

[0247] A “dialogue unit” is a module that can interact with the monitored object via voice or text to achieve information collection and feedback.

[0248] "Instruction unit" refers to a device module that generates and outputs specific action suggestions or operation prompts for an object based on the analysis results.

[0249] The "record sending unit" refers to the module that organizes event information, interaction history, and analysis results and forwards them to external communication devices.

[0250] "Environmental condition detection function" refers to the ability to sense and acquire physical parameters such as temperature and humidity in the environment and output detection results.

[0251] "Artificial intelligence unit" refers to an information processing module that is based on machine learning or deep learning algorithms and can automatically understand and process human language data to generate decision-making or guidance content.

[0252] A “guidance unit” refers to a unit that provides reassurance, guidance, or behavioral suggestions to the monitored object based on information analysis.

[0253] This invention relates to a comprehensive safety system based on multiple information acquisition, analysis, and interaction methods, suitable for preventing accidents caused by monitored objects (such as infants or children) left behind in autonomous vehicles. The system mainly includes a biometric detection image acquisition device, an environmental status detection terminal, an information processing device, an alarm unit, a communication unit, an artificial intelligence dialogue unit, a status analysis and instruction generation unit, and a multi-terminal synchronous notification recording and sending unit. To implement this invention, the following specific hardware and software can be used:

[0254] On the hardware side, the terminal can utilize a camera and human body sensor based on a development board (such as Arduino), as well as a portable device equipped with a temperature sensor (such as a Raspberry Pi with a temperature probe). Speakers, buzzers, and LEDs can also be integrated to meet multi-channel alarm requirements. The server portion can be deployed on general-purpose server equipment or cloud service platforms such as cloud computing infrastructure. Various information terminal devices can be smartphones, tablets, and other devices with network information receiving capabilities.

[0255] On the software side, the information processing device runs control and analysis programs to perform human figure recognition on the acquired images and threshold judgment on temperature data. The alarm unit and data communication unit can be implemented using an operating system (such as Linux) and network transmission libraries (such as MQTT and HTTP) on an embedded system. The server side deploys a data parsing module, a push notification system, a multi-terminal message distribution module, an AI dialogue system, and a sentiment analysis engine. The AI ​​dialogue unit can call generative AI models, such as open-domain natural language generation models, to process generated prompts, dialogue content, and emergency commands. The sentiment analysis engine uses a cloud-based sentiment API for user emotion recognition.

[0256] The workflow is as follows: The terminal periodically collects images and ambient temperature information from inside the vehicle. When it detects someone inside the vehicle and an abnormal temperature, it immediately alerts the user with an audible and visual alarm and sends the alarm information to the server. Upon receiving the alarm, the server immediately pushes a notification to multiple registered information terminals, with content such as "Warning: A monitored object has been detected inside the vehicle with an abnormal temperature. Please handle immediately!" The server also activates a natural language generation dialogue engine to initiate interaction with the monitored object inside the vehicle, such as "Hello, little friend, can you hear me?" If the user responds (e.g., "I'm next to the car door"), the server uses sentiment analysis and AI dialogue to further guide the user to take self-rescue measures, such as "Please try pressing the red button on the car door" or "Please knock on the car window to send a signal." The server synchronously distributes all alarm events, sentiment analysis results, and interaction content to multiple information terminals, including those for parents and administrators, in real time to ensure timely and coordinated rescue.

[0257] Here's a specific example: In the hot summer, after the driver gets out of the car, the terminal's human detection camera detects that a child is still inside, and the temperature sensor detects a temperature exceeding 35 degrees Celsius. The terminal immediately issues a voice warning and triggers a high-decibel alarm via buzzer, while simultaneously illuminating the warning lights. The alarm information is uploaded to the server via the 4G / 5G network. Upon receiving the alarm, the server pushes a warning message to the parents' and park management personnel's mobile phones within seconds, prompting them to go to the vehicle immediately. Subsequently, the server activates the AI ​​dialogue system to communicate with the child inside the car: "Hello, little friend, are you feeling alright?" If the child responds with "a little scared," the server, through emotion analysis, detects their anxiety and will send reassuring and instructive statements: "Don't worry, assistance is on its way. Please see if there are any red buttons you can press."

[0258] Here are some examples of prompts for generative artificial intelligence models:

[0259] Hello, little one, we've detected you're in a car, and it's a bit hot now. Please tell me your location. Are you feeling alright? Are there any buttons nearby that you can press?

[0260] Don't panic, someone will come to help you soon. Could you describe your surroundings to me?

[0261] Try tapping on a window or pressing any visible alarm button.

[0262] Through the above specific measures and interactive implementation, this invention can effectively and promptly detect abnormal conditions of the monitored objects inside the vehicle, automatically issue alarms, prompts, reassurances and guidance, achieve 360-degree intelligent vehicle safety protection, and significantly reduce the probability of danger to those left behind.

[0263] use Figure 14 The processing procedure is explained.

[0264] Step 1:

[0265] The terminal collects data periodically. It acquires in-vehicle image information via a biometric sensor camera and in-vehicle temperature data via a temperature sensor. The inputs are real-time in-vehicle image data and temperature data. The terminal performs human detection on the image data and numerical judgment on the temperature data, outputting "Is there a human present?" status information and a temperature value.

[0266] Step 2:

[0267] The terminal performs linkage judgment and alarm. Based on the detection results, the terminal determines whether there are people inside the vehicle and whether the temperature exceeds a preset threshold (e.g., 35 degrees Celsius). The input is the "Is there a human body?" status from step 1 and the temperature value. Data processing is performed as a judgment condition: if there is a person and the temperature is high, the terminal immediately triggers a speaker to issue a voice alarm, simultaneously activates a buzzer and flashing LED warning lights, and packets the alarm information (including the person's status, temperature value, timestamp, etc.). The output is the audible and visual alarm action and the alarm data packet.

[0268] Step 3:

[0269] The terminal sends alarm information to the server. The terminal sends the alarm data packet to the server in real time via a 4G / 5G or WiFi communication module. The input is the alarm data packet generated in step 2, and the output is the data packet upload action and the alarm content transmitted to the server.

[0270] Step 4:

[0271] The server receives alarms and pushes them to multiple devices. The server listens to the data stream reported by the terminals, parses the alarm information, and selects the alarm level based on the alarm type (e.g., "Children inside the vehicle and high temperature"). The input is an alarm data packet; the server parses and classifies the data, and the output is multiple push messages, such as "An anomaly detected inside the vehicle, please handle immediately," pushed to multiple registered user terminal devices.

[0272] Step 5:

[0273] The server initiates an AI dialogue. Upon receiving a high-priority alarm, the server generates interactive prompts for the monitored object (such as a child) by integrating an artificial intelligence model (such as a generative AI model). The input is the alarm information and system configuration, the processing is a natural language generation process, and the output is a dialogue-appropriate prompt, such as "Hello, little friend, can you hear me? How are you feeling now?" These prompts are broadcast to the user through the terminal's voice module or display screen.

[0274] Step 6:

[0275] Users interact with the dialogue system. Users (especially children) receive voice or text prompts from the terminal and provide voice or action feedback based on the situation. Input includes AI-generated prompts, the user's current environment, and personal status; output includes voice responses or button operation data.

[0276] Step 7:

[0277] The server analyzes user responses. Upon receiving user voice or text feedback, the server uses a sentiment analysis engine to determine the emotion of the user's content. For example, if it determines that the user is "a little scared," the system adjusts the prompt to a reassuring tone. The input is the user's response content; the processing involves sentiment analysis and intelligent sentence generation; the output is a new reassuring or operational guidance statement, such as, "Don't worry, someone will come to help you soon. Please see if you can find the red button."

[0278] Step 8:

[0279] The server issues specific operational instructions. Taking into account user feedback and spatial information, the server generates more specific self-rescue suggestions. The input consists of dialogue history, spatial information, and sentiment analysis results. After processing by a generative artificial intelligence model, the output is operational guidance, such as "Try pressing the red button near you, or knock on the window to attract attention outside."

[0280] Step 9:

[0281] The server synchronously reports event progress across multiple terminals. All alarms, dialogues, sentiment analysis, and user operation data are distributed in real-time to all registered guardians and management terminals. Inputs include alarm data, dialogue history, and analysis results; the server processes these into comprehensive information packages, and outputs multi-terminal push notifications and event log archiving.

[0282] Step 10:

[0283] The terminal and user continue to interact until the incident is resolved. The terminal continues to collect data, the server continuously adjusts the dialogue strategy and instructions, and the user can interact repeatedly until relevant personnel arrive to handle the situation and eliminate the security risk. The input is real-time environment and user feedback information, and the output is continuous dialogue and a final alarm clearance notification.

[0284] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0285] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0286] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0287] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0288] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0289] Second Implementation Method

[0290] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0291] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0292] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0293] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0294] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0295] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0296] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0297] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0298] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0299] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0300] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0301] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0302] Example 1

[0303] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0304] Application Example 1

[0305] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0306] Example 2

[0307] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0308] Application Example 2

[0309] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0310] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0311] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0312] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0313] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0314] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0315] Third Implementation Method

[0316] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0317] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0318] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0319] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0320] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0321] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0322] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0323] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0324] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0325] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0326] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0327] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0328] Example 1

[0329] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0330] Application Example 1

[0331] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0332] Example 2

[0333] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0334] Application Example 2

[0335] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0336] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0337] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 includes prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0338] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0339] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0340] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0341] Fourth Implementation Method

[0342] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0343] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0344] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0345] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0346] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0347] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0348] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0349] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0350] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0351] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0352] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0353] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0354] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0355] Example 1

[0356] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0357] Application Example 1

[0358] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0359] Example 2

[0360] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0361] Application Example 2

[0362] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0363] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input from the user representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0364] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0365] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0366] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0367] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0368] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0369] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0370] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0371] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0372] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and battery level, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps, for example, can be based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Brain Physiological Signal Analysis Systems for Emotions, Tokushima University, Doctoral Dissertation: https: / /

[0373] The map is generated using the index / / ci.nii.ac.jp / naid / 500000375379. In the left half of the emotion map, emotions belonging to the "response" region, where sensation is dominant, are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the "situation" region, where situational cognition is dominant, are arranged.

[0374] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0375] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0376] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0377] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0378] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0379] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0380] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0381] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0382] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0383] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0384] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0385] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0386] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0387] In addition, the following notes are provided in response to the above explanation.

[0388] Example 1

[0389] (Note 1)

[0390] An information processing system includes: a device for determining the presence or absence of an object in a space based on information acquired by a detection device; a device for monitoring the working status of a starting device and the removal status of operating components, and issuing an alarm by sound when an object remains under specific conditions; a device for sending the alarm information to an external information processing device; a device for pushing notifications to multiple terminal devices based on the alarm information via a communication network; a device for invoking a generative artificial intelligence model when the alarm information occurs, automatically generating dialogue content using pre-set prompts, and providing specific instructions such as questioning, manipulating, and rescuing the object in the space as appropriate; and a device for recording the dialogue content and status information, and forwarding it to the external information processing device.

[0391] (Note 2)

[0392] According to the information processing system described in Appendix 1, the system further includes a detection device for measuring spatial environment information, a device for receiving environmental information acquired by the detection device and determining anomalies when the information exceeds a predetermined threshold, and a device for issuing an alarm when an anomaly is determined, sending the alarm information to an external information processing device via a communication network, and pushing notifications to multiple terminal devices based on the alarm information.

[0393] (Note 3)

[0394] According to the information processing system described in Appendix 1, the system further includes a device for responding to the aforementioned alarm information or environmental information, invoking a generative artificial intelligence model, generating dialogue content for situation confirmation and corresponding guidance operations based on preset prompt statements, and sending the situation information and dialogue records collected during the dialogue process to an external information processing device.

[0395] Application Example 1

[0396] (Note 1)

[0397] An information processing system includes: a device for receiving sensor data from a detection device and detecting the presence of people in a space; a device for outputting an audio signal when a predetermined trigger condition is met and people are present in the space; a device for receiving temperature data from a measuring device and determining that the temperature is abnormal; a device for sending the above-mentioned abnormality detection information or personnel presence detection information to a communication device; a device for sending notifications to multiple information terminals registered in a storage device based on the sent information; a device for generating prompt content based on the above-mentioned abnormality detection information or personnel presence detection information using a generative artificial intelligence model; a device for engaging in dialogue and obtaining status information or response information based on the prompt content using natural language processing technology; a device for generating and outputting appropriate instruction content based on the obtained status information or response information; and a device for saving the status information and response information to a storage device.

[0398] (Note 2)

[0399] The information processing system according to Appendix 1 further includes means for making sentiment inferences based on prompts generated by a generative artificial intelligence model, acquired situation information, or response information.

[0400] (Note 3)

[0401] The information processing system according to Appendix 1 further includes a device for dynamically optimizing the content of instructions or notifications based on sentiment inference results and status information, and sending them to multiple information terminals.

[0402] Example 2

[0403] (Note 1)

[0404] An information processing system includes: a human perception information acquisition device; an information processing device for receiving human presence information obtained by the human perception information acquisition device and detecting the presence or absence of personnel in a vehicle space; a notification device for executing a voice notification when the vehicle power stops and personnel are detected, based on information obtained from a vehicle operating device; a communication device for sending alarm information to a remote information processing device via a communication network simultaneously with the audio notification; a notification integration device for sending alarm notifications to multiple registered information terminals based on alarm information from the remote information processing device; a communication control device for enabling data transmission and reception between the devices via a communication network; and an information management device for continuously accumulating and managing the acquired alarm content and status information by the information processing device.

[0405] (Note 2)

[0406] The information processing system according to Appendix 1 further includes a temperature detection information acquisition device configured as an environmental information acquisition device, an information processing device for receiving temperature information obtained by the temperature detection information acquisition device and determining abnormal temperatures by comparing it with a set threshold, a notification device for automatically executing an audio alarm notification when an abnormal temperature is determined, a communication device for sending alarm information to a remote information processing device via a communication network, and a notification integration device for sending temperature abnormality notifications to multiple registered information terminals based on alarm information from the remote information processing device.

[0407] (Note 3)

[0408] The information processing system according to Appendix 1 further includes an artificial intelligence processing device that performs dialogue processing using natural language processing technology when receiving alarm information, a dialogue support device that uses a generative artificial intelligence model generated by the artificial intelligence processing device to grasp the situation and perform emotional state analysis, and prompts the information terminal with the required support behavior or response steps, and an information sharing device for sending the situation information or emotional analysis results collected and generated by the dialogue support device to a remote information processing device or multiple information terminals.

[0409] Application Example 2

[0410] (Note 1)

[0411] An information processing system includes: an image acquisition device with biometric detection function; a discrimination unit for receiving data acquired by the image acquisition device and determining the presence of a monitored object in a space through the information processing device; an alarm unit for notifying the presence of a monitored object in the space via an audible signal when the movement of a mobile body stops or the entrance / exit is locked; a sending unit for sending alarm notification information to an information processing device with communication function; a transmission unit for transmitting notifications to multiple information terminal devices registered in a recording medium based on the sent alarm notification information; a dialogue unit for receiving response information from information terminal devices or information acquisition devices in the space, generating response content based on the response information, and adaptively providing guidance responses to the monitored object in the space; an instruction unit for performing situation analysis based on dialogue history information and response information, generating and providing individual operation guidance or behavioral instructions; and a recording and sending unit for sending situation data, dialogue history information, and analysis results continuously collected by the information acquisition device and the information processing device to an external communication device.

[0412] (Note 2)

[0413] The information processing system according to Appendix 1 further includes: a mobile terminal device with environmental status detection function; a discrimination unit for receiving environmental status data acquired by the terminal device and judging abnormal status based on a set threshold by the information processing device; an alarm unit for issuing an alarm notification via sound signal when an abnormal status is detected; a sending unit for sending the alarm notification information to an external communication device; and a transmission unit for transmitting notifications to multiple information terminal devices registered in a recording medium based on the transmitted alarm notification information.

[0414] (Note 3)

[0415] The information processing system according to Appendix 1 further includes: an artificial intelligence unit for receiving alarm notification information or dialogue history information and generating prompt content through information processing technology with automatic human language parsing; a guidance unit for judging the emotional information of the monitored object through the status data obtained from dialogue and response, and providing adaptive responses and operation guidance; and a sending unit for sending dialogue history information and analysis results to an external communication device.

Claims

1. An information processing system, characterized in that, include: A camera equipped with a human body sensor; A device for receiving data acquired by the camera and detecting the presence of people inside the vehicle; A device that sounds an alarm via voice when someone is detected inside a vehicle when the key is removed. Device for sending alarm information to external communication equipment; A device for sending notifications to multiple registered terminals based on the sent alarm information.

2. The information processing system according to claim 1, characterized in that, Also includes: Mobile devices used to monitor the temperature inside a vehicle; A device for receiving temperature data acquired by the mobile device and detecting abnormal temperatures; A device for issuing a voice alarm when an abnormal temperature is detected; Device for sending alarm information to external communication equipment; A device for sending notifications to multiple registered terminals based on the sent alarm information.

3. The information processing system according to claim 1, characterized in that, Also includes: A device for receiving the alarm information and engaging in dialogue using natural language processing technology; A device for gathering information through the dialogue and instructing on methods of seeking help; A means for sending the situation information to an external communication device.

Images