Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises an image acquisition means fixed on driving equipment; an analysis means for analyzing the image data acquired by the image acquisition means; an alarm means for sending an alarm to the driver on the basis of the data analyzed by the analysis means; and the storage means is used for automatically recording and storing recorded data when an accident occurs.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] Existing electric scooters make it difficult for drivers to keep track of the surrounding environment and potential risks while in motion. They lack effective real-time warning mechanisms and automatic accident data saving methods, which makes it difficult to react quickly in emergency situations and to handle effective evidence after an accident occurs. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides an information processing system comprising: an image acquisition means fixed to a driving device; an analysis means for parsing image data acquired by the image acquisition means; an alarm means for sending an alert to the driver based on the data parsed by the analysis means; and a storage means for automatically recording and saving data in the event of an accident. Through these means, real-time environmental monitoring, automatic risk warning, and automatic preservation of accident evidence during the electric scooter's operation are achieved, effectively improving driving safety and accident handling efficiency.

[0005] "Image acquisition means" refers to a device that can be fixed to driving equipment to collect real-time image data of the vehicle's surrounding environment, such as a camera or a mobile terminal with video recording capabilities.

[0006] "Analysis methods" refer to hardware or software modules used to analyze and process image data acquired through image acquisition methods and extract useful information such as target distance and speed, including devices that use artificial intelligence models for recognition and calculation.

[0007] "Alarm measures" refer to devices that can provide warning information to the driver based on the results of analysis, such as through displays, sounds, vibrations, etc.

[0008] "Storage means" refers to a device or module that automatically records and saves relevant data (such as video information before and after the accident) when an accident is detected, so as to facilitate subsequent querying and evidence collection.

[0009] "Driving equipment" refers to a means of transportation that is operated by a user, and in this invention, it mainly refers to personal mobility devices such as electric scooters.

[0010] "Mobile terminal" refers to a portable electronic device with camera, data processing and communication functions, such as a smartphone or a dedicated data acquisition device.

[0011] "Artificial intelligence model" refers to an algorithmic model built based on machine learning or deep learning technology for analysis and processing such as image recognition and distance measurement. Attached Figure Description

[0012] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0013] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0014] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0015] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0016] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0017] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0018] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0019] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0020] Figure 9 This represents an emotion map that maps multiple emotions.

[0021] Figure 10 This represents an emotion map that maps multiple emotions.

[0022] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0023] Figure 12This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0024] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0025] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0026] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0027] First, let me explain the terminology used in the following instructions.

[0028] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0029] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0030] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0031] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0033] First Implementation Method

[0034] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0035] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0036] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0038] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0041] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0043] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0044] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0045] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0046] Example 1

[0047] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0048] In traditional personal transportation, drivers struggle to identify and respond to potential environmental risks in real time, and in the event of an accident, they cannot quickly and automatically acquire, preserve, and process critical on-site data. This not only impacts driving safety but also complicates and inefficiently determines liability and processes insurance claims after an accident. Therefore, existing technologies cannot simultaneously achieve real-time assessment of driving safety risks, immediate early warnings, and automatic recording and management of accident data, failing to meet the growing demands for intelligent mobility and traffic safety.

[0049] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0050] In this invention, the server includes a data acquisition device, a processing device, a warning notification device, a record storage device, a communication device, an advanced analysis device, a control device, and a management device. This enables real-time perception and data acquisition of the personal vehicle's environment. Through a generative artificial intelligence model, the system automatically analyzes and recognizes patterns in the collected time-series and physical quantity data, and makes risk assessments and provides early warnings based on the evaluation results. When specific events occur (such as accidents or severe vibrations), the system can automatically record and save relevant data, and also possesses management and external information provision capabilities, thereby significantly improving driving safety, accident handling efficiency, and the level of intelligent data management.

[0051] "Data acquisition device" refers to a hardware device installed on information processing equipment for real-time acquisition of environmental information, driving status and related sensor data.

[0052] "Computational processing unit" refers to an electronic processing unit used to analyze, calculate and preprocess the acquired time-series data.

[0053] "Warning notification device" refers to an output device that notifies the user of risk assessment results in real time through visual, auditory, or tactile means.

[0054] "Record storage device" refers to hardware and its control system that can automatically record and store data in local storage media or external storage devices when predetermined conditions are met.

[0055] "Communication device" refers to a network communication module that enables wired or wireless data transmission between a terminal and a server or between a terminal and an external device.

[0056] "Advanced analysis device" refers to a processing unit that performs advanced analysis, pattern recognition, and judgment on collected time-series data and physical quantity data based on generative artificial intelligence models.

[0057] "Control device" refers to a logical decision-making unit that generates and distributes control signals based on analysis and risk assessment results, guiding other units to work collaboratively.

[0058] "Management device" refers to a control device that automatically manages the storage, organization, and external access permissions of relevant information when a specific event occurs.

[0059] "Generative artificial intelligence models" refer to artificial intelligence algorithm systems that can be trained on a large amount of data and automatically summarize patterns from the data through algorithms such as deep learning, and then be used to identify and judge targets.

[0060] "Time series data" refers to a collection of data acquired and arranged in chronological order, reflecting the state or changes of an object over time.

[0061] "Physical quantity data" refers to quantitative measurement values ​​used to describe physical states, motion characteristics, etc., including information such as velocity, acceleration, and position.

[0062] This invention can be implemented in the following specific ways.

[0063] This system includes data acquisition devices, computing and processing devices, warning and notification devices, record storage devices, communication devices, advanced analysis devices, control devices, and management devices installed on information processing equipment (such as smartphones or vehicle control units).

[0064] The terminal can be a smartphone, tablet, or embedded hardware integrated into personal transportation vehicles (such as electric scooters, bicycles, etc.). The terminal is equipped with a camera, accelerometer, and GPS module to collect video streams, location information, speed, vibration, and other physical quantity data of the surrounding environment. The data acquisition device collects the aforementioned environmental information and sensor signals in real time, while the processing device performs preliminary processing on the data, such as data format conversion and timestamp marking.

[0065] The terminal periodically uploads collected data to the server via communication devices (such as 4G / 5G / WLAN modules). The server possesses powerful computing and storage resources, enabling it to run advanced analytics. These advanced analytics rely on generative artificial intelligence models (such as neural network models trained using deep learning frameworks like TensorFlow and PyTorch) to perform feature extraction, target recognition, object tracking, and pattern determination on time-series video and physical quantity data. Specifically, the model can identify pedestrians, other vehicles, and fixed obstacles, and estimate the distance and movement trends between these objects and the user.

[0066] Based on the analysis results output by the advanced analytics device, the server executes a risk assessment algorithm through the control device. If a high risk is detected (such as being too close, traveling too fast, or encountering a sudden obstacle), a warning signal is immediately generated.

[0067] Upon receiving an alarm signal from the server, the terminal will alert the user in various ways through a warning notification device. For example, the terminal can display the alarm information on the screen with red text and icons, simultaneously play a voice prompt "Obstacle ahead, please slow down," and activate a vibration motor to remind the driver to take precautions. Users can reduce speed or avoid obstacles in a timely manner based on the alarm information, thus improving driving safety.

[0068] When the terminal detects an anomaly in the accelerometer (such as a severe impact or fall), the recording device automatically activates the accident recording mode, packaging and storing data from the time period before and after the accident. This data is then uploaded to the server via a communication device. The management device on the server is responsible for storing this critical accident data in the cloud (such as cloud storage hardware or an external data server) and providing data access permissions to insurance institutions, regulatory authorities, or the user as required.

[0069] The data analysis and AI processing solutions adopted in this invention can use general deep learning development platforms such as TensorFlow and PyTorch, and the hardware of the terminal and server can be selected according to different application scenarios, including information terminals with sufficient processing performance, computing servers and high-bandwidth data communication networks, to meet actual deployment requirements.

[0070] Specific examples:

[0071] A user attaches their smartphone to the handlebars of an electric scooter and opens a customized app. The app automatically initializes the camera and sensors before starting the ride. Video and speed information collected by the phone are uploaded to a server in real time. The server uses a generative artificial intelligence model to analyze the images and assess the risks. When a pedestrian is detected approaching and a collision risk is detected, the server immediately issues a sound, visual, and vibration warning via the app. Upon receiving the alert, the user slows down and successfully avoids danger. If a collision occurs due to an emergency, the phone automatically records video of the scene before and after the collision and uploads it to the cloud for subsequent accident investigation or insurance verification.

[0072] Here are some examples of prompts for generative artificial intelligence models:

[0073] I want to develop a system that provides safety warnings for electric scooter riders, identifying obstacles and pedestrians ahead in real time, promptly alerting users when there is a collision risk, and automatically recording and uploading data in the event of an accident. What hardware and AI models should I choose? How can data collection and risk assessment be specifically implemented?

[0074] Through the above methods, the system of the present invention can efficiently realize intelligent protection of personal vehicle driving safety, real-time risk management, and automatic collection and management of accident data.

[0075] use Figure 11 The processing flow is explained.

[0076] Step 1:

[0077] The user attaches a terminal (such as a smartphone) to their personal vehicle and launches a dedicated application. Input consists of the user's physical installation and application launch commands. Output is the completion of hardware initialization, with the camera, GPS, and accelerometer sensors entering a ready state, and the application displaying "System ready." During this process, the terminal activates various sensors and notifies the user that the device is available.

[0078] Step 2:

[0079] The terminal collects environmental video, location, and speed data in real time. Inputs include real-time images of the surrounding environment, GPS signals, and accelerometer data. The terminal timestamps video frames, synchronizes location and motion data, and outputs a tagged video stream and physical quantity data packets. Specifically, the camera captures one frame per second, and the sensors record speed and latitude / longitude once per second.

[0080] Step 3:

[0081] The terminal periodically uploads the collected composite data packets to the server via its communication module. The input is time-synchronized data packets, and the output is a data transmission request encapsulated with a secure protocol. The terminal performs batch data packaging, sends the data to the server via the wireless network, and displays the upload progress in real time.

[0082] Step 4:

[0083] After receiving data packets from the terminal, the server feeds video frames and physical quantity data into the generative artificial intelligence model. The input is the uploaded data packet (containing video frames, velocity, position, etc.). The server uses deep learning libraries (such as TensorFlow) to perform object detection, recognition, and distance and velocity estimation on each frame. The output is the analysis results, including the identified target category, relative distance, relative velocity, and risk score. Specifically, the server schedules hardware resources to perform batch calculations on the continuous data stream.

[0084] Step 5:

[0085] The server performs a risk assessment based on the AI ​​model's analysis results. The input includes target recognition and distance / velocity analysis results. The server executes a risk grading algorithm (e.g., determining the risk level based on distance / velocity thresholds), and the output is a risk assessment (e.g., a high-risk or low-risk label) and a corresponding warning signal. Specifically, the server generates a warning JSON message and prepares to push it to the terminal.

[0086] Step 6:

[0087] The server sends risk alarm signals to the terminal via the communication channel. The input is the risk alarm information calculated by the algorithm, and the output is a network datagram. The server sends real-time warnings according to priority and ensures that the message reaches the terminal.

[0088] Step 7:

[0089] Upon receiving a warning signal from the server, the terminal initiates a multimodal alert: a highlighted alarm with images and text pops up on the screen, the speaker plays a voice reminder "Please be careful ahead," and a brief vibration occurs. The input is the warning message pushed by the server, and the terminal outputs visual, auditory, and tactile feedback that the user can perceive. Specifically, the terminal activates the multimedia module and vibration motor to alert the user.

[0090] Step 8:

[0091] Users adjust their driving behavior based on terminal alerts. The input is the terminal's warning prompt, and the output is the actual driving action (such as deceleration, steering, or stopping). Users make judgments based on the prompts to improve driving safety.

[0092] Step 9:

[0093] When the terminal detects an anomaly such as a severe collision or fall via the accelerometer, it automatically enters accident data recording mode. The input is sudden acceleration data, and the output is a package of video and physical quantity data before and after the accident. The terminal automatically saves relevant data from the accident scene and uploads it to the server via the communication module.

[0094] Step 10:

[0095] After receiving incident data, the server saves the data to cloud storage via a management device for later retrieval. Input consists of incident data packets uploaded by terminals, and output includes data storage and external interfaces. The server automatically archives and encrypts incident data, which can be accessed by administrators and relevant organizations according to their permissions.

[0096] Application Example 1

[0097] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0098] With the widespread adoption of intelligent mobility devices, traffic accidents caused by driver inattention and delayed environmental awareness are gradually increasing. Most existing safety monitoring systems can only achieve basic video capture and simple alerts locally, lacking comprehensive real-time risk assessment and intelligent response to changes in driver emotional state. Furthermore, the storage and traceability of accident data are inadequate, failing to meet the demands for efficient, intelligent, and safe driver assistance systems. Therefore, providing a system capable of real-time comprehensive analysis of environmental risks, dynamically adjusting alert content and intensity, and automatically saving high-quality evidence data in the event of an accident is a pressing technical challenge for the industry.

[0099] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0100] In this invention, the server includes an information acquisition unit, a computing unit, an output unit, a storage unit, a sending unit, and a control unit on an external computing device. This enables real-time data acquisition of the driving environment and driver status, multi-dimensional environmental risk and emotional state analysis based on generative artificial intelligence models, intelligent customization of alarm content and methods, and automatic storage and remote cloud management of accident data, thereby effectively improving the safety of mobile travel and the accuracy of accident handling.

[0101] "Information acquisition unit" refers to hardware or software components used to acquire data from the target device and its surrounding environment, including but not limited to camera equipment, sensors, microphones, etc.

[0102] A “computing unit” refers to a module that processes, analyzes, and judges the collected information data. It can be implemented by a processor, microcontroller, or computing platform based on an artificial intelligence model.

[0103] "Output subject" refers to the object that receives the warning information and can take corresponding actions, usually the driver or end user.

[0104] "Output unit" refers to a device used to convey warning information to the output subject, including human-computer interaction media such as displays, speakers, and vibration devices.

[0105] "Storage unit" refers to a data carrier or storage module that can store information data or record files for a long time or temporarily, such as memory card, hard drive, solid-state storage device, etc.

[0106] A “transmission unit” refers to a hardware or software module used to transmit data or status information to an external computing device via a communication network.

[0107] "External computing device" refers to a data center, server, or cloud platform that is connected to a terminal via a network and has the ability to process data and make decisions.

[0108] The “control unit” refers to a functional module that can dynamically adjust and generate alarm content and parameters based on the analysis results and subject information, and achieve overall system coordination.

[0109] "Generative artificial intelligence models" refer to intelligent algorithm models that can autonomously generate analytical judgment results or risk assessments by learning from large amounts of data, such as deep neural networks.

[0110] "Physiological information" refers to data that reflects the current physiological or biological characteristics of the output subject, including but not limited to facial expressions, heart rate, voice and other information.

[0111] "Identification information" refers to data that can be used to identify the identity or status of the output subject, including image recognition results, speech recognition results, etc.

[0112] This invention provides an intelligent safety assistance system based on real-time data acquisition and a generative artificial intelligence model, applicable to safety monitoring and risk warning of various mobile target devices (such as electric scooters, bicycles, and automobiles). The specific embodiments of this invention will be described in detail below, covering aspects such as system structure, hardware and software configuration, data processing methods, application scenarios, and operational examples.

[0113] This system consists of terminal units, a server, and a communication network. The terminal unit is typically a portable terminal installed on the target device, such as a smartphone or dedicated smart terminal, integrating hardware such as a camera, accelerometer, GPS module, microphone, and display device. The terminal can collect environmental video, speed information, driver facial expressions, and voice data in real time, and then periodically upload this data to the server via 4G / 5G network or Wi-Fi after preprocessing.

[0114] The server includes high-performance computing units equipped with processors, graphics accelerator cards, and an AI computing platform. Its core software comprises data parsing and risk assessment programs that utilize generative AI models such as TensorFlow and Keras to process and analyze received multi-channel data. It leverages OpenCV for video processing and object detection and integrates industry-standard emotion recognition APIs to automatically identify the driver's psychological state. Furthermore, the server includes a control module for alarm generation and intelligent adjustment, and a cloud storage system (such as a public cloud object storage service) to store accident data.

[0115] The core functions of the system include:

[0116] 1. The terminal automatically acquires environmental video and information from various sensors, and combines audio and facial images to collect the driver's physiological signals without human intervention, ensuring the continuity and real-time nature of information collection.

[0117] 2. The terminal periodically compresses and packages the collected data and uploads it to the server.

[0118] 3. After receiving the data, the server first uses OpenCV to process the image, such as object segmentation, person detection, and distance calculation. Then, it calls a generative artificial intelligence model (such as a risk prediction model based on deep neural networks) to comprehensively analyze the distance between vehicles, the visibility of people and vehicles to hazards, and the distance to obstacles. At the same time, it analyzes the driver's emotional state and attention status (such as distraction, anxiety, tension, etc.).

[0119] 4. The server automatically generates alerts based on the interaction between environmental risks and the driver's psychological state, adjusting the content, method, and intensity of the alerts. For example, when the AI ​​determines that there is a pedestrian 1.3 meters ahead and the driver is in a tense state, the system can select a stronger sound and vibration mode than usual and highlight the alert in a pop-up window on the terminal screen.

[0120] 5. The terminal receives alarms in real time and alerts users through display, sound, motor vibration, and other means, effectively improving the speed of safety response.

[0121] 6. When the terminal detects a collision or abnormally strong acceleration, it automatically switches to accident recording and tracking mode, records video clips before and after the accident, and uploads them to the cloud along with relevant data to facilitate subsequent evidence collection and accident liability determination.

[0122] Examples of real-world application scenarios are as follows:

[0123] The driver attaches their smartphone to the electric scooter and opens an app. The camera then begins recording in real time, collecting acceleration and GPS data. If a pedestrian is detected 1.5 meters ahead, and the driver's facial expression shows anxiety, the server immediately generates a high-priority alarm, prompting the user to slow down via loud volume and strong vibration. In the event of a collision, all recorded data before and after the accident is uploaded to the cloud, allowing the management platform and insurance companies to remotely access the data.

[0124] The AI ​​component of this system supports the following example prompts to achieve training and application goals:

[0125] "Based on video stream and speed sensor data, analyze the risk of contact between the electric vehicle and the pedestrian, and automatically adjust the alarm intensity according to the driver's real-time facial expressions to generate corresponding warning prompts."

[0126] "Develop an AI system for safety monitoring of electric scooters that can detect the distance of surrounding objects in real time, recognize the driver's emotions, and automatically issue visual and voice warnings based on risks."

[0127] "Use OpenCV and TensorFlow to build an automatic accident recording, cloud storage and alarm triggering mechanism system, adapted to the vehicle terminal environment."

[0128] Through the aforementioned hardware and software collaboration and intelligent data processing mechanism, this invention can effectively improve the safety performance of mobile vehicles, realize proactive hazard prevention, driver status monitoring, and automatic accident tracing, and provide a solid technical foundation for modern intelligent transportation and smart travel.

[0129] use Figure 12 The processing flow is explained.

[0130] Step 1:

[0131] The user attaches a mobile terminal (such as a smartphone) to the target device and launches the security monitoring application. The input consists of user operation signals and the device's initialization status; the output is the terminal's ready-to-collect data operating mode. The user's specific actions are: clicking the app icon, granting camera and microphone permissions, and adjusting the device position according to the prompts.

[0132] Step 2:

[0133] The terminal collects environmental video data in real time via a camera, and simultaneously uses an accelerometer, GPS module, and microphone to collect data such as speed, acceleration, user voice, and facial expressions. The input is the raw sensor signals, and the output is a multimodal raw data stream with timestamps. Specifically, the terminal periodically reads data from various hardware sensors and buffers video frames and audio streams to a local cache.

[0134] Step 3:

[0135] The terminal performs preliminary compression and encryption on the collected data. The input is the collected raw multimodal data, and the output is the encoded and encrypted data packet. Specifically, the terminal encodes video frames using H.264, compresses audio and text data, and encrypts the data packet using AES.

[0136] Step 4:

[0137] The terminal periodically uploads processed data packets to the server via a wireless communication network (such as 4G / 5G or Wi-Fi). The input is local data packets, and the output is the data uploaded to the server. Specifically, the terminal uses a network API to periodically send POST requests to a specified interface on the server.

[0138] Step 5:

[0139] The server receives data packets from the terminal and stores them in a temporary data cache. The input is the data packet uploaded by the terminal, and the output is the server's raw data queue. Specifically, the server parses the HTTP request body, splits the multimodal data and stores it in a structured cache pool, and performs file integrity checks.

[0140] Step 6:

[0141] The server utilizes OpenCV and generative AI models (such as TensorFlow and Keras models) to analyze received video, audio, and sensor data. It calculates the distance to target objects, identifies humans and obstacles, and extracts emotional states from user facial expressions and speech. The input is multimodal data, which is processed into multidimensional feature vectors. The output includes risk assessment results and emotion recognition results. Specifically, the server performs frame-level detection of the distance and state of objects in front, extracts features such as speed, and uses AI models to obtain emotion classifications from facial expressions and speech signals.

[0142] Step 7:

[0143] The server generates alert commands based on the analysis results, including the alert type, priority, and notification method (sound, vibration, visual, etc.), and dynamically adjusts the alert intensity and frequency according to the user's current emotional state. The input is the risk assessment and emotion recognition output, and the output is an alert information data packet. Specifically, the server calls the alert generation module to generate specific commands based on thresholds and emotion weighting rules.

[0144] Step 8:

[0145] The server sends alarm signals to the corresponding terminals via API or WebSocket. The input is an alarm information data packet, and the output is the real-time alarm data sent to the terminal. Specifically, the server pushes a compressed command packet to the client immediately and waits for confirmation from the terminal.

[0146] Step 9:

[0147] Upon receiving an alarm signal, the terminal invokes its local display, speaker, and vibration motor modules to issue a multi-mode warning to the user. The input is the server's alarm command, and the output is an alarm event perceptible to the user. The terminal's specific actions include: a screen pop-up, a speaker voice prompt, or motor vibration, combined according to alarm priority.

[0148] Step 10:

[0149] When the terminal detects a collision or a strong acceleration fluctuation, it automatically triggers the accident recording mode, recording video and related data for the period before and after the accident. The input is real-time data from the acceleration sensor, the data is processed for event determination, and the output is an accident video data packet. The terminal's specific actions are: initiating circular buffer recording, saving data from several minutes before and after the accident point into a separate file.

[0150] Step 11:

[0151] The terminal uploads incident-related data packets to the server, which then saves this data to a cloud storage system. The input is the incident data packet, and the output is the incident file in cloud storage. Specifically, the terminal performs the following actions: ensuring reliable transmission of large files through fragmented uploads; and the server associates the file with metadata and stores it in distributed object storage.

[0152] Step 12:

[0153] Users can supplement their images and text descriptions through the application after an incident occurs. The server receives and binds these to the corresponding incident file. The input is the user's edited information, and the output is the updated incident record. Specifically, the user's actions are: uploading images or filling in remarks on the APP's incident handling page, which the server saves to the database in real time.

[0154] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0155] Example 2

[0156] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0157] Existing driving safety systems typically only focus on physical risks and cannot effectively identify and respond to changes in the user's emotional state and attention. They are unable to issue appropriate warnings in a timely manner for behavioral risks such as high stress and decreased attention. Furthermore, they lack automated data recording and evidence preservation mechanisms when accidents occur, making it difficult to determine liability afterward.

[0158] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0159] In this invention, the server includes an information acquisition unit, a processing unit, a notification unit, a storage unit, and a status recognition unit, wherein the processing unit is equipped with a generative artificial intelligence model. This enables real-time analysis of environmental information and the user's emotional state during driving, dynamic adjustment of notification schemes based on risks, and automatic collection and storage of relevant data in the event of an accident, thereby improving driving safety and accident tracing capabilities.

[0160] "Information acquisition unit" refers to a device that can acquire various raw data such as environmental information around the driving device and user status information, including hardware such as sensors, cameras, and microphones or combinations thereof.

[0161] "Processing unit" refers to a device or functional module that analyzes, judges, and processes the data collected by the information acquisition unit, and can be implemented using a combination of hardware and software.

[0162] A "notification unit" refers to a device that issues warnings or prompts to the user based on the processing results of the processing unit. This includes devices such as displays, speakers, and vibration motors, and is used to deliver notifications in a visual, auditory, or tactile manner.

[0163] A "storage unit" refers to a device used to record and store relevant data and information during the driving process and when an accident occurs. It can be a local storage device or work in conjunction with an external storage device.

[0164] A “state recognition unit” refers to a hardware, software, or a combination of both module that can recognize and assess a user’s emotional state, such as stress or attention, and can automatically adjust notification strategies based on the recognition results.

[0165] "Generative artificial intelligence models" refer to artificial intelligence algorithm models built on technologies such as deep learning and machine learning, which can autonomously perform feature analysis and risk assessment by learning from a large amount of data.

[0166] "Portable information processing device" refers to an electronic device that can be carried around and has information collection, processing and communication functions, such as smartphones and tablets.

[0167] "External information storage device" refers to a remote storage device that collaborates with a system through a communication network to store and manage large amounts of data information, including cloud storage platforms or network servers.

[0168] The invention can be implemented in the following specific ways.

[0169] This system includes an information acquisition unit, a processing unit, a notification unit, a storage unit, and a status recognition unit. Some functions can be implemented by terminals (such as smartphones, tablets, and other portable information processing devices), while others can be implemented by the server (data processing system).

[0170] The terminal can collect real-time data on the surrounding environment, location, speed, and the user's facial expressions and voice through integrated hardware such as cameras, microphones, GPS modules, and accelerometers. The collected data will be initially processed by the terminal and transmitted to a server via a mobile network. Information acquisition units typically include, but are not limited to, smart terminal hardware, such as cameras and sensor modules built into Android or iOS smartphones.

[0171] After receiving the aforementioned data, the server uses generative artificial intelligence models (such as models developed based on deep learning platforms like TensorFlow and PyTorch) and computer vision libraries like OpenCV to perform object detection, object tracking, and distance calculation on the video content during the driving process. The server can also extract features from uploaded multimodal emotional data such as facial expressions and speech, and use sentiment analysis models (such as facial expression recognition models and speech emotion recognition models) to assess the user's stress and attention levels.

[0172] The processing unit outputs a result that determines the driving risk level based on a preset algorithm. If the risk is high, the notification unit automatically adjusts the warning content and notification intensity according to the emotional state feedback from the state recognition unit, and provides multimodal warnings through the terminal's screen prompts, speaker audio, and vibration motor.

[0173] When an accident occurs, the terminal detects abnormal acceleration or sensor values ​​and automatically starts recording the accident, saving video footage and relevant sensor data for the period before and after the accident. The terminal uploads this data to a server, which collaborates with external information storage devices (such as cloud storage platforms) to securely store the accident data long-term for subsequent traceability or third-party verification.

[0174] Specific hardware examples include, but are not limited to: Android / Apple smartphones, compatible camera modules, MEMS accelerometers, GPS modules, and data communication modules. Software examples include: TensorFlow, PyTorch, OpenCV, face recognition SDK, speech recognition API, and cloud storage service interfaces.

[0175] Usage example:

[0176] Users attach their smartphones to the electric scooter and launch a specific application. The device automatically acquires camera footage, sensor data, and the user's emotional state. The server receives and analyzes the data in real time. If it detects a pedestrian too close and the user is under high stress, it immediately provides voice and vibration alerts through the device. In the event of a collision, the system automatically records video and synchronizes all data to the cloud for subsequent review and liability determination.

[0177] Example of generating AI model prompts:

[0178] "Please describe the system flow for electric scooter safety. Please describe the system functions step by step, taking into account the user, terminal, and server, and further explain through specific scenarios."

[0179] Through the above methods, the present invention can comprehensively realize intelligent monitoring and risk alerts for driving safety, and provide data support for tracing liability for accidents.

[0180] use Figure 13 The processing flow is explained.

[0181] Step 1:

[0182] Users attach portable information processing devices (such as smartphones) to the vehicle and launch a dedicated application.

[0183] Input: Smartphone or mobile application launch command.

[0184] Output: The application starts and is ready to run.

[0185] Users click the application icon through the interface, and the device requests relevant permissions such as camera, microphone, and GPS to prepare for subsequent data collection.

[0186] Step 2:

[0187] The terminal initializes hardware modules such as the camera, GPS, and accelerometer, and simultaneously activates the front-facing camera and microphone for emotional data collection.

[0188] Input: Terminal hardware component initialization command.

[0189] Output: The hardware module enters standby mode, ready to collect data.

[0190] The terminal calls the Camera API and related sensor APIs to achieve real-time data acquisition of the environment and users.

[0191] Step 3:

[0192] The terminal collects environmental video, location information, speed data, as well as the user's facial expressions and voice in real time, and uploads them to the server in batches at regular intervals (e.g., once per second).

[0193] Input: camera footage, location information, acceleration data, user audio and video data.

[0194] Output: A data packet containing multimodal data, sent to the server via HTTP POST.

[0195] The terminal collects and uploads data concurrently through multiple threads to ensure data continuity and timeliness.

[0196] Step 4:

[0197] After receiving the data uploaded by the terminal, the server calls the processing unit to analyze the image data, sensor data, and emotion data.

[0198] Input: Multimodal data packets (video streams, sensor data, audio data, facial images, etc.).

[0199] Outputs: Environmental risk assessment results and user stress determination results.

[0200] The server uses OpenCV for object recognition and distance measurement in the scene, and uses generative artificial intelligence models such as TensorFlow or PyTorch to assess stress and attention based on emotional characteristics.

[0201] Step 5:

[0202] The server determines the risk level based on the analysis results and decides whether to send a warning signal to the terminal, while automatically adjusting the intensity and frequency of the alarm according to the user's status.

[0203] Input: Risk assessment results, emotional state assessment results.

[0204] Output: A notification signal containing the warning type, intensity, and frequency.

[0205] The server generates notification content according to a custom warning policy and returns it to the terminal in real time via push service.

[0206] Step 6:

[0207] The terminal receives the warning command sent back by the server and immediately issues a warning to the user simultaneously through multiple means such as the screen, speaker, and vibration motor.

[0208] Input: Server notification signal.

[0209] Output: Visual, auditory, and tactile alarms received by the user.

[0210] The terminal displays a red pop-up window, a high-intensity sound alert, and is accompanied by vibration feedback to ensure that users are aware of danger in a timely manner.

[0211] Step 7:

[0212] After receiving a warning, users should take appropriate safety measures (such as slowing down or avoiding obstacles) based on the warning content.

[0213] Input: Multimodal alarm information.

[0214] Output: User actions (braking, steering, etc.).

[0215] Users react quickly to warnings and take actual control of the driving device to effectively avoid accidents.

[0216] Step 8:

[0217] The terminal continuously monitors acceleration sensor data. If a sudden high-intensity change occurs, it is automatically identified as an accident and switches to accident mode, starting accident recording.

[0218] Input: Real-time acceleration sensor data.

[0219] Output: Accident identification signal, recording on status.

[0220] The terminal enables the accident recording API to save pre- and post-accident videos and raw sensor data to local storage.

[0221] Step 9:

[0222] After completing the accident recording, the terminal uploads the accident data to the server through a secure channel, and the server further saves it to an external information storage device (such as a cloud storage platform).

[0223] Input: Accident video recordings, accident information data.

[0224] Output: Remotely stored accident evidence information.

[0225] The server automatically executes data storage interface calls to achieve remote and secure storage of recorded data.

[0226] Application Example 2

[0227] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0228] With the development of autonomous driving and intelligent transportation systems, existing technologies have the following shortcomings in ensuring driving safety: relying solely on a single data source, such as traditional in-vehicle cameras or sensors, cannot achieve dynamic recognition and response to the driver's emotional state; furthermore, in the event of an emergency, they cannot automatically and efficiently record and save key images and multimodal data in real time. In addition, existing early warning mechanisms lack adaptive adjustments to the driver's individual state (such as stress and fatigue), resulting in limited early warning effectiveness and failing to achieve true human-machine collaborative safety management. Therefore, there is an urgent need for a technical solution that can combine environmental information and driver emotions to dynamically adjust the alarm intensity and achieve intelligent accident data recording and management.

[0229] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0230] In this invention, the server includes an information acquisition device fixed to an information processing device, an information parsing device for parsing the information data, a warning output device for outputting warnings, an emotion analysis device for dynamic analysis of emotional states and adaptive adjustment of warnings, an automatic recording and storage device based on multi-sensor information, and a generative artificial intelligence analysis device. This enables multi-dimensional perception and joint analysis of the driving environment and the driver's emotional state, dynamically adjusting the warning method and intensity, and automatically saving multi-modal data in the event of an accident, effectively improving the safety management level in autonomous driving and intelligent transportation.

[0231] "Information acquisition device" refers to a hardware device that can be fixed on an information processing device to collect environmental information, user-related information or other external data in real time, such as a camera, microphone, sensor, etc.

[0232] "Information analysis device" refers to a software or hardware system used to analyze, preprocess, and extract features from data collected by information acquisition devices.

[0233] "Warning output device" refers to a device or module that outputs warning signals to the user through visual, auditory, or tactile means based on the information analysis results.

[0234] An "emotion analysis device" refers to a hardware and software system that analyzes data such as a user's facial expressions, voice, and body posture to determine the user's current emotional state and dynamically adjust the system's response accordingly.

[0235] "Information storage device" refers to a device or system that can automatically record and store multi-source data (such as video, audio, sensor data, etc.) related to an emergency, for the purpose of accident reproduction and subsequent analysis.

[0236] "Generative artificial intelligence analysis device" refers to a module or system that uses generative artificial intelligence models to extract features, analyze, fuse, and reason about collected information data.

[0237] "Information processing device" refers to computing equipment that undertakes functions such as data reception, storage, parsing, model operation, and instruction issuance, including but not limited to central processing units, servers, or integrated processing platforms.

[0238] "Portable electronic devices" refer to electronic terminal devices that can be carried by users and have the ability to collect, process, and communicate information, such as smartphones and tablets.

[0239] "Sensor" refers to a physical detection element or measurement and control device used to collect information such as vehicle driving status, environmental parameters, and changes in physical quantities in real time.

[0240] "User" refers to the operator of the system or an individual who interacts with the system, such as a vehicle driver.

[0241] The present invention can be implemented in the following ways.

[0242] This system includes an information acquisition device, an information analysis device, a warning output device, an emotion analysis device, an information storage device, and a generative artificial intelligence analysis device. The information acquisition device, such as a camera, microphone, accelerometer, and collision sensor, can be fixed to an information processing device (e.g., a server or portable electronic device) to acquire real-time environmental data and user status data. The information acquisition device can use common hardware components, such as high-resolution cameras, dynamic microphones, and multi-axis accelerometers, and works in conjunction with software and hardware drivers (such as a general-purpose camera SDK and audio acquisition driver) to complete data acquisition.

[0243] The terminal is responsible for collecting real-time images of the vehicle's surrounding environment, facial expressions and voice data of occupants, and periodically collecting motion information from sensors. The terminal transmits the collected data to the server in real time via wireless communication (e.g., Wi-Fi, 4G / 5G networks).

[0244] After receiving data from the terminal, the server uses an information analysis device to preprocess the image, audio, and sensor data. Commonly used software includes OpenCV (image processing) and common audio denoising libraries. The server further uses a generative artificial intelligence analysis device, based on object detection and analysis models built with TensorFlow and Keras, to identify the position, category, relative distance, and speed of objects in the environment. It also uses libraries such as dlib to recognize the driver's facial expressions and uses an emotion recognition model to determine the driver's emotional state (such as tension, distraction, fatigue, etc.).

[0245] The information storage device, in conjunction with sensors (such as acceleration sensors and collision sensors), automatically starts recording video and audio when an abnormality is detected in the vehicle (such as sudden braking or collision), and stores key data in the local storage unit while uploading it to a cloud storage platform (such as a general cloud storage service).

[0246] When the server determines that a security risk exists, the warning output device will generate multimodal warning information tailored to the current situation, such as visual warning images, voice broadcasts (using a TTS text-to-speech engine), and vibration alerts. The emotion analysis device can also dynamically adjust the intensity and frequency of warnings based on the driver's current emotional state, for example, increasing the volume or frequency of warnings for tense or distracted users.

[0247] Users simply need to attach a terminal device (such as a smartphone with a camera and sensors) to the vehicle's dashboard, install and launch the designated application, and the system can automatically collect data and complete all processing.

[0248] For example, in real-world applications, the terminal captures and transmits data in real time after startup. The server uses a generative artificial intelligence model to identify a pedestrian approaching from the distance. Calculating the distance to less than 1 meter and determining a safety risk, and assuming the driver is under high stress, the system displays a red alert on the terminal screen and issues a voice warning via a TTS (Text-to-Speech) engine: "Pedestrian ahead, please slow down immediately." Simultaneously, the phone vibrates as a warning. In the event of a collision, the terminal automatically saves audio and video data before and after the accident and uploads it to the cloud for insurance claims and liability determination.

[0249] Here is an example of a prompt statement for a generative artificial intelligence model:

[0250] Please design an autonomous driving safety alarm system based on the following scenario: When a pedestrian suddenly appears within 1 meter in front, analyze the driver's facial expression and emotional state in real time. If the driver is tense or distracted, automatically adjust the alarm sound and display mode. In the event of a collision, automatically record video and upload it to the cloud.

[0251] Autonomous vehicles collect real-time video of the surrounding environment and facial images of the driver. A server uses an AI model to identify distance to the vehicle and the driver's emotions, sending alerts to the terminal in case of danger. Please construct the complete data flow and corresponding processing steps (hardware and software), and provide examples.

[0252] use Figure 14 The processing flow is explained.

[0253] Step 1:

[0254] After the terminal is started, the user attaches the terminal to the vehicle's dashboard and opens the monitoring application.

[0255] Input: User actions, environment variables.

[0256] The terminal detects that it has been secured and initializes hardware such as the camera, microphone, and accelerometer, preparing to begin data acquisition.

[0257] Output: Hardware initialization complete status.

[0258] Step 2:

[0259] The terminal collects real-time image data of the vehicle's surroundings, the driver's facial expressions and voice data, as well as sensor data on the vehicle's motion status.

[0260] Input: Physical environment information, driver status.

[0261] The terminal packages the continuously captured video frames, audio streams, and real-time sensor data into a package, ready to send.

[0262] Output: Acquired video, audio, and raw sensor data.

[0263] Step 3:

[0264] The terminal sends the above data to the server in real time via a wireless network (such as Wi-Fi, 4G / 5G).

[0265] Input: The raw data stream collected by the terminal.

[0266] After being encrypted and processed by network protocols, the data is transmitted to the server as data packets.

[0267] Output: Data packets sent and transmission status.

[0268] Step 4:

[0269] The server receives data from the terminal and preprocesses video and audio data.

[0270] Input: Data packets from the terminal.

[0271] The server uses tools such as OpenCV to denoise and enhance images, and to denoise and convert audio formats to generate cleaned multimodal data.

[0272] Output: The preprocessed data stream.

[0273] Step 5:

[0274] The server uses generative artificial intelligence models (such as models based on TensorFlow and Keras) to perform object detection and behavior recognition on the processed data.

[0275] Input: Cleaned multimodal data.

[0276] The server uses AI models to identify objects ahead, calculate the distance and speed to vehicles, and combines this with driver facial expressions and voice analysis to determine their current emotional state.

[0277] Output: Object detection results, distance and speed parameters, and driver emotional state data.

[0278] Step 6:

[0279] The server assesses the risk level based on the identification results and generates corresponding early warning signals and warning parameters.

[0280] Input: Object detection and sentiment analysis results.

[0281] The server uses a rule engine or decision model to integrate information such as object distance and driver emotion to determine the level of danger and decide on the content and intensity of the warning.

[0282] Output: Warning signal, warning level parameter.

[0283] Step 7:

[0284] The server sends a warning signal back to the terminal via a network protocol.

[0285] Input: Warning signals and parameters generated by the server.

[0286] Data is sent to the terminal device via protocols such as WebSocket.

[0287] Output: Warning data packets received by the terminal.

[0288] Step 8:

[0289] The terminal issues warnings to the user using various methods such as visual, auditory, and tactile means, based on the content and parameters of the warning data packet.

[0290] Input: Warning data packet sent by the server.

[0291] A pop-up warning window appears on the terminal display, the TTS voice module reads a warning message, and the vibration motor activates vibration. The warning mode is adjusted according to the driver's mood.

[0292] Output: Multimodal alerts received by the user.

[0293] Step 9:

[0294] The terminal continuously monitors sensor data and automatically starts recording when it detects abnormal situations such as sudden braking or collision.

[0295] Input: Real-time sensor data.

[0296] The data processing logic determines whether a danger threshold has been reached; if so, it automatically saves video and audio clips from before and after the accident.

[0297] Output: Accident video file and trigger event flags.

[0298] Step 10:

[0299] The terminal encrypts and uploads relevant information such as accident recordings and sensor data to the cloud for later use.

[0300] Input: Accident-related information, video recordings, and data files.

[0301] The terminal uses a security protocol to upload data to cloud storage and generates a unique event number for subsequent tracing.

[0302] Output: Cloud storage confirmation feedback and file index.

[0303] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0304] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0305] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0306] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0307] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0308] Second Implementation Method

[0309] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0310] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0311] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0312] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0313] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0314] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0315] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0316] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0317] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0318] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0319] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0320] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0321] Example 1

[0322] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0323] Application Example 1

[0324] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0325] Example 2

[0326] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0327] Application Example 2

[0328] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0329] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0330] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0331] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0332] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0333] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0334] Third Implementation Method

[0335] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0336] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0337] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0338] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0339] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0340] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0341] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0342] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0343] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0344] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0345] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0346] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0347] Example 1

[0348] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0349] Application Example 1

[0350] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0351] Example 2

[0352] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0353] Application Example 2

[0354] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0355] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0356] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0357] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0358] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0359] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0360] Fourth Implementation Method

[0361] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0362] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0363] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0364] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0365] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0366] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0367] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0368] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0369] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0370] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0371] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0372] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0373] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0374] Example 1

[0375] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0376] Application Example 1

[0377] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0378] Example 2

[0379] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0380] Application Example 2

[0381] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0382] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0383] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0384] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0385] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0386] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0387] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0388] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0389] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0390] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0391] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0392] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0393] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0394] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0395] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0396] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0397] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0398] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0399] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0400] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0401] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0402] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0403] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0404] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0405] In addition, the following notes are provided in response to the above explanation.

[0406] Example 1

[0407] (Note 1)

[0408] An information processing system includes: a data acquisition device installed on an information processing device; a computational processing device for analyzing and processing time-series data acquired by the data acquisition device; a warning notification device for sending warning notifications to users in real time based on the information analyzed by the computational processing device; a record storage device for automatically starting recording under predetermined conditions and storing the acquired data in a storage medium or external storage device; a communication device for remotely transmitting time-series data of the analyzed object; an advanced analysis device for inputting received time-series data and physical quantity data into a generative artificial intelligence model for pattern recognition and judgment; a control device for generating warning signals based on risk assessment results and transmitting them to the warning notification device; and a management device for storing information before and after a specified time period when a specific event such as an accident occurs in a storage device, and providing relevant information to external organizations when needed.

[0409] (Note 2)

[0410] According to the information processing system described in Appendix 1, the data acquisition device includes a portable information terminal installed on the information processing equipment.

[0411] (Note 3)

[0412] According to the information processing system described in Appendix 1, the advanced analysis device includes a generative artificial intelligence model capable of estimating relative distance information, motion state, and spatial position between a person and an object.

[0413] Application Example 1

[0414] (Note 1)

[0415] An information processing system includes: an information acquisition unit installed on a target device; a processing unit for parsing information data acquired by the information acquisition unit; an output unit for issuing warning information to an output subject based on the data parsed by the processing unit; a storage unit for automatically recording and storing information data in a storage medium when an impact is detected; a sending unit for sending the information data and status information to an external computing device via a communication network; and a control unit on the external computing device for automatically controlling the content, intensity, and frequency of the warning information based on the parsing results of the information data and status information, combined with the physiological information or identification information of the output subject.

[0416] (Note 2)

[0417] According to the information processing system described in Appendix 1, the information acquisition unit is a portable terminal installed on the target device.

[0418] (Note 3)

[0419] According to the information processing system described in Appendix 1, the computing unit on the external computing device performs distance calculation between target objects, distance calculation of the identified body, distance calculation of obstacles based on a generative artificial intelligence model, and performs risk assessment processing based on the physiological information or identification information of the output subject.

[0420] Example 2

[0421] (Note 1)

[0422] An information processing system includes: an information acquisition unit installed on a driving device; a processing unit for processing the information acquired by the information acquisition unit; a notification unit for notifying a user based on the information processed by the processing unit; a storage unit for automatically acquiring and storing operating status records in the event of an accident; and a status recognition unit for recognizing the user's emotional state and adjusting the notification content, frequency, or intensity based on the recognition result. The processing unit includes a functional unit for analyzing information using a generative artificial intelligence model.

[0423] (Note 2)

[0424] According to the information processing system described in Appendix 1, the information acquisition unit includes a portable information processing device installed on the driving device.

[0425] (Note 3)

[0426] According to the information processing system described in Appendix 1, the storage unit includes the function of cooperating with an external information storage device through communication to save recorded information.

[0427] Application Example 2

[0428] (Note 1)

[0429] An information processing system includes: an information acquisition device fixed to an information processing device; an information parsing device for parsing information data acquired by the information acquisition device; a warning output device for outputting a warning to a user based on the information parsed by the information parsing device; an emotion parsing device for analyzing the user's emotional state and dynamically adjusting the intensity or frequency of the warning based on the emotional state; an information storage device for automatically recording and saving information to a storage medium when an emergency occurs based on event information collected by multiple sensors; and a generative artificial intelligence parsing device for extracting and parsing features from the information data using a generative artificial intelligence model.

[0430] (Note 2)

[0431] According to the information processing system described in Appendix 1, the information acquisition device is a portable electronic device fixed to the information processing device.

[0432] (Note 3)

[0433] According to the information processing system described in Appendix 1, the information analysis device and the emotion analysis device include a generative artificial intelligence model for simultaneously analyzing the relative distance and relative speed of the target object as well as the user's emotional changes.

Claims

1. An information processing system, characterized in that, include: Image acquisition methods fixed to driving equipment; A means of parsing image data acquired through the image acquisition means; An alarm means for sending an alarm to the driver based on the data parsed by the aforementioned parsing means; and A means of automatically recording and saving data in the event of an accident.

2. The information processing system according to claim 1, characterized in that, The image acquisition method is a mobile terminal fixed to the driving equipment.

3. The information processing system according to claim 1, characterized in that, The analytical methods include artificial intelligence models capable of analyzing distances between vehicles and between people and objects.

Images