System

Abstract

To provide a system capable of improving safety of an electric kickboard whose use is increased, preventing an accident, and quickly coping with the accident.SOLUTION: The system includes an image acquisition means fixed to an operation device, an analysis means for analyzing image data acquired by the image acquisition means, an alert means for issuing an alert to a driver on the basis of data analyzed by the analysis means, and a storage means for performing automatic recording and storing recorded data when an accident occurs.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As the use of electric scooters increases, so does the risk of traffic accidents. However, current electric scooters lack a system that can monitor the distance between vehicles and people in real time, recognize the risk of collision, and issue a timely alert to the driver. Furthermore, they lack a means to automatically collect and store evidence in the event of an accident. This can delay the collection of evidence needed for post-accident response and insurance processing, potentially putting drivers at a disadvantage. Therefore, there is a need for a system that can improve the safety of electric scooters, prevent accidents before they occur, and respond quickly when an accident does occur. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system that includes a video acquisition means fixed to a driving device, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, and a storage means for automatically recording and saving the recorded data when an accident occurs. This system acquires video using a mobile device fixed to the driving device and can analyze the distance between vehicles and the distance to people and objects using an AI model. Based on the analysis results, the system recognizes the risk of collision and issues an alert to the driver, and if an accident occurs, the system automatically records the video and stores the video data in the cloud, enabling the collection of evidence and rapid response after the accident.

[0006] "Driving device" means an electric scooter or similar personal transportation device that is operated by a user.

[0007] "Video acquisition means" refers to a device that is fixed to the driving equipment and acquires surrounding images in real time, and in many cases refers to the use of a smartphone camera or a dedicated camera.

[0008] "Analysis means" refers to the technical methods and algorithms for processing the data acquired by the image acquisition means and calculating the distance between vehicles, the distance to people, and the distance to objects, as well as the hardware and software for executing these.

[0009] "Alert means" refers to a method or device that issues a warning to the driver when there is a high risk of contact based on information obtained by the analysis means, and can use visual displays, sound, vibration, etc.

[0010] "Storage means" refers to the technical methods and devices for automatically recording and storing video data in the event of an accident, as well as the hardware and software for implementing such methods.

[0011] "Mobile device" refers to a portable electronic device such as a smartphone or tablet that has built-in or supports video acquisition means, analysis means, and alert means.

[0012] An "AI model" refers to an algorithm or system for data analysis that uses artificial intelligence technology, and is used in particular to recognize objects and calculate distances from video data. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] System Overview

[0035] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, an analysis unit, an alert unit, and a storage unit attached to the driving device. This system monitors the surrounding situation in real time while driving, alerts the driver if there is a high risk of contact, and automatically records and stores video data before and after an accident if one occurs.

[0036] System configuration

[0037] 1. Video acquisition method

[0038] The device uses a smartphone or dedicated camera attached to the driving device to capture real-time images of the surroundings, and this image data is sent to an analysis means.

[0039] 2. Analysis method

[0040] The server receives the video data sent from the device and analyzes it using an AI model. Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and objects, and evaluates the risk of collision based on speed information.

[0041] 3. Alert Methods

[0042] If the server determines that there is a high risk of contact based on the analysis results, it sends an alert signal to the terminal.

[0043] The device will then issue a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, etc.

[0044] 4. Preservation means

[0045] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[0046] The server stores the recorded video data in cloud storage, making it available as evidence to insurance companies and in legal proceedings.

[0047] A natural language description of the program's operation

[0048] Startup and preparation

[0049] The user attaches the smartphone to the electric kick scooter and launches the application.

[0050] The device will activate the camera and begin capturing video, and will also obtain speed information from the GPS sensor and accelerometer.

[0051] Data transmission and analysis

[0052] The device transmits the captured video and speed information to the server at regular intervals.

[0053] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0054] Alert generation and notification

[0055] If the server determines from the analysis results that there is a high risk of contact, it generates an alert signal and sends it to the terminal.

[0056] The device receives the alert signal and warns the driver visually, audibly, or by vibration, allowing the driver to acknowledge the alert and take necessary evasive action.

[0057] Accident response and data preservation

[0058] If the impact sensor detects a force above a certain level, the device will automatically switch to accident mode and begin recording.

[0059] The device sends the recorded data to a server, which then stores it in cloud storage, where it can be used for accident response and legal procedures.

[0060] Specific examples

[0061] System startup and start-up

[0062] The user attaches their smartphone to the electric kick scooter and launches the application.

[0063] The device will activate the camera and begin taking video and measuring speed.

[0064] Alert generation and avoidance actions

[0065] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[0066] The server determines that the risk of contact is high and generates an alert signal to send to the terminal.

[0067] The device will display the received alert on the screen and provide a voice prompt to warn you.

[0068] Users acknowledge the alert, slow down, and watch out for pedestrians.

[0069] Response in the event of an accident

[0070] The device comes into contact with an obstacle and the impact sensor is activated.

[0071] The device automatically records and saves footage before and after the accident.

[0072] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal procedures as needed.

[0073] In this way, each component works together to create a system that ensures the safety of the driver.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The user attaches the smartphone to the electric scooter and launches the corresponding application.

[0077] Step 2:

[0078] The device will activate the camera and begin capturing video in real time, while simultaneously obtaining speed information from the GPS sensor and accelerometer.

[0079] Step 3:

[0080] The device sends the captured video data and speed information to the server at regular intervals, which are usually set in units of a few seconds.

[0081] Step 4:

[0082] The server receives the video data and speed information sent from the terminal and stores them in temporary storage.

[0083] Step 5:

[0084] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0085] Step 6:

[0086] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal that includes the type of risk (vehicle, person, object) and the risk level (high, medium, low).

[0087] Step 7:

[0088] The server sends the generated alert signal to the terminal.

[0089] Step 8:

[0090] The terminal receives an alert signal from the server and issues a warning to the driver using a display device or speaker, such as a visual display, sound, or vibration.

[0091] Step 9:

[0092] Users can check the alerts from their devices and take evasive action by slowing down if necessary.

[0093] Step 10:

[0094] If the shock sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[0095] Step 11:

[0096] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[0097] Step 12:

[0098] The terminal transmits the recorded video data to the server.

[0099] Step 13:

[0100] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[0101] Step 14:

[0102] In the event of an accident, users can use the accident reporting feature within the application to enter more detailed information.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Previous technology had the problem of being unable to respond immediately to situations that required the driver's attention. Furthermore, when an accident occurred, video data serving as evidence was not acquired, making it impossible to quickly provide the information necessary for insurance companies and legal procedures. This resulted in issues such as insufficient driver safety and complicated, time-consuming post-accident procedures.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording and saving the recorded data when an accident occurs, a speed acquisition means for acquiring speed information from the driving equipment and sending it to the analysis means, and a risk assessment means for analyzing the received video data and speed information using an AI model and assessing the risk of contact. This makes it possible to quickly assess risks while driving and notify the driver, and automatically save important video data when an accident occurs.

[0108] "Driving device" refers to an electric scooter, bicycle, or other personal transportation device.

[0109] "Video acquisition means" refers to a camera or mobile device that is fixed to the driving equipment and captures the surrounding situation in real time.

[0110] "Analysis means" refers to a device or AI model that analyzes the video data acquired by the video acquisition means and performs object recognition and distance measurement.

[0111] "Alert means" refers to a device that issues a visual, audio, or vibration warning to the driver based on data obtained from the analysis means.

[0112] "Storage means" refers to a device or cloud storage that automatically starts recording when an accident occurs and saves the captured video data.

[0113] "Speed ​​acquisition means" refers to a sensor or device for acquiring speed information of driving equipment and transmitting it to analysis means.

[0114] "Risk assessment means" refers to a device or algorithm that assesses the risk of contact based on analyzed video data and speed information.

[0115] An "AI model" refers to an algorithm that uses machine learning and deep learning techniques to analyze image and video data and recognize specific patterns and objects.

[0116] "Real-time" refers to data being acquired and processed and analyzed with very little delay.

[0117] "Automatic recording" refers to a function that automatically starts recording when a sensor or other device detects certain conditions.

[0118] The present invention provides a system for improving safety in driving equipment such as electric kick scooters. The system includes an image acquisition unit, an analysis unit, an alert unit, a storage unit, a speed acquisition unit, and a risk assessment unit.

[0119] Video acquisition

[0120] The user attaches a mobile terminal (such as a smartphone) to the driving device and launches a dedicated application.

[0121] The device uses a camera to capture real-time images of the surrounding area, and collects location information from a GPS sensor and speed information from an acceleration sensor. This data is then sent to a server at regular intervals.

[0122] Data analysis

[0123] The server receives the video data and speed information transmitted from the terminal.

[0124] The server analyzes the video data using an AI model. Specifically, it calculates the distance between the vehicle and the person, and the distance between the object and the vehicle, and identifies the object. This can be done using deep learning libraries such as TENSORFLOW (registered trademark).

[0125] Risk assessment and alert notifications

[0126] The server evaluates the risk of contact based on the analysis results, and if it determines that the risk is high, it generates an alert signal and sends it to the device.

[0127] The device will then display a warning to the user based on the received alert signal, which can be visual (display on the smartphone screen), audible (audio alert), or tactile (vibration).

[0128] Incident detection and data storage

[0129] If the impact sensor detects a certain level of force, the device automatically switches to recording mode, recording video data before and after the accident and sending it to a server.

[0130] The server then stores the received recording data in cloud storage, which can be managed for insurance companies or legal procedures. Cloud storage such as Amazon S3 can be used.

[0131] Specific examples

[0132] System startup and start-up

[0133] The user attaches a smartphone to the electric scooter and launches the application, which causes the entire system to transition from standby to operational mode.

[0134] The device will be adjusted so that the camera is facing forward and will display a "Ready" message after launching the application.

[0135] Alert generation and avoidance actions

[0136] After the device sends the video footage, the server performs real-time analysis and generates an alert a few seconds later saying, "There is a pedestrian ahead."

[0137] The device will display a red warning on the smartphone screen and simultaneously play an audio alert saying "Be careful."

[0138] In response to this alert, the user slows down and checks the situation ahead.

[0139] Response in the event of an accident

[0140] The device goes into recording mode the moment the impact sensor detects contact with the kick scooter.

[0141] The device records a few seconds of data, including the immediately preceding footage, and automatically sends it to the server.

[0142] The server immediately uploads this data to cloud storage and provides it to insurance companies if required.

[0143] Prompt Sentence Examples

[0144] "I would like to develop a safety support system for electric scooters. I would like it to monitor the surroundings in real time while driving and send an alert to the driver if there is a high risk of collision. In the event of an accident, I would like it to automatically record and save video data before and after the accident. What kind of hardware and software would be required?"

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Step 1: System startup

[0147] The user attaches a smartphone to the electric scooter and launches a dedicated application. The input is the attachment of the smartphone and the launch of the application, and the output is the entire system transitioning to an operational state.

[0148] The device initializes the camera, GPS sensor, and accelerometer, and notifies the user that it is ready. Specifically, the device starts collecting data from each sensor and displays a "Ready" message on the application screen.

[0149] Step 2: Data Acquisition

[0150] The device starts capturing real-time images of its surroundings using its camera, with the input being the surrounding images and data from the GPS and accelerometer, and the output being packetized versions of these data.

[0151] The device periodically transmits the collected video data, location information, and speed information to a server. Specifically, the device periodically streams the data, which is then sent to the server via a high-speed internet connection.

[0152] Step 3: Video analysis

[0153] The server receives the video data and speed information sent from the device. The input is the data packets from the device, and the output is the analysis results from the AI ​​model.

[0154] The server analyzes the received data using an AI model. Specifically, it calculates the distance between the vehicle and the person, and the distance between the object and the vehicle, and then performs object identification. Specifically, the AI ​​model analyzes each frame of video and recognizes specific objects (pedestrians, vehicles, fixed objects, etc.).

[0155] Step 4: Risk assessment

[0156] The server evaluates the risk of contact based on the results of video analysis and speed information. The input is the analysis results of the AI ​​model and speed information, and the output is the risk assessment result.

[0157] If the server determines that the risk of contact is high, it generates an alert signal. Specifically, the server uses a scoring algorithm to evaluate the risk level and generates an alert signal if the risk exceeds a certain threshold.

[0158] Step 5: Alert Notification

[0159] The server sends the generated alert signal to the terminal. The input is the result of the risk assessment and the output is the alert signal.

[0160] The device will then provide a visual, audio, or vibration warning to the user based on the received alert signal, by displaying a warning message on the smartphone screen, playing an audio alert, and activating the vibration motor if necessary.

[0161] Step 6: Incident detection and data storage

[0162] The device automatically switches to recording mode when the impact sensor detects a certain amount of force. The input is a signal from the impact sensor, and the output is a command to start recording and save the data.

[0163] The device sends the recorded data to a server, which then stores it in cloud storage. Specifically, the device records the recorded data, including the most recent footage, and immediately sends it to the server. The server then uploads the received data to cloud storage, making it available to insurance companies and legal proceedings.

[0164] The above are the specific steps of the program processing of this system.

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] Traffic accidents are often caused by drivers' lack of attention or awareness of their surroundings. Therefore, there is a need for a system that can monitor the surrounding situation in real time while driving and issue alerts at the appropriate time. However, current technology makes it difficult to achieve this efficiently and accurately. In addition, recording and storing data in the event of an accident is also an important issue in ensuring driver safety.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording when an accident occurs and saving the recorded data, and a means for evaluating the risk of contact in real time and sending a warning to the automated driving system. This makes it possible to quickly evaluate the risk of contact while driving, issue an appropriate warning, and reliably save data when an accident occurs.

[0170] "Driving equipment" refers to any device used to assist in driving a vehicle.

[0171] "Image acquisition means" refers to a device that collects visual information using a camera, sensor, etc.

[0172] "Analysis means" refers to software or hardware for analyzing acquired data and extracting specific information.

[0173] "Alert means" refers to a device or function for issuing a warning to the driver or system based on the analysis results.

[0174] "Storage means" refers to a device or system that stores acquired data and makes it available for later use.

[0175] "Contact risk" refers to the possibility of colliding with other objects or people while driving.

[0176] "Real-time" refers to data acquisition and processing occurring immediately, without delay.

[0177] An "autonomous driving system" refers to an integrated system that enables a vehicle to drive autonomously without driver intervention.

[0178] System Overview

[0179] This invention is a system that combines a series of means installed in driving equipment to improve safety. Specifically, it is composed of video acquisition means, analysis means, alert means, and storage means. This makes it possible to evaluate the risk of contact while driving in real time, issue necessary warnings, and record and store data when an accident occurs.

[0180] Hardware and software used

[0181] 1. Hardware

[0182] Cameras: These are mounted on driving equipment for safety monitoring, with 360-degree cameras being particularly recommended.

[0183] AI-dedicated chip: Used for high-speed analysis of video data and sensor data.

[0184] Speed ​​sensor: Monitors the vehicle's speed.

[0185] Distance sensor: Measures the distance to surrounding objects and people.

[0186] Impact sensor: Detects collisions and impacts to detect the occurrence of an accident.

[0187] 2. Software

[0188] AI analysis module: Includes an AI model that uses TensorFlow / Keras, etc. to analyze the distance between vehicles, the distance between people, and the distance between objects.

[0189] Image Processing Module: Acquires and pre-processes camera images using OpenCV.

[0190] Data storage module: Has the function of uploading video and sensor data to cloud storage in the event of an accident.

[0191] Alert module: Provides visual, audio, or vibration warnings to the driver.

[0192] Natural language explanation of program processing

[0193] Server Processing

[0194] The server receives video and sensor data sent from the device and analyzes this data in real time using an AI analysis module. Based on the analysis results, if there is a high risk of contact, an alert signal is generated and sent to the device. Furthermore, if an accident occurs, video data before and after the accident is automatically saved to cloud storage.

[0195] Terminal handling

[0196] The device collects data from cameras and sensors attached to the driving equipment and sends it to a server. When it receives an alert signal from the server, it warns the driver visually, audibly, or by vibrating. When the impact sensor detects a force greater than a certain level, it switches to accident mode, starts recording, and sends the data to the server.

[0197] User operations

[0198] The user mounts their smartphone on the driving device and launches the application. The application provides an interface for acquiring camera and sensor data and sending it to the server. If necessary, the user can acknowledge alerts and take appropriate evasive action.

[0199] Specific examples

[0200] Risk detection while driving: The server detects approaching pedestrians ahead and sends a warning to the device when the distance is less than 1.5 meters. The device then issues a visual and audio alert to remind the user to be careful.

[0201] Accident data storage: When the device detects a collision, it automatically stores video data before and after the accident in cloud storage, which can be used for later review.

[0202] Prompt Sentence Examples

[0203] "Creating an AI model that assesses collision risk and provides necessary warnings to drivers"

[0204] "Development of a system for monitoring the surroundings of autonomous vehicles in real time"

[0205] "A program that automatically assesses the risk of an accident based on the analysis results and sends a warning to the autonomous driving system."

[0206] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0207] Step 1:

[0208] The user attaches the smartphone to the driving device and launches the application. The input is the smartphone and the application, and the output is the data collection status from the camera and sensors. Specifically, the user taps to launch the application and confirms that the smartphone is attached.

[0209] Step 2:

[0210] The device starts the camera and captures the surrounding video in real time. The input is the camera and the output is the captured time-series video data. Specifically, the application calls the camera module and starts streaming the video.

[0211] Step 3:

[0212] The device collects input data from the speed sensor and distance sensor. The input is the speed sensor and distance sensor, and the output is the current speed data and distance data to surrounding objects. Specifically, the device reads data from the sensor terminal and processes it within the application.

[0213] Step 4:

[0214] The device transmits the captured video and sensor data to the server at regular intervals. The input is the video and sensor data, and the output is the data transmitted to the server. Specifically, the device divides the data into packets and uploads them to the server via an internet connection.

[0215] Step 5:

[0216] The server analyzes the received video data and sensor data using an AI model. The input is the received video data and sensor data, and the output is risk assessment data as the analysis result. Specifically, the server runs an AI analysis module and calculates the distance between vehicles, people, and objects from the data.

[0217] Step 6:

[0218] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and sends it to the terminal.The input is risk assessment data and the output is an alert signal.Specifically, if the risk assessment exceeds a certain threshold, the server calls the alert generation module and sends an alert signal to the terminal.

[0219] Step 7:

[0220] The terminal issues a warning to the driver based on the alert signal received from the server. The input is the alert signal, and the output is a visual, audio, or vibration warning. Specifically, the terminal activates the warning module and issues a warning using the screen display, speaker, or vibration function.

[0221] Step 8:

[0222] If the impact sensor detects a force greater than a certain level, the device switches to accident mode and starts automatic recording. The input is the impact sensor data, and the output is the recorded video data. Specifically, the device activates the accident detection module and starts recording video.

[0223] Step 9:

[0224] The device sends the recorded video data to the server, which then stores it in cloud storage. The input is the recorded video data, and the output is the data stored in cloud storage. Specifically, the device sends the video data to the server via a transmission module, and the server uploads the received data to the cloud. This stored data can be used later for insurance companies or legal procedures if necessary.

[0225] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0226] System Overview

[0227] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, analysis unit, alert unit, storage unit, and an emotion engine that recognizes the user's emotions. This system monitors the surrounding environment in real time while driving, alerts the driver if there is a high risk of contact, and automatically records and stores video data before and after an accident if one occurs. The system also recognizes the user's emotional state and adjusts the intensity and frequency of alerts accordingly to provide appropriate warnings.

[0228] System configuration

[0229] 1. Video acquisition method

[0230] The device uses a smartphone or dedicated camera attached to the driving device to capture real-time images of the surroundings, and this image data is sent to an analysis means.

[0231] 2. Analysis method

[0232] The server receives the video data sent from the device and analyzes it using an AI model. Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and objects, and assesses the risk of collision based on speed information.

[0233] 3. Alert Methods

[0234] If the server determines that there is a high risk of contact based on the analysis results, it sends an alert signal to the terminal.

[0235] The device will then issue a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, etc.

[0236] 4. Preservation means

[0237] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[0238] The server stores the recorded video data in cloud storage, making it available as evidence to insurance companies and in legal proceedings.

[0239] 5. Emotion Engine

[0240] The device includes facial recognition technology to analyze the user's facial expressions and assess the driver's emotional state.

[0241] The terminal includes voice recognition technology to analyze the user's voice input and assess their emotional state.

[0242] The server provides a means to adjust the intensity and frequency of alerts based on the user's emotional state.

[0243] A natural language description of the program's operation

[0244] Startup and preparation

[0245] The user attaches the smartphone to the electric kick scooter and launches the application.

[0246] The device will activate the camera and begin capturing video, and will also obtain speed information from the GPS sensor and accelerometer.

[0247] The device initializes the emotion engine and captures the user's face and voice data in real time.

[0248] Data transmission and analysis

[0249] The device transmits the captured video, speed information, and the user's face and voice data to the server at regular intervals.

[0250] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0251] The server analyzes the user's emotional data and assesses their state of stress and attention while driving.

[0252] Alert generation and notification

[0253] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and adjusts the strength and frequency of the alert based on the user's emotional state.

[0254] The device will then issue a warning to the driver based on the received alert signal, which can be visual, audible, or vibration.

[0255] Accident response and data preservation

[0256] If the shock sensor detects a force greater than a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[0257] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[0258] The terminal transmits the recorded video data to the server.

[0259] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[0260] Specific examples

[0261] System startup and start-up

[0262] The user attaches their smartphone to the electric kick scooter and launches the application.

[0263] The device will activate its camera and begin taking video, measuring speed, and capturing emotional data.

[0264] Alert generation and avoidance actions

[0265] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[0266] The server determines that the risk of contact is high and sets an appropriate alert strength based on the user's emotional data. For example, if the user is in a high stress state, it selects a moderately strong voice alert.

[0267] The server generates an alert signal and sends it to the terminal.

[0268] The device will display the received alert on the screen and provide a voice prompt to warn you.

[0269] Users acknowledge the alert, slow down, and watch out for pedestrians.

[0270] Response in the event of an accident

[0271] The device comes into contact with an obstacle and the impact sensor is activated.

[0272] The device will automatically start recording and record footage for a certain period of time before and after the accident.

[0273] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal procedures as needed.

[0274] In this way, each component works together to ensure the safety of the driver. The addition of an emotion engine also enables the system to respond flexibly to the user's condition.

[0275] The processing flow will be explained below.

[0276] Step 1:

[0277] The user attaches the smartphone to the electric scooter and launches the application.

[0278] Step 2:

[0279] The device will activate the camera and begin capturing video in real time, while simultaneously obtaining speed information from the GPS sensor and accelerometer.

[0280] Step 3:

[0281] The device initializes the emotion engine, captures the user's face and voice data in real time, and uses facial and voice recognition technologies to analyze the user's emotional state from their facial expressions and words.

[0282] Step 4:

[0283] The device transmits the captured video data, speed information, and user emotion data to the server at regular intervals, usually set to a few seconds.

[0284] Step 5:

[0285] The server receives the video data, speed information, and emotion data sent from the device and stores them in temporary storage.

[0286] Step 6:

[0287] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0288] Step 7:

[0289] The server evaluates the user's emotional state analyzed by the emotion engine and adjusts the intensity and frequency of alerts based on whether the user is in an emotional state such as stress or distraction.

[0290] Step 8:

[0291] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal that includes the type of risk (vehicle, person, object) and the risk level (high, medium, low) according to the emotional state.

[0292] Step 9:

[0293] The server sends the generated alert signal to the terminal.

[0294] Step 10:

[0295] The terminal receives an alert signal from the server and issues a warning to the driver using a display device or speaker. The alert is notified by visual display, sound, vibration, etc. The warning is made stronger especially when the user is in a high stress state.

[0296] Step 11:

[0297] Users can check the alerts from their devices and take evasive action by slowing down if necessary.

[0298] Step 12:

[0299] If the shock sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[0300] Step 13:

[0301] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[0302] Step 14:

[0303] The terminal transmits the recorded video data to the server.

[0304] Step 15:

[0305] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[0306] Step 16:

[0307] In the event of an accident, users can use the accident reporting feature within the application to enter more detailed information.

[0308] In this way, each step works smoothly together to ensure the safety of the driver. The addition of an emotion engine enables the system to respond flexibly according to the user's psychological state.

[0309] Example 2

[0310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] Conventional electric scooter safety systems simply assess physical risks without considering the driver's situational awareness or stress level. As a result, they have been unable to adequately address the risk of accidents caused by high stress levels or reduced attention. Furthermore, in the event of an accident, automation of data storage is insufficient, which can lead to a lack of video evidence.

[0312] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording when an accident occurs and saving the recorded data, and an emotion recognition means for recognizing the user's emotional state and adjusting the intensity and frequency of the alert. This enables flexible warnings according to the user's emotional state and reliable data saving when an accident occurs.

[0313] "Image acquisition means" refers to a device or apparatus that is fixed to the driving equipment and that captures and acquires images of the surrounding area.

[0314] "Analysis means" refers to equipment or software that analyzes acquired video data and other sensor data to perform risk assessments, distance measurement, etc.

[0315] The "alert means" is a device or notification system for issuing a warning to the driver based on the analysis results.

[0316] "Storage means" refers to a system or device that automatically starts recording when an accident occurs and stores the recorded data.

[0317] "Emotion recognition means" refers to technology or a system that recognizes the user's emotional state and adjusts the intensity and frequency of alerts according to that state.

[0318] A "generative AI model" is an algorithm or framework that uses machine learning or deep learning to perform a specific task (e.g., distance measurement or risk assessment).

[0319] "Driving equipment" refers to means of transportation such as electric kick scooters, and in this case refers to equipment equipped with various sensors and systems to improve safety.

[0320] MODE FOR CARRYING OUT THE INVENTION

[0321] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, an analysis unit, an alert unit, a storage unit, and an emotion recognition unit that recognizes the user's emotions. This system monitors the surroundings in real time while driving, and issues an alert to the driver if there is a high risk of contact. In the event of an accident, the system automatically records and stores video data before and after the accident. The system also recognizes the user's emotional state and adjusts the intensity and frequency of alerts accordingly to provide appropriate warnings.

[0322] System configuration

[0323] 1. Video acquisition method

[0324] The terminal uses a mobile device (such as a smartphone or dedicated camera) attached to the driving equipment to capture images of the surroundings in real time. This image data is sent to the analysis means.

[0325] 2. Analysis method

[0326] The server receives the video data sent from the device and analyzes it using a generative AI model (e.g., TensorFlow or PyTorch). Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and other objects, and assesses the risk of contact based on speed information.

[0327] 3. Alert Methods

[0328] If the server determines that the risk of contact is high based on the analysis results, it sends an alert signal to the terminal.

[0329] The device issues a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, or the like.

[0330] 4. Preservation means

[0331] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[0332] The server stores the recorded video data in cloud storage (e.g., AWS (registered trademark) S3 or Google (registered trademark) Cloud Storage) so that it can be used as evidence by insurance companies or in legal proceedings.

[0333] 5. Emotion recognition means

[0334] The terminal includes facial recognition technology to perform facial expression analysis of the user and assess the emotional state of the driver.

[0335] The terminal includes voice recognition technology to analyze the user's voice input and assess their emotional state.

[0336] The server provides a means to adjust the intensity and frequency of alerts based on the user's emotional state.

[0337] Specific examples

[0338] System startup and start-up

[0339] The user attaches their smartphone to the electric kick scooter and launches the application.

[0340] The device will activate its camera and begin taking video, measuring speed, and capturing emotional data.

[0341] Alert generation and avoidance actions

[0342] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[0343] The server determines that the risk of contact is high and sets an appropriate alert strength based on the user's emotional data. For example, if the user is in a high stress state, a moderately strong voice alert is selected.

[0344] The server generates and sends an alert signal to the terminal.

[0345] The device will display the received alert on the screen and give an audio warning.

[0346] The user acknowledges the alert, slows down, and watches out for pedestrians.

[0347] Response in the event of an accident

[0348] The device comes into contact with an obstacle and the impact sensor is activated.

[0349] The device will automatically start recording and record footage for a certain period of time before and after the accident.

[0350] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal proceedings as needed.

[0351] Prompt Sentence Examples

[0352] Here are some example prompts to explain system behavior to a generative AI model:

[0353] "Please explain how the safety system for electric scooters works. Please use a concrete scenario to explain step by step the roles of the user, device, and server, as well as the system's functions."

[0354] In this way, a system that realizes safe driving is constructed by using specific hardware and software to process and calculate data.

[0355] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0356] System program processing flow

[0357] Step 1: Boot the system

[0358] The user attaches the smartphone to the electric kick scooter and launches the application.

[0359] Input: Electric scooter, smartphone, application startup operation

[0360] Output: System booting, initial setup complete

[0361] Specific operation: Place the device in the dedicated smartphone holder and tap the app icon to launch it.

[0362] Step 2: Initialize the video and sensors

[0363] The device will start the camera and begin recording video, and will also initialize the GPS sensor and acceleration sensor to begin acquiring location and speed information.

[0364] Input: Smartphone, camera, GPS sensor, accelerometer

[0365] Output: Real-time video data, position data, speed data

[0366] Specific behavior: Launches the camera app in the background and calls the sensor API to start the data stream.

[0367] Step 3: Activating the Emotion Engine

[0368] The device initializes the emotion engine and begins capturing the user's face and voice data in real time.

[0369] Input: Smartphone, face recognition library, voice recognition API

[0370] Output: Real-time emotion data

[0371] Specific behavior: Launches the face recognition library and speech recognition API, preparing for data capture.

[0372] Step 4: Send data

[0373] The terminal transmits the captured video, speed information, and the user's face and voice data to the server at regular intervals (for example, once per second).

[0374] Input: Real-time video data, location data, speed data, emotion data

[0375] Output: Data sent to the server

[0376] What it does: Sends data to the server using an HTTP POST request.

[0377] Step 5: Video Analysis

[0378] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0379] Input: Video data, position data, speed data

[0380] Output: Analysis results (distance data, risk assessment)

[0381] What it does: Identify objects in images and measure distances using computer vision libraries (e.g., OpenCV) and deep learning models (e.g., TensorFlow).

[0382] Step 6: Sentiment Analysis

[0383] The server analyzes the user's emotional data and evaluates their state of stress and attention while driving.

[0384] Input: Emotion data (face, voice)

[0385] Output: Analysis results (stress level, attention evaluation)

[0386] Specific operations: Facial features are analyzed using an expression analysis model, and vocal tone and speed are measured using an audio analysis model.

[0387] Step 7: Alert Generation

[0388] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and adjusts the strength and frequency of the alert based on the user's emotional state.

[0389] Input: Risk assessment results, emotion assessment results

[0390] Output: Alert signal (strength, frequency)

[0391] What it does: It quantifies the risk level by setting standard deviations and thresholds, then runs an algorithm that determines the type and intensity of the alert (visual, audio, vibration, etc.).

[0392] Step 8: Alert Notifications

[0393] The device issues a warning to the driver based on the received alert signal, notifying the driver of the alert through visual display, audio, vibration, or other means.

[0394] Input: Alert signal

[0395] Output: Warning to the driver (visual, audio, vibration)

[0396] Specific actions: Displays a warning message on the smartphone display, outputs an audio alert through the speaker, and activates the vibration motor.

[0397] Step 9: Detecting an incident

[0398] If the shock sensor detects a force greater than a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[0399] Input: Impact sensor data

[0400] Output: Switch to accident mode

[0401] Specific operation: Analyzes accelerometer data and sets a flag if a sudden change is detected.

[0402] Step 10: Record footage of the incident

[0403] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage of the time before and after the accident.

[0404] Input: Accident mode, camera video data

[0405] Output: Recording data

[0406] What it does: Uses the video recording API to save the video stream to a fixed disk or cloud storage.

[0407] Step 11: Data transmission and storage

[0408] The terminal transmits the recorded video data to the server.

[0409] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal procedures if necessary.

[0410] Input: Recording data

[0411] Output: Cloud saved data

[0412] Specific operation: Data is stored using a database management system (e.g., AWS S3 or Google Cloud Storage) while setting access permissions.

[0413] In this way, each step works together to provide a system that comprehensively supports driver safety.

[0414] (Application example 2)

[0415] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0416] While driving an autonomous vehicle, multiple sensors and analytical means are required to properly monitor the surrounding environment and the driver's state and ensure safe driving. However, these means operating individually are not sufficient. In particular, it is difficult to recognize the driver's emotional state in real time and issue appropriate warnings or alerts. Furthermore, it is necessary to respond quickly and accurately in the event of an accident. Therefore, this invention provides a system that recognizes the driver's emotional state and issues appropriate alerts to improve the safety of autonomous vehicles.

[0417] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0418] In this invention, the server includes a video capture means fixed to the driving equipment, an analysis means for analyzing the video data captured by the video capture means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, an emotion engine for recognizing the emotional state of the driver and adjusting the intensity and frequency of the alert, and a storage means for automatically recording when an accident occurs and saving the recorded data. This enables real-time monitoring while driving and flexible response according to the emotional state of the driver.

[0419] The "image acquisition means" is a device that is fixed to the driving equipment and captures the surrounding situation in real time.

[0420] The "analysis means" is a device or software for analyzing the video data acquired by the video acquisition means and calculating distance, speed, etc.

[0421] The "alert means" is a mechanism or system that issues a visual, audible, or tactile warning to the driver based on the information obtained by the analysis means.

[0422] An "emotion engine" is a mechanism or software that analyzes the driver's facial expressions and voice to recognize and evaluate their emotional state.

[0423] "Storage means" refers to a device or system that automatically starts recording when an accident occurs and records and stores the video data.

[0424] An "AI model" refers to a learning algorithm or computational model that uses artificial intelligence to analyze the distance and speed of objects from video data.

[0425] MODE FOR CARRYING OUT THE INVENTION

[0426] System Overview

[0427] The present invention provides a system for improving the safety of autonomous vehicles. The system includes a plurality of image acquisition means, analysis means, alert means, emotion engine, and storage means, all of which are fixed to a driving device. Specific embodiments of each means are described below.

[0428] Video acquisition method

[0429] The device uses multiple cameras to capture images of the surroundings in real time. This allows image data to be acquired from various viewpoints, such as the front and sides of the vehicle. This image data is then sent to subsequent analysis means. The image acquisition means includes high-resolution cameras and software control for these cameras.

[0430] Analysis means

[0431] The server receives the video data sent from the device and analyzes it using a generative AI model. Specifically, it calculates the distance between the vehicle and the pedestrian, and the distance between the vehicle and the object, and evaluates the risk of contact based on speed information. This analysis uses machine learning frameworks such as TensorFlow and Keras. It also uses dlib as facial recognition technology to evaluate the driver's emotional state.

[0432] Alert Method

[0433] If the server determines that there is a high risk of collision based on the analysis results, it generates an alert signal and sends it to the device. Based on the received alert signal, the device warns the driver by means of visual display, sound, vibration, etc. For example, a text-to-speech engine (TTS engine) is used for the sound alert.

[0434] Emotion Engine

[0435] The device uses a camera and microphone to capture the driver's face and voice data. This data is analyzed using an emotion recognition model to evaluate the driver's emotional state. A generative AI model adjusts the intensity and frequency of alerts according to the driver's stress and attention levels. This allows for flexible responses tailored to the driver's psychological state.

[0436] Preservation means

[0437] When the device detects a force exceeding a certain level using impact and acceleration sensors, it determines that an accident has occurred and automatically starts recording. This recorded data is stored in cloud storage via a server and can be used by insurance companies or in legal proceedings as needed. Since recording requires high-speed data transfer and large-capacity storage, a cloud service is used.

[0438] Specific examples

[0439] To activate the system, the user attaches their smartphone to the dashboard of the autonomous vehicle and launches the application. The device's camera captures video in real time and sends it to the server. The server analyzes the video data, detects the presence of pedestrians ahead, and issues an alert to the driver. Furthermore, if the driver is under high stress, a stronger audio alert is issued. In the event of an accident, the recorded data is stored in the cloud and necessary measures are taken.

[0440] Prompt Sentence Examples

[0441] "We will design an application that monitors the driver's emotional state in real time when an object (pedestrian) that poses a risk of contact suddenly appears within one meter ahead, and issues an appropriate alert to enable safe driving."

[0442] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0443] Step 1: The device activates the camera and captures video in real time.

[0444] Input: Real-time video data from the camera.

[0445] Output: The video data stream of the device.

[0446] Specific operation: A camera fixed to the terminal continuously captures video, and this video data is sent to subsequent processing steps.

[0447] Step 2: The terminal transmits the video data to the server.

[0448] Input: Real-time video data captured by the device.

[0449] Output: Video data stream to server.

[0450] Specific operation: The device transmits the acquired video data to the server in real time via wireless communication, etc. This communication uses networks such as Wi-Fi or LTE.

[0451] Step 3: The server performs data pre-processing on the received video data.

[0452] Input: Video data sent from the device.

[0453] Output: Pre-processed video data.

[0454] Specific operation: The server performs preprocessing such as sharpening, noise reduction, and resolution adjustment on the video data, thereby improving the accuracy of analysis.

[0455] Step 4: The server analyzes the preprocessed video data with the generative AI model.

[0456] Input: Preprocessed video data.

[0457] Output: Position and distance data of objects and people.

[0458] How it works: The server's generated AI model calculates and extracts information about the distance between vehicles and pedestrians and objects in the video. It also analyzes speed information.

[0459] Step 5: The server captures the driver's face and voice data and sends it to the emotion engine.

[0460] Input: Driver's face and voice data.

[0461] Output: Driver's emotional state data.

[0462] Specific operation: The device uses a camera and microphone to continuously capture the driver's facial expressions and voice, which are then analyzed by the emotion engine. The analysis results are sent to the server.

[0463] Step 6: The server evaluates the risk of contact based on the analysis results.

[0464] Input: Object distance data and driver emotional state data.

[0465] Output: Contact risk assessment data.

[0466] Specific operation: The server combines the distance data obtained by the generative AI model with the driver's emotional state to calculate the risk of collision. If the risk is high, it generates an alert signal.

[0467] Step 7: The server sends an alert signal to the terminal.

[0468] Input: Contact risk assessment data.

[0469] Output: Alert signal data packet.

[0470] Specific operation: If the server detects a high risk of collision, it generates an alert signal and sends it to the terminal to warn the driver visually, audibly, or by vibration.

[0471] Step 8: The terminal warns the driver based on the received alert signal.

[0472] Input: The alert signal sent from the server.

[0473] Output: Visual, audible, or tactile warning to the driver.

[0474] Specific actions: The device visually displays a warning message on the display, audibly plays an alert sound through the built-in speaker, and haptically activates the vibration motor.

[0475] Step 9: The device monitors sensor information such as impacts and sudden stops, and automatically starts recording when an accident occurs.

[0476] Input: Impact sensor and accelerometer data.

[0477] Output: Recorded data of the accident video.

[0478] Specific operation: The device constantly monitors information from the impact sensor and acceleration sensor, and automatically starts recording when it detects a force above a certain level. It records and saves footage of the accident before and after it occurs.

[0479] Step 10: The device uploads the recording data to cloud storage.

[0480] Input: Recording data.

[0481] Output: Saved data in cloud storage.

[0482] Specific operation: After the device has completed recording, it uploads the recorded data to cloud storage and manages it so that it can be used by insurance companies or for legal procedures if necessary.

[0483] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0484] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0485] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0486] [Second embodiment]

[0487] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0488] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0489] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0490] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0491] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0492] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0493] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0494] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0495] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0496] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0497] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0498] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0499] System Overview

[0500] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, an analysis unit, an alert unit, and a storage unit attached to the driving device. This system monitors the surrounding situation in real time while driving, alerts the driver if there is a high risk of contact, and automatically records and stores video data before and after an accident if one occurs.

[0501] System configuration

[0502] 1. Video acquisition method

[0503] The device uses a smartphone or dedicated camera attached to the driving device to capture real-time images of the surroundings, and this image data is sent to an analysis means.

[0504] 2. Analysis method

[0505] The server receives the video data sent from the device and analyzes it using an AI model. Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and objects, and evaluates the risk of collision based on speed information.

[0506] 3. Alert Methods

[0507] If the server determines that there is a high risk of contact based on the analysis results, it sends an alert signal to the terminal.

[0508] The device will then issue a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, etc.

[0509] 4. Preservation means

[0510] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[0511] The server stores the recorded video data in cloud storage, making it available as evidence to insurance companies and in legal proceedings.

[0512] A natural language description of the program's operation

[0513] Startup and preparation

[0514] The user attaches the smartphone to the electric kick scooter and launches the application.

[0515] The device will activate the camera and begin capturing video, and will also obtain speed information from the GPS sensor and accelerometer.

[0516] Data transmission and analysis

[0517] The device transmits the captured video and speed information to the server at regular intervals.

[0518] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0519] Alert generation and notification

[0520] If the server determines from the analysis results that there is a high risk of contact, it generates an alert signal and sends it to the terminal.

[0521] The device receives the alert signal and warns the driver visually, audibly, or by vibration, allowing the driver to acknowledge the alert and take necessary evasive action.

[0522] Accident response and data preservation

[0523] If the impact sensor detects a force above a certain level, the device will automatically switch to accident mode and begin recording.

[0524] The device sends the recorded data to a server, which then stores it in cloud storage, where it can be used for accident response and legal procedures.

[0525] Specific examples

[0526] System startup and start-up

[0527] The user attaches their smartphone to the electric kick scooter and launches the application.

[0528] The device will activate the camera and begin taking video and measuring speed.

[0529] Alert generation and avoidance actions

[0530] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[0531] The server determines that the risk of contact is high and generates an alert signal to send to the terminal.

[0532] The device will display the received alert on the screen and provide a voice prompt to warn you.

[0533] Users acknowledge the alert, slow down, and watch out for pedestrians.

[0534] Response in the event of an accident

[0535] The device comes into contact with an obstacle and the impact sensor is activated.

[0536] The device automatically records and saves footage before and after the accident.

[0537] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal procedures as needed.

[0538] In this way, each component works together to create a system that ensures the safety of the driver.

[0539] The processing flow will be explained below.

[0540] Step 1:

[0541] The user attaches the smartphone to the electric scooter and launches the corresponding application.

[0542] Step 2:

[0543] The device will activate the camera and begin capturing video in real time, while simultaneously obtaining speed information from the GPS sensor and accelerometer.

[0544] Step 3:

[0545] The device sends the captured video data and speed information to the server at regular intervals, which are usually set in units of a few seconds.

[0546] Step 4:

[0547] The server receives the video data and speed information sent from the terminal and stores them in temporary storage.

[0548] Step 5:

[0549] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0550] Step 6:

[0551] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal that includes the type of risk (vehicle, person, object) and the risk level (high, medium, low).

[0552] Step 7:

[0553] The server sends the generated alert signal to the terminal.

[0554] Step 8:

[0555] The terminal receives an alert signal from the server and issues a warning to the driver using a display device or speaker, such as a visual display, sound, or vibration.

[0556] Step 9:

[0557] Users can check the alerts from their devices and take evasive action by slowing down if necessary.

[0558] Step 10:

[0559] If the shock sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[0560] Step 11:

[0561] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[0562] Step 12:

[0563] The terminal transmits the recorded video data to the server.

[0564] Step 13:

[0565] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[0566] Step 14:

[0567] In the event of an accident, users can use the accident reporting feature within the application to enter more detailed information.

[0568] Example 1

[0569] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0570] Previous technology had the problem of being unable to respond immediately to situations that required the driver's attention. Furthermore, when an accident occurred, video data serving as evidence was not acquired, making it impossible to quickly provide the information necessary for insurance companies and legal procedures. This resulted in issues such as insufficient driver safety and complicated, time-consuming post-accident procedures.

[0571] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0572] In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording and saving the recorded data when an accident occurs, a speed acquisition means for acquiring speed information from the driving equipment and sending it to the analysis means, and a risk assessment means for analyzing the received video data and speed information using an AI model and assessing the risk of contact. This makes it possible to quickly assess risks while driving and notify the driver, and automatically save important video data when an accident occurs.

[0573] "Driving device" refers to an electric scooter, bicycle, or other personal transportation device.

[0574] "Video acquisition means" refers to a camera or mobile device that is fixed to the driving equipment and captures the surrounding situation in real time.

[0575] "Analysis means" refers to a device or AI model that analyzes the video data acquired by the video acquisition means and performs object recognition and distance measurement.

[0576] "Alert means" refers to a device that issues a visual, audio, or vibration warning to the driver based on data obtained from the analysis means.

[0577] "Storage means" refers to a device or cloud storage that automatically starts recording when an accident occurs and saves the captured video data.

[0578] "Speed ​​acquisition means" refers to a sensor or device for acquiring speed information of driving equipment and transmitting it to analysis means.

[0579] "Risk assessment means" refers to a device or algorithm that assesses the risk of contact based on analyzed video data and speed information.

[0580] An "AI model" refers to an algorithm that uses machine learning and deep learning techniques to analyze image and video data and recognize specific patterns and objects.

[0581] "Real-time" refers to data being acquired and processed and analyzed with very little delay.

[0582] "Automatic recording" refers to a function that automatically starts recording when a sensor or other device detects certain conditions.

[0583] The present invention provides a system for improving safety in driving equipment such as electric kick scooters. The system includes an image acquisition unit, an analysis unit, an alert unit, a storage unit, a speed acquisition unit, and a risk assessment unit.

[0584] Video acquisition

[0585] The user attaches a mobile terminal (such as a smartphone) to the driving device and launches a dedicated application.

[0586] The device uses a camera to capture real-time images of the surrounding area, and collects location information from a GPS sensor and speed information from an acceleration sensor. This data is then sent to a server at regular intervals.

[0587] Data analysis

[0588] The server receives the video data and speed information transmitted from the terminal.

[0589] The server analyzes the video data using an AI model. Specifically, it calculates the distance between the vehicle and the person, and the distance between the object and the vehicle, and identifies the object. This can be done using deep learning libraries such as TensorFlow.

[0590] Risk assessment and alert notifications

[0591] The server evaluates the risk of contact based on the analysis results, and if it determines that the risk is high, it generates an alert signal and sends it to the device.

[0592] The device will then display a warning to the user based on the received alert signal, which can be visual (display on the smartphone screen), audible (audio alert), or tactile (vibration).

[0593] Incident detection and data storage

[0594] If the impact sensor detects a certain level of force, the device automatically switches to recording mode, recording video data before and after the accident and sending it to a server.

[0595] The server then stores the received recording data in cloud storage, which can be managed for insurance companies or legal procedures. Cloud storage such as Amazon S3 can be used.

[0596] Specific examples

[0597] System startup and start-up

[0598] The user attaches a smartphone to the electric scooter and launches the application, which causes the entire system to transition from standby to operational mode.

[0599] The device will be adjusted so that the camera is facing forward and will display a "Ready" message after launching the application.

[0600] Alert generation and avoidance actions

[0601] After the device sends the video footage, the server performs real-time analysis and generates an alert a few seconds later saying, "There is a pedestrian ahead."

[0602] The device will display a red warning on the smartphone screen and simultaneously play an audio alert saying "Be careful."

[0603] In response to this alert, the user slows down and checks the situation ahead.

[0604] Response in the event of an accident

[0605] The device goes into recording mode the moment the impact sensor detects contact with the kick scooter.

[0606] The device records a few seconds of data, including the immediately preceding footage, and automatically sends it to the server.

[0607] The server immediately uploads this data to cloud storage and provides it to insurance companies if required.

[0608] Prompt Sentence Examples

[0609] "I would like to develop a safety support system for electric scooters. I would like it to monitor the surroundings in real time while driving and send an alert to the driver if there is a high risk of collision. In the event of an accident, I would like it to automatically record and save video data before and after the accident. What kind of hardware and software would be required?"

[0610] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0611] Step 1: System startup

[0612] The user attaches a smartphone to the electric scooter and launches a dedicated application. The input is the attachment of the smartphone and the launch of the application, and the output is the entire system transitioning to an operational state.

[0613] The device initializes the camera, GPS sensor, and accelerometer, and notifies the user that it is ready. Specifically, the device starts collecting data from each sensor and displays a "Ready" message on the application screen.

[0614] Step 2: Data Acquisition

[0615] The device starts capturing real-time images of its surroundings using its camera, with the input being the surrounding images and data from the GPS and accelerometer, and the output being packetized versions of these data.

[0616] The device periodically transmits the collected video data, location information, and speed information to a server. Specifically, the device periodically streams the data, which is then sent to the server via a high-speed internet connection.

[0617] Step 3: Video analysis

[0618] The server receives the video data and speed information sent from the device. The input is the data packets from the device, and the output is the analysis results from the AI ​​model.

[0619] The server analyzes the received data using an AI model. Specifically, it calculates the distance between the vehicle and the person, and the distance between the object and the vehicle, and then performs object identification. Specifically, the AI ​​model analyzes each frame of video and recognizes specific objects (pedestrians, vehicles, fixed objects, etc.).

[0620] Step 4: Risk assessment

[0621] The server evaluates the risk of contact based on the results of video analysis and speed information. The input is the analysis results of the AI ​​model and speed information, and the output is the risk assessment result.

[0622] If the server determines that the risk of contact is high, it generates an alert signal. Specifically, the server uses a scoring algorithm to evaluate the risk level and generates an alert signal if the risk exceeds a certain threshold.

[0623] Step 5: Alert Notification

[0624] The server sends the generated alert signal to the terminal. The input is the result of the risk assessment and the output is the alert signal.

[0625] The device will then provide a visual, audio, or vibration warning to the user based on the received alert signal, by displaying a warning message on the smartphone screen, playing an audio alert, and activating the vibration motor if necessary.

[0626] Step 6: Incident detection and data storage

[0627] The device automatically switches to recording mode when the impact sensor detects a certain amount of force. The input is a signal from the impact sensor, and the output is a command to start recording and save the data.

[0628] The device sends the recorded data to a server, which then stores it in cloud storage. Specifically, the device records the recorded data, including the most recent footage, and immediately sends it to the server. The server then uploads the received data to cloud storage, making it available to insurance companies and legal proceedings.

[0629] The above are the specific steps of the program processing of this system.

[0630] (Application example 1)

[0631] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0632] Traffic accidents are often caused by drivers' lack of attention or awareness of their surroundings. Therefore, there is a need for a system that can monitor the surrounding situation in real time while driving and issue alerts at the appropriate time. However, current technology makes it difficult to achieve this efficiently and accurately. In addition, recording and storing data in the event of an accident is also an important issue in ensuring driver safety.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0634] In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording when an accident occurs and saving the recorded data, and a means for evaluating the risk of contact in real time and sending a warning to the automated driving system. This makes it possible to quickly evaluate the risk of contact while driving, issue an appropriate warning, and reliably save data when an accident occurs.

[0635] "Driving equipment" refers to any device used to assist in driving a vehicle.

[0636] "Image acquisition means" refers to a device that collects visual information using a camera, sensor, etc.

[0637] "Analysis means" refers to software or hardware for analyzing acquired data and extracting specific information.

[0638] "Alert means" refers to a device or function for issuing a warning to the driver or system based on the analysis results.

[0639] "Storage means" refers to a device or system that stores acquired data and makes it available for later use.

[0640] "Contact risk" refers to the possibility of colliding with other objects or people while driving.

[0641] "Real-time" refers to data acquisition and processing occurring immediately, without delay.

[0642] An "autonomous driving system" refers to an integrated system that enables a vehicle to drive autonomously without driver intervention.

[0643] System Overview

[0644] This invention is a system that combines a series of means installed in driving equipment to improve safety. Specifically, it is composed of video acquisition means, analysis means, alert means, and storage means. This makes it possible to evaluate the risk of contact while driving in real time, issue necessary warnings, and record and store data when an accident occurs.

[0645] Hardware and software used

[0646] 1. Hardware

[0647] Cameras: These are mounted on driving equipment for safety monitoring, with 360-degree cameras being particularly recommended.

[0648] AI-dedicated chip: Used for high-speed analysis of video data and sensor data.

[0649] Speed ​​sensor: Monitors the vehicle's speed.

[0650] Distance sensor: Measures the distance to surrounding objects and people.

[0651] Impact sensor: Detects collisions and impacts to detect the occurrence of an accident.

[0652] 2. Software

[0653] AI analysis module: Includes an AI model that uses TensorFlow / Keras, etc. to analyze the distance between vehicles, the distance between people, and the distance between objects.

[0654] Image Processing Module: Acquires and pre-processes camera images using OpenCV.

[0655] Data storage module: Has the function of uploading video and sensor data to cloud storage in the event of an accident.

[0656] Alert module: Provides visual, audio, or vibration warnings to the driver.

[0657] Natural language explanation of program processing

[0658] Server Processing

[0659] The server receives video and sensor data sent from the device and analyzes this data in real time using an AI analysis module. Based on the analysis results, if there is a high risk of contact, an alert signal is generated and sent to the device. Furthermore, if an accident occurs, video data before and after the accident is automatically saved to cloud storage.

[0660] Terminal handling

[0661] The device collects data from cameras and sensors attached to the driving equipment and sends it to a server. When it receives an alert signal from the server, it warns the driver visually, audibly, or by vibrating. When the impact sensor detects a force greater than a certain level, it switches to accident mode, starts recording, and sends the data to the server.

[0662] User operations

[0663] The user mounts their smartphone on the driving device and launches the application. The application provides an interface for acquiring camera and sensor data and sending it to the server. If necessary, the user can acknowledge alerts and take appropriate evasive action.

[0664] Specific examples

[0665] Risk detection while driving: The server detects approaching pedestrians ahead and sends a warning to the device when the distance is less than 1.5 meters. The device then issues a visual and audio alert to remind the user to be careful.

[0666] Accident data storage: When the device detects a collision, it automatically stores video data before and after the accident in cloud storage, which can be used for later review.

[0667] Prompt Sentence Examples

[0668] "Creating an AI model that assesses collision risk and provides necessary warnings to drivers"

[0669] "Development of a system for monitoring the surroundings of autonomous vehicles in real time"

[0670] "A program that automatically assesses the risk of an accident based on the analysis results and sends a warning to the autonomous driving system."

[0671] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0672] Step 1:

[0673] The user attaches the smartphone to the driving device and launches the application. The input is the smartphone and the application, and the output is the data collection status from the camera and sensors. Specifically, the user taps to launch the application and confirms that the smartphone is attached.

[0674] Step 2:

[0675] The device starts the camera and captures the surrounding video in real time. The input is the camera and the output is the captured time-series video data. Specifically, the application calls the camera module and starts streaming the video.

[0676] Step 3:

[0677] The device collects input data from the speed sensor and distance sensor. The input is the speed sensor and distance sensor, and the output is the current speed data and distance data to surrounding objects. Specifically, the device reads data from the sensor terminal and processes it within the application.

[0678] Step 4:

[0679] The device transmits the captured video and sensor data to the server at regular intervals. The input is the video and sensor data, and the output is the data transmitted to the server. Specifically, the device divides the data into packets and uploads them to the server via an internet connection.

[0680] Step 5:

[0681] The server analyzes the received video data and sensor data using an AI model. The input is the received video data and sensor data, and the output is risk assessment data as the analysis result. Specifically, the server runs an AI analysis module and calculates the distance between vehicles, people, and objects from the data.

[0682] Step 6:

[0683] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and sends it to the terminal.The input is risk assessment data and the output is an alert signal.Specifically, if the risk assessment exceeds a certain threshold, the server calls the alert generation module and sends an alert signal to the terminal.

[0684] Step 7:

[0685] The terminal issues a warning to the driver based on the alert signal received from the server. The input is the alert signal, and the output is a visual, audio, or vibration warning. Specifically, the terminal activates the warning module and issues a warning using the screen display, speaker, or vibration function.

[0686] Step 8:

[0687] If the impact sensor detects a force greater than a certain level, the device switches to accident mode and starts automatic recording. The input is the impact sensor data, and the output is the recorded video data. Specifically, the device activates the accident detection module and starts recording video.

[0688] Step 9:

[0689] The device sends the recorded video data to the server, which then stores it in cloud storage. The input is the recorded video data, and the output is the data stored in cloud storage. Specifically, the device sends the video data to the server via a transmission module, and the server uploads the received data to the cloud. This stored data can be used later for insurance companies or legal procedures if necessary.

[0690] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0691] System Overview

[0692] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, analysis unit, alert unit, storage unit, and an emotion engine that recognizes the user's emotions. This system monitors the surrounding environment in real time while driving, alerts the driver if there is a high risk of contact, and automatically records and stores video data before and after an accident if one occurs. The system also recognizes the user's emotional state and adjusts the intensity and frequency of alerts accordingly to provide appropriate warnings.

[0693] System configuration

[0694] 1. Video acquisition method

[0695] The device uses a smartphone or dedicated camera attached to the driving device to capture real-time images of the surroundings, and this image data is sent to an analysis means.

[0696] 2. Analysis method

[0697] The server receives the video data sent from the device and analyzes it using an AI model. Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and objects, and assesses the risk of collision based on speed information.

[0698] 3. Alert Methods

[0699] If the server determines that there is a high risk of contact based on the analysis results, it sends an alert signal to the terminal.

[0700] The device will then issue a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, etc.

[0701] 4. Preservation means

[0702] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[0703] The server stores the recorded video data in cloud storage, making it available as evidence to insurance companies and in legal proceedings.

[0704] 5. Emotion Engine

[0705] The device includes facial recognition technology to analyze the user's facial expressions and assess the driver's emotional state.

[0706] The terminal includes voice recognition technology to analyze the user's voice input and assess their emotional state.

[0707] The server provides a means to adjust the intensity and frequency of alerts based on the user's emotional state.

[0708] A natural language description of the program's operation

[0709] Startup and preparation

[0710] The user attaches the smartphone to the electric kick scooter and launches the application.

[0711] The device will activate the camera and begin capturing video, and will also obtain speed information from the GPS sensor and accelerometer.

[0712] The device initializes the emotion engine and captures the user's face and voice data in real time.

[0713] Data transmission and analysis

[0714] The device transmits the captured video, speed information, and the user's face and voice data to the server at regular intervals.

[0715] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0716] The server analyzes the user's emotional data and assesses their state of stress and attention while driving.

[0717] Alert generation and notification

[0718] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and adjusts the strength and frequency of the alert based on the user's emotional state.

[0719] The device will then issue a warning to the driver based on the received alert signal, which can be visual, audible, or vibration.

[0720] Accident response and data preservation

[0721] If the shock sensor detects a force greater than a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[0722] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[0723] The terminal transmits the recorded video data to the server.

[0724] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[0725] Specific examples

[0726] System startup and start-up

[0727] The user attaches their smartphone to the electric kick scooter and launches the application.

[0728] The device will activate its camera and begin taking video, measuring speed, and capturing emotional data.

[0729] Alert generation and avoidance actions

[0730] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[0731] The server determines that the risk of contact is high and sets an appropriate alert strength based on the user's emotional data. For example, if the user is in a high stress state, it selects a moderately strong voice alert.

[0732] The server generates an alert signal and sends it to the terminal.

[0733] The device will display the received alert on the screen and provide a voice prompt to warn you.

[0734] Users acknowledge the alert, slow down, and watch out for pedestrians.

[0735] Response in the event of an accident

[0736] The device comes into contact with an obstacle and the impact sensor is activated.

[0737] The device will automatically start recording and record footage for a certain period of time before and after the accident.

[0738] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal procedures as needed.

[0739] In this way, each component works together to ensure the safety of the driver. The addition of an emotion engine also enables the system to respond flexibly to the user's condition.

[0740] The processing flow will be explained below.

[0741] Step 1:

[0742] The user attaches the smartphone to the electric scooter and launches the application.

[0743] Step 2:

[0744] The device will activate the camera and begin capturing video in real time, while simultaneously obtaining speed information from the GPS sensor and accelerometer.

[0745] Step 3:

[0746] The device initializes the emotion engine, captures the user's face and voice data in real time, and uses facial and voice recognition technologies to analyze the user's emotional state from their facial expressions and words.

[0747] Step 4:

[0748] The device transmits the captured video data, speed information, and user emotion data to the server at regular intervals, usually set to a few seconds.

[0749] Step 5:

[0750] The server receives the video data, speed information, and emotion data sent from the device and stores them in temporary storage.

[0751] Step 6:

[0752] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0753] Step 7:

[0754] The server evaluates the user's emotional state analyzed by the emotion engine and adjusts the intensity and frequency of alerts based on whether the user is in an emotional state such as stress or distraction.

[0755] Step 8:

[0756] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal that includes the type of risk (vehicle, person, object) and the risk level (high, medium, low) according to the emotional state.

[0757] Step 9:

[0758] The server sends the generated alert signal to the terminal.

[0759] Step 10:

[0760] The terminal receives an alert signal from the server and issues a warning to the driver using a display device or speaker. The alert is notified by visual display, sound, vibration, etc. The warning is made stronger especially when the user is in a high stress state.

[0761] Step 11:

[0762] Users can check the alerts from their devices and take evasive action by slowing down if necessary.

[0763] Step 12:

[0764] If the shock sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[0765] Step 13:

[0766] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[0767] Step 14:

[0768] The terminal transmits the recorded video data to the server.

[0769] Step 15:

[0770] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[0771] Step 16:

[0772] In the event of an accident, users can use the accident reporting feature within the application to enter more detailed information.

[0773] In this way, each step works smoothly together to ensure the safety of the driver. The addition of an emotion engine enables the system to respond flexibly according to the user's psychological state.

[0774] Example 2

[0775] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] Conventional electric scooter safety systems simply assess physical risks without considering the driver's situational awareness or stress level. As a result, they have been unable to adequately address the risk of accidents caused by high stress levels or reduced attention. Furthermore, in the event of an accident, automation of data storage is insufficient, which can lead to a lack of video evidence.

[0777] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording when an accident occurs and saving the recorded data, and an emotion recognition means for recognizing the user's emotional state and adjusting the intensity and frequency of the alert. This enables flexible warnings according to the user's emotional state and reliable data saving when an accident occurs.

[0778] "Image acquisition means" refers to a device or apparatus that is fixed to the driving equipment and that captures and acquires images of the surrounding area.

[0779] "Analysis means" refers to equipment or software that analyzes acquired video data and other sensor data to perform risk assessments, distance measurement, etc.

[0780] The "alert means" is a device or notification system for issuing a warning to the driver based on the analysis results.

[0781] "Storage means" refers to a system or device that automatically starts recording when an accident occurs and stores the recorded data.

[0782] "Emotion recognition means" refers to technology or a system that recognizes the user's emotional state and adjusts the intensity and frequency of alerts according to that state.

[0783] A "generative AI model" is an algorithm or framework that uses machine learning or deep learning to perform a specific task (e.g., distance measurement or risk assessment).

[0784] "Driving equipment" refers to means of transportation such as electric kick scooters, and in this case refers to equipment equipped with various sensors and systems to improve safety.

[0785] MODE FOR CARRYING OUT THE INVENTION

[0786] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, an analysis unit, an alert unit, a storage unit, and an emotion recognition unit that recognizes the user's emotions. This system monitors the surroundings in real time while driving, and issues an alert to the driver if there is a high risk of contact. In the event of an accident, the system automatically records and stores video data before and after the accident. The system also recognizes the user's emotional state and adjusts the intensity and frequency of alerts accordingly to provide appropriate warnings.

[0787] System configuration

[0788] 1. Video acquisition method

[0789] The terminal uses a mobile device (such as a smartphone or dedicated camera) attached to the driving equipment to capture images of the surroundings in real time. This image data is sent to the analysis means.

[0790] 2. Analysis method

[0791] The server receives the video data sent from the device and analyzes it using a generative AI model (e.g., TensorFlow or PyTorch). Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and other objects, and assesses the risk of contact based on speed information.

[0792] 3. Alert Methods

[0793] If the server determines that the risk of contact is high based on the analysis results, it sends an alert signal to the terminal.

[0794] The device issues a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, or the like.

[0795] 4. Preservation means

[0796] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[0797] The server stores the recorded video data in cloud storage (e.g., AWS S3 or Google Cloud Storage) and makes it available to insurance companies and as evidence in legal proceedings.

[0798] 5. Emotion recognition means

[0799] The terminal includes facial recognition technology to perform facial expression analysis of the user and assess the emotional state of the driver.

[0800] The terminal includes voice recognition technology to analyze the user's voice input and assess their emotional state.

[0801] The server provides a means to adjust the intensity and frequency of alerts based on the user's emotional state.

[0802] Specific examples

[0803] System startup and start-up

[0804] The user attaches their smartphone to the electric kick scooter and launches the application.

[0805] The device will activate its camera and begin taking video, measuring speed, and capturing emotional data.

[0806] Alert generation and avoidance actions

[0807] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[0808] The server determines that the risk of contact is high and sets an appropriate alert strength based on the user's emotional data. For example, if the user is in a high stress state, a moderately strong voice alert is selected.

[0809] The server generates and sends an alert signal to the terminal.

[0810] The device will display the received alert on the screen and give an audio warning.

[0811] The user acknowledges the alert, slows down, and watches out for pedestrians.

[0812] Response in the event of an accident

[0813] The device comes into contact with an obstacle and the impact sensor is activated.

[0814] The device will automatically start recording and record footage for a certain period of time before and after the accident.

[0815] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal proceedings as needed.

[0816] Prompt Sentence Examples

[0817] Here are some example prompts to explain system behavior to a generative AI model:

[0818] "Please explain how the safety system for electric scooters works. Please use a concrete scenario to explain step by step the roles of the user, device, and server, as well as the system's functions."

[0819] In this way, a system that realizes safe driving is constructed by using specific hardware and software to process and calculate data.

[0820] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0821] System program processing flow

[0822] Step 1: Boot the system

[0823] The user attaches the smartphone to the electric kick scooter and launches the application.

[0824] Input: Electric scooter, smartphone, application startup operation

[0825] Output: System booting, initial setup complete

[0826] Specific operation: Place the device in the dedicated smartphone holder and tap the app icon to launch it.

[0827] Step 2: Initialize the video and sensors

[0828] The device will start the camera and begin recording video, and will also initialize the GPS sensor and acceleration sensor to begin acquiring location and speed information.

[0829] Input: Smartphone, camera, GPS sensor, accelerometer

[0830] Output: Real-time video data, position data, speed data

[0831] Specific behavior: Launches the camera app in the background and calls the sensor API to start the data stream.

[0832] Step 3: Activating the Emotion Engine

[0833] The device initializes the emotion engine and begins capturing the user's face and voice data in real time.

[0834] Input: Smartphone, face recognition library, voice recognition API

[0835] Output: Real-time emotion data

[0836] Specific behavior: Launches the face recognition library and speech recognition API, preparing for data capture.

[0837] Step 4: Send data

[0838] The terminal transmits the captured video, speed information, and the user's face and voice data to the server at regular intervals (for example, once per second).

[0839] Input: Real-time video data, location data, speed data, emotion data

[0840] Output: Data sent to the server

[0841] What it does: Sends data to the server using an HTTP POST request.

[0842] Step 5: Video Analysis

[0843] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0844] Input: Video data, position data, speed data

[0845] Output: Analysis results (distance data, risk assessment)

[0846] What it does: Identify objects in images and measure distances using computer vision libraries (e.g., OpenCV) and deep learning models (e.g., TensorFlow).

[0847] Step 6: Sentiment Analysis

[0848] The server analyzes the user's emotional data and evaluates their state of stress and attention while driving.

[0849] Input: Emotion data (face, voice)

[0850] Output: Analysis results (stress level, attention evaluation)

[0851] Specific operations: Facial features are analyzed using an expression analysis model, and vocal tone and speed are measured using an audio analysis model.

[0852] Step 7: Alert Generation

[0853] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and adjusts the strength and frequency of the alert based on the user's emotional state.

[0854] Input: Risk assessment results, emotion assessment results

[0855] Output: Alert signal (strength, frequency)

[0856] What it does: It quantifies the risk level by setting standard deviations and thresholds, then runs an algorithm that determines the type and intensity of the alert (visual, audio, vibration, etc.).

[0857] Step 8: Alert Notifications

[0858] The device issues a warning to the driver based on the received alert signal, notifying the driver of the alert through visual display, audio, vibration, or other means.

[0859] Input: Alert signal

[0860] Output: Warning to the driver (visual, audio, vibration)

[0861] Specific actions: Displays a warning message on the smartphone display, outputs an audio alert through the speaker, and activates the vibration motor.

[0862] Step 9: Detecting an incident

[0863] If the shock sensor detects a force greater than a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[0864] Input: Impact sensor data

[0865] Output: Switch to accident mode

[0866] Specific operation: Analyzes accelerometer data and sets a flag if a sudden change is detected.

[0867] Step 10: Record footage of the incident

[0868] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage of the time before and after the accident.

[0869] Input: Accident mode, camera video data

[0870] Output: Recording data

[0871] What it does: Uses the video recording API to save the video stream to a fixed disk or cloud storage.

[0872] Step 11: Data transmission and storage

[0873] The terminal transmits the recorded video data to the server.

[0874] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal procedures if necessary.

[0875] Input: Recording data

[0876] Output: Cloud saved data

[0877] Specific operation: Data is stored using a database management system (e.g., AWS S3 or Google Cloud Storage) while setting access permissions.

[0878] In this way, each step works together to provide a system that comprehensively supports driver safety.

[0879] (Application example 2)

[0880] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0881] While driving an autonomous vehicle, multiple sensors and analytical means are required to properly monitor the surrounding environment and the driver's state and ensure safe driving. However, these means operating individually are not sufficient. In particular, it is difficult to recognize the driver's emotional state in real time and issue appropriate warnings or alerts. Furthermore, it is necessary to respond quickly and accurately in the event of an accident. Therefore, this invention provides a system that recognizes the driver's emotional state and issues appropriate alerts to improve the safety of autonomous vehicles.

[0882] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0883] In this invention, the server includes a video capture means fixed to the driving equipment, an analysis means for analyzing the video data captured by the video capture means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, an emotion engine for recognizing the emotional state of the driver and adjusting the intensity and frequency of the alert, and a storage means for automatically recording when an accident occurs and saving the recorded data. This enables real-time monitoring while driving and flexible response according to the emotional state of the driver.

[0884] The "image acquisition means" is a device that is fixed to the driving equipment and captures the surrounding situation in real time.

[0885] The "analysis means" is a device or software for analyzing the video data acquired by the video acquisition means and calculating distance, speed, etc.

[0886] The "alert means" is a mechanism or system that issues a visual, audible, or tactile warning to the driver based on the information obtained by the analysis means.

[0887] An "emotion engine" is a mechanism or software that analyzes the driver's facial expressions and voice to recognize and evaluate their emotional state.

[0888] "Storage means" refers to a device or system that automatically starts recording when an accident occurs and records and stores the video data.

[0889] An "AI model" refers to a learning algorithm or computational model that uses artificial intelligence to analyze the distance and speed of objects from video data.

[0890] MODE FOR CARRYING OUT THE INVENTION

[0891] System Overview

[0892] The present invention provides a system for improving the safety of autonomous vehicles. The system includes a plurality of image acquisition means, analysis means, alert means, emotion engine, and storage means, all of which are fixed to a driving device. Specific embodiments of each means are described below.

[0893] Video acquisition method

[0894] The device uses multiple cameras to capture images of the surroundings in real time. This allows image data to be acquired from various viewpoints, such as the front and sides of the vehicle. This image data is then sent to subsequent analysis means. The image acquisition means includes high-resolution cameras and software control for these cameras.

[0895] Analysis means

[0896] The server receives the video data sent from the device and analyzes it using a generative AI model. Specifically, it calculates the distance between the vehicle and the pedestrian, and the distance between the vehicle and the object, and evaluates the risk of contact based on speed information. This analysis uses machine learning frameworks such as TensorFlow and Keras. It also uses dlib as facial recognition technology to evaluate the driver's emotional state.

[0897] Alert Method

[0898] If the server determines that there is a high risk of collision based on the analysis results, it generates an alert signal and sends it to the device. Based on the received alert signal, the device warns the driver by means of visual display, sound, vibration, etc. For example, a text-to-speech engine (TTS engine) is used for the sound alert.

[0899] Emotion Engine

[0900] The device uses a camera and microphone to capture the driver's face and voice data. This data is analyzed using an emotion recognition model to evaluate the driver's emotional state. A generative AI model adjusts the intensity and frequency of alerts according to the driver's stress and attention levels. This allows for flexible responses tailored to the driver's psychological state.

[0901] Preservation means

[0902] When the device detects a force exceeding a certain level using impact and acceleration sensors, it determines that an accident has occurred and automatically starts recording. This recorded data is stored in cloud storage via a server and can be used by insurance companies or in legal proceedings as needed. Since recording requires high-speed data transfer and large-capacity storage, a cloud service is used.

[0903] Specific examples

[0904] To activate the system, the user attaches their smartphone to the dashboard of the autonomous vehicle and launches the application. The device's camera captures video in real time and sends it to the server. The server analyzes the video data, detects the presence of pedestrians ahead, and issues an alert to the driver. Furthermore, if the driver is under high stress, a stronger audio alert is issued. In the event of an accident, the recorded data is stored in the cloud and necessary measures are taken.

[0905] Prompt Sentence Examples

[0906] "We will design an application that monitors the driver's emotional state in real time when an object (pedestrian) that poses a risk of contact suddenly appears within one meter ahead, and issues an appropriate alert to enable safe driving."

[0907] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0908] Step 1: The device activates the camera and captures video in real time.

[0909] Input: Real-time video data from the camera.

[0910] Output: The video data stream of the device.

[0911] Specific operation: A camera fixed to the terminal continuously captures video, and this video data is sent to subsequent processing steps.

[0912] Step 2: The terminal transmits the video data to the server.

[0913] Input: Real-time video data captured by the device.

[0914] Output: Video data stream to server.

[0915] Specific operation: The device transmits the acquired video data to the server in real time via wireless communication, etc. This communication uses networks such as Wi-Fi or LTE.

[0916] Step 3: The server performs data pre-processing on the received video data.

[0917] Input: Video data sent from the device.

[0918] Output: Pre-processed video data.

[0919] Specific operation: The server performs preprocessing such as sharpening, noise reduction, and resolution adjustment on the video data, thereby improving the accuracy of analysis.

[0920] Step 4: The server analyzes the preprocessed video data with the generative AI model.

[0921] Input: Preprocessed video data.

[0922] Output: Position and distance data of objects and people.

[0923] How it works: The server's generated AI model calculates and extracts information about the distance between vehicles and pedestrians and objects in the video. It also analyzes speed information.

[0924] Step 5: The server captures the driver's face and voice data and sends it to the emotion engine.

[0925] Input: Driver's face and voice data.

[0926] Output: Driver's emotional state data.

[0927] Specific operation: The device uses a camera and microphone to continuously capture the driver's facial expressions and voice, which are then analyzed by the emotion engine. The analysis results are sent to the server.

[0928] Step 6: The server evaluates the risk of contact based on the analysis results.

[0929] Input: Object distance data and driver emotional state data.

[0930] Output: Contact risk assessment data.

[0931] Specific operation: The server combines the distance data obtained by the generative AI model with the driver's emotional state to calculate the risk of collision. If the risk is high, it generates an alert signal.

[0932] Step 7: The server sends an alert signal to the terminal.

[0933] Input: Contact risk assessment data.

[0934] Output: Alert signal data packet.

[0935] Specific operation: If the server detects a high risk of collision, it generates an alert signal and sends it to the terminal to warn the driver visually, audibly, or by vibration.

[0936] Step 8: The terminal warns the driver based on the received alert signal.

[0937] Input: The alert signal sent from the server.

[0938] Output: Visual, audible, or tactile warning to the driver.

[0939] Specific actions: The device visually displays a warning message on the display, audibly plays an alert sound through the built-in speaker, and haptically activates the vibration motor.

[0940] Step 9: The device monitors sensor information such as impacts and sudden stops, and automatically starts recording when an accident occurs.

[0941] Input: Impact sensor and accelerometer data.

[0942] Output: Recorded data of the accident video.

[0943] Specific operation: The device constantly monitors information from the impact sensor and acceleration sensor, and automatically starts recording when it detects a force above a certain level. It records and saves footage of the accident before and after it occurs.

[0944] Step 10: The device uploads the recording data to cloud storage.

[0945] Input: Recording data.

[0946] Output: Saved data in cloud storage.

[0947] Specific operation: After the device has completed recording, it uploads the recorded data to cloud storage and manages it so that it can be used by insurance companies or for legal procedures if necessary.

[0948] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0949] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0950] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0951] [Third embodiment]

[0952] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0953] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0954] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0955] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0956] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0957] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0958] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0959] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0960] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0961] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0962] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0963] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0964] System Overview

[0965] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, an analysis unit, an alert unit, and a storage unit attached to the driving device. This system monitors the surrounding situation in real time while driving, alerts the driver if there is a high risk of contact, and automatically records and stores video data before and after an accident if one occurs.

[0966] System configuration

[0967] 1. Video acquisition method

[0968] The device uses a smartphone or dedicated camera attached to the driving device to capture real-time images of the surroundings, and this image data is sent to an analysis means.

[0969] 2. Analysis method

[0970] The server receives the video data sent from the device and analyzes it using an AI model. Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and objects, and evaluates the risk of collision based on speed information.

[0971] 3. Alert Methods

[0972] If the server determines that there is a high risk of contact based on the analysis results, it sends an alert signal to the terminal.

[0973] The device will then issue a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, etc.

[0974] 4. Preservation means

[0975] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[0976] The server stores the recorded video data in cloud storage, making it available as evidence to insurance companies and in legal proceedings.

[0977] A natural language description of the program's operation

[0978] Startup and preparation

[0979] The user attaches the smartphone to the electric kick scooter and launches the application.

[0980] The device will activate the camera and begin capturing video, and will also obtain speed information from the GPS sensor and accelerometer.

[0981] Data transmission and analysis

[0982] The device transmits the captured video and speed information to the server at regular intervals.

[0983] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[0984] Alert generation and notification

[0985] If the server determines from the analysis results that there is a high risk of contact, it generates an alert signal and sends it to the terminal.

[0986] The device receives the alert signal and warns the driver visually, audibly, or by vibration, allowing the driver to acknowledge the alert and take necessary evasive action.

[0987] Accident response and data preservation

[0988] If the impact sensor detects a force above a certain level, the device will automatically switch to accident mode and begin recording.

[0989] The device sends the recorded data to a server, which then stores it in cloud storage, where it can be used for accident response and legal procedures.

[0990] Specific examples

[0991] System startup and start-up

[0992] The user attaches their smartphone to the electric kick scooter and launches the application.

[0993] The device will activate the camera and begin taking video and measuring speed.

[0994] Alert generation and avoidance actions

[0995] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[0996] The server determines that the risk of contact is high and generates an alert signal to send to the terminal.

[0997] The device will display the received alert on the screen and provide a voice prompt to warn you.

[0998] Users acknowledge the alert, slow down, and watch out for pedestrians.

[0999] Response in the event of an accident

[1000] The device comes into contact with an obstacle and the impact sensor is activated.

[1001] The device automatically records and saves footage before and after the accident.

[1002] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal procedures as needed.

[1003] In this way, each component works together to create a system that ensures the safety of the driver.

[1004] The processing flow will be explained below.

[1005] Step 1:

[1006] The user attaches the smartphone to the electric scooter and launches the corresponding application.

[1007] Step 2:

[1008] The device will activate the camera and begin capturing video in real time, while simultaneously obtaining speed information from the GPS sensor and accelerometer.

[1009] Step 3:

[1010] The device sends the captured video data and speed information to the server at regular intervals, which are usually set in units of a few seconds.

[1011] Step 4:

[1012] The server receives the video data and speed information sent from the terminal and stores them in temporary storage.

[1013] Step 5:

[1014] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1015] Step 6:

[1016] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal that includes the type of risk (vehicle, person, object) and the risk level (high, medium, low).

[1017] Step 7:

[1018] The server sends the generated alert signal to the terminal.

[1019] Step 8:

[1020] The terminal receives an alert signal from the server and issues a warning to the driver using a display device or speaker, such as a visual display, sound, or vibration.

[1021] Step 9:

[1022] Users can check the alerts from their devices and take evasive action by slowing down if necessary.

[1023] Step 10:

[1024] If the shock sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[1025] Step 11:

[1026] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[1027] Step 12:

[1028] The terminal transmits the recorded video data to the server.

[1029] Step 13:

[1030] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[1031] Step 14:

[1032] In the event of an accident, users can use the accident reporting feature within the application to enter more detailed information.

[1033] Example 1

[1034] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1035] Previous technology had the problem of being unable to respond immediately to situations that required the driver's attention. Furthermore, when an accident occurred, video data serving as evidence was not acquired, making it impossible to quickly provide the information necessary for insurance companies and legal procedures. This resulted in issues such as insufficient driver safety and complicated, time-consuming post-accident procedures.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1037] In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording and saving the recorded data when an accident occurs, a speed acquisition means for acquiring speed information from the driving equipment and sending it to the analysis means, and a risk assessment means for analyzing the received video data and speed information using an AI model and assessing the risk of contact. This makes it possible to quickly assess risks while driving and notify the driver, and automatically save important video data when an accident occurs.

[1038] "Driving device" refers to an electric scooter, bicycle, or other personal transportation device.

[1039] "Video acquisition means" refers to a camera or mobile device that is fixed to the driving equipment and captures the surrounding situation in real time.

[1040] "Analysis means" refers to a device or AI model that analyzes the video data acquired by the video acquisition means and performs object recognition and distance measurement.

[1041] "Alert means" refers to a device that issues a visual, audio, or vibration warning to the driver based on data obtained from the analysis means.

[1042] "Storage means" refers to a device or cloud storage that automatically starts recording when an accident occurs and saves the captured video data.

[1043] "Speed ​​acquisition means" refers to a sensor or device for acquiring speed information of driving equipment and transmitting it to analysis means.

[1044] "Risk assessment means" refers to a device or algorithm that assesses the risk of contact based on analyzed video data and speed information.

[1045] An "AI model" refers to an algorithm that uses machine learning and deep learning techniques to analyze image and video data and recognize specific patterns and objects.

[1046] "Real-time" refers to data being acquired and processed and analyzed with very little delay.

[1047] "Automatic recording" refers to a function that automatically starts recording when a sensor or other device detects certain conditions.

[1048] The present invention provides a system for improving safety in driving equipment such as electric kick scooters. The system includes an image acquisition unit, an analysis unit, an alert unit, a storage unit, a speed acquisition unit, and a risk assessment unit.

[1049] Video acquisition

[1050] The user attaches a mobile terminal (such as a smartphone) to the driving device and launches a dedicated application.

[1051] The device uses a camera to capture real-time images of the surrounding area, and collects location information from a GPS sensor and speed information from an acceleration sensor. This data is then sent to a server at regular intervals.

[1052] Data analysis

[1053] The server receives the video data and speed information transmitted from the terminal.

[1054] The server analyzes the video data using an AI model. Specifically, it calculates the distance between the vehicle and the person, and the distance between the object and the vehicle, and identifies the object. This can be done using deep learning libraries such as TensorFlow.

[1055] Risk assessment and alert notifications

[1056] The server evaluates the risk of contact based on the analysis results, and if it determines that the risk is high, it generates an alert signal and sends it to the device.

[1057] The device will then display a warning to the user based on the received alert signal, which can be visual (display on the smartphone screen), audible (audio alert), or tactile (vibration).

[1058] Incident detection and data storage

[1059] If the impact sensor detects a certain level of force, the device automatically switches to recording mode, recording video data before and after the accident and sending it to a server.

[1060] The server then stores the received recording data in cloud storage, which can be managed for insurance companies or legal procedures. Cloud storage such as Amazon S3 can be used.

[1061] Specific examples

[1062] System startup and start-up

[1063] The user attaches a smartphone to the electric scooter and launches the application, which causes the entire system to transition from standby to operational mode.

[1064] The device will be adjusted so that the camera is facing forward and will display a "Ready" message after launching the application.

[1065] Alert generation and avoidance actions

[1066] After the device sends the video footage, the server performs real-time analysis and generates an alert a few seconds later saying, "There is a pedestrian ahead."

[1067] The device will display a red warning on the smartphone screen and simultaneously play an audio alert saying "Be careful."

[1068] In response to this alert, the user slows down and checks the situation ahead.

[1069] Response in the event of an accident

[1070] The device goes into recording mode the moment the impact sensor detects contact with the kick scooter.

[1071] The device records a few seconds of data, including the immediately preceding footage, and automatically sends it to the server.

[1072] The server immediately uploads this data to cloud storage and provides it to insurance companies if required.

[1073] Prompt Sentence Examples

[1074] "I would like to develop a safety support system for electric scooters. I would like it to monitor the surroundings in real time while driving and send an alert to the driver if there is a high risk of collision. In the event of an accident, I would like it to automatically record and save video data before and after the accident. What kind of hardware and software would be required?"

[1075] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1076] Step 1: System startup

[1077] The user attaches a smartphone to the electric scooter and launches a dedicated application. The input is the attachment of the smartphone and the launch of the application, and the output is the entire system transitioning to an operational state.

[1078] The device initializes the camera, GPS sensor, and accelerometer, and notifies the user that it is ready. Specifically, the device starts collecting data from each sensor and displays a "Ready" message on the application screen.

[1079] Step 2: Data Acquisition

[1080] The device starts capturing real-time images of its surroundings using its camera, with the input being the surrounding images and data from the GPS and accelerometer, and the output being packetized versions of these data.

[1081] The device periodically transmits the collected video data, location information, and speed information to a server. Specifically, the device periodically streams the data, which is then sent to the server via a high-speed internet connection.

[1082] Step 3: Video analysis

[1083] The server receives the video data and speed information sent from the device. The input is the data packets from the device, and the output is the analysis results from the AI ​​model.

[1084] The server analyzes the received data using an AI model. Specifically, it calculates the distance between the vehicle and the person, and the distance between the object and the vehicle, and then performs object identification. Specifically, the AI ​​model analyzes each frame of video and recognizes specific objects (pedestrians, vehicles, fixed objects, etc.).

[1085] Step 4: Risk assessment

[1086] The server evaluates the risk of contact based on the results of video analysis and speed information. The input is the analysis results of the AI ​​model and speed information, and the output is the risk assessment result.

[1087] If the server determines that the risk of contact is high, it generates an alert signal. Specifically, the server uses a scoring algorithm to evaluate the risk level and generates an alert signal if the risk exceeds a certain threshold.

[1088] Step 5: Alert Notification

[1089] The server sends the generated alert signal to the terminal. The input is the result of the risk assessment and the output is the alert signal.

[1090] The device will then provide a visual, audio, or vibration warning to the user based on the received alert signal, by displaying a warning message on the smartphone screen, playing an audio alert, and activating the vibration motor if necessary.

[1091] Step 6: Incident detection and data storage

[1092] The device automatically switches to recording mode when the impact sensor detects a certain amount of force. The input is a signal from the impact sensor, and the output is a command to start recording and save the data.

[1093] The device sends the recorded data to a server, which then stores it in cloud storage. Specifically, the device records the recorded data, including the most recent footage, and immediately sends it to the server. The server then uploads the received data to cloud storage, making it available to insurance companies and legal proceedings.

[1094] The above are the specific steps of the program processing of this system.

[1095] (Application example 1)

[1096] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1097] Traffic accidents are often caused by drivers' lack of attention or awareness of their surroundings. Therefore, there is a need for a system that can monitor the surrounding situation in real time while driving and issue alerts at the appropriate time. However, current technology makes it difficult to achieve this efficiently and accurately. In addition, recording and storing data in the event of an accident is also an important issue in ensuring driver safety.

[1098] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1099] In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording when an accident occurs and saving the recorded data, and a means for evaluating the risk of contact in real time and sending a warning to the automated driving system. This makes it possible to quickly evaluate the risk of contact while driving, issue an appropriate warning, and reliably save data when an accident occurs.

[1100] "Driving equipment" refers to any device used to assist in driving a vehicle.

[1101] "Image acquisition means" refers to a device that collects visual information using a camera, sensor, etc.

[1102] "Analysis means" refers to software or hardware for analyzing acquired data and extracting specific information.

[1103] "Alert means" refers to a device or function for issuing a warning to the driver or system based on the analysis results.

[1104] "Storage means" refers to a device or system that stores acquired data and makes it available for later use.

[1105] "Contact risk" refers to the possibility of colliding with other objects or people while driving.

[1106] "Real-time" refers to data acquisition and processing occurring immediately, without delay.

[1107] An "autonomous driving system" refers to an integrated system that enables a vehicle to drive autonomously without driver intervention.

[1108] System Overview

[1109] This invention is a system that combines a series of means installed in driving equipment to improve safety. Specifically, it is composed of video acquisition means, analysis means, alert means, and storage means. This makes it possible to evaluate the risk of contact while driving in real time, issue necessary warnings, and record and store data when an accident occurs.

[1110] Hardware and software used

[1111] 1. Hardware

[1112] Cameras: These are mounted on driving equipment for safety monitoring, with 360-degree cameras being particularly recommended.

[1113] AI-dedicated chip: Used for high-speed analysis of video data and sensor data.

[1114] Speed ​​sensor: Monitors the vehicle's speed.

[1115] Distance sensor: Measures the distance to surrounding objects and people.

[1116] Impact sensor: Detects collisions and impacts to detect the occurrence of an accident.

[1117] 2. Software

[1118] AI analysis module: Includes an AI model that uses TensorFlow / Keras, etc. to analyze the distance between vehicles, the distance between people, and the distance between objects.

[1119] Image Processing Module: Acquires and pre-processes camera images using OpenCV.

[1120] Data storage module: Has the function of uploading video and sensor data to cloud storage in the event of an accident.

[1121] Alert module: Provides visual, audio, or vibration warnings to the driver.

[1122] Natural language explanation of program processing

[1123] Server Processing

[1124] The server receives video and sensor data sent from the device and analyzes this data in real time using an AI analysis module. Based on the analysis results, if there is a high risk of contact, an alert signal is generated and sent to the device. Furthermore, if an accident occurs, video data before and after the accident is automatically saved to cloud storage.

[1125] Terminal handling

[1126] The device collects data from cameras and sensors attached to the driving equipment and sends it to a server. When it receives an alert signal from the server, it warns the driver visually, audibly, or by vibrating. When the impact sensor detects a force greater than a certain level, it switches to accident mode, starts recording, and sends the data to the server.

[1127] User operations

[1128] The user mounts their smartphone on the driving device and launches the application. The application provides an interface for acquiring camera and sensor data and sending it to the server. If necessary, the user can acknowledge alerts and take appropriate evasive action.

[1129] Specific examples

[1130] Risk detection while driving: The server detects approaching pedestrians ahead and sends a warning to the device when the distance is less than 1.5 meters. The device then issues a visual and audio alert to remind the user to be careful.

[1131] Accident data storage: When the device detects a collision, it automatically stores video data before and after the accident in cloud storage, which can be used for later review.

[1132] Prompt Sentence Examples

[1133] "Creating an AI model that assesses collision risk and provides necessary warnings to drivers"

[1134] "Development of a system for monitoring the surroundings of autonomous vehicles in real time"

[1135] "A program that automatically assesses the risk of an accident based on the analysis results and sends a warning to the autonomous driving system."

[1136] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1137] Step 1:

[1138] The user attaches the smartphone to the driving device and launches the application. The input is the smartphone and the application, and the output is the data collection status from the camera and sensors. Specifically, the user taps to launch the application and confirms that the smartphone is attached.

[1139] Step 2:

[1140] The device starts the camera and captures the surrounding video in real time. The input is the camera and the output is the captured time-series video data. Specifically, the application calls the camera module and starts streaming the video.

[1141] Step 3:

[1142] The device collects input data from the speed sensor and distance sensor. The input is the speed sensor and distance sensor, and the output is the current speed data and distance data to surrounding objects. Specifically, the device reads data from the sensor terminal and processes it within the application.

[1143] Step 4:

[1144] The device transmits the captured video and sensor data to the server at regular intervals. The input is the video and sensor data, and the output is the data transmitted to the server. Specifically, the device divides the data into packets and uploads them to the server via an internet connection.

[1145] Step 5:

[1146] The server analyzes the received video data and sensor data using an AI model. The input is the received video data and sensor data, and the output is risk assessment data as the analysis result. Specifically, the server runs an AI analysis module and calculates the distance between vehicles, people, and objects from the data.

[1147] Step 6:

[1148] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and sends it to the terminal.The input is risk assessment data and the output is an alert signal.Specifically, if the risk assessment exceeds a certain threshold, the server calls the alert generation module and sends an alert signal to the terminal.

[1149] Step 7:

[1150] The terminal issues a warning to the driver based on the alert signal received from the server. The input is the alert signal, and the output is a visual, audio, or vibration warning. Specifically, the terminal activates the warning module and issues a warning using the screen display, speaker, or vibration function.

[1151] Step 8:

[1152] If the impact sensor detects a force greater than a certain level, the device switches to accident mode and starts automatic recording. The input is the impact sensor data, and the output is the recorded video data. Specifically, the device activates the accident detection module and starts recording video.

[1153] Step 9:

[1154] The device sends the recorded video data to the server, which then stores it in cloud storage. The input is the recorded video data, and the output is the data stored in cloud storage. Specifically, the device sends the video data to the server via a transmission module, and the server uploads the received data to the cloud. This stored data can be used later for insurance companies or legal procedures if necessary.

[1155] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1156] System Overview

[1157] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, analysis unit, alert unit, storage unit, and an emotion engine that recognizes the user's emotions. This system monitors the surrounding environment in real time while driving, alerts the driver if there is a high risk of contact, and automatically records and stores video data before and after an accident if one occurs. The system also recognizes the user's emotional state and adjusts the intensity and frequency of alerts accordingly to provide appropriate warnings.

[1158] System configuration

[1159] 1. Video acquisition method

[1160] The device uses a smartphone or dedicated camera attached to the driving device to capture real-time images of the surroundings, and this image data is sent to an analysis means.

[1161] 2. Analysis method

[1162] The server receives the video data sent from the device and analyzes it using an AI model. Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and objects, and assesses the risk of collision based on speed information.

[1163] 3. Alert Methods

[1164] If the server determines that there is a high risk of contact based on the analysis results, it sends an alert signal to the terminal.

[1165] The device will then issue a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, etc.

[1166] 4. Preservation means

[1167] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[1168] The server stores the recorded video data in cloud storage, making it available as evidence to insurance companies and in legal proceedings.

[1169] 5. Emotion Engine

[1170] The device includes facial recognition technology to analyze the user's facial expressions and assess the driver's emotional state.

[1171] The terminal includes voice recognition technology to analyze the user's voice input and assess their emotional state.

[1172] The server provides a means to adjust the intensity and frequency of alerts based on the user's emotional state.

[1173] A natural language description of the program's operation

[1174] Startup and preparation

[1175] The user attaches the smartphone to the electric kick scooter and launches the application.

[1176] The device will activate the camera and begin capturing video, and will also obtain speed information from the GPS sensor and accelerometer.

[1177] The device initializes the emotion engine and captures the user's face and voice data in real time.

[1178] Data transmission and analysis

[1179] The device transmits the captured video, speed information, and the user's face and voice data to the server at regular intervals.

[1180] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1181] The server analyzes the user's emotional data and assesses their state of stress and attention while driving.

[1182] Alert generation and notification

[1183] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and adjusts the strength and frequency of the alert based on the user's emotional state.

[1184] The device will then issue a warning to the driver based on the received alert signal, which can be visual, audible, or vibration.

[1185] Accident response and data preservation

[1186] If the shock sensor detects a force greater than a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[1187] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[1188] The terminal transmits the recorded video data to the server.

[1189] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[1190] Specific examples

[1191] System startup and start-up

[1192] The user attaches their smartphone to the electric kick scooter and launches the application.

[1193] The device will activate its camera and begin taking video, measuring speed, and capturing emotional data.

[1194] Alert generation and avoidance actions

[1195] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[1196] The server determines that the risk of contact is high and sets an appropriate alert strength based on the user's emotional data. For example, if the user is in a high stress state, it selects a moderately strong voice alert.

[1197] The server generates an alert signal and sends it to the terminal.

[1198] The device will display the received alert on the screen and provide a voice prompt to warn you.

[1199] Users acknowledge the alert, slow down, and watch out for pedestrians.

[1200] Response in the event of an accident

[1201] The device comes into contact with an obstacle and the impact sensor is activated.

[1202] The device will automatically start recording and record footage for a certain period of time before and after the accident.

[1203] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal procedures as needed.

[1204] In this way, each component works together to ensure the safety of the driver. The addition of an emotion engine also enables the system to respond flexibly to the user's condition.

[1205] The processing flow will be explained below.

[1206] Step 1:

[1207] The user attaches the smartphone to the electric scooter and launches the application.

[1208] Step 2:

[1209] The device will activate the camera and begin capturing video in real time, while simultaneously obtaining speed information from the GPS sensor and accelerometer.

[1210] Step 3:

[1211] The device initializes the emotion engine, captures the user's face and voice data in real time, and uses facial and voice recognition technologies to analyze the user's emotional state from their facial expressions and words.

[1212] Step 4:

[1213] The device transmits the captured video data, speed information, and user emotion data to the server at regular intervals, usually set to a few seconds.

[1214] Step 5:

[1215] The server receives the video data, speed information, and emotion data sent from the device and stores them in temporary storage.

[1216] Step 6:

[1217] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1218] Step 7:

[1219] The server evaluates the user's emotional state analyzed by the emotion engine and adjusts the intensity and frequency of alerts based on whether the user is in an emotional state such as stress or distraction.

[1220] Step 8:

[1221] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal that includes the type of risk (vehicle, person, object) and the risk level (high, medium, low) according to the emotional state.

[1222] Step 9:

[1223] The server sends the generated alert signal to the terminal.

[1224] Step 10:

[1225] The terminal receives an alert signal from the server and issues a warning to the driver using a display device or speaker. The alert is notified by visual display, sound, vibration, etc. The warning is made stronger especially when the user is in a high stress state.

[1226] Step 11:

[1227] Users can check the alerts from their devices and take evasive action by slowing down if necessary.

[1228] Step 12:

[1229] If the shock sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[1230] Step 13:

[1231] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[1232] Step 14:

[1233] The terminal transmits the recorded video data to the server.

[1234] Step 15:

[1235] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[1236] Step 16:

[1237] In the event of an accident, users can use the accident reporting feature within the application to enter more detailed information.

[1238] In this way, each step works smoothly together to ensure the safety of the driver. The addition of an emotion engine enables the system to respond flexibly according to the user's psychological state.

[1239] Example 2

[1240] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1241] Conventional electric scooter safety systems simply assess physical risks without considering the driver's situational awareness or stress level. As a result, they have been unable to adequately address the risk of accidents caused by high stress levels or reduced attention. Furthermore, in the event of an accident, automation of data storage is insufficient, which can lead to a lack of video evidence.

[1242] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording when an accident occurs and saving the recorded data, and an emotion recognition means for recognizing the user's emotional state and adjusting the intensity and frequency of the alert. This enables flexible warnings according to the user's emotional state and reliable data saving when an accident occurs.

[1243] "Image acquisition means" refers to a device or apparatus that is fixed to the driving equipment and that captures and acquires images of the surrounding area.

[1244] "Analysis means" refers to equipment or software that analyzes acquired video data and other sensor data to perform risk assessments, distance measurement, etc.

[1245] The "alert means" is a device or notification system for issuing a warning to the driver based on the analysis results.

[1246] "Storage means" refers to a system or device that automatically starts recording when an accident occurs and stores the recorded data.

[1247] "Emotion recognition means" refers to technology or a system that recognizes the user's emotional state and adjusts the intensity and frequency of alerts according to that state.

[1248] A "generative AI model" is an algorithm or framework that uses machine learning or deep learning to perform a specific task (e.g., distance measurement or risk assessment).

[1249] "Driving equipment" refers to means of transportation such as electric kick scooters, and in this case refers to equipment equipped with various sensors and systems to improve safety.

[1250] MODE FOR CARRYING OUT THE INVENTION

[1251] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, an analysis unit, an alert unit, a storage unit, and an emotion recognition unit that recognizes the user's emotions. This system monitors the surroundings in real time while driving, and issues an alert to the driver if there is a high risk of contact. In the event of an accident, the system automatically records and stores video data before and after the accident. The system also recognizes the user's emotional state and adjusts the intensity and frequency of alerts accordingly to provide appropriate warnings.

[1252] System configuration

[1253] 1. Video acquisition method

[1254] The terminal uses a mobile device (such as a smartphone or dedicated camera) attached to the driving equipment to capture images of the surroundings in real time. This image data is sent to the analysis means.

[1255] 2. Analysis method

[1256] The server receives the video data sent from the device and analyzes it using a generative AI model (e.g., TensorFlow or PyTorch). Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and other objects, and assesses the risk of contact based on speed information.

[1257] 3. Alert Methods

[1258] If the server determines that the risk of contact is high based on the analysis results, it sends an alert signal to the terminal.

[1259] The device issues a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, or the like.

[1260] 4. Preservation means

[1261] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[1262] The server stores the recorded video data in cloud storage (e.g., AWS S3 or Google Cloud Storage) and makes it available to insurance companies and as evidence in legal proceedings.

[1263] 5. Emotion recognition means

[1264] The terminal includes facial recognition technology to perform facial expression analysis of the user and assess the emotional state of the driver.

[1265] The terminal includes voice recognition technology to analyze the user's voice input and assess their emotional state.

[1266] The server provides a means to adjust the intensity and frequency of alerts based on the user's emotional state.

[1267] Specific examples

[1268] System startup and start-up

[1269] The user attaches their smartphone to the electric kick scooter and launches the application.

[1270] The device will activate its camera and begin taking video, measuring speed, and capturing emotional data.

[1271] Alert generation and avoidance actions

[1272] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[1273] The server determines that the risk of contact is high and sets an appropriate alert strength based on the user's emotional data. For example, if the user is in a high stress state, a moderately strong voice alert is selected.

[1274] The server generates and sends an alert signal to the terminal.

[1275] The device will display the received alert on the screen and give an audio warning.

[1276] The user acknowledges the alert, slows down, and watches out for pedestrians.

[1277] Response in the event of an accident

[1278] The device comes into contact with an obstacle and the impact sensor is activated.

[1279] The device will automatically start recording and record footage for a certain period of time before and after the accident.

[1280] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal proceedings as needed.

[1281] Prompt Sentence Examples

[1282] Here are some example prompts to explain system behavior to a generative AI model:

[1283] "Please explain how the safety system for electric scooters works. Please use a concrete scenario to explain step by step the roles of the user, device, and server, as well as the system's functions."

[1284] In this way, a system that realizes safe driving is constructed by using specific hardware and software to process and calculate data.

[1285] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1286] System program processing flow

[1287] Step 1: Boot the system

[1288] The user attaches the smartphone to the electric kick scooter and launches the application.

[1289] Input: Electric scooter, smartphone, application startup operation

[1290] Output: System booting, initial setup complete

[1291] Specific operation: Place the device in the dedicated smartphone holder and tap the app icon to launch it.

[1292] Step 2: Initialize the video and sensors

[1293] The device will start the camera and begin recording video, and will also initialize the GPS sensor and acceleration sensor to begin acquiring location and speed information.

[1294] Input: Smartphone, camera, GPS sensor, accelerometer

[1295] Output: Real-time video data, position data, speed data

[1296] Specific behavior: Launches the camera app in the background and calls the sensor API to start the data stream.

[1297] Step 3: Activating the Emotion Engine

[1298] The device initializes the emotion engine and begins capturing the user's face and voice data in real time.

[1299] Input: Smartphone, face recognition library, voice recognition API

[1300] Output: Real-time emotion data

[1301] Specific behavior: Launches the face recognition library and speech recognition API, preparing for data capture.

[1302] Step 4: Send data

[1303] The terminal transmits the captured video, speed information, and the user's face and voice data to the server at regular intervals (for example, once per second).

[1304] Input: Real-time video data, location data, speed data, emotion data

[1305] Output: Data sent to the server

[1306] What it does: Sends data to the server using an HTTP POST request.

[1307] Step 5: Video Analysis

[1308] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1309] Input: Video data, position data, speed data

[1310] Output: Analysis results (distance data, risk assessment)

[1311] What it does: Identify objects in images and measure distances using computer vision libraries (e.g., OpenCV) and deep learning models (e.g., TensorFlow).

[1312] Step 6: Sentiment Analysis

[1313] The server analyzes the user's emotional data and evaluates their state of stress and attention while driving.

[1314] Input: Emotion data (face, voice)

[1315] Output: Analysis results (stress level, attention evaluation)

[1316] Specific operations: Facial features are analyzed using an expression analysis model, and vocal tone and speed are measured using an audio analysis model.

[1317] Step 7: Alert Generation

[1318] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and adjusts the strength and frequency of the alert based on the user's emotional state.

[1319] Input: Risk assessment results, emotion assessment results

[1320] Output: Alert signal (strength, frequency)

[1321] What it does: It quantifies the risk level by setting standard deviations and thresholds, then runs an algorithm that determines the type and intensity of the alert (visual, audio, vibration, etc.).

[1322] Step 8: Alert Notifications

[1323] The device issues a warning to the driver based on the received alert signal, notifying the driver of the alert through visual display, audio, vibration, or other means.

[1324] Input: Alert signal

[1325] Output: Warning to the driver (visual, audio, vibration)

[1326] Specific actions: Displays a warning message on the smartphone display, outputs an audio alert through the speaker, and activates the vibration motor.

[1327] Step 9: Detecting an incident

[1328] If the shock sensor detects a force greater than a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[1329] Input: Impact sensor data

[1330] Output: Switch to accident mode

[1331] Specific operation: Analyzes accelerometer data and sets a flag if a sudden change is detected.

[1332] Step 10: Record footage of the incident

[1333] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage of the time before and after the accident.

[1334] Input: Accident mode, camera video data

[1335] Output: Recording data

[1336] What it does: Uses the video recording API to save the video stream to a fixed disk or cloud storage.

[1337] Step 11: Data transmission and storage

[1338] The terminal transmits the recorded video data to the server.

[1339] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal procedures if necessary.

[1340] Input: Recording data

[1341] Output: Cloud saved data

[1342] Specific operation: Data is stored using a database management system (e.g., AWS S3 or Google Cloud Storage) while setting access permissions.

[1343] In this way, each step works together to provide a system that comprehensively supports driver safety.

[1344] (Application example 2)

[1345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1346] While driving an autonomous vehicle, multiple sensors and analytical means are required to properly monitor the surrounding environment and the driver's state and ensure safe driving. However, these means operating individually are not sufficient. In particular, it is difficult to recognize the driver's emotional state in real time and issue appropriate warnings or alerts. Furthermore, it is necessary to respond quickly and accurately in the event of an accident. Therefore, this invention provides a system that recognizes the driver's emotional state and issues appropriate alerts to improve the safety of autonomous vehicles.

[1347] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1348] In this invention, the server includes a video capture means fixed to the driving equipment, an analysis means for analyzing the video data captured by the video capture means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, an emotion engine for recognizing the emotional state of the driver and adjusting the intensity and frequency of the alert, and a storage means for automatically recording when an accident occurs and saving the recorded data. This enables real-time monitoring while driving and flexible response according to the emotional state of the driver.

[1349] The "image acquisition means" is a device that is fixed to the driving equipment and captures the surrounding situation in real time.

[1350] The "analysis means" is a device or software for analyzing the video data acquired by the video acquisition means and calculating distance, speed, etc.

[1351] The "alert means" is a mechanism or system that issues a visual, audible, or tactile warning to the driver based on the information obtained by the analysis means.

[1352] An "emotion engine" is a mechanism or software that analyzes the driver's facial expressions and voice to recognize and evaluate their emotional state.

[1353] "Storage means" refers to a device or system that automatically starts recording when an accident occurs and records and stores the video data.

[1354] An "AI model" refers to a learning algorithm or computational model that uses artificial intelligence to analyze the distance and speed of objects from video data.

[1355] MODE FOR CARRYING OUT THE INVENTION

[1356] System Overview

[1357] The present invention provides a system for improving the safety of autonomous vehicles. The system includes a plurality of image acquisition means, analysis means, alert means, emotion engine, and storage means, all of which are fixed to a driving device. Specific embodiments of each means are described below.

[1358] Video acquisition method

[1359] The device uses multiple cameras to capture images of the surroundings in real time. This allows image data to be acquired from various viewpoints, such as the front and sides of the vehicle. This image data is then sent to subsequent analysis means. The image acquisition means includes high-resolution cameras and software control for these cameras.

[1360] Analysis means

[1361] The server receives the video data sent from the device and analyzes it using a generative AI model. Specifically, it calculates the distance between the vehicle and the pedestrian, and the distance between the vehicle and the object, and evaluates the risk of contact based on speed information. This analysis uses machine learning frameworks such as TensorFlow and Keras. It also uses dlib as facial recognition technology to evaluate the driver's emotional state.

[1362] Alert Method

[1363] If the server determines that there is a high risk of collision based on the analysis results, it generates an alert signal and sends it to the device. Based on the received alert signal, the device warns the driver by means of visual display, sound, vibration, etc. For example, a text-to-speech engine (TTS engine) is used for the sound alert.

[1364] Emotion Engine

[1365] The device uses a camera and microphone to capture the driver's face and voice data. This data is analyzed using an emotion recognition model to evaluate the driver's emotional state. A generative AI model adjusts the intensity and frequency of alerts according to the driver's stress and attention levels. This allows for flexible responses tailored to the driver's psychological state.

[1366] Preservation means

[1367] When the device detects a force exceeding a certain level using impact and acceleration sensors, it determines that an accident has occurred and automatically starts recording. This recorded data is stored in cloud storage via a server and can be used by insurance companies or in legal proceedings as needed. Since recording requires high-speed data transfer and large-capacity storage, a cloud service is used.

[1368] Specific examples

[1369] To activate the system, the user attaches their smartphone to the dashboard of the autonomous vehicle and launches the application. The device's camera captures video in real time and sends it to the server. The server analyzes the video data, detects the presence of pedestrians ahead, and issues an alert to the driver. Furthermore, if the driver is under high stress, a stronger audio alert is issued. In the event of an accident, the recorded data is stored in the cloud and necessary measures are taken.

[1370] Prompt Sentence Examples

[1371] "We will design an application that monitors the driver's emotional state in real time when an object (pedestrian) that poses a risk of contact suddenly appears within one meter ahead, and issues an appropriate alert to enable safe driving."

[1372] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1373] Step 1: The device activates the camera and captures video in real time.

[1374] Input: Real-time video data from the camera.

[1375] Output: The video data stream of the device.

[1376] Specific operation: A camera fixed to the terminal continuously captures video, and this video data is sent to subsequent processing steps.

[1377] Step 2: The terminal transmits the video data to the server.

[1378] Input: Real-time video data captured by the device.

[1379] Output: Video data stream to server.

[1380] Specific operation: The device transmits the acquired video data to the server in real time via wireless communication, etc. This communication uses networks such as Wi-Fi or LTE.

[1381] Step 3: The server performs data pre-processing on the received video data.

[1382] Input: Video data sent from the device.

[1383] Output: Pre-processed video data.

[1384] Specific operation: The server performs preprocessing such as sharpening, noise reduction, and resolution adjustment on the video data, thereby improving the accuracy of analysis.

[1385] Step 4: The server analyzes the preprocessed video data with the generative AI model.

[1386] Input: Preprocessed video data.

[1387] Output: Position and distance data of objects and people.

[1388] How it works: The server's generated AI model calculates and extracts information about the distance between vehicles and pedestrians and objects in the video. It also analyzes speed information.

[1389] Step 5: The server captures the driver's face and voice data and sends it to the emotion engine.

[1390] Input: Driver's face and voice data.

[1391] Output: Driver's emotional state data.

[1392] Specific operation: The device uses a camera and microphone to continuously capture the driver's facial expressions and voice, which are then analyzed by the emotion engine. The analysis results are sent to the server.

[1393] Step 6: The server evaluates the risk of contact based on the analysis results.

[1394] Input: Object distance data and driver emotional state data.

[1395] Output: Contact risk assessment data.

[1396] Specific operation: The server combines the distance data obtained by the generative AI model with the driver's emotional state to calculate the risk of collision. If the risk is high, it generates an alert signal.

[1397] Step 7: The server sends an alert signal to the terminal.

[1398] Input: Contact risk assessment data.

[1399] Output: Alert signal data packet.

[1400] Specific operation: If the server detects a high risk of collision, it generates an alert signal and sends it to the terminal to warn the driver visually, audibly, or by vibration.

[1401] Step 8: The terminal warns the driver based on the received alert signal.

[1402] Input: The alert signal sent from the server.

[1403] Output: Visual, audible, or tactile warning to the driver.

[1404] Specific actions: The device visually displays a warning message on the display, audibly plays an alert sound through the built-in speaker, and haptically activates the vibration motor.

[1405] Step 9: The device monitors sensor information such as impacts and sudden stops, and automatically starts recording when an accident occurs.

[1406] Input: Impact sensor and accelerometer data.

[1407] Output: Recorded data of the accident video.

[1408] Specific operation: The device constantly monitors information from the impact sensor and acceleration sensor, and automatically starts recording when it detects a force above a certain level. It records and saves footage of the accident before and after it occurs.

[1409] Step 10: The device uploads the recording data to cloud storage.

[1410] Input: Recording data.

[1411] Output: Saved data in cloud storage.

[1412] Specific operation: After the device has completed recording, it uploads the recorded data to cloud storage and manages it so that it can be used by insurance companies or for legal procedures if necessary.

[1413] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1414] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1415] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1416] [Fourth embodiment]

[1417] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1418] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1419] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1420] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1421] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1422] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1423] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1424] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1425] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1426] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1427] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1428] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1429] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1430] System Overview

[1431] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, an analysis unit, an alert unit, and a storage unit attached to the driving device. This system monitors the surrounding situation in real time while driving, alerts the driver if there is a high risk of contact, and automatically records and stores video data before and after an accident if one occurs.

[1432] System configuration

[1433] 1. Video acquisition method

[1434] The device uses a smartphone or dedicated camera attached to the driving device to capture real-time images of the surroundings, and this image data is sent to an analysis means.

[1435] 2. Analysis method

[1436] The server receives the video data sent from the device and analyzes it using an AI model. Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and objects, and evaluates the risk of collision based on speed information.

[1437] 3. Alert Methods

[1438] If the server determines that there is a high risk of contact based on the analysis results, it sends an alert signal to the terminal.

[1439] The device will then issue a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, etc.

[1440] 4. Preservation means

[1441] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[1442] The server stores the recorded video data in cloud storage, making it available as evidence to insurance companies and in legal proceedings.

[1443] A natural language description of the program's operation

[1444] Startup and preparation

[1445] The user attaches the smartphone to the electric kick scooter and launches the application.

[1446] The device will activate the camera and begin capturing video, and will also obtain speed information from the GPS sensor and accelerometer.

[1447] Data transmission and analysis

[1448] The device transmits the captured video and speed information to the server at regular intervals.

[1449] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1450] Alert generation and notification

[1451] If the server determines from the analysis results that there is a high risk of contact, it generates an alert signal and sends it to the terminal.

[1452] The device receives the alert signal and warns the driver visually, audibly, or by vibration, allowing the driver to acknowledge the alert and take necessary evasive action.

[1453] Accident response and data preservation

[1454] If the impact sensor detects a force above a certain level, the device will automatically switch to accident mode and begin recording.

[1455] The device sends the recorded data to a server, which then stores it in cloud storage, where it can be used for accident response and legal procedures.

[1456] Specific examples

[1457] System startup and start-up

[1458] The user attaches their smartphone to the electric kick scooter and launches the application.

[1459] The device will activate the camera and begin taking video and measuring speed.

[1460] Alert generation and avoidance actions

[1461] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[1462] The server determines that the risk of contact is high and generates an alert signal to send to the terminal.

[1463] The device will display the received alert on the screen and provide a voice prompt to warn you.

[1464] Users acknowledge the alert, slow down, and watch out for pedestrians.

[1465] Response in the event of an accident

[1466] The device comes into contact with an obstacle and the impact sensor is activated.

[1467] The device automatically records and saves footage before and after the accident.

[1468] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal procedures as needed.

[1469] In this way, each component works together to create a system that ensures the safety of the driver.

[1470] The processing flow will be explained below.

[1471] Step 1:

[1472] The user attaches the smartphone to the electric scooter and launches the corresponding application.

[1473] Step 2:

[1474] The device will activate the camera and begin capturing video in real time, while simultaneously obtaining speed information from the GPS sensor and accelerometer.

[1475] Step 3:

[1476] The device sends the captured video data and speed information to the server at regular intervals, which are usually set in units of a few seconds.

[1477] Step 4:

[1478] The server receives the video data and speed information sent from the terminal and stores them in temporary storage.

[1479] Step 5:

[1480] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1481] Step 6:

[1482] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal that includes the type of risk (vehicle, person, object) and the risk level (high, medium, low).

[1483] Step 7:

[1484] The server sends the generated alert signal to the terminal.

[1485] Step 8:

[1486] The terminal receives an alert signal from the server and issues a warning to the driver using a display device or speaker, such as a visual display, sound, or vibration.

[1487] Step 9:

[1488] Users can check the alerts from their devices and take evasive action by slowing down if necessary.

[1489] Step 10:

[1490] If the shock sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[1491] Step 11:

[1492] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[1493] Step 12:

[1494] The terminal transmits the recorded video data to the server.

[1495] Step 13:

[1496] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[1497] Step 14:

[1498] In the event of an accident, users can use the accident reporting feature within the application to enter more detailed information.

[1499] Example 1

[1500] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1501] Previous technology had the problem of being unable to respond immediately to situations that required the driver's attention. Furthermore, when an accident occurred, video data serving as evidence was not acquired, making it impossible to quickly provide the information necessary for insurance companies and legal procedures. This resulted in issues such as insufficient driver safety and complicated, time-consuming post-accident procedures.

[1502] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1503] In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording and saving the recorded data when an accident occurs, a speed acquisition means for acquiring speed information from the driving equipment and sending it to the analysis means, and a risk assessment means for analyzing the received video data and speed information using an AI model and assessing the risk of contact. This makes it possible to quickly assess risks while driving and notify the driver, and automatically save important video data when an accident occurs.

[1504] "Driving device" refers to an electric scooter, bicycle, or other personal transportation device.

[1505] "Video acquisition means" refers to a camera or mobile device that is fixed to the driving equipment and captures the surrounding situation in real time.

[1506] "Analysis means" refers to a device or AI model that analyzes the video data acquired by the video acquisition means and performs object recognition and distance measurement.

[1507] "Alert means" refers to a device that issues a visual, audio, or vibration warning to the driver based on data obtained from the analysis means.

[1508] "Storage means" refers to a device or cloud storage that automatically starts recording when an accident occurs and saves the captured video data.

[1509] "Speed ​​acquisition means" refers to a sensor or device for acquiring speed information of driving equipment and transmitting it to analysis means.

[1510] "Risk assessment means" refers to a device or algorithm that assesses the risk of contact based on analyzed video data and speed information.

[1511] An "AI model" refers to an algorithm that uses machine learning and deep learning techniques to analyze image and video data and recognize specific patterns and objects.

[1512] "Real-time" refers to data being acquired and processed and analyzed with very little delay.

[1513] "Automatic recording" refers to a function that automatically starts recording when a sensor or other device detects certain conditions.

[1514] The present invention provides a system for improving safety in driving equipment such as electric kick scooters. The system includes an image acquisition unit, an analysis unit, an alert unit, a storage unit, a speed acquisition unit, and a risk assessment unit.

[1515] Video acquisition

[1516] The user attaches a mobile terminal (such as a smartphone) to the driving device and launches a dedicated application.

[1517] The device uses a camera to capture real-time images of the surrounding area, and collects location information from a GPS sensor and speed information from an acceleration sensor. This data is then sent to a server at regular intervals.

[1518] Data analysis

[1519] The server receives the video data and speed information transmitted from the terminal.

[1520] The server analyzes the video data using an AI model. Specifically, it calculates the distance between the vehicle and the person, and the distance between the object and the vehicle, and identifies the object. This can be done using deep learning libraries such as TensorFlow.

[1521] Risk assessment and alert notifications

[1522] The server evaluates the risk of contact based on the analysis results, and if it determines that the risk is high, it generates an alert signal and sends it to the device.

[1523] The device will then display a warning to the user based on the received alert signal, which can be visual (display on the smartphone screen), audible (audio alert), or tactile (vibration).

[1524] Incident detection and data storage

[1525] If the impact sensor detects a certain level of force, the device automatically switches to recording mode, recording video data before and after the accident and sending it to a server.

[1526] The server then stores the received recording data in cloud storage, which can be managed for insurance companies or legal procedures. Cloud storage such as Amazon S3 can be used.

[1527] Specific examples

[1528] System startup and start-up

[1529] The user attaches a smartphone to the electric scooter and launches the application, which causes the entire system to transition from standby to operational mode.

[1530] The device will be adjusted so that the camera is facing forward and will display a "Ready" message after launching the application.

[1531] Alert generation and avoidance actions

[1532] After the device sends the video footage, the server performs real-time analysis and generates an alert a few seconds later saying, "There is a pedestrian ahead."

[1533] The device will display a red warning on the smartphone screen and simultaneously play an audio alert saying "Be careful."

[1534] In response to this alert, the user slows down and checks the situation ahead.

[1535] Response in the event of an accident

[1536] The device goes into recording mode the moment the impact sensor detects contact with the kick scooter.

[1537] The device records a few seconds of data, including the immediately preceding footage, and automatically sends it to the server.

[1538] The server immediately uploads this data to cloud storage and provides it to insurance companies if required.

[1539] Prompt Sentence Examples

[1540] "I would like to develop a safety support system for electric scooters. I would like it to monitor the surroundings in real time while driving and send an alert to the driver if there is a high risk of collision. In the event of an accident, I would like it to automatically record and save video data before and after the accident. What kind of hardware and software would be required?"

[1541] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1542] Step 1: System startup

[1543] The user attaches a smartphone to the electric scooter and launches a dedicated application. The input is the attachment of the smartphone and the launch of the application, and the output is the entire system transitioning to an operational state.

[1544] The device initializes the camera, GPS sensor, and accelerometer, and notifies the user that it is ready. Specifically, the device starts collecting data from each sensor and displays a "Ready" message on the application screen.

[1545] Step 2: Data Acquisition

[1546] The device starts capturing real-time images of its surroundings using its camera, with the input being the surrounding images and data from the GPS and accelerometer, and the output being packetized versions of these data.

[1547] The device periodically transmits the collected video data, location information, and speed information to a server. Specifically, the device periodically streams the data, which is then sent to the server via a high-speed internet connection.

[1548] Step 3: Video analysis

[1549] The server receives the video data and speed information sent from the device. The input is the data packets from the device, and the output is the analysis results from the AI ​​model.

[1550] The server analyzes the received data using an AI model. Specifically, it calculates the distance between the vehicle and the person, and the distance between the object and the vehicle, and then performs object identification. Specifically, the AI ​​model analyzes each frame of video and recognizes specific objects (pedestrians, vehicles, fixed objects, etc.).

[1551] Step 4: Risk assessment

[1552] The server evaluates the risk of contact based on the results of video analysis and speed information. The input is the analysis results of the AI ​​model and speed information, and the output is the risk assessment result.

[1553] If the server determines that the risk of contact is high, it generates an alert signal. Specifically, the server uses a scoring algorithm to evaluate the risk level and generates an alert signal if the risk exceeds a certain threshold.

[1554] Step 5: Alert Notification

[1555] The server sends the generated alert signal to the terminal. The input is the result of the risk assessment and the output is the alert signal.

[1556] The device will then provide a visual, audio, or vibration warning to the user based on the received alert signal, by displaying a warning message on the smartphone screen, playing an audio alert, and activating the vibration motor if necessary.

[1557] Step 6: Incident detection and data storage

[1558] The device automatically switches to recording mode when the impact sensor detects a certain amount of force. The input is a signal from the impact sensor, and the output is a command to start recording and save the data.

[1559] The device sends the recorded data to a server, which then stores it in cloud storage. Specifically, the device records the recorded data, including the most recent footage, and immediately sends it to the server. The server then uploads the received data to cloud storage, making it available to insurance companies and legal proceedings.

[1560] The above are the specific steps of the program processing of this system.

[1561] (Application example 1)

[1562] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1563] Traffic accidents are often caused by drivers' lack of attention or awareness of their surroundings. Therefore, there is a need for a system that can monitor the surrounding situation in real time while driving and issue alerts at the appropriate time. However, current technology makes it difficult to achieve this efficiently and accurately. In addition, recording and storing data in the event of an accident is also an important issue in ensuring driver safety.

[1564] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1565] In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording when an accident occurs and saving the recorded data, and a means for evaluating the risk of contact in real time and sending a warning to the automated driving system. This makes it possible to quickly evaluate the risk of contact while driving, issue an appropriate warning, and reliably save data when an accident occurs.

[1566] "Driving equipment" refers to any device used to assist in driving a vehicle.

[1567] "Image acquisition means" refers to a device that collects visual information using a camera, sensor, etc.

[1568] "Analysis means" refers to software or hardware for analyzing acquired data and extracting specific information.

[1569] "Alert means" refers to a device or function for issuing a warning to the driver or system based on the analysis results.

[1570] "Storage means" refers to a device or system that stores acquired data and makes it available for later use.

[1571] "Contact risk" refers to the possibility of colliding with other objects or people while driving.

[1572] "Real-time" refers to data acquisition and processing occurring immediately, without delay.

[1573] An "autonomous driving system" refers to an integrated system that enables a vehicle to drive autonomously without driver intervention.

[1574] System Overview

[1575] This invention is a system that combines a series of means installed in driving equipment to improve safety. Specifically, it is composed of video acquisition means, analysis means, alert means, and storage means. This makes it possible to evaluate the risk of contact while driving in real time, issue necessary warnings, and record and store data when an accident occurs.

[1576] Hardware and software used

[1577] 1. Hardware

[1578] Cameras: These are mounted on driving equipment for safety monitoring, with 360-degree cameras being particularly recommended.

[1579] AI-dedicated chip: Used for high-speed analysis of video data and sensor data.

[1580] Speed ​​sensor: Monitors the vehicle's speed.

[1581] Distance sensor: Measures the distance to surrounding objects and people.

[1582] Impact sensor: Detects collisions and impacts to detect the occurrence of an accident.

[1583] 2. Software

[1584] AI analysis module: Includes an AI model that uses TensorFlow / Keras, etc. to analyze the distance between vehicles, the distance between people, and the distance between objects.

[1585] Image Processing Module: Acquires and pre-processes camera images using OpenCV.

[1586] Data storage module: Has the function of uploading video and sensor data to cloud storage in the event of an accident.

[1587] Alert module: Provides visual, audio, or vibration warnings to the driver.

[1588] Natural language explanation of program processing

[1589] Server Processing

[1590] The server receives video and sensor data sent from the device and analyzes this data in real time using an AI analysis module. Based on the analysis results, if there is a high risk of contact, an alert signal is generated and sent to the device. Furthermore, if an accident occurs, video data before and after the accident is automatically saved to cloud storage.

[1591] Terminal handling

[1592] The device collects data from cameras and sensors attached to the driving equipment and sends it to a server. When it receives an alert signal from the server, it warns the driver visually, audibly, or by vibrating. When the impact sensor detects a force greater than a certain level, it switches to accident mode, starts recording, and sends the data to the server.

[1593] User operations

[1594] The user mounts their smartphone on the driving device and launches the application. The application provides an interface for acquiring camera and sensor data and sending it to the server. If necessary, the user can acknowledge alerts and take appropriate evasive action.

[1595] Specific examples

[1596] Risk detection while driving: The server detects approaching pedestrians ahead and sends a warning to the device when the distance is less than 1.5 meters. The device then issues a visual and audio alert to remind the user to be careful.

[1597] Accident data storage: When the device detects a collision, it automatically stores video data before and after the accident in cloud storage, which can be used for later review.

[1598] Prompt Sentence Examples

[1599] "Creating an AI model that assesses collision risk and provides necessary warnings to drivers"

[1600] "Development of a system for monitoring the surroundings of autonomous vehicles in real time"

[1601] "A program that automatically assesses the risk of an accident based on the analysis results and sends a warning to the autonomous driving system."

[1602] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1603] Step 1:

[1604] The user attaches the smartphone to the driving device and launches the application. The input is the smartphone and the application, and the output is the data collection status from the camera and sensors. Specifically, the user taps to launch the application and confirms that the smartphone is attached.

[1605] Step 2:

[1606] The device starts the camera and captures the surrounding video in real time. The input is the camera and the output is the captured time-series video data. Specifically, the application calls the camera module and starts streaming the video.

[1607] Step 3:

[1608] The device collects input data from the speed sensor and distance sensor. The input is the speed sensor and distance sensor, and the output is the current speed data and distance data to surrounding objects. Specifically, the device reads data from the sensor terminal and processes it within the application.

[1609] Step 4:

[1610] The device transmits the captured video and sensor data to the server at regular intervals. The input is the video and sensor data, and the output is the data transmitted to the server. Specifically, the device divides the data into packets and uploads them to the server via an internet connection.

[1611] Step 5:

[1612] The server analyzes the received video data and sensor data using an AI model. The input is the received video data and sensor data, and the output is risk assessment data as the analysis result. Specifically, the server runs an AI analysis module and calculates the distance between vehicles, people, and objects from the data.

[1613] Step 6:

[1614] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and sends it to the terminal.The input is risk assessment data and the output is an alert signal.Specifically, if the risk assessment exceeds a certain threshold, the server calls the alert generation module and sends an alert signal to the terminal.

[1615] Step 7:

[1616] The terminal issues a warning to the driver based on the alert signal received from the server. The input is the alert signal, and the output is a visual, audio, or vibration warning. Specifically, the terminal activates the warning module and issues a warning using the screen display, speaker, or vibration function.

[1617] Step 8:

[1618] If the impact sensor detects a force greater than a certain level, the device switches to accident mode and starts automatic recording. The input is the impact sensor data, and the output is the recorded video data. Specifically, the device activates the accident detection module and starts recording video.

[1619] Step 9:

[1620] The device sends the recorded video data to the server, which then stores it in cloud storage. The input is the recorded video data, and the output is the data stored in cloud storage. Specifically, the device sends the video data to the server via a transmission module, and the server uploads the received data to the cloud. This stored data can be used later for insurance companies or legal procedures if necessary.

[1621] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1622] System Overview

[1623] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, analysis unit, alert unit, storage unit, and an emotion engine that recognizes the user's emotions. This system monitors the surrounding environment in real time while driving, alerts the driver if there is a high risk of contact, and automatically records and stores video data before and after an accident if one occurs. The system also recognizes the user's emotional state and adjusts the intensity and frequency of alerts accordingly to provide appropriate warnings.

[1624] System configuration

[1625] 1. Video acquisition method

[1626] The device uses a smartphone or dedicated camera attached to the driving device to capture real-time images of the surroundings, and this image data is sent to an analysis means.

[1627] 2. Analysis method

[1628] The server receives the video data sent from the device and analyzes it using an AI model. Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and objects, and assesses the risk of collision based on speed information.

[1629] 3. Alert Methods

[1630] If the server determines that there is a high risk of contact based on the analysis results, it sends an alert signal to the terminal.

[1631] The device will then issue a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, etc.

[1632] 4. Preservation means

[1633] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[1634] The server stores the recorded video data in cloud storage, making it available as evidence to insurance companies and in legal proceedings.

[1635] 5. Emotion Engine

[1636] The device includes facial recognition technology to analyze the user's facial expressions and assess the driver's emotional state.

[1637] The terminal includes voice recognition technology to analyze the user's voice input and assess their emotional state.

[1638] The server provides a means to adjust the intensity and frequency of alerts based on the user's emotional state.

[1639] A natural language description of the program's operation

[1640] Startup and preparation

[1641] The user attaches the smartphone to the electric kick scooter and launches the application.

[1642] The device will activate the camera and begin capturing video, and will also obtain speed information from the GPS sensor and accelerometer.

[1643] The device initializes the emotion engine and captures the user's face and voice data in real time.

[1644] Data transmission and analysis

[1645] The device transmits the captured video, speed information, and the user's face and voice data to the server at regular intervals.

[1646] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1647] The server analyzes the user's emotional data and assesses their state of stress and attention while driving.

[1648] Alert generation and notification

[1649] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and adjusts the strength and frequency of the alert based on the user's emotional state.

[1650] The device will then issue a warning to the driver based on the received alert signal, which can be visual, audible, or vibration.

[1651] Accident response and data preservation

[1652] If the shock sensor detects a force greater than a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[1653] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[1654] The terminal transmits the recorded video data to the server.

[1655] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[1656] Specific examples

[1657] System startup and start-up

[1658] The user attaches their smartphone to the electric kick scooter and launches the application.

[1659] The device will activate its camera and begin taking video, measuring speed, and capturing emotional data.

[1660] Alert generation and avoidance actions

[1661] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[1662] The server determines that the risk of contact is high and sets an appropriate alert strength based on the user's emotional data. For example, if the user is in a high stress state, it selects a moderately strong voice alert.

[1663] The server generates an alert signal and sends it to the terminal.

[1664] The device will display the received alert on the screen and provide a voice prompt to warn you.

[1665] Users acknowledge the alert, slow down, and watch out for pedestrians.

[1666] Response in the event of an accident

[1667] The device comes into contact with an obstacle and the impact sensor is activated.

[1668] The device will automatically start recording and record footage for a certain period of time before and after the accident.

[1669] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal procedures as needed.

[1670] In this way, each component works together to ensure the safety of the driver. The addition of an emotion engine also enables the system to respond flexibly to the user's condition.

[1671] The processing flow will be explained below.

[1672] Step 1:

[1673] The user attaches the smartphone to the electric scooter and launches the application.

[1674] Step 2:

[1675] The device will activate the camera and begin capturing video in real time, while simultaneously obtaining speed information from the GPS sensor and accelerometer.

[1676] Step 3:

[1677] The device initializes the emotion engine, captures the user's face and voice data in real time, and uses facial and voice recognition technologies to analyze the user's emotional state from their facial expressions and words.

[1678] Step 4:

[1679] The device transmits the captured video data, speed information, and user emotion data to the server at regular intervals, usually set to a few seconds.

[1680] Step 5:

[1681] The server receives the video data, speed information, and emotion data sent from the device and stores them in temporary storage.

[1682] Step 6:

[1683] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1684] Step 7:

[1685] The server evaluates the user's emotional state analyzed by the emotion engine and adjusts the intensity and frequency of alerts based on whether the user is in an emotional state such as stress or distraction.

[1686] Step 8:

[1687] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal that includes the type of risk (vehicle, person, object) and the risk level (high, medium, low) according to the emotional state.

[1688] Step 9:

[1689] The server sends the generated alert signal to the terminal.

[1690] Step 10:

[1691] The terminal receives an alert signal from the server and issues a warning to the driver using a display device or speaker. The alert is notified by visual display, sound, vibration, etc. The warning is made stronger especially when the user is in a high stress state.

[1692] Step 11:

[1693] Users can check the alerts from their devices and take evasive action by slowing down if necessary.

[1694] Step 12:

[1695] If the shock sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[1696] Step 13:

[1697] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage from before and after the accident.

[1698] Step 14:

[1699] The terminal transmits the recorded video data to the server.

[1700] Step 15:

[1701] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal proceedings if necessary.

[1702] Step 16:

[1703] In the event of an accident, users can use the accident reporting feature within the application to enter more detailed information.

[1704] In this way, each step works smoothly together to ensure the safety of the driver. The addition of an emotion engine enables the system to respond flexibly according to the user's psychological state.

[1705] Example 2

[1706] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1707] Conventional electric scooter safety systems simply assess physical risks without considering the driver's situational awareness or stress level. As a result, they have been unable to adequately address the risk of accidents caused by high stress levels or reduced attention. Furthermore, in the event of an accident, automation of data storage is insufficient, which can lead to a lack of video evidence.

[1708] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a video acquisition means fixed to the driving equipment, an analysis means for analyzing the video data acquired by the video acquisition means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, a storage means for automatically recording when an accident occurs and saving the recorded data, and an emotion recognition means for recognizing the user's emotional state and adjusting the intensity and frequency of the alert. This enables flexible warnings according to the user's emotional state and reliable data saving when an accident occurs.

[1709] "Image acquisition means" refers to a device or apparatus that is fixed to the driving equipment and that captures and acquires images of the surrounding area.

[1710] "Analysis means" refers to equipment or software that analyzes acquired video data and other sensor data to perform risk assessments, distance measurement, etc.

[1711] The "alert means" is a device or notification system for issuing a warning to the driver based on the analysis results.

[1712] "Storage means" refers to a system or device that automatically starts recording when an accident occurs and stores the recorded data.

[1713] "Emotion recognition means" refers to technology or a system that recognizes the user's emotional state and adjusts the intensity and frequency of alerts according to that state.

[1714] A "generative AI model" is an algorithm or framework that uses machine learning or deep learning to perform a specific task (e.g., distance measurement or risk assessment).

[1715] "Driving equipment" refers to means of transportation such as electric kick scooters, and in this case refers to equipment equipped with various sensors and systems to improve safety.

[1716] MODE FOR CARRYING OUT THE INVENTION

[1717] To improve the safety of electric scooters, the present invention provides a system that includes a video capture unit, an analysis unit, an alert unit, a storage unit, and an emotion recognition unit that recognizes the user's emotions. This system monitors the surroundings in real time while driving, and issues an alert to the driver if there is a high risk of contact. In the event of an accident, the system automatically records and stores video data before and after the accident. The system also recognizes the user's emotional state and adjusts the intensity and frequency of alerts accordingly to provide appropriate warnings.

[1718] System configuration

[1719] 1. Video acquisition method

[1720] The terminal uses a mobile device (such as a smartphone or dedicated camera) attached to the driving equipment to capture images of the surroundings in real time. This image data is sent to the analysis means.

[1721] 2. Analysis method

[1722] The server receives the video data sent from the device and analyzes it using a generative AI model (e.g., TensorFlow or PyTorch). Specifically, it calculates the distance between the vehicle and other vehicles, the distance between people and other objects, and assesses the risk of contact based on speed information.

[1723] 3. Alert Methods

[1724] If the server determines that the risk of contact is high based on the analysis results, it sends an alert signal to the terminal.

[1725] The device issues a warning to the driver based on the received alert signal, which may be in the form of a visual display, sound, vibration, or the like.

[1726] 4. Preservation means

[1727] If the impact sensor or acceleration sensor detects a force above a certain level, the device will determine that an accident has occurred and automatically begin recording.

[1728] The server stores the recorded video data in cloud storage (e.g., AWS S3 or Google Cloud Storage) and makes it available to insurance companies and as evidence in legal proceedings.

[1729] 5. Emotion recognition means

[1730] The terminal includes facial recognition technology to perform facial expression analysis of the user and assess the emotional state of the driver.

[1731] The terminal includes voice recognition technology to analyze the user's voice input and assess their emotional state.

[1732] The server provides a means to adjust the intensity and frequency of alerts based on the user's emotional state.

[1733] Specific examples

[1734] System startup and start-up

[1735] The user attaches their smartphone to the electric kick scooter and launches the application.

[1736] The device will activate its camera and begin taking video, measuring speed, and capturing emotional data.

[1737] Alert generation and avoidance actions

[1738] Based on the video transmitted by the device, the server detects that a pedestrian is approaching ahead (at a distance of approximately 1.5 meters).

[1739] The server determines that the risk of contact is high and sets an appropriate alert strength based on the user's emotional data. For example, if the user is in a high stress state, a moderately strong voice alert is selected.

[1740] The server generates and sends an alert signal to the terminal.

[1741] The device will display the received alert on the screen and give an audio warning.

[1742] The user acknowledges the alert, slows down, and watches out for pedestrians.

[1743] Response in the event of an accident

[1744] The device comes into contact with an obstacle and the impact sensor is activated.

[1745] The device will automatically start recording and record footage for a certain period of time before and after the accident.

[1746] The server uploads the recorded data to cloud storage and manages it for insurance companies or legal proceedings as needed.

[1747] Prompt Sentence Examples

[1748] Here are some example prompts to explain system behavior to a generative AI model:

[1749] "Please explain how the safety system for electric scooters works. Please use a concrete scenario to explain step by step the roles of the user, device, and server, as well as the system's functions."

[1750] In this way, a system that realizes safe driving is constructed by using specific hardware and software to process and calculate data.

[1751] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1752] System program processing flow

[1753] Step 1: Boot the system

[1754] The user attaches the smartphone to the electric kick scooter and launches the application.

[1755] Input: Electric scooter, smartphone, application startup operation

[1756] Output: System booting, initial setup complete

[1757] Specific operation: Place the device in the dedicated smartphone holder and tap the app icon to launch it.

[1758] Step 2: Initialize the video and sensors

[1759] The device will start the camera and begin recording video, and will also initialize the GPS sensor and acceleration sensor to begin acquiring location and speed information.

[1760] Input: Smartphone, camera, GPS sensor, accelerometer

[1761] Output: Real-time video data, position data, speed data

[1762] Specific behavior: Launches the camera app in the background and calls the sensor API to start the data stream.

[1763] Step 3: Activating the Emotion Engine

[1764] The device initializes the emotion engine and begins capturing the user's face and voice data in real time.

[1765] Input: Smartphone, face recognition library, voice recognition API

[1766] Output: Real-time emotion data

[1767] Specific behavior: Launches the face recognition library and speech recognition API, preparing for data capture.

[1768] Step 4: Send data

[1769] The terminal transmits the captured video, speed information, and the user's face and voice data to the server at regular intervals (for example, once per second).

[1770] Input: Real-time video data, location data, speed data, emotion data

[1771] Output: Data sent to the server

[1772] What it does: Sends data to the server using an HTTP POST request.

[1773] Step 5: Video Analysis

[1774] The server analyzes the received video data using an AI model, calculating the distance between the vehicle and other vehicles, as well as the distance between people and objects, and assessing the risk of contact based on speed information.

[1775] Input: Video data, position data, speed data

[1776] Output: Analysis results (distance data, risk assessment)

[1777] What it does: Identify objects in images and measure distances using computer vision libraries (e.g., OpenCV) and deep learning models (e.g., TensorFlow).

[1778] Step 6: Sentiment Analysis

[1779] The server analyzes the user's emotional data and evaluates their state of stress and attention while driving.

[1780] Input: Emotion data (face, voice)

[1781] Output: Analysis results (stress level, attention evaluation)

[1782] Specific operations: Facial features are analyzed using an expression analysis model, and vocal tone and speed are measured using an audio analysis model.

[1783] Step 7: Alert Generation

[1784] If the server determines that the risk of contact is high based on the analysis results, it generates an alert signal and adjusts the strength and frequency of the alert based on the user's emotional state.

[1785] Input: Risk assessment results, emotion assessment results

[1786] Output: Alert signal (strength, frequency)

[1787] What it does: It quantifies the risk level by setting standard deviations and thresholds, then runs an algorithm that determines the type and intensity of the alert (visual, audio, vibration, etc.).

[1788] Step 8: Alert Notifications

[1789] The device issues a warning to the driver based on the received alert signal, notifying the driver of the alert through visual display, audio, vibration, or other means.

[1790] Input: Alert signal

[1791] Output: Warning to the driver (visual, audio, vibration)

[1792] Specific actions: Displays a warning message on the smartphone display, outputs an audio alert through the speaker, and activates the vibration motor.

[1793] Step 9: Detecting an incident

[1794] If the shock sensor detects a force greater than a certain level, the device will determine that an accident has occurred and automatically switch to accident mode.

[1795] Input: Impact sensor data

[1796] Output: Switch to accident mode

[1797] Specific operation: Analyzes accelerometer data and sets a flag if a sudden change is detected.

[1798] Step 10: Record footage of the incident

[1799] The device starts recording video for a certain period of time (for example, 5 minutes) from the moment it switches to incident mode. The recorded data includes footage of the time before and after the accident.

[1800] Input: Accident mode, camera video data

[1801] Output: Recording data

[1802] What it does: Uses the video recording API to save the video stream to a fixed disk or cloud storage.

[1803] Step 11: Data transmission and storage

[1804] The terminal transmits the recorded video data to the server.

[1805] The server stores the transmitted video data in cloud storage, where it can be used by insurance companies or for legal procedures if necessary.

[1806] Input: Recording data

[1807] Output: Cloud saved data

[1808] Specific operation: Data is stored using a database management system (e.g., AWS S3 or Google Cloud Storage) while setting access permissions.

[1809] In this way, each step works together to provide a system that comprehensively supports driver safety.

[1810] (Application example 2)

[1811] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1812] While driving an autonomous vehicle, multiple sensors and analytical means are required to properly monitor the surrounding environment and the driver's state and ensure safe driving. However, these means operating individually are not sufficient. In particular, it is difficult to recognize the driver's emotional state in real time and issue appropriate warnings or alerts. Furthermore, it is necessary to respond quickly and accurately in the event of an accident. Therefore, this invention provides a system that recognizes the driver's emotional state and issues appropriate alerts to improve the safety of autonomous vehicles.

[1813] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1814] In this invention, the server includes a video capture means fixed to the driving equipment, an analysis means for analyzing the video data captured by the video capture means, an alert means for issuing an alert to the driver based on the data analyzed by the analysis means, an emotion engine for recognizing the emotional state of the driver and adjusting the intensity and frequency of the alert, and a storage means for automatically recording when an accident occurs and saving the recorded data. This enables real-time monitoring while driving and flexible response according to the emotional state of the driver.

[1815] The "image acquisition means" is a device that is fixed to the driving equipment and captures the surrounding situation in real time.

[1816] The "analysis means" is a device or software for analyzing the video data acquired by the video acquisition means and calculating distance, speed, etc.

[1817] The "alert means" is a mechanism or system that issues a visual, audible, or tactile warning to the driver based on the information obtained by the analysis means.

[1818] An "emotion engine" is a mechanism or software that analyzes the driver's facial expressions and voice to recognize and evaluate their emotional state.

[1819] "Storage means" refers to a device or system that automatically starts recording when an accident occurs and records and stores the video data.

[1820] An "AI model" refers to a learning algorithm or computational model that uses artificial intelligence to analyze the distance and speed of objects from video data.

[1821] MODE FOR CARRYING OUT THE INVENTION

[1822] System Overview

[1823] The present invention provides a system for improving the safety of autonomous vehicles. The system includes a plurality of image acquisition means, analysis means, alert means, emotion engine, and storage means, all of which are fixed to a driving device. Specific embodiments of each means are described below.

[1824] Video acquisition method

[1825] The device uses multiple cameras to capture images of the surroundings in real time. This allows image data to be acquired from various viewpoints, such as the front and sides of the vehicle. This image data is then sent to subsequent analysis means. The image acquisition means includes high-resolution cameras and software control for these cameras.

[1826] Analysis means

[1827] The server receives the video data sent from the device and analyzes it using a generative AI model. Specifically, it calculates the distance between the vehicle and the pedestrian, and the distance between the vehicle and the object, and evaluates the risk of contact based on speed information. This analysis uses machine learning frameworks such as TensorFlow and Keras. It also uses dlib as facial recognition technology to evaluate the driver's emotional state.

[1828] Alert Method

[1829] If the server determines that there is a high risk of collision based on the analysis results, it generates an alert signal and sends it to the device. Based on the received alert signal, the device warns the driver by means of visual display, sound, vibration, etc. For example, a text-to-speech engine (TTS engine) is used for the sound alert.

[1830] Emotion Engine

[1831] The device uses a camera and microphone to capture the driver's face and voice data. This data is analyzed using an emotion recognition model to evaluate the driver's emotional state. A generative AI model adjusts the intensity and frequency of alerts according to the driver's stress and attention levels. This allows for flexible responses tailored to the driver's psychological state.

[1832] Preservation means

[1833] When the device detects a force exceeding a certain level using impact and acceleration sensors, it determines that an accident has occurred and automatically starts recording. This recorded data is stored in cloud storage via a server and can be used by insurance companies or in legal proceedings as needed. Since recording requires high-speed data transfer and large-capacity storage, a cloud service is used.

[1834] Specific examples

[1835] To activate the system, the user attaches their smartphone to the dashboard of the autonomous vehicle and launches the application. The device's camera captures video in real time and sends it to the server. The server analyzes the video data, detects the presence of pedestrians ahead, and issues an alert to the driver. Furthermore, if the driver is under high stress, a stronger audio alert is issued. In the event of an accident, the recorded data is stored in the cloud and necessary measures are taken.

[1836] Prompt Sentence Examples

[1837] "We will design an application that monitors the driver's emotional state in real time when an object (pedestrian) that poses a risk of contact suddenly appears within one meter ahead, and issues an appropriate alert to enable safe driving."

[1838] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1839] Step 1: The device activates the camera and captures video in real time.

[1840] Input: Real-time video data from the camera.

[1841] Output: The video data stream of the device.

[1842] Specific operation: A camera fixed to the terminal continuously captures video, and this video data is sent to subsequent processing steps.

[1843] Step 2: The terminal transmits the video data to the server.

[1844] Input: Real-time video data captured by the device.

[1845] Output: Video data stream to server.

[1846] Specific operation: The device transmits the acquired video data to the server in real time via wireless communication, etc. This communication uses networks such as Wi-Fi or LTE.

[1847] Step 3: The server performs data pre-processing on the received video data.

[1848] Input: Video data sent from the device.

[1849] Output: Pre-processed video data.

[1850] Specific operation: The server performs preprocessing such as sharpening, noise reduction, and resolution adjustment on the video data, thereby improving the accuracy of analysis.

[1851] Step 4: The server analyzes the preprocessed video data with the generative AI model.

[1852] Input: Preprocessed video data.

[1853] Output: Position and distance data of objects and people.

[1854] How it works: The server's generated AI model calculates and extracts information about the distance between vehicles and pedestrians and objects in the video. It also analyzes speed information.

[1855] Step 5: The server captures the driver's face and voice data and sends it to the emotion engine.

[1856] Input: Driver's face and voice data.

[1857] Output: Driver's emotional state data.

[1858] Specific operation: The device uses a camera and microphone to continuously capture the driver's facial expressions and voice, which are then analyzed by the emotion engine. The analysis results are sent to the server.

[1859] Step 6: The server evaluates the risk of contact based on the analysis results.

[1860] Input: Object distance data and driver emotional state data.

[1861] Output: Contact risk assessment data.

[1862] Specific operation: The server combines the distance data obtained by the generative AI model with the driver's emotional state to calculate the risk of collision. If the risk is high, it generates an alert signal.

[1863] Step 7: The server sends an alert signal to the terminal.

[1864] Input: Contact risk assessment data.

[1865] Output: Alert signal data packet.

[1866] Specific operation: If the server detects a high risk of collision, it generates an alert signal and sends it to the terminal to warn the driver visually, audibly, or by vibration.

[1867] Step 8: The terminal warns the driver based on the received alert signal.

[1868] Input: The alert signal sent from the server.

[1869] Output: Visual, audible, or tactile warning to the driver.

[1870] Specific actions: The device visually displays a warning message on the display, audibly plays an alert sound through the built-in speaker, and haptically activates the vibration motor.

[1871] Step 9: The device monitors sensor information such as impacts and sudden stops, and automatically starts recording when an accident occurs.

[1872] Input: Impact sensor and accelerometer data.

[1873] Output: Recorded data of the accident video.

[1874] Specific operation: The device constantly monitors information from the impact sensor and acceleration sensor, and automatically starts recording when it detects a force above a certain level. It records and saves footage of the accident before and after it occurs.

[1875] Step 10: The device uploads the recording data to cloud storage.

[1876] Input: Recording data.

[1877] Output: Saved data in cloud storage.

[1878] Specific operation: After the device has completed recording, it uploads the recorded data to cloud storage and manages it so that it can be used by insurance companies or for legal procedures if necessary.

[1879] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1880] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1881] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1882] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1883] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1884] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1885] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1886] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1887] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1888] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1889] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1890] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1891] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1892] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1893] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1894] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1895] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1896] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1897] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1898] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1899] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1900] The following is further disclosed regarding the above embodiment.

[1901] (Claim 1)

[1902] an image capturing means fixed to the driving device;

[1903] analysis means for analyzing the video data acquired by the video acquisition means;

[1904] alert means for issuing an alert to a driver based on the data analyzed by the analysis means;

[1905] A storage means for automatically recording when an accident occurs and saving the recorded data;

[1906] A system including:

[1907] (Claim 2)

[1908] 2. The system of claim 1, wherein the image capture means is a mobile terminal fixed to the driving equipment.

[1909] (Claim 3)

[1910] The system according to claim 1, wherein the analysis means includes an AI model capable of analyzing the distance between vehicles, the distance between people, and the distance between objects.

[1911] "Example 1"

[1912] (Claim 1)

[1913] an image capturing means fixed to the driving device;

[1914] analysis means for analyzing the video data acquired by the video acquisition means;

[1915] alert means for issuing an alert to a driver based on the data analyzed by the analysis means;

[1916] A storage means for automatically recording when an accident occurs and saving the recorded data;

[1917] a speed acquisition means for acquiring speed information of the driving device and transmitting it to an analysis means;

[1918] a risk assessment means for analyzing the received video data and speed information using an AI model and assessing a contact risk;

[1919] A system including:

[1920] (Claim 2)

[1921] 2. The system of claim 1, wherein the image capture means is a mobile terminal fixed to the driving equipment.

[1922] (Claim 3)

[1923] The system according to claim 1, wherein the analysis means includes an AI model capable of analyzing the distance between vehicles, the distance between people, and the distance between objects.

[1924] "Application Example 1"

[1925] (Claim 1)

[1926] an image capturing means fixed to the driving device;

[1927] analysis means for analyzing the video data acquired by the video acquisition means;

[1928] alert means for issuing an alert to a driver based on the data analyzed by the analysis means;

[1929] A storage means for automatically recording when an accident occurs and saving the recorded data;

[1930] A means to assess collision risk in real time and send a warning to the autonomous driving system;

[1931] A system including:

[1932] (Claim 2)

[1933] 2. The system of claim 1, wherein the image capture means is a mobile terminal fixed to the driving equipment.

[1934] (Claim 3)

[1935] The system according to claim 1, wherein the analysis means includes an AI model capable of analyzing the distance between vehicles, the distance between people, and the distance between objects.

[1936] "Example 2: Combining Emotion Engines"

[1937] (Claim 1)

[1938] an image capturing means fixed to the driving device;

[1939] analysis means for analyzing the video data acquired by the video acquisition means;

[1940] alert means for issuing an alert to a driver based on the data analyzed by the analysis means;

[1941] A storage means for automatically recording when an accident occurs and saving the recorded data;

[1942] an emotion recognition means for recognizing the emotional state of a user and adjusting the intensity and frequency of alerts;

[1943] A system including:

[1944] (Claim 2)

[1945] 2. The system according to claim 1, wherein the image acquisition means is a mobile terminal fixed to the driving equipment.

[1946] (Claim 3)

[1947] The system according to claim 1, wherein the analysis means includes a generative AI model capable of analyzing the distance between vehicles, the distance between people, and the distance between objects.

[1948] "Application example 2 when combining emotion engines"

[1949] (Claim 1)

[1950] an image capturing means fixed to the driving device;

[1951] analysis means for analyzing the video data acquired by the video acquisition means;

[1952] alert means for issuing an alert to a driver based on the data analyzed by the analysis means;

[1953] An emotion engine that recognizes the driver's emotional state and adjusts the intensity and frequency of alerts;

[1954] A storage means for automatically recording when an accident occurs and saving the recorded data;

[1955] A system including:

[1956] (Claim 2)

[1957] 2. The system of claim 1, wherein the image capture means is a mobile terminal fixed to the driving equipment.

[1958] (Claim 3)

[1959] 2. The system according to claim 1, wherein the analysis means includes an AI model capable of analyzing the distance to an object located ahead. [Explanation of symbols]

[1960] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. an image capturing means fixed to the driving device; analysis means for analyzing the video data acquired by the video acquisition means; an alert means for issuing an alert to a driver based on the data analyzed by the analysis means; A storage means for automatically recording when an accident occurs and saving the recorded data; A system including:

2. 2. The system of claim 1, wherein the image capture means is a mobile terminal fixed to the driving equipment.

3. The system according to claim 1 , wherein the analysis means includes an AI model capable of analyzing the distance between vehicles, the distance between people, and the distance between objects.

Images