system

Abstract

Provide a system. 【Solution means】 A mobile body safety management system for analyzing the collision risk caused by multiple mobile bodies and transmitting an alert, Camera means for acquiring images, Image analysis means for identifying an object in the image, Distance measurement means for calculating the distance to the identified object, Evaluation means for evaluating the risk of collision based on the distance and the speed data of the mobile body, Alert means for transmitting an alert to the driver based on the risk, Recording means for recording and storing the front and rear images when the alert is transmitted, Communication means for uploading the recorded data to the cloud, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to improve the safety during travel of a moving body, particularly an electric kick scooter or a bicycle. In recent years, the use of these moving bodies has increased rapidly, and the accident rate has also increased accordingly. However, existing safety countermeasure systems are expensive and require dedicated hardware, so they are often difficult for users to introduce. Therefore, there is a need to develop an inexpensive and easily available safety management system.

Means for Solving the Problems

[0005] This invention provides a system that attaches a smartphone or tablet camera to a moving object and uses image analysis technology to measure the distance to surrounding moving objects and obstacles, thereby pre-assessing the risk of collision. Using the acquired video data, the system combines camera, image analysis, distance measurement, evaluation, alert, recording, and communication means to provide real-time safety information to the driver. Furthermore, if network connectivity is unavailable, data is temporarily stored using local storage, and past recorded data is organized and provided via cloud connectivity to meet the needs of insurance companies and other organizations. This makes it possible to improve the safety of moving objects inexpensively and efficiently.

[0006] A "mobile vehicle safety management system" is a collection of devices and technologies designed to improve safety while a vehicle is in motion, and is particularly applicable to small mobile vehicles such as electric scooters and bicycles.

[0007] "Camera equipment" refers to a device for acquiring images, specifically a camera built into a smartphone or tablet attached to a mobile device.

[0008] "Image analysis means" refers to the technologies and algorithms used to identify objects from images acquired by a camera, and is used for object detection, identification, and tracking of specific targets.

[0009] "Distance measurement means" refers to the techniques and algorithms used to calculate the distance to an object recognized by image analysis, utilizing methods such as depth estimation.

[0010] "Evaluation methods" refer to technologies and algorithms that determine the risk of collision based on measured distance and moving object speed data.

[0011] "Alert means" refers to means used to inform a driver of the risk of collision, and includes visual or auditory warnings.

[0012] "Recording means" refers to a device or process that records and saves video when an alert is issued, and has the function of recording before and after an accident.

[0013] "Communication methods" refer to technologies and protocols used to transmit recorded data and other information to the cloud or other devices.

[0014] "Local storage means" refers to devices and technologies for temporarily storing data when communication means are unavailable, and uploading the data when cloud connectivity becomes available.

[0015] "Organization methods" refer to the processes and technologies used to organize video data for provision to insurance companies and related organizations, to organize information related to specific events, and to provide it in the required format. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9]Shows an emotion map to which a plurality of emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be described.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention provides a mobile safety management system that uses a smartphone attached to a mobile device such as an electric scooter or bicycle. This system achieves the following functions by installing a specific application on the smartphone.

[0038] Camera methods and image analysis

[0039] The device uses a smartphone camera mounted on the front of the moving object to capture real-time video. The video is continuously captured, and an image analysis algorithm recognizes objects. This algorithm identifies specific objects by enclosing them in a rectangle and makes decisions based on the traffic situation.

[0040] Distance measurement and evaluation

[0041] The device utilizes depth estimation technology to estimate the distance to recognized objects. Distance and velocity data are used to assess collision risk, and alerts are generated in high-risk scenarios.

[0042] Alert issuance

[0043] To alert the user, the device uses both audio alarms and screen displays. For example, if there is a vehicle rapidly approaching ahead, a warning sound will be emitted and a message such as "Caution! Obstacle ahead" will be displayed on the screen.

[0044] Recording and saving

[0045] The device automatically records video footage immediately before and after an alert is triggered, making it available as evidence of the accident. This recorded data is temporarily stored in the smartphone's internal storage.

[0046] Cloud storage of data

[0047] Once network connectivity is restored, the device will transfer the recorded data to a secure cloud server. The server will organize the received data and store it for easy access.

[0048] Providing data to insurance companies

[0049] If necessary, the server provides organized data to insurance companies and related organizations. The data contains information useful for accident investigation and is prepared to meet legal requirements.

[0050] This system utilizes existing smartphones, keeping implementation costs down while effectively improving the safety of mobile devices. Its ease of setup and operation makes it a suitable solution for a wide range of users.

[0051] The following describes the processing flow.

[0052] Step 1:

[0053] The device activates the smartphone's camera and acquires video in real time. The acquired video is saved as a series of frames and prepared for image analysis.

[0054] Step 2:

[0055] The device applies an image analysis algorithm to each captured frame to identify objects on the road. This algorithm recognizes specific objects (vehicles, pedestrians, obstacles) by enclosing them in a rectangle.

[0056] Step 3:

[0057] The device performs distance estimation for identified objects. Using depth estimation technology, it accurately measures the distance from the camera to the object and updates this information in real time.

[0058] Step 4:

[0059] The device uses measured distance and velocity data to assess the movement of objects and the likelihood of collision. A threshold is set, and if it determines that there is a risk exceeding that value, it is classified as high risk.

[0060] Step 5:

[0061] The device generates an alert if it is assessed as high-risk. It sends an audible alarm and an on-screen warning message to the user to draw their attention.

[0062] Step 6:

[0063] The device continuously records video from the moment an alert is issued, documenting the situation immediately before and after the event. This data is temporarily stored on the device.

[0064] Step 7:

[0065] The device uploads the recorded data to the cloud server as soon as it establishes a network connection. The data is transmitted along with a timestamp and identification information, and is stored for later reference.

[0066] Step 8:

[0067] The server receives and organizes the uploaded video data. If requested by insurance companies or related organizations, it prepares the data to be provided in a format suitable for verification.

[0068] (Example 1)

[0069] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0070] There is a need to provide an effective mobile safety management system that reduces the risk of traffic accidents and near misses and improves driver safety. However, existing systems are insufficient in real time object recognition, distance measurement, and risk assessment, and there are also challenges in the proper management and provision of video data in the event of an accident.

[0071] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0072] In this invention, the server includes a shooting means for acquiring video footage, an analysis means for identifying objects in the video footage, and an estimation means for calculating the distance to the identified objects. This makes it possible to accurately recognize the situation around a moving vehicle in real time and to quickly assess the risk of collision.

[0073] "Shooting means" refers to a device or function for acquiring images of the surroundings of a moving object in real time.

[0074] "Analysis means" refers to a device or algorithm used to identify objects in acquired video footage and determine their location and type.

[0075] "Estimation means" refers to a device or calculation method that calculates the distance to an object identified by an analysis means and derives an accurate relative position.

[0076] "Evaluation means" refers to a device or process that evaluates the risk of collision based on the distance of an identified object and the velocity data of a moving object.

[0077] "Warning means" refers to a device or method used to alert the operator based on the results obtained by the evaluation means.

[0078] "Recording means" refers to a device or mechanism for recording and storing video footage before and after a warning signal is issued.

[0079] "Transmission means" refers to a device or technology for transferring recorded video data to other devices or servers via a communication network.

[0080] A "storage medium" is a device or physical storage capable of temporarily storing video footage recorded by a recording means.

[0081] "Management means" refers to a device or method for organizing, appropriately storing, and utilizing recorded data related to a specific event.

[0082] The system for implementing this invention utilizes a smartphone as a terminal attached to a mobile device to manage traffic safety. This system is operated using specific hardware and software to perform several key functions.

[0083] The device utilizes the camera built into the smartphone as a means of capturing images, acquiring real-time video of the direction of movement of a moving object. This video data is processed by an image analysis algorithm as an analytical tool, identifying objects such as vehicles and pedestrians. By using, for example, a deep learning model for this analysis, specific objects can be identified with high accuracy.

[0084] Furthermore, the terminal uses estimation means to calculate the distance to the identified object. Distance estimation utilizes depth estimation technology using machine learning models, providing accurate distance information in real time. This allows the terminal to use evaluation means to assess the risk of collision with the object, and if a high level of risk is detected, it can use warning means to alert the user with an audio alarm or screen display.

[0085] Furthermore, the device can use recording means to automatically record video before and after an alert is issued and temporarily save it to the smartphone's storage medium. This data is then transferred to the server using transmission means after the network connection is restored. The server securely stores this recorded data, organizes it using management means as needed, and provides it to public institutions and insurance companies.

[0086] As a concrete example, suppose a user is delivering goods by bicycle and a pedestrian suddenly appears in front of them. The device instantly recognizes the pedestrian, estimates the distance, assesses the risk of collision, and issues a warning, allowing the user to take evasive action quickly.

[0087] An example of a prompt to input into a generative AI model is a question such as, "In a safety management system using a smartphone attached to an electric scooter, please explain the methods for recognizing objects moving in front and measuring their distance." By utilizing this prompt, it is possible to deepen the understanding of the system's mechanisms and performance through the generative AI model.

[0088] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0089] Step 1:

[0090] The device uses the smartphone's camera to acquire real-time video of the area in front of a moving object. The input is continuous image data from the camera, and the output is video data for analysis. In this process, the camera captures high-resolution video and saves it directly to memory, providing the data necessary for subsequent analysis steps.

[0091] Step 2:

[0092] The terminal processes the acquired video data using an analysis device and applies an image analysis algorithm to identify objects in the video. The input is the video data which is the output of step 1, and the output is data on the location and type of the identified objects. Specifically, it uses a machine learning model to recognize a particular object and draws a rectangle around its location.

[0093] Step 3:

[0094] The terminal uses estimation methods to calculate the distance to the identified object. The input is the object identification data from step 2, and the output is the estimated distance to the object. In this step, depth estimation is performed by combining image disparity information and algorithms to calculate the exact distance to the object.

[0095] Step 4:

[0096] The terminal uses an evaluation method to assess collision risk based on the calculated distance and moving object velocity data. The input is the distance and velocity data from step 3, and the output is the risk level evaluation result. Specifically, the program compares a threshold with the actual data and determines that the risk level is high if the probability of collision exceeds a certain level.

[0097] Step 5:

[0098] The terminal uses the warning means to issue an alert to the user based on the output of the evaluation means. The input is the evaluation result from step 4, and the output is an audio alarm and a warning message displayed on the screen to the user. In this case, a warning sound is emitted from the speaker and a specific warning message is displayed on the display.

[0099] Step 6:

[0100] The terminal records video before and after a warning is issued using a recording device and temporarily saves it to a local storage medium. The input is the video data from step 1, and the output is the recorded file data. The terminal selects and efficiently saves video from a certain period before and after the alert occurs.

[0101] Step 7:

[0102] The server, upon recovery of network connectivity, instructs the transmission method to transfer the recorded data to the cloud. The input is the temporary data stored in step 6, and the output is the data stored in the cloud. During this process, the data is encrypted to ensure the security of the transfer.

[0103] Step 8:

[0104] The server organizes the recorded data using management tools and provides the records to public institutions or insurance companies as needed. The input is the cloud-stored data from step 7, and the output is the organized evidence data. At this stage, the server organizes the relevant data and creates accurate verification materials.

[0105] (Application Example 1)

[0106] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0107] Effective means of improving home safety are needed for the safety management of consumer robots used in homes. However, existing technologies do not adequately monitor the safety of children and pets, and detecting and alerting to dangers tailored to the home environment is a particular challenge.

[0108] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0109] In this invention, the server includes a camera means for acquiring video footage, an image analysis means for identifying objects in the video footage, and a measuring means for calculating the distance to the identified objects. This enables effective safety monitoring and immediate alert generation by consumer robots within the home.

[0110] A "mobile device" is a self-propelled device used to move people or objects, and specifically includes electric scooters and bicycles.

[0111] A "filming device" is a device used to acquire video data, and generally refers to a camera.

[0112] An "image analysis device" is a device that processes video data acquired by a camera to identify specific objects.

[0113] A "measuring device" is a device used to calculate the distance to an identified object, and includes rangefinders and depth sensors.

[0114] An "evaluation device" is a device used to determine the likelihood and risk of a collision based on measured distance and speed data.

[0115] A "warning device" is a device that alerts users based on the risks determined by an evaluation device, and has the function of issuing warnings through voice or display.

[0116] A "recording device" is a device that saves video footage before and after a warning is issued, and it uses memory or storage to hold the data.

[0117] A "communication device" is a device used to transmit acquired data and video to an external server or cloud.

[0118] A "monitoring device" is a device used by consumer robots to monitor the safety of children and pets within a home environment, and has the function of detecting risks in specific situations.

[0119] The system for implementing this invention is primarily intended to monitor the safety of children and pets in a home environment using consumer robots. The system mainly consists of the following hardware and software.

[0120] First, the imaging device mounted on the robot acquires video data of the surrounding environment in real time. This imaging device generally refers to a camera, and for example, a Raspberry Pi Camera Module can be used. The captured video data is processed by an image analysis device to identify specific objects, such as furniture or moving objects (children or pets). Software libraries such as OpenCV are used for image analysis.

[0121] For identified objects, a measuring device calculates the distance. Depth sensors and similar devices are used for this purpose. Based on the measurement results, an evaluation device assesses the risk of collision and issues a warning if necessary. The warning is displayed on the robot's voice speaker and display.

[0122] If a warning is issued, the recording device saves video data before and after the warning. This data is then used as evidence to later review the situation. The saved data is temporarily stored in local storage and uploaded to an external cloud server by the communication device as soon as network connectivity is restored. For example, AWS® S3 is used as the cloud server.

[0123] A concrete example is a scenario where a robot moves around the home, and while a child is playing, the robot detects nearby furniture that might fall over and warns the child with a voice message. Such functions can help prevent accidents in the home.

[0124] An example of a prompt is, "Please explain the procedure for a robot to monitor its surroundings in the home and issue warnings when it detects danger. Please also explain how to save the recordings to the cloud." This prompt is used when inputting data into the AI ​​model.

[0125] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0126] Step 1:

[0127] The terminal uses a camera to capture the surrounding environment in real time and acquire video data. The input is video data from the camera, and the output is frame data of that video. These frames serve as basic data for analysis.

[0128] Step 2:

[0129] The terminal uses an image analysis device to process the input video data and recognize specific objects. Here, OpenCV is used to identify specific objects (e.g., children or pets). The identification result outputs the type of object and its coordinate information.

[0130] Step 3:

[0131] The terminal uses the coordinate information of an object obtained from an image analysis device as input and calculates the distance to the object using a measuring device. Depth sensor data is used in this calculation. The distance data is output and used for risk assessment.

[0132] Step 4:

[0133] The terminal inputs distance data from the measuring device and the current speed information of the moving object into the evaluation device to assess the collision risk. If a risk exceeding a certain threshold is detected, the risk is output.

[0134] Step 5:

[0135] If the evaluation device outputs a high-risk signal to the user, the warning device will issue a warning. This includes an audio alert and a message displayed on the screen to draw the user's attention.

[0136] Step 6:

[0137] When a warning is issued, the recording device becomes active and saves video footage for several seconds before and after the warning. In this case, the input is real-time video frames, and the output is the saved video clip.

[0138] Step 7:

[0139] The server uploads the recorded video data to the cloud server via a communication device as soon as network connectivity is restored. The cloud storage stores the video clips in an easily organized format.

[0140] Step 8:

[0141] Based on the stored video data, the server uses a generative AI model to construct generated output corresponding to specific scenarios and prompts when necessary, which can then be used for further analysis and decision-making.

[0142] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0143] This invention aims to improve safety more effectively in a safety management system for mobile devices such as electric scooters and bicycles by taking into account the user's emotional state. This system is centered around a smartphone and consists of the following main elements.

[0144] Camera methods and image analysis

[0145] The device uses the camera built into the smartphone to acquire real-time forward-facing video of the moving object. This video is then analyzed using image processing technology to identify objects on the road and perform object recognition. The recognized objects, along with distance measurement, are used for safety evaluation of the system.

[0146] Introducing an emotional engine

[0147] The device incorporates an emotion engine to assess the user's emotional state. This engine analyzes the user's facial expressions and voice via the smartphone's camera or other sensors to understand their emotions. For example, the emotion engine recognizes whether the user is tense or relaxed.

[0148] Emotion-based warning adjustment

[0149] The device automatically adjusts the content of alerts based on the emotional state recognized by the emotion engine. For example, if the user is relaxed, it will issue a stronger warning than usual to draw their attention. On the other hand, if the user is already stressed, it will adjust the warning to be more concise.

[0150] Recording and saving data

[0151] The device automatically records video when danger is detected, along with the user's emotional state at that time. The recorded video and emotional data are temporarily stored on the device and used later for analysis or as evidence.

[0152] Cloud storage and organization of data

[0153] Once a network connection is established, the device uploads the recorded data to a cloud server. The server then properly organizes the received data and prepares it for provision to insurance companies and related organizations as needed.

[0154] This configuration dramatically improves the safety of mobile devices and enables flexible responses that take into account changes in the user's emotions. As a result, accident prevention and rapid response in the event of an accident can be achieved.

[0155] The following describes the processing flow.

[0156] Step 1:

[0157] The device activates the smartphone's camera and acquires real-time images of the area in front. The camera is fixed in the direction of the moving object's movement, providing the latest visual information at all times.

[0158] Step 2:

[0159] The device inputs the acquired video into an image analysis algorithm to perform object recognition. This identifies vehicles, pedestrians, and obstacles on the road and determines the location of each object.

[0160] Step 3:

[0161] The device measures the distance to the identified object. This process uses depth estimation technology to calculate the distance from the camera position to the object.

[0162] Step 4:

[0163] The device uses an emotion engine to perform facial recognition and voice analysis to evaluate the user's emotional state. The user's emotional state is then classified as, for example, comfortable, tense, or anxious.

[0164] Step 5:

[0165] The device assesses the level of driving risk based on distance data and emotional state. A threshold is set, and if it determines that the risk of collision is high, the level of danger is assessed.

[0166] Step 6:

[0167] The device generates alerts based on the assessed level of danger and the user's emotions. Strong warnings are issued when the user is relaxed, while gentler warnings are given when the user is stressed.

[0168] Step 7:

[0169] The device records video footage before and after an alert is issued, along with data on the user's emotional state. This data can be used for later analysis or as evidence in the event of an accident.

[0170] Step 8:

[0171] Once a network connection is established, the server receives the video and emotional data transmitted from the terminals into the cloud and organizes it. If requested by insurance companies or related organizations, the data is prepared in the necessary format.

[0172] (Example 2)

[0173] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0174] In recent years, the increase in traffic accidents involving moving vehicles has become a social problem. One of the contributing factors is thought to be decreased driver attention and emotional changes. However, conventional safety management systems are mainly based on physical information and do not take into account the driver's emotional state. Therefore, in order to improve the safety of moving vehicles, it is necessary to take into account the emotional state of the user, but current technology does not adequately address this.

[0175] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0176] In this invention, the server includes a visual sensor means for acquiring video footage, an emotion sensor means for acquiring data for analyzing emotional states, and a warning means for adjusting and issuing warnings based on the evaluated risk and emotional state, in order to manage the safety of a moving object. This makes it possible to adjust the content of warnings to take into account the user's emotional state, in addition to evaluating the physical risk.

[0177] "Visual sensor means" refers to a device for acquiring images of the surroundings of a moving object, and typically refers to a device that collects visual information using a camera or the like.

[0178] "Image processing means" refers to technology or equipment for analyzing acquired video data and identifying and extracting specific objects.

[0179] "Measurement means" refers to devices or algorithms used to calculate the distance of an object from analyzed video data.

[0180] An "emotional sensor device" is a device used to acquire a user's emotional state, and typically refers to a device that uses a camera or microphone to collect facial expressions and sounds.

[0181] "Emotional analysis means" refers to technologies and algorithms that analyze a user's emotional state based on data obtained from emotional sensor means.

[0182] "Evaluation means" refers to technologies and systems for determining physical hazards based on data such as the speed of a moving object and its distance from an object.

[0183] "Warning measures" refer to devices and systems that issue appropriate warnings to drivers based on assessed risks and emotional states.

[0184] "Recording means" refers to devices or systems for recording and storing video footage and related data when a warning is issued.

[0185] "Communication means" refers to a network connection used to transmit recorded data to remote data storage.

[0186] "Memory device" refers to an internal storage device used to temporarily store recorded data when network connectivity is unavailable.

[0187] "Data organization methods" refer to technologies and systems that organize record data related to specific conditions and convert it into a format suitable for provision to relevant organizations.

[0188] This invention provides a system for managing the safety of mobile objects, offering a device that enhances safety more effectively by considering the user's emotional state. The central element of the system is a mobile information terminal such as a smartphone. The specific configuration and functions of the system are described below.

[0189] The device first uses the camera built into the smartphone as a visual sensor to acquire real-time video of the direction of travel of the moving object. This allows monitoring of the surrounding environment while driving. The acquired video data is analyzed by computer vision algorithms (e.g., OpenCV or TENSORFLOW®) as an image processing means to identify objects on the road (e.g., vehicles, pedestrians, bicycles, etc.).

[0190] The measurement method utilizes stereo vision and laser rangefinders to calculate the distance to the target object. This allows for the assessment of potential hazards along the path of a moving object. Furthermore, to consider the user's emotional state, the terminal incorporates emotion sensors that collect the user's facial expressions and voice through cameras and microphones. The emotion analysis method uses machine learning models to analyze this data and quickly infer the user's emotional state.

[0191] The evaluation system assesses the risk of collision based on distance data and moving object speed data. Based on this evaluation result and the results of sentiment analysis, the warning system automatically adjusts the warning content and issues an alert to the driver. A concise and understated warning can be issued to a stressed user, while a more emphasized warning can be issued to a relaxed user to draw their attention.

[0192] Furthermore, the device has recording capabilities to record and temporarily store video and related data when a warning is issued. If a network connection is available via communication means, this recorded data can be uploaded to a cloud server to prevent data loss.

[0193] The server organizes the uploaded data and prepares it for provision to relevant organizations and insurance companies. Because the data is converted into the appropriate format by the data organization tools, this contributes to rapid and effective risk management.

[0194] For example, if a user is riding an electric scooter and the device detects a child's bicycle ahead, while simultaneously analyzing the user's emotional state as "relaxed," an enhanced warning sound will be emitted. An example of a prompt message would be, "We want to develop a safety management system that analyzes the emotional state of a user while they are on the move and adjusts the warning content in real time."

[0195] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0196] Step 1:

[0197] The device uses the camera built into the smartphone to acquire real-time video of its surroundings. At this stage, the camera collects visual information as digital data. The input is a video stream from the camera, and the output is image data for each frame.

[0198] Step 2:

[0199] The terminal passes the acquired video data to a computer vision algorithm that uses it as an image processing tool. In this process, for example, OpenCV or TensorFlow is used to identify specific objects in the video. The input is the image data obtained in step 1, and the output is identification data including the position and type of the object.

[0200] Step 3:

[0201] The terminal measures the distance to an object using stereo vision or a laser rangefinder based on identification data. The input is the identification data from step 2, and the output is distance information to the object. This distance data is used for subsequent hazard assessment.

[0202] Step 4:

[0203] The device uses the smartphone's emotion sensors, namely the camera and microphone, to collect data on the user's facial expressions and voice. The input is audio and image data, and the output is raw data indicating the user's emotional state.

[0204] Step 5:

[0205] The device uses emotion analysis tools to analyze the raw data obtained in step 4 and uses a machine learning model to estimate the user's emotional state. The input is raw emotion data, and the output is the estimated emotional state (e.g., tense, relaxed).

[0206] Step 6:

[0207] The terminal uses distance data from the measurement device and velocity data of the moving object as input to evaluate the risk of collision using the evaluation device. The output is the risk assessment result, which shows the potential probability of collision numerically or as a category.

[0208] Step 7:

[0209] The terminal adjusts the warning content using warning mechanisms based on the aforementioned risk assessment results and sentiment analysis results, and issues an alert to the user. The input is the risk assessment results and sentiment state, and the output is an appropriately adjusted warning signal.

[0210] Step 8:

[0211] The terminal uses recording devices to record and store surrounding video and related data when a warning is issued. The input is the trigger event for the warning, and the output is the recorded data stored in storage.

[0212] Step 9:

[0213] The terminal uploads recorded data to the cloud server when a network becomes available via communication means. The input is the stored recorded data, and the output is the data securely stored on the cloud.

[0214] (Application Example 2)

[0215] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0216] Current autonomous driving systems do not incorporate safety management that takes into account the emotional state of the occupants, resulting in insufficient warnings and operational adjustments based on the occupants' psychological condition. Furthermore, there is a lack of technology to appropriately utilize the impact of changes in occupants' emotions on safety. Therefore, there is a need to analyze the occupants' emotional state in real time and optimize the operation of the autonomous driving system based on that analysis.

[0217] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0218] In this invention, the server includes an image acquisition means for acquiring video footage, a visual analysis means for identifying objects within the video footage, and an emotion adjustment means for evaluating the emotional state of the crew using an emotion analysis means and adjusting the content of the warning based on the emotional state. This makes it possible to issue warnings and adjust operations while taking the crew's emotions into consideration.

[0219] "Mobility devices" refer to mechanical devices used to transport people or goods, such as electric scooters, bicycles, and self-driving vehicles.

[0220] A "safety management system" is a system designed to improve safety in the operation of mobile devices, and it performs risk analysis and issues warnings.

[0221] An "image acquisition means" is a device that acquires video footage of the surroundings of a mobile device in real time and provides it for analysis in other systems.

[0222] "Visual analysis means" refers to technology that identifies objects within acquired video footage and analyzes that information to utilize it for safety management.

[0223] A "distance measuring means" is a method that calculates the distance between an object identified in the video and other objects, and provides data for evaluating the risk of collision.

[0224] An "evaluation tool" is the part that analyzes a situation based on multiple pieces of information and has the function of determining risks and countermeasures.

[0225] A "warning system" is a means of ensuring safety by providing users with appropriate warnings based on the assessed risks.

[0226] "Emotional analysis means" refers to technology that analyzes the emotional state of crew members from their facial expressions and voices, and reflects this in the operation of the system.

[0227] "Emotional adjustment mechanisms" are functions that dynamically adjust warnings and operating conditions by taking into account the emotional state of the crew, thereby supporting safe operation.

[0228] A "recording device" is a device used to record video and related data when a specific event occurs, for later analysis or use as evidence.

[0229] "Communication means" refers to communication technology used to transmit recorded data to a remote server for analysis and storage.

[0230] The system that realizes this invention mainly consists of a terminal mounted on a mobile device. The terminal combines image acquisition means, visual analysis means, distance measurement means, emotion analysis means, warning issuing means, recording means, and communication means to perform comprehensive safety management.

[0231] The terminal uses its built-in camera to acquire real-time images of the area ahead while the mobile device is in operation. This image data is input to a visual analysis system, which uses image processing technology (e.g., OpenCV) to identify objects around the mobile device and evaluate their positions and distances. Next, a distance measuring system calculates the distance to the identified objects and determines the urgency of a potential collision.

[0232] Meanwhile, the terminal acquires the occupant's emotional state and diagnoses their emotions through facial expressions and voice using emotion analysis tools. Emotion recognition software such as EmoVu is used for this emotion analysis. Depending on whether the occupant is relaxed or stressed, the warning system adjusts the content of the warning and provides appropriate feedback to the occupant.

[0233] In the event of a dangerous incident, the device records video footage immediately before and after the incident using recording devices and saves it chronologically. This data is uploaded to a cloud server via communication devices when the internet is available, allowing for later analysis and use as evidence.

[0234] As a concrete example of the above process, if the occupants feel uneasy while driving on the road, the terminal can reduce the vehicle's speed and display a visual alert to draw their attention. Conversely, if they are relaxed, the alert will be less pronounced, and the vehicle will continue operating at a normal speed.

[0235] An example of a prompt in the use of a generative AI model is as follows: "Design an application that uses a smartphone camera to analyze the emotions of passengers in an autonomous vehicle in real time and contribute to safe driving." This prompt allows for the design of a function that considers how the AI ​​will link the emotions of the occupants with the vehicle's operation.

[0236] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0237] Step 1:

[0238] The device captures video in front of it in real time using an image acquisition device. The input is video data from the camera, and the output is this video data being sent to the next analysis stage. Specifically, the device continuously acquires images at a rate of several frames per second.

[0239] Step 2:

[0240] The terminal identifies objects from input video data using visual analysis tools. The input is previously acquired video data, and the output is the identification of objects and their location information. In this step, the terminal uses libraries such as OpenCV to extract and identify the boundaries and features of objects.

[0241] Step 3:

[0242] The terminal calculates the distance to an object identified through distance measurement means. The input is the object's position information, the current position and velocity data of the moving device, and the output is the distance calculated based on this information. Specifically, measurements are performed using trigonometry or built-in sensors (e.g., LiDAR).

[0243] Step 4:

[0244] The terminal analyzes distance and speed data using an evaluation tool to assess the risk of collision. The input is calculated distance and speed information, and the output is the risk level (e.g., low, medium, high). The terminal uses this information to determine the urgency of the situation.

[0245] Step 5:

[0246] The device evaluates the user's emotions using emotion analysis tools. The input is the user's video and audio data, and the output is their emotional state (e.g., relaxed, stressed). Here, EmoVu is used to analyze changes in the user's facial muscles.

[0247] Step 6:

[0248] The device uses a warning system to issue warnings based on the assessed emotional state and risk of collision. The input is the emotional state and risk level, and the output is a tailored warning message. Specifically, the user is provided with audio and visual alerts.

[0249] Step 7:

[0250] If a hazard is detected, the device uses recording equipment to record video before and after the event. Input consists of video footage for a set period and a trigger for the hazardous event; output is the saved video clip. The recording is stored in local storage.

[0251] Step 8:

[0252] The terminal uploads recorded data to a cloud server via a communication method. The input is the saved video data, and the output is the completion of data transfer to the remote server. It checks the network connection and sends data if available.

[0253] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0254] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0255] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0256] [Second Embodiment]

[0257] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0258] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0259] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0260] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0261] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0262] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0263] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0264] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0265] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0266] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0267] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0268] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0269] This invention implements a mobile safety management system using a smartphone attached to a mobile device such as an electric scooter or bicycle. This system achieves the following functions by installing a specific application on the smartphone.

[0270] Camera methods and image analysis

[0271] The device uses a smartphone camera mounted on the front of the moving object to capture real-time video. The video is continuously captured, and an image analysis algorithm recognizes objects. This algorithm identifies specific objects by enclosing them in rectangles and makes decisions based on the traffic situation.

[0272] Distance measurement and evaluation

[0273] The device utilizes depth estimation technology to estimate the distance to recognized objects. Distance and velocity data are used to assess collision risk, and alerts are generated in high-risk scenarios.

[0274] Alert issuance

[0275] To alert the user, the device uses both audio alarms and screen displays. For example, if there is a vehicle rapidly approaching ahead, a warning sound will be emitted and a message such as "Caution! Obstacle ahead" will be displayed on the screen.

[0276] Recording and saving

[0277] The device automatically records video footage immediately before and after an alert is triggered, making it available as evidence of the accident. This recorded data is temporarily stored in the smartphone's internal storage.

[0278] Cloud storage of data

[0279] Once network connectivity is restored, the device will transfer the recorded data to a secure cloud server. The server will organize the received data and store it for easy access.

[0280] Providing data to insurance companies

[0281] If necessary, the server provides organized data to insurance companies and related organizations. The data contains information useful for accident investigation and is prepared to meet legal requirements.

[0282] By leveraging an existing smartphone, this system can reduce the introduction cost while enhancing the safety of moving objects. With its simple settings and operations, it provides a solution suitable for a wide range of users.

[0283] The following is an explanation of the processing flow.

[0284] Step 1:

[0285] The terminal activates the camera of the smartphone and acquires real-time video. The acquired video is saved as continuous frames to prepare for image analysis.

[0286] Step 2:

[0287] The terminal applies an image analysis algorithm to each captured frame to identify objects on the road. This algorithm recognizes specific objects (vehicles, pedestrians, obstacles) by surrounding them with rectangles.

[0288] Step 3:

[0289] The terminal performs distance estimation for the identified objects. Using depth estimation technology, it accurately measures the distance from the camera to the objects and updates it in real time.

[0290] Step 4:

[0291] The terminal evaluates the movement and collision potential of the objects using the measured distance and speed data. By setting a threshold value, if it is determined that there is a risk exceeding that value, the risk level is judged to be high.

[0292] Step 5:

[0293] When the terminal evaluates the risk level as high, it generates an alert. It sends an auditory alarm and a warning message on the screen to the user to prompt attention.

[0294] Step 6:

[0295] The device continuously records video from the moment an alert is issued, documenting the situation immediately before and after the event. This data is temporarily stored on the device.

[0296] Step 7:

[0297] The device uploads the recorded data to the cloud server as soon as it establishes a network connection. The data is transmitted along with a timestamp and identification information, and is stored for later reference.

[0298] Step 8:

[0299] The server receives and organizes the uploaded video data. If requested by insurance companies or related organizations, it prepares the data to be provided in a format suitable for verification.

[0300] (Example 1)

[0301] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0302] There is a need to provide an effective mobile safety management system that reduces the risk of traffic accidents and near misses and improves driver safety. However, existing systems are insufficient in real time object recognition, distance measurement, and risk assessment, and there are also challenges in the proper management and provision of video data in the event of an accident.

[0303] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0304] In this invention, the server includes imaging means for acquiring an image, analysis means for identifying an object in the image, and estimation means for calculating the distance to the identified object. Thereby, it becomes possible to accurately recognize the situation around the moving body during driving in real time and quickly evaluate the collision risk.

[0305] The "imaging means" is a device or function for acquiring an image of the surroundings of the moving body in real time.

[0306] The "analysis means" is a device or algorithm used to identify an object in the acquired image and specify its position and type.

[0307] The "estimation means" is a device or calculation method for calculating the distance to the object identified by the analysis means and deriving an accurate relative position.

[0308] The "evaluation means" is a device or process for evaluating the risk of collision based on the distance of the identified object and the speed data of the moving body.

[0309] The "warning means" is a device or method used to alert the operator based on the result obtained by the evaluation means.

[0310] The "recording means" is a device or mechanism for recording and storing the images before and after the warning means is triggered.

[0311] The "transmission means" is a device or technology for transferring the recorded video data to other devices or servers through a communication network.

[0312] The "storage medium" is a device or physical storage that can temporarily store the images recorded by the recording means.

[0313] The "management means" is a device or method for organizing the recorded data related to a specific event and storing and utilizing it appropriately.

[0314] The system for implementing this invention utilizes a smartphone as a terminal attached to a mobile device to manage traffic safety. This system is operated using specific hardware and software to perform several key functions.

[0315] The device utilizes the camera built into the smartphone as a means of capturing images, acquiring real-time video of the direction of movement of a moving object. This video data is processed by an image analysis algorithm as an analytical tool, identifying objects such as vehicles and pedestrians. By using, for example, a deep learning model for this analysis, specific objects can be identified with high accuracy.

[0316] Furthermore, the terminal uses estimation means to calculate the distance to the identified object. Distance estimation utilizes depth estimation technology using machine learning models, providing accurate distance information in real time. This allows the terminal to use evaluation means to assess the risk of collision with the object, and if a high level of risk is detected, it can use warning means to alert the user with an audio alarm or screen display.

[0317] Furthermore, the device can use recording means to automatically record video before and after an alert is issued and temporarily save it to the smartphone's storage medium. This data is then transferred to the server using transmission means after the network connection is restored. The server securely stores this recorded data, organizes it using management means as needed, and provides it to public institutions and insurance companies.

[0318] As a concrete example, suppose a user is delivering goods by bicycle and a pedestrian suddenly appears in front of them. The device instantly recognizes the pedestrian, estimates the distance, assesses the risk of collision, and issues a warning, allowing the user to take evasive action quickly.

[0319] An example of a prompt to input into a generative AI model is a question such as, "In a safety management system using a smartphone attached to an electric scooter, please explain the methods for recognizing objects moving in front and measuring their distance." By utilizing this prompt, it is possible to deepen the understanding of the system's mechanisms and performance through the generative AI model.

[0320] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0321] Step 1:

[0322] The device uses the smartphone's camera to acquire real-time video of the area in front of a moving object. The input is continuous image data from the camera, and the output is video data for analysis. In this process, the camera captures high-resolution video and saves it directly to memory, providing the data necessary for subsequent analysis steps.

[0323] Step 2:

[0324] The terminal processes the acquired video data using an analysis device and applies an image analysis algorithm to identify objects in the video. The input is the video data which is the output of step 1, and the output is data on the location and type of the identified objects. Specifically, it uses a machine learning model to recognize a particular object and draws a rectangle around its location.

[0325] Step 3:

[0326] The terminal uses estimation methods to calculate the distance to the identified object. The input is the object identification data from step 2, and the output is the estimated distance to the object. In this step, depth estimation is performed by combining image disparity information and algorithms to calculate the exact distance to the object.

[0327] Step 4:

[0328] The terminal uses an evaluation method to assess collision risk based on the calculated distance and moving object velocity data. The input is the distance and velocity data from step 3, and the output is the risk level evaluation result. Specifically, the program compares a threshold with the actual data and determines that the risk level is high if the probability of collision exceeds a certain level.

[0329] Step 5:

[0330] The terminal uses the warning means to issue an alert to the user based on the output of the evaluation means. The input is the evaluation result from step 4, and the output is an audio alarm and a warning message displayed on the screen to the user. In this case, a warning sound is emitted from the speaker and a specific warning message is displayed on the display.

[0331] Step 6:

[0332] The terminal records video before and after a warning is issued using a recording device and temporarily saves it to a local storage medium. The input is the video data from step 1, and the output is the recorded file data. The terminal selects and efficiently saves video from a certain period before and after the alert occurs.

[0333] Step 7:

[0334] The server, upon recovery of network connectivity, instructs the transmission method to transfer the recorded data to the cloud. The input is the temporary data stored in step 6, and the output is the data stored in the cloud. During this process, the data is encrypted to ensure the security of the transfer.

[0335] Step 8:

[0336] The server organizes the recorded data using management tools and provides the records to public institutions or insurance companies as needed. The input is the cloud-stored data from step 7, and the output is the organized evidence data. At this stage, the server organizes the relevant data and creates accurate verification materials.

[0337] (Application Example 1)

[0338] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0339] Effective means of improving home safety are needed for the safety management of consumer robots used in homes. However, existing technologies do not adequately monitor the safety of children and pets, and detecting and alerting to dangers tailored to the home environment is a particular challenge.

[0340] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0341] In this invention, the server includes a camera means for acquiring video footage, an image analysis means for identifying objects in the video footage, and a measuring means for calculating the distance to the identified objects. This enables effective safety monitoring and immediate alert generation by consumer robots within the home.

[0342] A "mobile device" is a self-propelled device used to move people or objects, and specifically includes electric scooters and bicycles.

[0343] A "filming device" is a device used to acquire video data, and generally refers to a camera.

[0344] An "image analysis device" is a device that processes video data acquired by a camera to identify specific objects.

[0345] A "measuring device" is a device used to calculate the distance to an identified object, and includes rangefinders and depth sensors.

[0346] An "evaluation device" is a device used to determine the likelihood and risk of a collision based on measured distance and speed data.

[0347] A "warning device" is a device that alerts users based on the risks determined by an evaluation device, and has the function of issuing warnings through voice or display.

[0348] A "recording device" is a device that saves video footage before and after a warning is issued, and it uses memory or storage to hold the data.

[0349] A "communication device" is a device used to transmit acquired data and video to an external server or cloud.

[0350] A "monitoring device" is a device used by consumer robots to monitor the safety of children and pets within a home environment, and has the function of detecting risks in specific situations.

[0351] The system for implementing this invention is primarily intended to monitor the safety of children and pets in a home environment using consumer robots. The system mainly consists of the following hardware and software.

[0352] First, the imaging device mounted on the robot acquires video data of the surrounding environment in real time. This imaging device generally refers to a camera, and for example, a Raspberry Pi Camera Module can be used. The captured video data is processed by an image analysis device to identify specific objects, such as furniture or moving objects (children or pets). Software libraries such as OpenCV are used for image analysis.

[0353] For identified objects, a measuring device calculates the distance. Depth sensors and similar devices are used for this purpose. Based on the measurement results, an evaluation device assesses the risk of collision and issues a warning if necessary. The warning is displayed on the robot's voice speaker and display.

[0354] If a warning is issued, the recording device saves video data before and after the warning. This data is then used as evidence to later review the situation. The saved data is temporarily stored in local storage and uploaded to an external cloud server by the communication device as soon as network connectivity is restored. For example, AWS S3 is used as the cloud server.

[0355] A concrete example is a scenario where a robot moves around the home, and while a child is playing, the robot detects nearby furniture that might fall over and warns the child with a voice message. Such functions can help prevent accidents in the home.

[0356] An example of a prompt is, "Please explain the procedure for a robot to monitor its surroundings in the home and issue warnings when it detects danger. Please also explain how to save the recordings to the cloud." This prompt is used when inputting data into the AI ​​model.

[0357] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0358] Step 1:

[0359] The terminal uses a camera to capture the surrounding environment in real time and acquire video data. The input is video data from the camera, and the output is frame data of that video. These frames serve as basic data for analysis.

[0360] Step 2:

[0361] The terminal uses an image analysis device to process the input video data and recognize specific objects. Here, OpenCV is used to identify specific objects (e.g., children or pets). The identification result outputs the type of object and its coordinate information.

[0362] Step 3:

[0363] The terminal uses the coordinate information of an object obtained from an image analysis device as input and calculates the distance to the object using a measuring device. Depth sensor data is used in this calculation. The distance data is output and used for risk assessment.

[0364] Step 4:

[0365] The terminal inputs distance data from the measuring device and the current speed information of the moving object into the evaluation device to assess the collision risk. If a risk exceeding a certain threshold is detected, the risk is output.

[0366] Step 5:

[0367] If the evaluation device outputs a high-risk signal to the user, the warning device will issue a warning. This includes an audio alert and a message displayed on the screen to draw the user's attention.

[0368] Step 6:

[0369] When a warning is issued, the recording device becomes active and saves video footage for several seconds before and after the warning. In this case, the input is real-time video frames, and the output is the saved video clip.

[0370] Step 7:

[0371] The server uploads the recorded video data to the cloud server via a communication device as soon as network connectivity is restored. The cloud storage stores the video clips in an easily organized format.

[0372] Step 8:

[0373] Based on the stored video data, the server uses a generative AI model to construct generated output corresponding to specific scenarios and prompts when necessary, which can then be used for further analysis and decision-making.

[0374] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0375] This invention aims to improve safety more effectively in a safety management system for mobile devices such as electric scooters and bicycles by taking into account the user's emotional state. This system is centered around a smartphone and consists of the following main elements.

[0376] Camera methods and image analysis

[0377] The device uses the camera built into the smartphone to acquire real-time forward-facing video of the moving object. This video is then analyzed using image processing technology to identify objects on the road and perform object recognition. The recognized objects, along with distance measurement, are used for safety evaluation of the system.

[0378] Introducing an emotional engine

[0379] The device incorporates an emotion engine to assess the user's emotional state. This engine analyzes the user's facial expressions and voice via the smartphone's camera or other sensors to understand their emotions. For example, the emotion engine recognizes whether the user is tense or relaxed.

[0380] Emotion-based warning adjustment

[0381] The device automatically adjusts the content of alerts based on the emotional state recognized by the emotion engine. For example, if the user is relaxed, it will issue a stronger warning than usual to draw their attention. On the other hand, if the user is already stressed, it will adjust the warning to be more concise.

[0382] Recording and saving data

[0383] The device automatically records video when danger is detected, along with the user's emotional state at that time. The recorded video and emotional data are temporarily stored on the device and used later for analysis or as evidence.

[0384] Cloud storage and organization of data

[0385] Once a network connection is established, the device uploads the recorded data to a cloud server. The server then properly organizes the received data and prepares it for provision to insurance companies and related organizations as needed.

[0386] This configuration dramatically improves the safety of mobile devices and enables flexible responses that take into account changes in the user's emotions. As a result, accident prevention and rapid response in the event of an accident can be achieved.

[0387] The following describes the processing flow.

[0388] Step 1:

[0389] The device activates the smartphone's camera and acquires real-time images of the area in front. The camera is fixed in the direction of the moving object's movement, providing the latest visual information at all times.

[0390] Step 2:

[0391] The device inputs the acquired video into an image analysis algorithm to perform object recognition. This identifies vehicles, pedestrians, and obstacles on the road and determines the location of each object.

[0392] Step 3:

[0393] The device measures the distance to the identified object. This process uses depth estimation technology to calculate the distance from the camera position to the object.

[0394] Step 4:

[0395] The device uses an emotion engine to perform facial recognition and voice analysis to evaluate the user's emotional state. The user's emotional state is then classified as, for example, comfortable, tense, or anxious.

[0396] Step 5:

[0397] The device assesses the level of driving risk based on distance data and emotional state. A threshold is set, and if it determines that the risk of collision is high, the level of danger is assessed.

[0398] Step 6:

[0399] The device generates alerts based on the assessed level of danger and the user's emotions. Strong warnings are issued when the user is relaxed, while gentler warnings are given when the user is stressed.

[0400] Step 7:

[0401] The device records video footage before and after an alert is issued, along with data on the user's emotional state. This data can be used for later analysis or as evidence in the event of an accident.

[0402] Step 8:

[0403] Once a network connection is established, the server receives the video and emotional data transmitted from the terminals into the cloud and organizes it. If requested by insurance companies or related organizations, the data is prepared in the necessary format.

[0404] (Example 2)

[0405] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0406] In recent years, the increase in traffic accidents involving moving vehicles has become a social problem. One of the contributing factors is thought to be decreased driver attention and emotional changes. However, conventional safety management systems are mainly based on physical information and do not take into account the driver's emotional state. Therefore, in order to improve the safety of moving vehicles, it is necessary to take into account the emotional state of the user, but current technology does not adequately address this.

[0407] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0408] In this invention, the server includes a visual sensor means for acquiring video footage, an emotion sensor means for acquiring data for analyzing emotional states, and a warning means for adjusting and issuing warnings based on the evaluated risk and emotional state, in order to manage the safety of a moving object. This makes it possible to adjust the content of warnings to take into account the user's emotional state, in addition to evaluating the physical risk.

[0409] "Visual sensor means" refers to a device for acquiring images of the surroundings of a moving object, and typically refers to a device that collects visual information using a camera or the like.

[0410] "Image processing means" refers to technology or equipment for analyzing acquired video data and identifying and extracting specific objects.

[0411] "Measurement means" refers to devices or algorithms used to calculate the distance of an object from analyzed video data.

[0412] An "emotional sensor device" is a device used to acquire a user's emotional state, and typically refers to a device that uses a camera or microphone to collect facial expressions and sounds.

[0413] "Emotional analysis means" refers to technologies and algorithms that analyze a user's emotional state based on data obtained from emotional sensor means.

[0414] "Evaluation means" refers to technologies and systems for determining physical hazards based on data such as the speed of a moving object and its distance from an object.

[0415] "Warning measures" refer to devices and systems that issue appropriate warnings to drivers based on assessed risks and emotional states.

[0416] "Recording means" refers to devices or systems for recording and storing video footage and related data when a warning is issued.

[0417] "Communication means" refers to a network connection used to transmit recorded data to remote data storage.

[0418] "Memory device" refers to an internal storage device used to temporarily store recorded data when network connectivity is unavailable.

[0419] "Data organization methods" refer to technologies and systems that organize record data related to specific conditions and convert it into a format suitable for provision to relevant organizations.

[0420] This invention provides a system for managing the safety of mobile objects, offering a device that enhances safety more effectively by considering the user's emotional state. The central element of the system is a mobile information terminal such as a smartphone. The specific configuration and functions of the system are described below.

[0421] The device first uses the camera built into the smartphone as a visual sensor to acquire real-time video of the direction of travel of the moving object. This allows monitoring of the surrounding environment while driving. The acquired video data is analyzed by computer vision algorithms (e.g., OpenCV or TensorFlow), which are image processing tools, to identify objects on the road (e.g., vehicles, pedestrians, bicycles, etc.).

[0422] The measurement method utilizes stereo vision and laser rangefinders to calculate the distance to the target object. This allows for the assessment of potential hazards along the path of a moving object. Furthermore, to consider the user's emotional state, the terminal incorporates emotion sensors that collect the user's facial expressions and voice through cameras and microphones. The emotion analysis method uses machine learning models to analyze this data and quickly infer the user's emotional state.

[0423] The evaluation system assesses the risk of collision based on distance data and moving object speed data. Based on this evaluation result and the results of sentiment analysis, the warning system automatically adjusts the warning content and issues an alert to the driver. A concise and understated warning can be issued to a stressed user, while a more emphasized warning can be issued to a relaxed user to draw their attention.

[0424] Furthermore, the device has recording capabilities to record and temporarily store video and related data when a warning is issued. If a network connection is available via communication means, this recorded data can be uploaded to a cloud server to prevent data loss.

[0425] The server organizes the uploaded data and prepares it for provision to relevant organizations and insurance companies. Because the data is converted into the appropriate format by the data organization tools, this contributes to rapid and effective risk management.

[0426] For example, if a user is riding an electric scooter and the device detects a child's bicycle ahead, while simultaneously analyzing the user's emotional state as "relaxed," an enhanced warning sound will be emitted. An example of a prompt message would be, "We want to develop a safety management system that analyzes the emotional state of a user while they are on the move and adjusts the warning content in real time."

[0427] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0428] Step 1:

[0429] The device uses the camera built into the smartphone to acquire real-time video of its surroundings. At this stage, the camera collects visual information as digital data. The input is a video stream from the camera, and the output is image data for each frame.

[0430] Step 2:

[0431] The terminal passes the acquired video data to a computer vision algorithm that uses it as an image processing tool. In this process, for example, OpenCV or TensorFlow is used to identify specific objects in the video. The input is the image data obtained in step 1, and the output is identification data including the position and type of the object.

[0432] Step 3:

[0433] The terminal measures the distance to an object using stereo vision or a laser rangefinder based on identification data. The input is the identification data from step 2, and the output is distance information to the object. This distance data is used for subsequent hazard assessment.

[0434] Step 4:

[0435] The device uses the smartphone's emotion sensors, namely the camera and microphone, to collect data on the user's facial expressions and voice. The input is audio and image data, and the output is raw data indicating the user's emotional state.

[0436] Step 5:

[0437] The device uses emotion analysis tools to analyze the raw data obtained in step 4 and uses a machine learning model to estimate the user's emotional state. The input is raw emotion data, and the output is the estimated emotional state (e.g., tense, relaxed).

[0438] Step 6:

[0439] The terminal uses distance data from the measurement device and velocity data of the moving object as input to evaluate the risk of collision using the evaluation device. The output is the risk assessment result, which shows the potential probability of collision numerically or as a category.

[0440] Step 7:

[0441] The terminal adjusts the warning content using warning mechanisms based on the aforementioned risk assessment results and sentiment analysis results, and issues an alert to the user. The input is the risk assessment results and sentiment state, and the output is an appropriately adjusted warning signal.

[0442] Step 8:

[0443] The terminal uses recording devices to record and store surrounding video and related data when a warning is issued. The input is the trigger event for the warning, and the output is the recorded data stored in storage.

[0444] Step 9:

[0445] The terminal uploads recorded data to the cloud server when a network becomes available via communication means. The input is the stored recorded data, and the output is the data securely stored on the cloud.

[0446] (Application Example 2)

[0447] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0448] Current autonomous driving systems do not incorporate safety management that takes into account the emotional state of the occupants, resulting in insufficient warnings and operational adjustments based on the occupants' psychological condition. Furthermore, there is a lack of technology to appropriately utilize the impact of changes in occupants' emotions on safety. Therefore, there is a need to analyze the occupants' emotional state in real time and optimize the operation of the autonomous driving system based on that analysis.

[0449] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0450] In this invention, the server includes an image acquisition means for acquiring video footage, a visual analysis means for identifying objects within the video footage, and an emotion adjustment means for evaluating the emotional state of the crew using an emotion analysis means and adjusting the content of the warning based on the emotional state. This makes it possible to issue warnings and adjust operations while taking the crew's emotions into consideration.

[0451] "Mobility devices" refer to mechanical devices used to transport people or goods, such as electric scooters, bicycles, and self-driving vehicles.

[0452] A "safety management system" is a system designed to improve safety in the operation of mobile devices, and it performs risk analysis and issues warnings.

[0453] An "image acquisition means" is a device that acquires video footage of the surroundings of a mobile device in real time and provides it for analysis in other systems.

[0454] "Visual analysis means" refers to technology that identifies objects within acquired video footage and analyzes that information to utilize it for safety management.

[0455] A "distance measuring means" is a method that calculates the distance between an object identified in the video and other objects, and provides data for evaluating the risk of collision.

[0456] An "evaluation tool" is the part that analyzes a situation based on multiple pieces of information and has the function of determining risks and countermeasures.

[0457] A "warning system" is a means of ensuring safety by providing users with appropriate warnings based on the assessed risks.

[0458] "Emotional analysis means" refers to technology that analyzes the emotional state of crew members from their facial expressions and voices, and reflects this in the operation of the system.

[0459] "Emotional adjustment mechanisms" are functions that dynamically adjust warnings and operating conditions by taking into account the emotional state of the crew, thereby supporting safe operation.

[0460] A "recording device" is a device used to record video and related data when a specific event occurs, for later analysis or use as evidence.

[0461] "Communication means" refers to communication technology used to transmit recorded data to a remote server for analysis and storage.

[0462] The system that realizes this invention mainly consists of a terminal mounted on a mobile device. The terminal combines image acquisition means, visual analysis means, distance measurement means, emotion analysis means, warning issuing means, recording means, and communication means to perform comprehensive safety management.

[0463] The terminal uses its built-in camera to acquire real-time images of the area ahead while the mobile device is in operation. This image data is input to a visual analysis system, which uses image processing technology (e.g., OpenCV) to identify objects around the mobile device and evaluate their positions and distances. Next, a distance measuring system calculates the distance to the identified objects and determines the urgency of a potential collision.

[0464] Meanwhile, the terminal acquires the occupant's emotional state and diagnoses their emotions through facial expressions and voice using emotion analysis tools. Emotion recognition software such as EmoVu is used for this emotion analysis. Depending on whether the occupant is relaxed or stressed, the warning system adjusts the content of the warning and provides appropriate feedback to the occupant.

[0465] In the event of a dangerous incident, the device records video footage immediately before and after the incident using recording devices and saves it chronologically. This data is uploaded to a cloud server via communication devices when the internet is available, allowing for later analysis and use as evidence.

[0466] As a concrete example of the above process, if the occupants feel uneasy while driving on the road, the terminal can reduce the vehicle's speed and display a visual alert to draw their attention. Conversely, if they are relaxed, the alert will be less pronounced, and the vehicle will continue operating at a normal speed.

[0467] An example of a prompt in the use of a generative AI model is as follows: "Design an application that uses a smartphone camera to analyze the emotions of passengers in an autonomous vehicle in real time and contribute to safe driving." This prompt allows for the design of a function that considers how the AI ​​will link the emotions of the occupants with the vehicle's operation.

[0468] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0469] Step 1:

[0470] The device captures video in front of it in real time using an image acquisition device. The input is video data from the camera, and the output is this video data being sent to the next analysis stage. Specifically, the device continuously acquires images at a rate of several frames per second.

[0471] Step 2:

[0472] The terminal identifies objects from input video data using visual analysis tools. The input is previously acquired video data, and the output is the identification of objects and their location information. In this step, the terminal uses libraries such as OpenCV to extract and identify the boundaries and features of objects.

[0473] Step 3:

[0474] The terminal calculates the distance to an object identified through distance measurement means. The input is the object's position information, the current position and velocity data of the moving device, and the output is the distance calculated based on this information. Specifically, measurements are performed using trigonometry or built-in sensors (e.g., LiDAR).

[0475] Step 4:

[0476] The terminal analyzes distance and speed data using an evaluation tool to assess the risk of collision. The input is calculated distance and speed information, and the output is the risk level (e.g., low, medium, high). The terminal uses this information to determine the urgency of the situation.

[0477] Step 5:

[0478] The device evaluates the user's emotions using emotion analysis tools. The input is the user's video and audio data, and the output is their emotional state (e.g., relaxed, stressed). Here, EmoVu is used to analyze changes in the user's facial muscles.

[0479] Step 6:

[0480] The device uses a warning system to issue warnings based on the assessed emotional state and risk of collision. The input is the emotional state and risk level, and the output is a tailored warning message. Specifically, the user is provided with audio and visual alerts.

[0481] Step 7:

[0482] If a hazard is detected, the device uses recording equipment to record video before and after the event. Input consists of video footage for a set period and a trigger for the hazardous event; output is the saved video clip. The recording is stored in local storage.

[0483] Step 8:

[0484] The terminal uploads recorded data to a cloud server via a communication method. The input is the saved video data, and the output is the completion of data transfer to the remote server. It checks the network connection and sends data if available.

[0485] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0486] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0487] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0488] [Third Embodiment]

[0489] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0490] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0491] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0492] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0493] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0494] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0495] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0496] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0497] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0498] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0499] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0500] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0501] This invention provides a mobile safety management system that uses a smartphone attached to a mobile device such as an electric scooter or bicycle. This system achieves the following functions by installing a specific application on the smartphone.

[0502] Camera methods and image analysis

[0503] The device uses a smartphone camera mounted on the front of the moving object to capture real-time video. The video is continuously captured, and an image analysis algorithm recognizes objects. This algorithm identifies specific objects by enclosing them in rectangles and makes decisions based on the traffic situation.

[0504] Distance measurement and evaluation

[0505] The device utilizes depth estimation technology to estimate the distance to recognized objects. Distance and velocity data are used to assess collision risk, and alerts are generated in high-risk scenarios.

[0506] Alert issuance

[0507] To alert the user, the device uses both audio alarms and screen displays. For example, if there is a vehicle rapidly approaching ahead, a warning sound will be emitted and a message such as "Caution! Obstacle ahead" will be displayed on the screen.

[0508] Recording and saving

[0509] The device automatically records video footage immediately before and after an alert is triggered, making it available as evidence of the accident. This recorded data is temporarily stored in the smartphone's internal storage.

[0510] Cloud storage of data

[0511] Once network connectivity is restored, the device will transfer the recorded data to a secure cloud server. The server will organize the received data and store it for easy access.

[0512] Providing data to insurance companies

[0513] If necessary, the server provides organized data to insurance companies and related organizations. The data contains information useful for accident investigation and is prepared to meet legal requirements.

[0514] This system utilizes existing smartphones, keeping implementation costs down while effectively improving the safety of mobile devices. Its ease of setup and operation makes it a suitable solution for a wide range of users.

[0515] The following describes the processing flow.

[0516] Step 1:

[0517] The device activates the smartphone's camera and acquires video in real time. The acquired video is saved as a series of frames and prepared for image analysis.

[0518] Step 2:

[0519] The device applies an image analysis algorithm to each captured frame to identify objects on the road. This algorithm recognizes specific objects (vehicles, pedestrians, obstacles) by enclosing them in a rectangle.

[0520] Step 3:

[0521] The device performs distance estimation for identified objects. Using depth estimation technology, it accurately measures the distance from the camera to the object and updates this information in real time.

[0522] Step 4:

[0523] The device uses measured distance and velocity data to assess the movement of objects and the likelihood of collision. A threshold is set, and if it determines that there is a risk exceeding that value, it is classified as high risk.

[0524] Step 5:

[0525] The device generates an alert if it is assessed as high-risk. It sends an audible alarm and an on-screen warning message to the user to draw their attention.

[0526] Step 6:

[0527] The device continuously records video from the moment an alert is issued, documenting the situation immediately before and after the event. This data is temporarily stored on the device.

[0528] Step 7:

[0529] The device uploads the recorded data to the cloud server as soon as it establishes a network connection. The data is transmitted along with a timestamp and identification information, and is stored for later reference.

[0530] Step 8:

[0531] The server receives and organizes the uploaded video data. If requested by insurance companies or related organizations, it prepares the data to be provided in a format suitable for verification.

[0532] (Example 1)

[0533] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0534] There is a need to provide an effective mobile safety management system that reduces the risk of traffic accidents and near misses and improves driver safety. However, existing systems are insufficient in real time object recognition, distance measurement, and risk assessment, and there are also challenges in the proper management and provision of video data in the event of an accident.

[0535] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0536] In this invention, the server includes a shooting means for acquiring video footage, an analysis means for identifying objects in the video footage, and an estimation means for calculating the distance to the identified objects. This makes it possible to accurately recognize the situation around a moving vehicle in real time and to quickly assess the risk of collision.

[0537] "Shooting means" refers to a device or function for acquiring images of the surroundings of a moving object in real time.

[0538] "Analysis means" refers to a device or algorithm used to identify objects in acquired video footage and determine their location and type.

[0539] "Estimation means" refers to a device or calculation method that calculates the distance to an object identified by an analysis means and derives an accurate relative position.

[0540] "Evaluation means" refers to a device or process that evaluates the risk of collision based on the distance of an identified object and the velocity data of a moving object.

[0541] "Warning means" refers to a device or method used to alert the operator based on the results obtained by the evaluation means.

[0542] "Recording means" refers to a device or mechanism for recording and storing video footage before and after a warning signal is issued.

[0543] "Transmission means" refers to a device or technology for transferring recorded video data to other devices or servers via a communication network.

[0544] A "storage medium" is a device or physical storage capable of temporarily storing video footage recorded by a recording means.

[0545] "Management means" refers to a device or method for organizing, appropriately storing, and utilizing recorded data related to a specific event.

[0546] The system for implementing this invention utilizes a smartphone as a terminal attached to a mobile device to manage traffic safety. This system is operated using specific hardware and software to perform several key functions.

[0547] The device utilizes the camera built into the smartphone as a means of capturing images, acquiring real-time video of the direction of movement of a moving object. This video data is processed by an image analysis algorithm as an analytical tool, identifying objects such as vehicles and pedestrians. By using, for example, a deep learning model for this analysis, specific objects can be identified with high accuracy.

[0548] Furthermore, the terminal uses estimation means to calculate the distance to the identified object. Distance estimation utilizes depth estimation technology using machine learning models, providing accurate distance information in real time. This allows the terminal to use evaluation means to assess the risk of collision with the object, and if a high level of risk is detected, it can use warning means to alert the user with an audio alarm or screen display.

[0549] Furthermore, the device can use recording means to automatically record video before and after an alert is issued and temporarily save it to the smartphone's storage medium. This data is then transferred to the server using transmission means after the network connection is restored. The server securely stores this recorded data, organizes it using management means as needed, and provides it to public institutions and insurance companies.

[0550] As a concrete example, suppose a user is delivering goods by bicycle and a pedestrian suddenly appears in front of them. The device instantly recognizes the pedestrian, estimates the distance, assesses the risk of collision, and issues a warning, allowing the user to take evasive action quickly.

[0551] An example of a prompt to input into a generative AI model is a question such as, "In a safety management system using a smartphone attached to an electric scooter, please explain the methods for recognizing objects moving in front and measuring their distance." By utilizing this prompt, it is possible to deepen the understanding of the system's mechanisms and performance through the generative AI model.

[0552] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0553] Step 1:

[0554] The device uses the smartphone's camera to acquire real-time video of the area in front of a moving object. The input is continuous image data from the camera, and the output is video data for analysis. In this process, the camera captures high-resolution video and saves it directly to memory, providing the data necessary for subsequent analysis steps.

[0555] Step 2:

[0556] The terminal processes the acquired video data using an analysis device and applies an image analysis algorithm to identify objects in the video. The input is the video data which is the output of step 1, and the output is data on the location and type of the identified objects. Specifically, it uses a machine learning model to recognize a particular object and draws a rectangle around its location.

[0557] Step 3:

[0558] The terminal uses estimation methods to calculate the distance to the identified object. The input is the object identification data from step 2, and the output is the estimated distance to the object. In this step, depth estimation is performed by combining image disparity information and algorithms to calculate the exact distance to the object.

[0559] Step 4:

[0560] The terminal uses an evaluation method to assess collision risk based on the calculated distance and moving object velocity data. The input is the distance and velocity data from step 3, and the output is the risk level evaluation result. Specifically, the program compares a threshold with the actual data and determines that the risk level is high if the probability of collision exceeds a certain level.

[0561] Step 5:

[0562] The terminal uses the warning means to issue an alert to the user based on the output of the evaluation means. The input is the evaluation result from step 4, and the output is an audio alarm and a warning message displayed on the screen to the user. In this case, a warning sound is emitted from the speaker and a specific warning message is displayed on the display.

[0563] Step 6:

[0564] The terminal records video before and after a warning is issued using a recording device and temporarily saves it to a local storage medium. The input is the video data from step 1, and the output is the recorded file data. The terminal selects and efficiently saves video from a certain period before and after the alert occurs.

[0565] Step 7:

[0566] The server, upon recovery of network connectivity, instructs the transmission method to transfer the recorded data to the cloud. The input is the temporary data stored in step 6, and the output is the data stored in the cloud. During this process, the data is encrypted to ensure the security of the transfer.

[0567] Step 8:

[0568] The server organizes the recorded data using management tools and provides the records to public institutions or insurance companies as needed. The input is the cloud-stored data from step 7, and the output is the organized evidence data. At this stage, the server organizes the relevant data and creates accurate verification materials.

[0569] (Application Example 1)

[0570] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] Effective means of improving home safety are needed for the safety management of consumer robots used in homes. However, existing technologies do not adequately monitor the safety of children and pets, and detecting and alerting to dangers tailored to the home environment is a particular challenge.

[0572] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0573] In this invention, the server includes a camera means for acquiring video footage, an image analysis means for identifying objects in the video footage, and a measuring means for calculating the distance to the identified objects. This enables effective safety monitoring and immediate alert generation by consumer robots within the home.

[0574] A "mobile device" is a self-propelled device used to move people or objects, and specifically includes electric scooters and bicycles.

[0575] A "filming device" is a device used to acquire video data, and generally refers to a camera.

[0576] An "image analysis device" is a device that processes video data acquired by a camera to identify specific objects.

[0577] A "measuring device" is a device used to calculate the distance to an identified object, and includes rangefinders and depth sensors.

[0578] An "evaluation device" is a device used to determine the likelihood and risk of a collision based on measured distance and speed data.

[0579] A "warning device" is a device that alerts users based on the risks determined by an evaluation device, and has the function of issuing warnings through voice or display.

[0580] A "recording device" is a device that saves video footage before and after a warning is issued, and it uses memory or storage to hold the data.

[0581] A "communication device" is a device used to transmit acquired data and video to an external server or cloud.

[0582] A "monitoring device" is a device used by consumer robots to monitor the safety of children and pets within a home environment, and has the function of detecting risks in specific situations.

[0583] The system for implementing this invention is primarily intended to monitor the safety of children and pets in a home environment using consumer robots. The system mainly consists of the following hardware and software.

[0584] First, the imaging device mounted on the robot acquires video data of the surrounding environment in real time. This imaging device generally refers to a camera, and for example, a Raspberry Pi Camera Module can be used. The captured video data is processed by an image analysis device to identify specific objects, such as furniture or moving objects (children or pets). Software libraries such as OpenCV are used for image analysis.

[0585] For identified objects, a measuring device calculates the distance. Depth sensors and similar devices are used for this purpose. Based on the measurement results, an evaluation device assesses the risk of collision and issues a warning if necessary. The warning is displayed on the robot's voice speaker and display.

[0586] If a warning is issued, the recording device saves video data before and after the warning. This data is then used as evidence to later review the situation. The saved data is temporarily stored in local storage and uploaded to an external cloud server by the communication device as soon as network connectivity is restored. For example, AWS S3 is used as the cloud server.

[0587] A concrete example is a scenario where a robot moves around the home, and while a child is playing, the robot detects nearby furniture that might fall over and warns the child with a voice message. Such functions can help prevent accidents in the home.

[0588] An example of a prompt is, "Please explain the procedure for a robot to monitor its surroundings in the home and issue warnings when it detects danger. Please also explain how to save the recordings to the cloud." This prompt is used when inputting data into the AI ​​model.

[0589] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0590] Step 1:

[0591] The terminal uses a camera to capture the surrounding environment in real time and acquire video data. The input is video data from the camera, and the output is frame data of that video. These frames serve as basic data for analysis.

[0592] Step 2:

[0593] The terminal uses an image analysis device to process the input video data and recognize specific objects. Here, OpenCV is used to identify specific objects (e.g., children or pets). The identification result outputs the type of object and its coordinate information.

[0594] Step 3:

[0595] The terminal uses the coordinate information of an object obtained from an image analysis device as input and calculates the distance to the object using a measuring device. Depth sensor data is used in this calculation. The distance data is output and used for risk assessment.

[0596] Step 4:

[0597] The terminal inputs distance data from the measuring device and the current speed information of the moving object into the evaluation device to assess the collision risk. If a risk exceeding a certain threshold is detected, the risk is output.

[0598] Step 5:

[0599] If the evaluation device outputs a high-risk signal to the user, the warning device will issue a warning. This includes an audio alert and a message displayed on the screen to draw the user's attention.

[0600] Step 6:

[0601] When a warning is issued, the recording device becomes active and saves video footage for several seconds before and after the warning. In this case, the input is real-time video frames, and the output is the saved video clip.

[0602] Step 7:

[0603] The server uploads the recorded video data to the cloud server via a communication device as soon as network connectivity is restored. The cloud storage stores the video clips in an easily organized format.

[0604] Step 8:

[0605] Based on the stored video data, the server uses a generative AI model to construct generated output corresponding to specific scenarios and prompts when necessary, which can then be used for further analysis and decision-making.

[0606] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0607] This invention aims to improve safety more effectively in a safety management system for mobile devices such as electric scooters and bicycles by taking into account the user's emotional state. This system is centered around a smartphone and consists of the following main elements.

[0608] Camera methods and image analysis

[0609] The device uses the camera built into the smartphone to acquire real-time forward-facing video of the moving object. This video is then analyzed using image processing technology to identify objects on the road and perform object recognition. The recognized objects, along with distance measurement, are used for safety evaluation of the system.

[0610] Introducing an emotional engine

[0611] The device incorporates an emotion engine to assess the user's emotional state. This engine analyzes the user's facial expressions and voice via the smartphone's camera or other sensors to understand their emotions. For example, the emotion engine recognizes whether the user is tense or relaxed.

[0612] Emotion-based warning adjustment

[0613] The device automatically adjusts the content of alerts based on the emotional state recognized by the emotion engine. For example, if the user is relaxed, it will issue a stronger warning than usual to draw their attention. On the other hand, if the user is already stressed, it will adjust the warning to be more concise.

[0614] Recording and saving data

[0615] The device automatically records video when danger is detected, along with the user's emotional state at that time. The recorded video and emotional data are temporarily stored on the device and used later for analysis or as evidence.

[0616] Cloud storage and organization of data

[0617] Once a network connection is established, the device uploads the recorded data to a cloud server. The server then properly organizes the received data and prepares it for provision to insurance companies and related organizations as needed.

[0618] This configuration dramatically improves the safety of mobile devices and enables flexible responses that take into account changes in the user's emotions. As a result, accident prevention and rapid response in the event of an accident can be achieved.

[0619] The following describes the processing flow.

[0620] Step 1:

[0621] The device activates the smartphone's camera and acquires real-time images of the area in front. The camera is fixed in the direction of the moving object's movement, providing the latest visual information at all times.

[0622] Step 2:

[0623] The device inputs the acquired video into an image analysis algorithm to perform object recognition. This identifies vehicles, pedestrians, and obstacles on the road and determines the location of each object.

[0624] Step 3:

[0625] The device measures the distance to the identified object. This process uses depth estimation technology to calculate the distance from the camera position to the object.

[0626] Step 4:

[0627] The device uses an emotion engine to perform facial recognition and voice analysis to evaluate the user's emotional state. The user's emotional state is then classified as, for example, comfortable, tense, or anxious.

[0628] Step 5:

[0629] The device assesses the level of driving risk based on distance data and emotional state. A threshold is set, and if it determines that the risk of collision is high, the level of danger is assessed.

[0630] Step 6:

[0631] The device generates alerts based on the assessed level of danger and the user's emotions. Strong warnings are issued when the user is relaxed, while gentler warnings are given when the user is stressed.

[0632] Step 7:

[0633] The device records video footage before and after an alert is issued, along with data on the user's emotional state. This data can be used for later analysis or as evidence in the event of an accident.

[0634] Step 8:

[0635] Once a network connection is established, the server receives the video and emotional data transmitted from the terminals into the cloud and organizes it. If requested by insurance companies or related organizations, the data is prepared in the necessary format.

[0636] (Example 2)

[0637] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0638] In recent years, the increase in traffic accidents involving moving vehicles has become a social problem. One of the contributing factors is thought to be decreased driver attention and emotional changes. However, conventional safety management systems are mainly based on physical information and do not take into account the driver's emotional state. Therefore, in order to improve the safety of moving vehicles, it is necessary to take into account the emotional state of the user, but current technology does not adequately address this.

[0639] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0640] In this invention, the server includes a visual sensor means for acquiring video footage, an emotion sensor means for acquiring data for analyzing emotional states, and a warning means for adjusting and issuing warnings based on the evaluated risk and emotional state, in order to manage the safety of a moving object. This makes it possible to adjust the content of warnings to take into account the user's emotional state, in addition to evaluating the physical risk.

[0641] "Visual sensor means" refers to a device for acquiring images of the surroundings of a moving object, and typically refers to a device that collects visual information using a camera or the like.

[0642] "Image processing means" refers to technology or equipment for analyzing acquired video data and identifying and extracting specific objects.

[0643] "Measurement means" refers to devices or algorithms used to calculate the distance of an object from analyzed video data.

[0644] An "emotional sensor device" is a device used to acquire a user's emotional state, and typically refers to a device that uses a camera or microphone to collect facial expressions and sounds.

[0645] "Emotional analysis means" refers to technologies and algorithms that analyze a user's emotional state based on data obtained from emotional sensor means.

[0646] "Evaluation means" refers to technologies and systems for determining physical hazards based on data such as the speed of a moving object and its distance from an object.

[0647] "Warning measures" refer to devices and systems that issue appropriate warnings to drivers based on assessed risks and emotional states.

[0648] "Recording means" refers to devices or systems for recording and storing video footage and related data when a warning is issued.

[0649] "Communication means" refers to a network connection used to transmit recorded data to remote data storage.

[0650] "Memory device" refers to an internal storage device used to temporarily store recorded data when network connectivity is unavailable.

[0651] "Data organization methods" refer to technologies and systems that organize record data related to specific conditions and convert it into a format suitable for provision to relevant organizations.

[0652] This invention provides a system for managing the safety of mobile objects, offering a device that enhances safety more effectively by considering the user's emotional state. The central element of the system is a mobile information terminal such as a smartphone. The specific configuration and functions of the system are described below.

[0653] The device first uses the camera built into the smartphone as a visual sensor to acquire real-time video of the direction of travel of the moving object. This allows monitoring of the surrounding environment while driving. The acquired video data is analyzed by computer vision algorithms (e.g., OpenCV or TensorFlow), which are image processing tools, to identify objects on the road (e.g., vehicles, pedestrians, bicycles, etc.).

[0654] The measurement method utilizes stereo vision and laser rangefinders to calculate the distance to the target object. This allows for the assessment of potential hazards along the path of a moving object. Furthermore, to consider the user's emotional state, the terminal incorporates emotion sensors that collect the user's facial expressions and voice through cameras and microphones. The emotion analysis method uses machine learning models to analyze this data and quickly infer the user's emotional state.

[0655] The evaluation system assesses the risk of collision based on distance data and moving object speed data. Based on this evaluation result and the results of sentiment analysis, the warning system automatically adjusts the warning content and issues an alert to the driver. A concise and understated warning can be issued to a stressed user, while a more emphasized warning can be issued to a relaxed user to draw their attention.

[0656] Furthermore, the device has recording capabilities to record and temporarily store video and related data when a warning is issued. If a network connection is available via communication means, this recorded data can be uploaded to a cloud server to prevent data loss.

[0657] The server organizes the uploaded data and prepares it for provision to relevant organizations and insurance companies. Because the data is converted into the appropriate format by the data organization tools, this contributes to rapid and effective risk management.

[0658] For example, if a user is riding an electric scooter and the device detects a child's bicycle ahead, while simultaneously analyzing the user's emotional state as "relaxed," an enhanced warning sound will be emitted. An example of a prompt message would be, "We want to develop a safety management system that analyzes the emotional state of a user while they are on the move and adjusts the warning content in real time."

[0659] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0660] Step 1:

[0661] The device uses the camera built into the smartphone to acquire real-time video of its surroundings. At this stage, the camera collects visual information as digital data. The input is a video stream from the camera, and the output is image data for each frame.

[0662] Step 2:

[0663] The terminal passes the acquired video data to a computer vision algorithm that uses it as an image processing tool. In this process, for example, OpenCV or TensorFlow is used to identify specific objects in the video. The input is the image data obtained in step 1, and the output is identification data including the position and type of the object.

[0664] Step 3:

[0665] The terminal measures the distance to an object using stereo vision or a laser rangefinder based on identification data. The input is the identification data from step 2, and the output is distance information to the object. This distance data is used for subsequent hazard assessment.

[0666] Step 4:

[0667] The device uses the smartphone's emotion sensors, namely the camera and microphone, to collect data on the user's facial expressions and voice. The input is audio and image data, and the output is raw data indicating the user's emotional state.

[0668] Step 5:

[0669] The device uses emotion analysis tools to analyze the raw data obtained in step 4 and uses a machine learning model to estimate the user's emotional state. The input is raw emotion data, and the output is the estimated emotional state (e.g., tense, relaxed).

[0670] Step 6:

[0671] The terminal uses distance data from the measurement device and velocity data of the moving object as input to evaluate the risk of collision using the evaluation device. The output is the risk assessment result, which shows the potential probability of collision numerically or as a category.

[0672] Step 7:

[0673] The terminal adjusts the warning content using warning mechanisms based on the aforementioned risk assessment results and sentiment analysis results, and issues an alert to the user. The input is the risk assessment results and sentiment state, and the output is an appropriately adjusted warning signal.

[0674] Step 8:

[0675] The terminal uses recording devices to record and store surrounding video and related data when a warning is issued. The input is the trigger event for the warning, and the output is the recorded data stored in storage.

[0676] Step 9:

[0677] The terminal uploads recorded data to the cloud server when a network becomes available via communication means. The input is the stored recorded data, and the output is the data securely stored on the cloud.

[0678] (Application Example 2)

[0679] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0680] Current autonomous driving systems do not incorporate safety management that takes into account the emotional state of the occupants, resulting in insufficient warnings and operational adjustments based on the occupants' psychological condition. Furthermore, there is a lack of technology to appropriately utilize the impact of changes in occupants' emotions on safety. Therefore, there is a need to analyze the occupants' emotional state in real time and optimize the operation of the autonomous driving system based on that analysis.

[0681] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0682] In this invention, the server includes an image acquisition means for acquiring video footage, a visual analysis means for identifying objects within the video footage, and an emotion adjustment means for evaluating the emotional state of the crew using an emotion analysis means and adjusting the content of the warning based on the emotional state. This makes it possible to issue warnings and adjust operations while taking the crew's emotions into consideration.

[0683] "Mobility devices" refer to mechanical devices used to transport people or goods, such as electric scooters, bicycles, and self-driving vehicles.

[0684] A "safety management system" is a system designed to improve safety in the operation of mobile devices, and it performs risk analysis and issues warnings.

[0685] An "image acquisition means" is a device that acquires video footage of the surroundings of a mobile device in real time and provides it for analysis in other systems.

[0686] "Visual analysis means" refers to technology that identifies objects within acquired video footage and analyzes that information to utilize it for safety management.

[0687] A "distance measuring means" is a method that calculates the distance between an object identified in the video and other objects, and provides data for evaluating the risk of collision.

[0688] An "evaluation tool" is the part that analyzes a situation based on multiple pieces of information and has the function of determining risks and countermeasures.

[0689] A "warning system" is a means of ensuring safety by providing users with appropriate warnings based on the assessed risks.

[0690] "Emotional analysis means" refers to technology that analyzes the emotional state of crew members from their facial expressions and voices, and reflects this in the operation of the system.

[0691] "Emotional adjustment mechanisms" are functions that dynamically adjust warnings and operating conditions by taking into account the emotional state of the crew, thereby supporting safe operation.

[0692] A "recording device" is a device used to record video and related data when a specific event occurs, for later analysis or use as evidence.

[0693] "Communication means" refers to communication technology used to transmit recorded data to a remote server for analysis and storage.

[0694] The system that realizes this invention mainly consists of a terminal mounted on a mobile device. The terminal combines image acquisition means, visual analysis means, distance measurement means, emotion analysis means, warning issuing means, recording means, and communication means to perform comprehensive safety management.

[0695] The terminal uses its built-in camera to acquire real-time images of the area ahead while the mobile device is in operation. This image data is input to a visual analysis system, which uses image processing technology (e.g., OpenCV) to identify objects around the mobile device and evaluate their positions and distances. Next, a distance measuring system calculates the distance to the identified objects and determines the urgency of a potential collision.

[0696] Meanwhile, the terminal acquires the occupant's emotional state and diagnoses their emotions through facial expressions and voice using emotion analysis tools. Emotion recognition software such as EmoVu is used for this emotion analysis. Depending on whether the occupant is relaxed or stressed, the warning system adjusts the content of the warning and provides appropriate feedback to the occupant.

[0697] In the event of a dangerous incident, the device records video footage immediately before and after the incident using recording devices and saves it chronologically. This data is uploaded to a cloud server via communication devices when the internet is available, allowing for later analysis and use as evidence.

[0698] As a concrete example of the above process, if the occupants feel uneasy while driving on the road, the terminal can reduce the vehicle's speed and display a visual alert to draw their attention. Conversely, if they are relaxed, the alert will be less pronounced, and the vehicle will continue operating at a normal speed.

[0699] An example of a prompt in the use of a generative AI model is as follows: "Design an application that uses a smartphone camera to analyze the emotions of passengers in an autonomous vehicle in real time and contribute to safe driving." This prompt allows for the design of a function that considers how the AI ​​will link the emotions of the occupants with the vehicle's operation.

[0700] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0701] Step 1:

[0702] The device captures video in front of it in real time using an image acquisition device. The input is video data from the camera, and the output is this video data being sent to the next analysis stage. Specifically, the device continuously acquires images at a rate of several frames per second.

[0703] Step 2:

[0704] The terminal identifies objects from input video data using visual analysis tools. The input is previously acquired video data, and the output is the identification of objects and their location information. In this step, the terminal uses libraries such as OpenCV to extract and identify the boundaries and features of objects.

[0705] Step 3:

[0706] The terminal calculates the distance to an object identified through distance measurement means. The input is the object's position information, the current position and velocity data of the moving device, and the output is the distance calculated based on this information. Specifically, measurements are performed using trigonometry or built-in sensors (e.g., LiDAR).

[0707] Step 4:

[0708] The terminal analyzes distance and speed data using an evaluation tool to assess the risk of collision. The input is calculated distance and speed information, and the output is the risk level (e.g., low, medium, high). The terminal uses this information to determine the urgency of the situation.

[0709] Step 5:

[0710] The device evaluates the user's emotions using emotion analysis tools. The input is the user's video and audio data, and the output is their emotional state (e.g., relaxed, stressed). Here, EmoVu is used to analyze changes in the user's facial muscles.

[0711] Step 6:

[0712] The device uses a warning system to issue warnings based on the assessed emotional state and risk of collision. The input is the emotional state and risk level, and the output is a tailored warning message. Specifically, the user is provided with audio and visual alerts.

[0713] Step 7:

[0714] If a hazard is detected, the device uses recording equipment to record video before and after the event. Input consists of video footage for a set period and a trigger for the hazardous event; output is the saved video clip. The recording is stored in local storage.

[0715] Step 8:

[0716] The terminal uploads recorded data to a cloud server via a communication method. The input is the saved video data, and the output is the completion of data transfer to the remote server. It checks the network connection and sends data if available.

[0717] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0718] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0719] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0720] [Fourth Embodiment]

[0721] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0722] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0723] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0724] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0725] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0726] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0727] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0728] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0729] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0730] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0731] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0732] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0733] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0734] This invention implements a mobile safety management system using a smartphone attached to a mobile device such as an electric scooter or bicycle. This system achieves the following functions by installing a specific application on the smartphone.

[0735] Camera methods and image analysis

[0736] The device uses a smartphone camera mounted on the front of the moving object to capture real-time video. The video is continuously captured, and an image analysis algorithm recognizes objects. This algorithm identifies specific objects by enclosing them in rectangles and makes decisions based on the traffic situation.

[0737] Distance measurement and evaluation

[0738] The device utilizes depth estimation technology to estimate the distance to recognized objects. Distance and velocity data are used to assess collision risk, and alerts are generated in high-risk scenarios.

[0739] Alert issuance

[0740] To alert the user, the device uses both audio alarms and screen displays. For example, if there is a vehicle rapidly approaching ahead, a warning sound will be emitted and a message such as "Caution! Obstacle ahead" will be displayed on the screen.

[0741] Recording and saving

[0742] The device automatically records video footage immediately before and after an alert is triggered, making it available as evidence of the accident. This recorded data is temporarily stored in the smartphone's internal storage.

[0743] Cloud storage of data

[0744] Once network connectivity is restored, the device will transfer the recorded data to a secure cloud server. The server will organize the received data and store it for easy access.

[0745] Providing data to insurance companies

[0746] If necessary, the server provides organized data to insurance companies and related organizations. The data contains information useful for accident investigation and is prepared to meet legal requirements.

[0747] This system utilizes existing smartphones, keeping implementation costs down while effectively improving the safety of mobile devices. Its ease of setup and operation makes it a suitable solution for a wide range of users.

[0748] The following describes the processing flow.

[0749] Step 1:

[0750] The device activates the smartphone's camera and acquires video in real time. The acquired video is saved as a series of frames and prepared for image analysis.

[0751] Step 2:

[0752] The device applies an image analysis algorithm to each captured frame to identify objects on the road. This algorithm recognizes specific objects (vehicles, pedestrians, obstacles) by enclosing them in a rectangle.

[0753] Step 3:

[0754] The device performs distance estimation for identified objects. Using depth estimation technology, it accurately measures the distance from the camera to the object and updates this information in real time.

[0755] Step 4:

[0756] The device uses measured distance and velocity data to assess the movement of objects and the likelihood of collision. A threshold is set, and if it determines that there is a risk exceeding that value, it is classified as high risk.

[0757] Step 5:

[0758] The device generates an alert if it is assessed as high-risk. It sends an audible alarm and an on-screen warning message to the user to draw their attention.

[0759] Step 6:

[0760] The device continuously records video from the moment an alert is issued, documenting the situation immediately before and after the event. This data is temporarily stored on the device.

[0761] Step 7:

[0762] The device uploads the recorded data to the cloud server as soon as it establishes a network connection. The data is transmitted along with a timestamp and identification information, and is stored for later reference.

[0763] Step 8:

[0764] The server receives and organizes the uploaded video data. If requested by insurance companies or related organizations, it prepares the data to be provided in a format suitable for verification.

[0765] (Example 1)

[0766] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0767] There is a need to provide an effective mobile safety management system that reduces the risk of traffic accidents and near misses and improves driver safety. However, existing systems are insufficient in real time object recognition, distance measurement, and risk assessment, and there are also challenges in the proper management and provision of video data in the event of an accident.

[0768] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0769] In this invention, the server includes a shooting means for acquiring video footage, an analysis means for identifying objects in the video footage, and an estimation means for calculating the distance to the identified objects. This makes it possible to accurately recognize the situation around a moving vehicle in real time and to quickly assess the risk of collision.

[0770] "Shooting means" refers to a device or function for acquiring images of the surroundings of a moving object in real time.

[0771] "Analysis means" refers to a device or algorithm used to identify objects in acquired video footage and determine their location and type.

[0772] "Estimation means" refers to a device or calculation method that calculates the distance to an object identified by an analysis means and derives an accurate relative position.

[0773] "Evaluation means" refers to a device or process that evaluates the risk of collision based on the distance of an identified object and the velocity data of a moving object.

[0774] "Warning means" refers to a device or method used to alert the operator based on the results obtained by the evaluation means.

[0775] "Recording means" refers to a device or mechanism for recording and storing video footage before and after a warning signal is issued.

[0776] "Transmission means" refers to a device or technology for transferring recorded video data to other devices or servers via a communication network.

[0777] A "storage medium" is a device or physical storage capable of temporarily storing video footage recorded by a recording means.

[0778] "Management means" refers to a device or method for organizing, appropriately storing, and utilizing recorded data related to a specific event.

[0779] The system for implementing this invention utilizes a smartphone as a terminal attached to a mobile device to manage traffic safety. This system is operated using specific hardware and software to perform several key functions.

[0780] The device utilizes the camera built into the smartphone as a means of capturing images, acquiring real-time video of the direction of movement of a moving object. This video data is processed by an image analysis algorithm as an analytical tool, identifying objects such as vehicles and pedestrians. By using, for example, a deep learning model for this analysis, specific objects can be identified with high accuracy.

[0781] Furthermore, the terminal uses estimation means to calculate the distance to the identified object. Distance estimation utilizes depth estimation technology using machine learning models, providing accurate distance information in real time. This allows the terminal to use evaluation means to assess the risk of collision with the object, and if a high level of risk is detected, it can use warning means to alert the user with an audio alarm or screen display.

[0782] Furthermore, the device can use recording means to automatically record video before and after an alert is issued and temporarily save it to the smartphone's storage medium. This data is then transferred to the server using transmission means after the network connection is restored. The server securely stores this recorded data, organizes it using management means as needed, and provides it to public institutions and insurance companies.

[0783] As a concrete example, suppose a user is delivering goods by bicycle and a pedestrian suddenly appears in front of them. The device instantly recognizes the pedestrian, estimates the distance, assesses the risk of collision, and issues a warning, allowing the user to take evasive action quickly.

[0784] An example of a prompt to input into a generative AI model is a question such as, "In a safety management system using a smartphone attached to an electric scooter, please explain the methods for recognizing objects moving in front and measuring their distance." By utilizing this prompt, it is possible to deepen the understanding of the system's mechanisms and performance through the generative AI model.

[0785] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0786] Step 1:

[0787] The device uses the smartphone's camera to acquire real-time video of the area in front of a moving object. The input is continuous image data from the camera, and the output is video data for analysis. In this process, the camera captures high-resolution video and saves it directly to memory, providing the data necessary for subsequent analysis steps.

[0788] Step 2:

[0789] The terminal processes the acquired video data using an analysis device and applies an image analysis algorithm to identify objects in the video. The input is the video data which is the output of step 1, and the output is data on the location and type of the identified objects. Specifically, it uses a machine learning model to recognize a particular object and draws a rectangle around its location.

[0790] Step 3:

[0791] The terminal uses estimation methods to calculate the distance to the identified object. The input is the object identification data from step 2, and the output is the estimated distance to the object. In this step, depth estimation is performed by combining image disparity information and algorithms to calculate the exact distance to the object.

[0792] Step 4:

[0793] The terminal uses an evaluation method to assess collision risk based on the calculated distance and moving object velocity data. The input is the distance and velocity data from step 3, and the output is the risk level evaluation result. Specifically, the program compares a threshold with the actual data and determines that the risk level is high if the probability of collision exceeds a certain level.

[0794] Step 5:

[0795] The terminal uses the warning means to issue an alert to the user based on the output of the evaluation means. The input is the evaluation result from step 4, and the output is an audio alarm and a warning message displayed on the screen to the user. In this case, a warning sound is emitted from the speaker and a specific warning message is displayed on the display.

[0796] Step 6:

[0797] The terminal records video before and after a warning is issued using a recording device and temporarily saves it to a local storage medium. The input is the video data from step 1, and the output is the recorded file data. The terminal selects and efficiently saves video from a certain period before and after the alert occurs.

[0798] Step 7:

[0799] The server, upon recovery of network connectivity, instructs the transmission method to transfer the recorded data to the cloud. The input is the temporary data stored in step 6, and the output is the data stored in the cloud. During this process, the data is encrypted to ensure the security of the transfer.

[0800] Step 8:

[0801] The server organizes the recorded data using management tools and provides the records to public institutions or insurance companies as needed. The input is the cloud-stored data from step 7, and the output is the organized evidence data. At this stage, the server organizes the relevant data and creates accurate verification materials.

[0802] (Application Example 1)

[0803] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0804] Effective means of improving home safety are needed for the safety management of consumer robots used in homes. However, existing technologies do not adequately monitor the safety of children and pets, and detecting and alerting to dangers tailored to the home environment is a particular challenge.

[0805] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0806] In this invention, the server includes a camera means for acquiring video footage, an image analysis means for identifying objects in the video footage, and a measuring means for calculating the distance to the identified objects. This enables effective safety monitoring and immediate alert generation by consumer robots within the home.

[0807] A "mobile device" is a self-propelled device used to move people or objects, and specifically includes electric scooters and bicycles.

[0808] A "filming device" is a device used to acquire video data, and generally refers to a camera.

[0809] An "image analysis device" is a device that processes video data acquired by a camera to identify specific objects.

[0810] A "measuring device" is a device used to calculate the distance to an identified object, and includes rangefinders and depth sensors.

[0811] An "evaluation device" is a device used to determine the likelihood and risk of a collision based on measured distance and speed data.

[0812] A "warning device" is a device that alerts users based on the risks determined by an evaluation device, and has the function of issuing warnings through voice or display.

[0813] A "recording device" is a device that saves video footage before and after a warning is issued, and it uses memory or storage to hold the data.

[0814] A "communication device" is a device used to transmit acquired data and video to an external server or cloud.

[0815] A "monitoring device" is a device used by consumer robots to monitor the safety of children and pets within a home environment, and has the function of detecting risks in specific situations.

[0816] The system for implementing this invention is primarily intended to monitor the safety of children and pets in a home environment using consumer robots. The system mainly consists of the following hardware and software.

[0817] First, the imaging device mounted on the robot acquires video data of the surrounding environment in real time. This imaging device generally refers to a camera, and for example, a Raspberry Pi Camera Module can be used. The captured video data is processed by an image analysis device to identify specific objects, such as furniture or moving objects (children or pets). Software libraries such as OpenCV are used for image analysis.

[0818] For identified objects, a measuring device calculates the distance. Depth sensors and similar devices are used for this purpose. Based on the measurement results, an evaluation device assesses the risk of collision and issues a warning if necessary. The warning is displayed on the robot's voice speaker and display.

[0819] If a warning is issued, the recording device saves video data before and after the warning. This data is then used as evidence to later review the situation. The saved data is temporarily stored in local storage and uploaded to an external cloud server by the communication device as soon as network connectivity is restored. For example, AWS S3 is used as the cloud server.

[0820] A concrete example is a scenario where a robot moves around the home, and while a child is playing, the robot detects nearby furniture that might fall over and warns the child with a voice message. Such functions can help prevent accidents in the home.

[0821] An example of a prompt is, "Please explain the procedure for a robot to monitor its surroundings in the home and issue warnings when it detects danger. Please also explain how to save the recordings to the cloud." This prompt is used when inputting data into the AI ​​model.

[0822] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0823] Step 1:

[0824] The terminal uses a camera to capture the surrounding environment in real time and acquire video data. The input is video data from the camera, and the output is frame data of that video. These frames serve as basic data for analysis.

[0825] Step 2:

[0826] The terminal uses an image analysis device to process the input video data and recognize specific objects. Here, OpenCV is used to identify specific objects (e.g., children or pets). The identification result outputs the type of object and its coordinate information.

[0827] Step 3:

[0828] The terminal uses the coordinate information of an object obtained from an image analysis device as input and calculates the distance to the object using a measuring device. Depth sensor data is used in this calculation. The distance data is output and used for risk assessment.

[0829] Step 4:

[0830] The terminal inputs distance data from the measuring device and the current speed information of the moving object into the evaluation device to assess the collision risk. If a risk exceeding a certain threshold is detected, the risk is output.

[0831] Step 5:

[0832] If the evaluation device outputs a high-risk signal to the user, the warning device will issue a warning. This includes an audio alert and a message displayed on the screen to draw the user's attention.

[0833] Step 6:

[0834] When a warning is issued, the recording device becomes active and saves video footage for several seconds before and after the warning. In this case, the input is real-time video frames, and the output is the saved video clip.

[0835] Step 7:

[0836] The server uploads the recorded video data to the cloud server via a communication device as soon as network connectivity is restored. The cloud storage stores the video clips in an easily organized format.

[0837] Step 8:

[0838] Based on the stored video data, the server uses a generative AI model to construct generated output corresponding to specific scenarios and prompts when necessary, which can then be used for further analysis and decision-making.

[0839] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0840] This invention aims to improve safety more effectively in a safety management system for mobile devices such as electric scooters and bicycles by taking into account the user's emotional state. This system is centered around a smartphone and consists of the following main elements.

[0841] Camera methods and image analysis

[0842] The device uses the camera built into the smartphone to acquire real-time forward-facing video of the moving object. This video is then analyzed using image processing technology to identify objects on the road and perform object recognition. The recognized objects, along with distance measurement, are used for safety evaluation of the system.

[0843] Introducing an emotional engine

[0844] The device incorporates an emotion engine to assess the user's emotional state. This engine analyzes the user's facial expressions and voice via the smartphone's camera or other sensors to understand their emotions. For example, the emotion engine recognizes whether the user is tense or relaxed.

[0845] Emotion-based warning adjustment

[0846] The device automatically adjusts the content of alerts based on the emotional state recognized by the emotion engine. For example, if the user is relaxed, it will issue a stronger warning than usual to draw their attention. On the other hand, if the user is already stressed, it will adjust the warning to be more concise.

[0847] Recording and saving data

[0848] The device automatically records video when danger is detected, along with the user's emotional state at that time. The recorded video and emotional data are temporarily stored on the device and used later for analysis or as evidence.

[0849] Cloud storage and organization of data

[0850] Once a network connection is established, the device uploads the recorded data to a cloud server. The server then properly organizes the received data and prepares it for provision to insurance companies and related organizations as needed.

[0851] This configuration dramatically improves the safety of mobile devices and enables flexible responses that take into account changes in the user's emotions. As a result, accident prevention and rapid response in the event of an accident can be achieved.

[0852] The following describes the processing flow.

[0853] Step 1:

[0854] The device activates the smartphone's camera and acquires real-time images of the area in front. The camera is fixed in the direction of the moving object's movement, providing the latest visual information at all times.

[0855] Step 2:

[0856] The device inputs the acquired video into an image analysis algorithm to perform object recognition. This identifies vehicles, pedestrians, and obstacles on the road and determines the location of each object.

[0857] Step 3:

[0858] The device measures the distance to the identified object. This process uses depth estimation technology to calculate the distance from the camera position to the object.

[0859] Step 4:

[0860] The device uses an emotion engine to perform facial recognition and voice analysis to evaluate the user's emotional state. The user's emotional state is then classified as, for example, comfortable, tense, or anxious.

[0861] Step 5:

[0862] The device assesses the level of driving risk based on distance data and emotional state. A threshold is set, and if it determines that the risk of collision is high, the level of danger is assessed.

[0863] Step 6:

[0864] The device generates alerts based on the assessed level of danger and the user's emotions. Strong warnings are issued when the user is relaxed, while gentler warnings are given when the user is stressed.

[0865] Step 7:

[0866] The device records video footage before and after an alert is issued, along with data on the user's emotional state. This data can be used for later analysis or as evidence in the event of an accident.

[0867] Step 8:

[0868] Once a network connection is established, the server receives the video and emotional data transmitted from the terminals into the cloud and organizes it. If requested by insurance companies or related organizations, the data is prepared in the necessary format.

[0869] (Example 2)

[0870] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0871] In recent years, the increase in traffic accidents involving moving vehicles has become a social problem. One of the contributing factors is thought to be decreased driver attention and emotional changes. However, conventional safety management systems are mainly based on physical information and do not take into account the driver's emotional state. Therefore, in order to improve the safety of moving vehicles, it is necessary to take into account the emotional state of the user, but current technology does not adequately address this.

[0872] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0873] In this invention, the server includes a visual sensor means for acquiring video footage, an emotion sensor means for acquiring data for analyzing emotional states, and a warning means for adjusting and issuing warnings based on the evaluated risk and emotional state, in order to manage the safety of a moving object. This makes it possible to adjust the content of warnings to take into account the user's emotional state, in addition to evaluating the physical risk.

[0874] "Visual sensor means" refers to a device for acquiring images of the surroundings of a moving object, and typically refers to a device that collects visual information using a camera or the like.

[0875] "Image processing means" refers to technology or equipment for analyzing acquired video data and identifying and extracting specific objects.

[0876] "Measurement means" refers to devices or algorithms used to calculate the distance of an object from analyzed video data.

[0877] An "emotional sensor device" is a device used to acquire a user's emotional state, and typically refers to a device that uses a camera or microphone to collect facial expressions and sounds.

[0878] "Emotional analysis means" refers to technologies and algorithms that analyze a user's emotional state based on data obtained from emotional sensor means.

[0879] "Evaluation means" refers to technologies and systems for determining physical hazards based on data such as the speed of a moving object and its distance from an object.

[0880] "Warning measures" refer to devices and systems that issue appropriate warnings to drivers based on assessed risks and emotional states.

[0881] "Recording means" refers to devices or systems for recording and storing video footage and related data when a warning is issued.

[0882] "Communication means" refers to a network connection used to transmit recorded data to remote data storage.

[0883] "Memory device" refers to an internal storage device used to temporarily store recorded data when network connectivity is unavailable.

[0884] "Data organization methods" refer to technologies and systems that organize record data related to specific conditions and convert it into a format suitable for provision to relevant organizations.

[0885] This invention provides a system for managing the safety of mobile objects, offering a device that enhances safety more effectively by considering the user's emotional state. The central element of the system is a mobile information terminal such as a smartphone. The specific configuration and functions of the system are described below.

[0886] The device first uses the camera built into the smartphone as a visual sensor to acquire real-time video of the direction of travel of the moving object. This allows monitoring of the surrounding environment while driving. The acquired video data is analyzed by computer vision algorithms (e.g., OpenCV or TensorFlow), which are image processing tools, to identify objects on the road (e.g., vehicles, pedestrians, bicycles, etc.).

[0887] The measurement method utilizes stereo vision and laser rangefinders to calculate the distance to the target object. This allows for the assessment of potential hazards along the path of a moving object. Furthermore, to consider the user's emotional state, the terminal incorporates emotion sensors that collect the user's facial expressions and voice through cameras and microphones. The emotion analysis method uses machine learning models to analyze this data and quickly infer the user's emotional state.

[0888] The evaluation system assesses the risk of collision based on distance data and moving object speed data. Based on this evaluation result and the results of sentiment analysis, the warning system automatically adjusts the warning content and issues an alert to the driver. A concise and understated warning can be issued to a stressed user, while a more emphasized warning can be issued to a relaxed user to draw their attention.

[0889] Furthermore, the device has recording capabilities to record and temporarily store video and related data when a warning is issued. If a network connection is available via communication means, this recorded data can be uploaded to a cloud server to prevent data loss.

[0890] The server organizes the uploaded data and prepares it for provision to relevant organizations and insurance companies. Because the data is converted into the appropriate format by the data organization tools, this contributes to rapid and effective risk management.

[0891] For example, if a user is riding an electric scooter and the device detects a child's bicycle ahead, while simultaneously analyzing the user's emotional state as "relaxed," an enhanced warning sound will be emitted. An example of a prompt message would be, "We want to develop a safety management system that analyzes the emotional state of a user while they are on the move and adjusts the warning content in real time."

[0892] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0893] Step 1:

[0894] The device uses the camera built into the smartphone to acquire real-time video of its surroundings. At this stage, the camera collects visual information as digital data. The input is a video stream from the camera, and the output is image data for each frame.

[0895] Step 2:

[0896] The terminal passes the acquired video data to a computer vision algorithm that uses it as an image processing tool. In this process, for example, OpenCV or TensorFlow is used to identify specific objects in the video. The input is the image data obtained in step 1, and the output is identification data including the position and type of the object.

[0897] Step 3:

[0898] The terminal measures the distance to an object using stereo vision or a laser rangefinder based on identification data. The input is the identification data from step 2, and the output is distance information to the object. This distance data is used for subsequent hazard assessment.

[0899] Step 4:

[0900] The device uses the smartphone's emotion sensors, namely the camera and microphone, to collect data on the user's facial expressions and voice. The input is audio and image data, and the output is raw data indicating the user's emotional state.

[0901] Step 5:

[0902] The device uses emotion analysis tools to analyze the raw data obtained in step 4 and uses a machine learning model to estimate the user's emotional state. The input is raw emotion data, and the output is the estimated emotional state (e.g., tense, relaxed).

[0903] Step 6:

[0904] The terminal uses distance data from the measurement device and velocity data of the moving object as input to evaluate the risk of collision using the evaluation device. The output is the risk assessment result, which shows the potential probability of collision numerically or as a category.

[0905] Step 7:

[0906] The terminal adjusts the warning content using warning mechanisms based on the aforementioned risk assessment results and sentiment analysis results, and issues an alert to the user. The input is the risk assessment results and sentiment state, and the output is an appropriately adjusted warning signal.

[0907] Step 8:

[0908] The terminal uses recording devices to record and store surrounding video and related data when a warning is issued. The input is the trigger event for the warning, and the output is the recorded data stored in storage.

[0909] Step 9:

[0910] The terminal uploads recorded data to the cloud server when a network becomes available via communication means. The input is the stored recorded data, and the output is the data securely stored on the cloud.

[0911] (Application Example 2)

[0912] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0913] Current autonomous driving systems do not incorporate safety management that takes into account the emotional state of the occupants, resulting in insufficient warnings and operational adjustments based on the occupants' psychological condition. Furthermore, there is a lack of technology to appropriately utilize the impact of changes in occupants' emotions on safety. Therefore, there is a need to analyze the occupants' emotional state in real time and optimize the operation of the autonomous driving system based on that analysis.

[0914] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0915] In this invention, the server includes an image acquisition means for acquiring video footage, a visual analysis means for identifying objects within the video footage, and an emotion adjustment means for evaluating the emotional state of the crew using an emotion analysis means and adjusting the content of the warning based on the emotional state. This makes it possible to issue warnings and adjust operations while taking the crew's emotions into consideration.

[0916] "Mobility devices" refer to mechanical devices used to transport people or goods, such as electric scooters, bicycles, and self-driving vehicles.

[0917] A "safety management system" is a system designed to improve safety in the operation of mobile devices, and it performs risk analysis and issues warnings.

[0918] An "image acquisition means" is a device that acquires video footage of the surroundings of a mobile device in real time and provides it for analysis in other systems.

[0919] "Visual analysis means" refers to technology that identifies objects within acquired video footage and analyzes that information to utilize it for safety management.

[0920] A "distance measuring means" is a method that calculates the distance between an object identified in the video and other objects, and provides data for evaluating the risk of collision.

[0921] An "evaluation tool" is the part that analyzes a situation based on multiple pieces of information and has the function of determining risks and countermeasures.

[0922] A "warning system" is a means of ensuring safety by providing users with appropriate warnings based on the assessed risks.

[0923] "Emotional analysis means" refers to technology that analyzes the emotional state of crew members from their facial expressions and voices, and reflects this in the operation of the system.

[0924] "Emotional adjustment mechanisms" are functions that dynamically adjust warnings and operating conditions by taking into account the emotional state of the crew, thereby supporting safe operation.

[0925] A "recording device" is a device used to record video and related data when a specific event occurs, for later analysis or use as evidence.

[0926] "Communication means" refers to communication technology used to transmit recorded data to a remote server for analysis and storage.

[0927] The system that realizes this invention mainly consists of a terminal mounted on a mobile device. The terminal combines image acquisition means, visual analysis means, distance measurement means, emotion analysis means, warning issuing means, recording means, and communication means to perform comprehensive safety management.

[0928] The terminal uses its built-in camera to acquire real-time images of the area ahead while the mobile device is in operation. This image data is input to a visual analysis system, which uses image processing technology (e.g., OpenCV) to identify objects around the mobile device and evaluate their positions and distances. Next, a distance measuring system calculates the distance to the identified objects and determines the urgency of a potential collision.

[0929] Meanwhile, the terminal acquires the occupant's emotional state and diagnoses their emotions through facial expressions and voice using emotion analysis tools. Emotion recognition software such as EmoVu is used for this emotion analysis. Depending on whether the occupant is relaxed or stressed, the warning system adjusts the content of the warning and provides appropriate feedback to the occupant.

[0930] In the event of a dangerous incident, the device records video footage immediately before and after the incident using recording devices and saves it chronologically. This data is uploaded to a cloud server via communication devices when the internet is available, allowing for later analysis and use as evidence.

[0931] As a concrete example of the above process, if the occupants feel uneasy while driving on the road, the terminal can reduce the vehicle's speed and display a visual alert to draw their attention. Conversely, if they are relaxed, the alert will be less pronounced, and the vehicle will continue operating at a normal speed.

[0932] An example of a prompt in the use of a generative AI model is as follows: "Design an application that uses a smartphone camera to analyze the emotions of passengers in an autonomous vehicle in real time and contribute to safe driving." This prompt allows for the design of a function that considers how the AI ​​will link the emotions of the occupants with the vehicle's operation.

[0933] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0934] Step 1:

[0935] The device captures video in front of it in real time using an image acquisition device. The input is video data from the camera, and the output is this video data being sent to the next analysis stage. Specifically, the device continuously acquires images at a rate of several frames per second.

[0936] Step 2:

[0937] The terminal identifies objects from input video data using visual analysis tools. The input is previously acquired video data, and the output is the identification of objects and their location information. In this step, the terminal uses libraries such as OpenCV to extract and identify the boundaries and features of objects.

[0938] Step 3:

[0939] The terminal calculates the distance to an object identified through distance measurement means. The input is the object's position information, the current position and velocity data of the moving device, and the output is the distance calculated based on this information. Specifically, measurements are performed using trigonometry or built-in sensors (e.g., LiDAR).

[0940] Step 4:

[0941] The terminal analyzes distance and speed data using an evaluation tool to assess the risk of collision. The input is calculated distance and speed information, and the output is the risk level (e.g., low, medium, high). The terminal uses this information to determine the urgency of the situation.

[0942] Step 5:

[0943] The device evaluates the user's emotions using emotion analysis tools. The input is the user's video and audio data, and the output is their emotional state (e.g., relaxed, stressed). Here, EmoVu is used to analyze changes in the user's facial muscles.

[0944] Step 6:

[0945] The device uses a warning system to issue warnings based on the assessed emotional state and risk of collision. The input is the emotional state and risk level, and the output is a tailored warning message. Specifically, the user is provided with audio and visual alerts.

[0946] Step 7:

[0947] If a hazard is detected, the device uses recording equipment to record video before and after the event. Input consists of video footage for a set period and a trigger for the hazardous event; output is the saved video clip. The recording is stored in local storage.

[0948] Step 8:

[0949] The terminal uploads recorded data to a cloud server via a communication method. The input is the saved video data, and the output is the completion of data transfer to the remote server. It checks the network connection and sends data if available.

[0950] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0951] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0952] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0953] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0954] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0955] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0956] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0957] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0958] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0959] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0960] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0961] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0962] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0963] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0964] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0965] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0966] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0967] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0968] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0969] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0970] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0971] The following is further disclosed regarding the embodiments described above.

[0972] (Claim 1)

[0973] A mobile safety management system for analyzing collision risks involving multiple moving objects and issuing alerts,

[0974] A camera means for acquiring video,

[0975] Image analysis means for identifying objects in the aforementioned video,

[0976] Distance measuring means for calculating the distance to the identified object,

[0977] An evaluation means for evaluating the risk of collision based on the aforementioned distance and the velocity data of the moving object,

[0978] Based on the aforementioned risks, an alert means is provided to send an alert to the driver,

[0979] A recording means for recording and saving video footage before and after the aforementioned alert is issued,

[0980] A communication means for uploading the aforementioned recorded data to the cloud,

[0981] A system that includes this.

[0982] (Claim 2)

[0983] The system according to claim 1, further comprising a local storage means for managing requests to temporarily save video from the recording means when network connectivity is unavailable.

[0984] (Claim 3)

[0985] The system according to claim 1, further comprising means for organizing recorded data related to a specific event for provision to insurance companies and related organizations.

[0986] "Example 1"

[0987] (Claim 1)

[0988] A means of capturing video,

[0989] Analysis means for identifying objects in the aforementioned video,

[0990] Estimation means for calculating the distance to the identified object,

[0991] An evaluation means for evaluating the risk of collision based on the aforementioned distance and the velocity data of the moving object,

[0992] Based on the aforementioned risks, a warning means is provided to issue an alert to the operator,

[0993] A recording means for recording and saving video footage before and after the aforementioned alert is issued,

[0994] A transmission means for transferring the aforementioned recorded data to a communication network,

[0995] A system that includes this.

[0996] (Claim 2)

[0997] The system according to claim 1, further comprising a storage medium for temporarily storing video from the recording means when network connectivity is unavailable.

[0998] (Claim 3)

[0999] The system according to claim 1, further comprising management means for organizing recorded data related to specific events for provision to public institutions.

[1000] "Application Example 1"

[1001] (Claim 1)

[1002] A safety management system for analyzing the risk of collisions between multiple moving objects and issuing alerts,

[1003] A camera that acquires images,

[1004] An image analysis device for identifying objects in the aforementioned video,

[1005] A measuring device for calculating the distance to the identified object,

[1006] An evaluation device that evaluates the risk of collision based on the aforementioned distance and the velocity data of the moving object,

[1007] Based on the aforementioned risks, a warning device is provided that issues an alert to the operator,

[1008] A recording device that records and saves video footage before and after the aforementioned alert is issued,

[1009] A communication device that uploads the aforementioned recorded data to the cloud,

[1010] A monitoring device that can be installed in consumer robots used in home environments and further equipped with functions to monitor the safety of children and pets,

[1011] A system that includes this.

[1012] (Claim 2)

[1013] The system according to claim 1, further comprising a local storage device for managing requests to temporarily save video from the recording device when network connectivity is unavailable.

[1014] (Claim 3)

[1015] The system according to claim 1, further comprising an organizing device for organizing recorded data related to a specific event for provision to insurance companies and related organizations.

[1016] "Example 2 of combining an emotion engine"

[1017] (Claim 1)

[1018] A system for managing the safety of moving objects,

[1019] A visual sensor means for acquiring images,

[1020] Image processing means for identifying objects in the aforementioned video,

[1021] A measuring means for calculating the distance to the identified object,

[1022] An emotion sensor means for acquiring data to analyze emotional states,

[1023] A means of sentiment analysis for evaluating the user's emotional state,

[1024] An evaluation means for evaluating the risk of collision based on the aforementioned distance and the velocity data of the moving object,

[1025] A warning system that adjusts and issues warnings based on the assessed risk and emotional state,

[1026] A recording means for recording and saving video footage and related data before and after the aforementioned warning is issued,

[1027] A communication means for uploading the aforementioned recorded data to data storage,

[1028] A system that includes this.

[1029] (Claim 2)

[1030] The system according to claim 1, further comprising a storage means for temporarily storing data in the recording means when network connectivity is unavailable.

[1031] (Claim 3)

[1032] The system according to claim 1, further comprising a data organization means for organizing record data related to specific conditions and providing it to relevant organizations.

[1033] "Application example 2 when combining with an emotional engine"

[1034] (Claim 1)

[1035] A safety management system for analyzing collision risks involving multiple mobile devices and issuing warnings,

[1036] Image acquisition means for acquiring video,

[1037] A visual analysis means for identifying an object in the aforementioned video,

[1038] Distance measuring means for calculating the distance to the identified object,

[1039] An evaluation means for evaluating the risk of collision based on the aforementioned distance and the speed information of the moving device,

[1040] A warning issuing means for issuing a warning to the occupants based on the assessed risk,

[1041] An emotion adjustment means that evaluates the emotional state of the crew using an emotion analysis means and adjusts the content of the warning based on the said emotional state,

[1042] A recording means for recording and saving video footage before and after the aforementioned warning is issued,

[1043] A communication means for transmitting the aforementioned recorded data to a remote server,

[1044] A system that includes this.

[1045] (Claim 2)

[1046] The system according to claim 1, further comprising a storage device for managing requests to temporarily store video from the recording device when network connectivity is difficult.

[1047] (Claim 3)

[1048] The system according to claim 1, further comprising means for organizing record data related to specific events for provision to insurance institutions and related organizations. [Explanation of symbols]

[1049] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A mobile safety management system for analyzing collision risks involving multiple moving objects and issuing alerts, A camera means for acquiring video, Image analysis means for identifying objects in the aforementioned video, Distance measuring means for calculating the distance to the identified object, An evaluation means for evaluating the risk of collision based on the aforementioned distance and the velocity data of the moving object, Based on the aforementioned risks, an alert means is provided to send an alert to the driver, A recording means for recording and saving video footage before and after the aforementioned alert is issued, A communication means for uploading the aforementioned recorded data to the cloud, A system that includes this.

2. The system according to claim 1, further comprising a local storage means for managing requests to temporarily save video from the recording means when network connectivity is unavailable.

3. The system according to claim 1, further comprising means for organizing recorded data related to a specific event for provision to insurance companies and related organizations.

Images