system

Abstract

Provide a system. [Solution] Equipped with a camera device, analysis means for analyzing video data acquired by the camera device; an audio generating means for generating an audio alert based on the analyzed data; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, many people rely on external visual information when walking or driving in their daily lives. However, visually impaired people and those who are distracted can have difficulty recognizing surrounding dangers and responding appropriately. It is also difficult to grasp the surrounding situation in real time, which can lead to accidents or dangerous situations without noticing traffic signals or obstacles. To solve these problems, a system is needed that can grasp the surrounding situation in real time and provide audio alerts at the appropriate time. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a system including a camera device, an analysis means for analyzing video data acquired by the camera device, and a voice generation means for generating a voice alert based on the analyzed data. The system also includes an object recognition means for recognizing surrounding objects based on the video data acquired by the camera device, a risk assessment means for evaluating risk based on the objects recognized by the object recognition means, and a means for generating a voice alert based on the evaluation result of the risk assessment means. The system further includes a signal recognition means for recognizing traffic signals based on the video data acquired by the camera device, and a means for generating a voice alert when the recognized traffic signal is in a specific state. This allows the user to be notified of surrounding dangers and signal information by voice in real time, supporting safe behavior.

[0006] "Camera device" refers to a device for acquiring video data.

[0007] "Analysis means" refers to a device or means for processing video data acquired by a camera device and extracting and analyzing specific information.

[0008] "Audio generating means" refers to a device or means for generating an audio alert based on the analyzed data and notifying the user thereof.

[0009] "Object recognition means" refers to a device or means for identifying an object in an image based on image data acquired by a camera device.

[0010] The "risk assessment means" refers to a device or means for assessing risk or danger based on information about an object recognized by the object recognition means.

[0011] "Signal recognition means" refers to a device or means for identifying the status of a traffic signal based on video data acquired by a camera device.

[0012] "Audio Alert" means a warning or notification provided to a user audibly.

[0013] "User" refers to an individual who uses the camera-enabled audio glasses. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system that includes a camera device, analyzes video data, and generates a voice alert. Hereinafter, an embodiment of the present invention will be described.

[0036] System Configuration

[0037] The terminal is equipped with a camera device that captures images of the surroundings in real time. The captured image data is processed by an analysis means to extract specific information (e.g., object identification and traffic signal recognition). Based on the analysis results, an audio generation means generates an audio alert to notify the user.

[0038] System Operation

[0039] 1. When a user wears the camera-equipped audio glasses, the device activates the camera and starts capturing video data continuously.

[0040] 2. The device sends the captured video data to the analysis means, and the object recognition means identifies objects in the video, for example, using a specific algorithm to detect cars, pedestrians, and other obstacles.

[0041] 3. The terminal performs risk assessment using the danger assessment means based on the object recognized by the object recognition means, for example, if an approaching vehicle is within a dangerous distance, the terminal evaluates the information.

[0042] 4. The device uses a signal recognition means to identify traffic signals in the video. Specifically, it identifies the color of the signal (red, yellow, green) and determines its status.

[0043] 5. The device uses the voice generation means to generate necessary voice alerts based on the analysis results, for example, when a dangerous object is detected or a red light is recognized, it generates a corresponding voice message to notify the user.

[0044] Specific examples

[0045] Suppose a user is walking down a busy street. The device's camera captures video and its object recognition system detects vehicles and pedestrians in the video. If a pedestrian suddenly tries to cross the road, the device immediately assesses this risk and generates a voice alert saying, "There is a pedestrian ahead. Please be careful."

[0046] Next, when the user approaches an intersection and the traffic light recognition means recognizes that the traffic light is red, the device provides a voice alert saying, "Red light. Please wait." When the light turns green, the device generates a voice alert saying, "Green light. Please proceed."

[0047] The system provides users with real-time audio information about their surroundings, helping to ensure the safety of people with visual impairments and those who are distracted.

[0048] The processing flow will be explained below.

[0049] Step 1:

[0050] The device initializes the camera and starts capturing video data, which specifically means configuring the camera device and starting continuous frame capture.

[0051] Step 2:

[0052] The device sends the captured video frames to the analysis means. The analysis means receives the video data and detects objects in the frames using the object recognition means. For example, it applies a specific algorithm (e.g., YOLO, Haar Cascades, etc.) to identify objects (pedestrians, cars, obstacles, etc.).

[0053] Step 3:

[0054] The terminal evaluates the risk using the danger evaluation means based on the output of the object recognition means. If a certain distance or speed is exceeded, it can be judged as "dangerous." The evaluation result is passed to the next processing step.

[0055] Step 4:

[0056] The device uses a signal recognition means to identify traffic signals from video frames captured by the camera. Specifically, it recognizes the color of the signal and determines its status (red, yellow, green). For example, it uses color region extraction and pattern recognition to identify the color of the signal.

[0057] Step 5:

[0058] The device generates an appropriate voice alert using the voice generation means based on the analysis results from the danger assessment means and the signal recognition means. For example, when a red light is recognized or a dangerous object is detected, it generates a voice message such as "Red light" or "Dangerous object present," respectively.

[0059] Step 6:

[0060] The device notifies the user of the generated voice alert by activating a speech synthesis engine and providing a voice alert to the user in real time, allowing the user to take appropriate action.

[0061] Step 7:

[0062] Based on the audio alerts, the user can take appropriate action depending on the surrounding situation, for example, stopping when hearing the audio alert for a red light, or crossing the road when hearing the audio alert for a green light.

[0063] This series of steps allows the system to recognize surrounding hazards and traffic signals in real time and provide appropriate audio alerts to the user.

[0064] Example 1

[0065] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0066] In modern society, people with visual impairments and those who are distracted need to be able to travel safely. However, current safety assistance devices often have difficulty accurately recognizing their surroundings in real time and providing timely warnings. They also lack the ability to accurately recognize traffic signal status and approaching dangerous objects and communicate them safely. These situations make it difficult to ensure the safety of people with visual impairments.

[0067] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0068] In this invention, the server includes a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, a risk assessment means for determining the surrounding situation in real time, a signal recognition means for determining the status of traffic signals, and a notification means for notifying the user of the voice alert, thereby enabling the user to accurately grasp the surrounding situation in real time and receive a warning at an appropriate time.

[0069] A "camera device" is a device for acquiring video data.

[0070] The "analysis means" is a device or software that processes the video data acquired by the camera device and extracts specific information.

[0071] "Audio generating means" refers to a device or software that generates an audio alert based on the data obtained by the analyzing means.

[0072] "Risk assessment means" means a device or software that assesses the surrounding situation in real time, in particular by performing risk assessment based on recognized objects and traffic signals.

[0073] "Signal recognition means" refers to a device or software that recognizes the state of a traffic signal (red, yellow, green) based on acquired image data.

[0074] A "notification means" is a device or software for notifying a user of a generated audio alert.

[0075] This invention is a system that uses a device equipped with a camera to capture and analyze surrounding video data in real time, generating audio alerts, primarily for the purpose of ensuring the safety of people with visual impairments and those who are distracted.

[0076] Hardware and software used

[0077] Device: For example, audio glasses with a camera

[0078] Camera device: Camera built into the device

[0079] Analysis tools: High-performance processor and dedicated object recognition algorithms (e.g., YOLO, TENSORFLOW®)

[0080] Voice generation means: Text-to-Speech (TTS) software (e.g., Google® Cloud Text-to-Speech, Amazon Polly)

[0081] Hazard Assessment Tools: Risk Assessment Algorithms

[0082] Signal Recognition: An algorithm for identifying the color of traffic signals

[0083] Notification Method: Speaker or earphone to notify the user of audio alerts

[0084] System Configuration

[0085] 1. The terminal uses a camera device to capture video data of the user's surroundings in real time.

[0086] 2. The captured video data is sent to the device's built-in analysis means, which uses an object recognition algorithm (e.g., YOLO).

[0087] 3. The device uses analytical means to identify objects in the video (cars, pedestrians, traffic lights, etc.).

[0088] 4. Based on the identified object, the danger assessment means assesses the risk, for example, if an approaching car is at a dangerous distance, the information is assessed immediately.

[0089] 5. The device uses signal recognition means to identify traffic signals in the video, for example, by determining the color of the signal (red, yellow, green) and determining its status.

[0090] 6. Finally, the necessary audio alert is generated using the audio generation means based on the analysis results, and the generated audio alert is notified to the user through the notification means.

[0091] Specific examples

[0092] Consider a scenario where a user is walking in a busy downtown area. The device's camera captures video and the acquired data is sent to an analysis means (e.g., the YOLO algorithm). The object recognition means detects vehicles and pedestrians, and the risk assessment means immediately assesses the risk. If a pedestrian suddenly tries to cross the road, the device generates a voice alert saying, "There is a pedestrian ahead. Please be careful," and notifies the user through a notification means.

[0093] Next, when the user approaches an intersection with a traffic light, the traffic light recognition means determines the color of the traffic light, and when it recognizes a red light, it generates a voice alert saying, "Red light. Please wait." When the light turns green, it provides an alert saying, "Green light. Please proceed."

[0094] Prompt Sentence Examples

[0095] The generative AI model generates a voice alert by providing a prompt like this:

[0096] There is a pedestrian ahead. Please be careful.

[0097] Red light, please wait. Green light, please go ahead.

[0098] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0099] Step 1:

[0100] The user wears audio glasses with a camera.

[0101] Input: Audio glasses with camera

[0102] Output: System boot

[0103] Specific operation: When the user puts on the audio glasses and turns on the power switch, the system starts up.

[0104] Step 2:

[0105] The terminal activates the camera device and captures the video data.

[0106] Input: Powered camera device

[0107] Output: Real-time video data

[0108] What it does: The camera device begins capturing video of the surroundings in real time, for example, capturing a continuous 360-degree view.

[0109] Step 3:

[0110] The terminal transmits the captured video data to the analysis means.

[0111] Input: Real-time video data

[0112] Output: Video data sent to analysis means

[0113] Specific operation: The video data captured by the camera is sent to the internal high-performance processor, which allows it to proceed to the next stage of analysis.

[0114] Step 4:

[0115] The terminal uses object recognition means to identify objects in the video.

[0116] Input: Transmitted video data

[0117] Output: Recognized object data

[0118] How it works: Object recognition methods such as the YOLO algorithm analyze video data and identify objects such as cars and pedestrians. The results are updated for each frame.

[0119] Step 5:

[0120] The terminal performs a risk assessment using a risk assessment means.

[0121] Input: Recognized object data

[0122] Output: The result of the risk assessment

[0123] Specific operation: Based on data such as the speed and distance of the recognized object, the risk assessment method evaluates the possibility of contact. For example, it immediately determines whether an approaching vehicle is within a dangerous distance.

[0124] Step 6:

[0125] The terminal uses the signal recognition means to identify the traffic signal.

[0126] Input: Video data

[0127] Output: Traffic light status (color)

[0128] Specific operation: The signal recognition means detects traffic signals from the video data and distinguishes their colors (red, yellow, green). For example, it analyzes the traffic light and recognizes a red signal.

[0129] Step 7:

[0130] The terminal generates an audio alert using the audio generating means to notify the user.

[0131] Input: Risk assessment results, traffic signal status

[0132] Output: Audio alert

[0133] Specific operation: A voice generation means (e.g., Google Cloud Text-to-Speech) generates a voice message based on the results of risk assessment and traffic signal recognition. This voice alert is delivered to the user through a notification means. For example, if there is a high risk of an approaching vehicle, the user will be notified by a voice saying, "There is a vehicle ahead. Please be careful."

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] In recent years, there has been a demand for improved safety in autonomous vehicles, but collisions with people and obstacles remain frequent. Technology that can properly recognize the surrounding situation in real time and respond immediately is lacking, especially in complex situations such as busy streets and intersections. There is also a lack of effective means to notify drivers of emergency situations using voice alerts. This calls for new technologies to improve the safety of autonomous vehicles and prevent accidents.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server is equipped with a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, an object identification means for identifying objects from the video data in real time, a risk assessment means for assessing a risk within a specific distance based on the identified objects, and a means for generating a voice alert when the risk level exceeds an assessment standard. This enables the server to recognize the situation around the autonomous vehicle in real time and immediately provide a voice alert in an emergency, thereby improving safety.

[0139] A "camera device" is a device for acquiring video data.

[0140] The "analysis means" is a means having a function for processing the acquired video data and extracting specific information.

[0141] The "audio generating means" is a means having the function of generating an audio alert based on the analyzed data.

[0142] The "object identification means" is a means having a function for identifying an object from video data in real time.

[0143] A "danger assessment means" is a means having the function of assessing danger within a certain distance based on an identified object.

[0144] The "signal recognition means" is a means having a function of recognizing traffic signals based on image data acquired by a camera device.

[0145] "Real-time" refers to processing or reacting immediately, without any time delay.

[0146] "Audio alert" refers to a warning or guidance given to the user by voice.

[0147] "Evaluation criteria" refers to the numerical values ​​and conditions that serve as the criteria when the risk assessment means makes an assessment.

[0148] "Analyzed data" refers to data after it has been processed by an analytical means.

[0149] The present invention relates to a system for improving the safety of autonomous vehicles.

[0150] System Configuration

[0151] The system includes a camera device, an analysis means, a sound generation means, an object identification means, a risk assessment means, and a signal recognition means.

[0152] 1. Camera equipment

[0153] This camera device acquires surrounding video data in real time. In particular, a camera installed in an autonomous vehicle is used. A specific example of a camera is an in-vehicle camera system.

[0154] 2. Analysis method

[0155] The acquired video data is processed by the analysis means, which uses libraries such as OpenCV and TensorFlow. The analysis means works in conjunction with the object identification means to recognize specific objects such as pedestrians and other vehicles.

[0156] 3. Voice Generation Method

[0157] Based on the analyzed data, audio alerts are generated using gTTS (Google Text-to-Speech) and other technologies, which are then sent to the user via a speaker in real time.

[0158] 4. Object Identification Method

[0159] It uses deep learning models such as TensorFlow to identify objects in video data in real time, such as pedestrians, other vehicles, and road signs.

[0160] 5. Risk Assessment Tools

[0161] Based on the identified object, it evaluates the object's potential danger to the autonomous vehicle, based on criteria such as a specific distance, speed, direction, etc. If the danger exceeds the criteria, an audio alert is generated.

[0162] 6. Signal recognition means

[0163] The system recognizes traffic signals based on the video data acquired by the camera device. This recognition means also works in conjunction with the analysis means to recognize the traffic light status (red, yellow, green) and provide that information to the user.

[0164] Specific examples

[0165] As a specific example, consider a case where an autonomous vehicle approaches an intersection. A camera device captures video of the area around the intersection, and an object identification means identifies pedestrians and other vehicles. If a pedestrian is assessed as approaching the vehicle, the risk assessment means immediately determines this information and generates an audio alert saying, "Pedestrian approaching." If the traffic light is red, an audio alert saying, "Red light. Please wait" is also generated.

[0166] Prompt Sentence Examples

[0167] Below is an example of a prompt for a program in a system for autonomous vehicles that recognizes the surrounding situation in real time and provides a voice alert in the event of an emergency.

[0168] Prompt: Write a program for a system that uses dashcam footage to detect pedestrians near intersections in real time and provide audio alerts.

[0169] This allows the present invention to improve safety by providing real-time awareness of the situation around an autonomous vehicle and providing immediate audio alerts in the event of an emergency.

[0170] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0171] Step 1:

[0172] The device activates a camera device and captures video data of the surroundings in real time. The input is video data from the camera. The camera captures the surroundings at a high frame rate and continuously collects the video data. The output is video data for each frame.

[0173] Step 2:

[0174] The device sends the captured video data to the analysis means. The input is the video data obtained in step 1. The video data is preprocessed using the OpenCV library and converted into an appropriate format. This process allows the analysis means to process the data efficiently. The output is the preprocessed video data.

[0175] Step 3:

[0176] The analysis means uses the preprocessed video data to perform object identification using the object identification means. The preprocessed video data is used as input. Object identification is performed using deep learning using a generative AI model from TensorFlow. This process identifies specific objects such as pedestrians and vehicles from each frame of video. The output is information about the identified objects (position, type).

[0177] Step 4:

[0178] The terminal performs a risk assessment using the risk assessment means based on the object information identified by the object identification means. The identified object information is input. Based on this information, the distance to the object, relative speed, direction, etc. are evaluated. Here, an evaluation standard is set, and if the risk is high, a flag is raised as a high risk. The risk assessment result is obtained as the output.

[0179] Step 5:

[0180] The terminal uses the voice generation means to generate the necessary voice alert based on the result of the danger assessment means. The input is the danger assessment result. Using gTTS (Google Text-to-Speech), a voice message is generated according to the assessment result. For example, if a pedestrian is approaching, a message such as "Pedestrian approaching" is generated, and if a stop is required at a red light, a message such as "Red light. Please wait" is generated. The output is the generated voice alert (voice file).

[0181] Step 6:

[0182] The device notifies the user of the generated audio alert. It has as input the generated audio alert. It plays the audio file through the speaker to provide a real-time warning to the user. It has as output the audio alert delivered to the user.

[0183] This allows the device to monitor the autonomous vehicle's surroundings in real time and generate and provide instant voice alerts when necessary, improving driver safety.

[0184] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0185] The present invention relates to a system that includes a camera device, analyzes video data, and generates an audio alert. The present invention also relates to a system that includes emotion recognition means that recognizes a user's emotion and adjusts the content and tone of the audio alert accordingly. Hereinafter, an embodiment of the present invention will be described.

[0186] System Configuration

[0187] The device is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data, including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0188] System Operation

[0189] 1. When a user puts on the camera-equipped audio glasses, the device starts the camera device and starts capturing video data, including not only the surrounding scenery but also the user's face.

[0190] 2. The device sends the video data to the analysis means, which first uses the object recognition means to identify objects in the video (e.g., pedestrians, cars, and other obstacles). Based on this, the risk assessment means assesses the risk.

[0191] 3. The device then uses its signal recognition means to identify traffic signals in the video. Specifically, it recognizes the color of the signal (red, yellow, green) and determines its status.

[0192] 4. The device further uses emotion recognition means to recognize emotions from the user's facial expressions, including algorithms for identifying emotional states such as smiling, angry, surprised, etc.

[0193] 5. The device will comprehensively evaluate the analysis results from the object recognition means, signal recognition means, and emotion recognition means and generate the necessary audio alert. For example, if a dangerous object is detected and the user is startled, the tone of the audio alert will be adjusted to be softer.

[0194] 6. The device then generates and sends the generated audio alert to the user in real time, adjusting the tone and content of the audio notification based on the emotion recognition results.

[0195] Specific examples

[0196] Suppose a user is walking in a busy shopping district. The device's camera captures video, and the object recognition means detects vehicles and pedestrians in the video. If the risk assessment means detects a car suddenly approaching at an intersection, it determines this as "danger."

[0197] The device also analyzes the user's facial expression using emotion recognition means and recognizes that the user is nervous. In this case, the device generates a voice alert in a calm tone saying, "Please be careful." If the traffic light recognition means recognizes a red light, the device generates a voice alert saying, "The light is red, please wait," to notify the user.

[0198] In this way, the present invention takes into account the user's emotional state, recognizes surrounding hazards and traffic signals in real time, and provides tailored audio alerts to the user, supporting safe and secure behavior.

[0199] The processing flow will be explained below.

[0200] Step 1:

[0201] The device initializes the camera and starts capturing video data, capturing images of the surrounding scenery and the user's face in real time.

[0202] Step 2:

[0203] The terminal transmits the captured video frames to the analysis means, which receives them and uses the object recognition means to detect objects (e.g., cars, pedestrians, and obstacles) in the frames.

[0204] Step 3:

[0205] The terminal uses the risk assessment means to assess the risk based on the output of the object recognition means. For example, if an approaching vehicle is within a certain distance, it is judged to be "dangerous."

[0206] Step 4:

[0207] The device uses a signal recognition means to analyze traffic signals in the video frames, recognize the signal color (red, yellow, green) and determine its status. This information is used in the next step.

[0208] Step 5:

[0209] The device analyzes the user's facial expressions using an emotion recognition means. The emotion recognition means identifies emotions (e.g., tension, surprise, joy) from the user's facial expressions in the video. The device then determines the user's emotional state from the facial expressions.

[0210] Step 6:

[0211] The device combines analysis results from object recognition, signal recognition, and emotion recognition methods, so if the user is nervous, for example, it will soften the content and tone of the audio alerts.

[0212] Step 7:

[0213] The terminal generates an appropriate audio alert using the audio generation means based on the integration result. For example, if a dangerous object is detected and the user is startled, the terminal generates an audio alert saying "Be careful" in a calm tone.

[0214] Step 8:

[0215] The device generates audio alerts and notifies the user in real time, driving a speech synthesis engine to play audio with adjusted tone and content to inform the user of the situation.

[0216] Step 9:

[0217] Users can take appropriate action based on the audio alerts provided, such as stopping at a red light when necessary or being aware of hazards.

[0218] This step allows the device to analyze the surrounding video in real time and provide audio alerts that take into account the user's emotional state, with the aim of ensuring the user's safety and peace of mind.

[0219] Example 2

[0220] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0221] Conventional systems can analyze video data captured by cameras and generate audio alerts, but they are unable to generate appropriate alerts based on the user's emotions. This creates the risk of the user being surprised or making a wrong decision. Furthermore, comprehensive risk assessment is not adequately performed when recognizing traffic signals and surrounding objects.

[0222] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0223] In this invention, the server is equipped with a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, and an emotion recognition means for recognizing the user's emotion and adjusting the content and tone of the voice alert accordingly, thereby enabling the server to generate and notify an appropriate voice alert in real time according to the user's emotional state.

[0224] "Camera device" refers to a device for acquiring video data, capturing the user's surroundings and facial expressions.

[0225] "Analysis means" refers to a function or device for processing video data acquired by a camera device and performing object recognition, signal recognition, and emotion recognition.

[0226] "Voice generation means" refers to a function or device for generating a voice alert based on the analyzed data, and notifying the user of necessary information by voice.

[0227] "Emotion recognition means" refers to a function or device for analyzing the user's facial expression data and identifying the user's emotional state (e.g., smile, anger, surprise).

[0228] "Object recognition means" refers to a function or device for identifying surrounding objects (e.g., pedestrians, automobiles, and other obstacles) based on video data.

[0229] The "risk assessment means" refers to a function or device for conducting risk assessment and determining danger based on the object recognized by the object recognition means.

[0230] "Signal recognition means" refers to a function or device for recognizing traffic signals (e.g., red, yellow, green) based on video data.

[0231] MODE FOR CARRYING OUT THE INVENTION

[0232] The present invention relates to a system that includes a camera device, analyzes video data, and generates an audio alert. The present invention also relates to a system that includes emotion recognition means that recognizes a user's emotion and adjusts the content and tone of the audio alert accordingly. Hereinafter, embodiments of the present invention will be described in detail.

[0233] System Configuration

[0234] The device is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data, including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0235] System Operation

[0236] When a user puts on the camera-equipped audio glasses, the device activates the camera and begins capturing video data. This includes not only the surrounding scenery but also the user's face. The device then sends the video data to the analysis means. The analysis means first uses the object recognition means to identify objects in the video (e.g., pedestrians, cars, and other obstacles). Based on this, the risk assessment means evaluates the risk. Next, the signal recognition means identifies traffic signals in the video. Specifically, it recognizes the color of the signal (red, yellow, green) and determines its status. The emotion recognition means also recognizes emotions from the user's facial expressions. This includes algorithms for identifying emotional states such as smiling, anger, and surprise. The device comprehensively evaluates the analysis results from the object recognition means, signal recognition means, and emotion recognition means and generates the necessary audio alert. The generated audio alert is then notified to the user in real time. The tone and content of the audio alert are adjusted according to the emotion recognition results.

[0237] Specific hardware and software names used

[0238] 1. Camera device: A device that captures the user's surroundings and facial expressions in real time.

[0239] 2. Analysis method: A computer for analyzing video data, using object recognition algorithms such as TensorFlow and YOLOv3, image processing libraries such as OpenCV, and deep learning algorithms such as the Facial Emotion Recognition (FER) model.

[0240] 3. Voice generation means: Has the function of generating a voice alert based on the analysis results.

[0241] 4. Scene capture and facial recognition software: Software used to capture and analyze video data in real time.

[0242] Specific examples

[0243] Example 1:

[0244] Assume that a user is walking in a busy shopping district. At this time, the device's camera device captures video, and the object recognition means detects vehicles and pedestrians in the video. If the risk assessment means detects a car suddenly approaching at an intersection, it judges this to be "danger." The device also analyzes the user's facial expression using the emotion recognition means and recognizes that the user looks nervous. In this case, the device generates a voice alert in a calm tone saying, "Be careful." If the traffic light recognition means recognizes a red light, it generates a voice alert saying, "The light is red, please wait," to notify the user.

[0245] Example 2:

[0246] Assume that a user is walking in a park. The device's camera captures video, and the object recognition means detects children and dogs playing. If the danger assessment means determines that there is no particular danger, the device generates a voice alert in a normal tone saying, "Be careful of your surroundings." The device also recognizes through its emotion recognition means that the user is relaxed. In this case, the voice alert is provided in a calm tone while still raising caution.

[0247] Prompt Sentence Examples

[0248] "Generate an audio alert if a car approaches quickly while walking downtown. If the user is nervous, tone down the tone of the alert."

[0249] "Generate audio alerts to get your attention based on your surroundings while walking through the park. If the user is relaxed, notify them in a gentle tone."

[0250] The present invention supports safe and secure behavior by taking into account the user's emotional state in real time, recognizing surrounding dangers and traffic signal information, and providing appropriate audio alerts.

[0251] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0252] Step 1:

[0253] When a user puts on the camera-equipped audio glasses, the device activates the camera and starts capturing video data. The input is a video of the user's environment and face, and the output is real-time video data. This video data includes the user's facial expressions and the surrounding scenery. Specifically, the camera captures video at a rate of 30 frames per second (fps).

[0254] Step 2:

[0255] The terminal transmits the captured video data to the analysis means. The input is real-time video data, and the output is data transmission to the analysis means. The video data is temporarily stored in a buffer, and individual frames are analyzed sequentially. Specifically, the video data is stored in a buffer and prepared for analysis.

[0256] Step 3:

[0257] The device's analysis means uses an object recognition algorithm (e.g., YOLOv3) to identify objects in the video. The input is each frame of video data stored in the buffer, and the output is information about the recognized objects. Specifically, it detects objects such as cars and pedestrians and identifies their location and type. For example, a specific set of pixels in a frame is recognized as a "car," and that information is passed on to the next step.

[0258] Step 4:

[0259] The device's analysis method uses image processing libraries such as OpenCV to identify traffic signals in the video. The input is each frame of video data stored in the buffer, and the output is the recognized signal color (red, yellow, green). Specifically, it detects the location of the traffic signal and analyzes its color from pixel values. For example, if a "red light" is detected, that information influences the next step.

[0260] Step 5:

[0261] The device's analysis means analyzes the user's facial expression data using an emotion recognition algorithm (e.g., Facial Emotion Recognition). The input is the video data of the user's face stored in a buffer, and the output is the recognized emotion (e.g., smile, anger, surprise). Specifically, the emotion of "surprise" is recognized from the movement of the user's eyebrows and the shape of their mouth.

[0262] Step 6:

[0263] The device comprehensively evaluates the analysis results obtained from the object recognition means, signal recognition means, and emotion recognition means, and generates the necessary voice alert. The input is the analysis result from each recognition means, and the output is the generated voice alert. For example, if a dangerous object is detected and the user is startled, a gentle voice alert such as "It's dangerous, please be careful" is generated.

[0264] Step 7:

[0265] The terminal notifies the user of the generated audio alert in real time. The input is the generated audio alert, and the output is a notification to the user as an audio alert. In specific operations, the audio generating means generates an audio alert with a predetermined tone, which is notified through the user's audio glasses.

[0266] (Application example 2)

[0267] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0268] Conventional camera devices and video analysis systems can analyze captured video data and provide audio alerts to warn users of danger and other dangers. However, these systems are often limited in their effectiveness because they do not take into account the user's emotional state. Furthermore, detecting suspicious individuals or dangerous objects through simple object recognition alone may not be sufficient. Therefore, there is a need for systems that can recognize the user's emotional state in real time and appropriately adjust the tone and content of alerts based on that information.

[0269] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server is equipped with a camera device and includes analysis means for analyzing video data acquired by the camera device, audio generation means for generating an audio alert based on the analyzed data, emotion recognition means for recognizing the user's emotion by the analysis means, and means for adjusting the tone and content of the audio alert based on the emotional state recognized by the emotion recognition means. This enables appropriate and effective audio alerts to be generated in accordance with the user's emotional state.

[0270] A "camera device" is an optical device for acquiring video data.

[0271] "Video data" refers to a series of image information captured by a camera device.

[0272] "Analysis means" refers to equipment or software that processes acquired video data and extracts and analyzes specific information.

[0273] "Audio generating means" refers to a device or software that generates an audio alert based on the data obtained by the analyzing means.

[0274] The "emotion recognition means" is a device or software for analyzing the facial expressions and movements of the user in the video data and determining the user's emotional state.

[0275] "Tone" refers to the manner of delivery and tone of voice of the audio alert, which is adjusted according to emotion and urgency.

[0276] "Content" refers to the specific message or information in the audio alert.

[0277] "Object recognition means" refers to a device or software for identifying objects in video data and extracting information about them.

[0278] The "risk assessment means" is a device or software for assessing potential danger based on the information obtained by the object recognition means and determining the risk level.

[0279] A "signal recognition means" is a device or software for identifying a traffic signal within the video data and determining whether it is in a particular state.

[0280] The present invention relates to a system that includes a camera device, analyzes captured video data, and generates an audio alert, and further relates to a system that recognizes a user's emotions and adjusts the content and tone of the audio alert according to the emotions.

[0281] System Configuration

[0282] The terminal is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a danger assessment means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0283] Hardware and software used

[0284] Hardware:

[0285] Camera device (e.g. webcam)

[0286] Microphone and speaker unit

[0287] software:

[0288] TensorFlow (object recognition and emotion recognition models)

[0289] OpenCV (camera image capture and processing)

[0290] gTTS (Google Text-to-Speech, generating audio alerts)

[0291] Data processing and calculation

[0292] The angle is initially input from the video data captured by the camera. The video data is analyzed by the analysis means in the following steps:

[0293] 1. Object recognition:

[0294] OpenCV is used to detect objects in the environment, and a TensorFlow model is used to identify specific objects.

[0295] 2. Risk Assessment:

[0296] The system evaluates the risk level based on the objects identified by the object recognition means. For example, if it detects suspicious movements, it will determine that the risk is high.

[0297] 3. Signal recognition:

[0298] A signal recognition means identifies traffic signals in the video data and identifies their status (red, yellow, green).

[0299] 4. Emotion recognition:

[0300] Using a TensorFlow model, we recognize the user's emotional state from the captured facial expressions, thereby determining whether the user is surprised or relaxed.

[0301] 5. Audio alert generation:

[0302] Based on the analysis results and the user's emotional state, gTTS is used to generate an audio alert with the optimal tone and content.

[0303] Specific examples

[0304] As a concrete example of how the system works, consider the case where a suspicious individual breaks into a room while the user is inside. The camera captures the scene, and the object recognition means identifies the presence of the suspicious individual. The danger assessment means determines that the risk is high, and the emotion recognition means detects the user's surprised facial expression. Based on the results of this analysis, gTTS generates a calm-toned voice alert, such as "Please be careful, a suspicious individual has been detected," to notify the user.

[0305] Prompt Sentence Examples

[0306] If the user is startled, create a program that will alert them in a calm tone, saying "Caution, a suspicious person has been detected."

[0307] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0308] Step 1:

[0309] The device activates the camera device and captures video data in real time. The input includes the surrounding scenery and the user's face. The output is the captured video data.

[0310] Step 2:

[0311] The device receives video data captured using OpenCV and performs object recognition. The input video data is analyzed using an object recognition model (TensorFlow). The output is information about the identified objects. Specifically, it detects moving objects and distinctive shapes in the video and classifies them as objects.

[0312] Step 3:

[0313] The terminal uses the output of object recognition to perform risk assessment. The input is the result of object recognition, and the risk is assessed by the risk assessment means. The output is the risk level assessment result. Specifically, if suspicious movement is detected, the situation is judged to be "high risk."

[0314] Step 4:

[0315] The device uses analysis means to extract the user's facial expressions from the video data and perform emotion recognition. The input is the captured video data, which is analyzed by an emotion recognition model (TensorFlow). The output is the user's emotional state. Specifically, it identifies emotions such as whether the user is surprised, nervous, or relaxed.

[0316] Step 5:

[0317] The terminal uses a signal recognition means to identify traffic signals in the video data. The input is the captured video data, and the analysis means determines the color and state of the signal. The output is the signal state (e.g., red, yellow, green). Specifically, if a red signal is displayed, it is determined to be a "stop" signal.

[0318] Step 6:

[0319] The device integrates the results of object recognition, danger assessment, emotion recognition, and signal recognition, and generates a voice alert using a voice generation means. The input is the analysis results, and gTTS is used to create a voice alert with an appropriate tone and content. The output is the generated voice alert. Specifically, if the user is surprised, it generates an alert with a calm tone such as "Be careful, a suspicious person has been detected."

[0320] Step 7:

[0321] The device notifies the user of the generated audio alert. The input is the audio alert data, and the output is a warning message that is audibly provided to the user. Specifically, the audio alert is played using a speaker.

[0322] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0323] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0324] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0325] [Second embodiment]

[0326] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0327] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0328] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0329] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0330] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0331] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0332] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0333] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0334] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0335] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0336] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0337] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0338] The present invention relates to a system that includes a camera device, analyzes video data, and generates a voice alert. Hereinafter, an embodiment of the present invention will be described.

[0339] System Configuration

[0340] The terminal is equipped with a camera device that captures images of the surroundings in real time. The captured image data is processed by an analysis means to extract specific information (e.g., object identification and traffic signal recognition). Based on the analysis results, an audio generation means generates an audio alert to notify the user.

[0341] System Operation

[0342] 1. When a user wears the camera-equipped audio glasses, the device activates the camera and starts capturing video data continuously.

[0343] 2. The device sends the captured video data to the analysis means, and the object recognition means identifies objects in the video, for example, using a specific algorithm to detect cars, pedestrians, and other obstacles.

[0344] 3. The terminal performs risk assessment using the danger assessment means based on the object recognized by the object recognition means, for example, if an approaching vehicle is within a dangerous distance, the terminal evaluates the information.

[0345] 4. The device uses a signal recognition means to identify traffic signals in the video. Specifically, it identifies the color of the signal (red, yellow, green) and determines its status.

[0346] 5. The device uses the voice generation means to generate necessary voice alerts based on the analysis results, for example, when a dangerous object is detected or a red light is recognized, it generates a corresponding voice message to notify the user.

[0347] Specific examples

[0348] Suppose a user is walking down a busy street. The device's camera captures video and its object recognition system detects vehicles and pedestrians in the video. If a pedestrian suddenly tries to cross the road, the device immediately assesses this risk and generates a voice alert saying, "There is a pedestrian ahead. Please be careful."

[0349] Next, when the user approaches an intersection and the traffic light recognition means recognizes that the traffic light is red, the device provides a voice alert saying, "Red light. Please wait." When the light turns green, the device generates a voice alert saying, "Green light. Please proceed."

[0350] The system provides users with real-time audio information about their surroundings, helping to ensure the safety of people with visual impairments and those who are distracted.

[0351] The processing flow will be explained below.

[0352] Step 1:

[0353] The device initializes the camera and starts capturing video data, which specifically means configuring the camera device and starting continuous frame capture.

[0354] Step 2:

[0355] The device sends the captured video frames to the analysis means. The analysis means receives the video data and detects objects in the frames using the object recognition means. For example, it applies a specific algorithm (e.g., YOLO, Haar Cascades, etc.) to identify objects (pedestrians, cars, obstacles, etc.).

[0356] Step 3:

[0357] The terminal evaluates the risk using the danger evaluation means based on the output of the object recognition means. If a certain distance or speed is exceeded, it can be judged as "dangerous." The evaluation result is passed to the next processing step.

[0358] Step 4:

[0359] The device uses a signal recognition means to identify traffic signals from video frames captured by the camera. Specifically, it recognizes the color of the signal and determines its status (red, yellow, green). For example, it uses color region extraction and pattern recognition to identify the color of the signal.

[0360] Step 5:

[0361] The device generates an appropriate voice alert using the voice generation means based on the analysis results from the danger assessment means and the signal recognition means. For example, when a red light is recognized or a dangerous object is detected, it generates a voice message such as "Red light" or "Dangerous object present," respectively.

[0362] Step 6:

[0363] The device notifies the user of the generated voice alert by activating a speech synthesis engine and providing a voice alert to the user in real time, allowing the user to take appropriate action.

[0364] Step 7:

[0365] Based on the audio alerts, the user can take appropriate action depending on the surrounding situation, for example, stopping when hearing the audio alert for a red light, or crossing the road when hearing the audio alert for a green light.

[0366] This series of steps allows the system to recognize surrounding hazards and traffic signals in real time and provide appropriate audio alerts to the user.

[0367] Example 1

[0368] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0369] In modern society, people with visual impairments and those who are distracted need to be able to travel safely. However, current safety assistance devices often have difficulty accurately recognizing their surroundings in real time and providing timely warnings. They also lack the ability to accurately recognize traffic signal status and approaching dangerous objects and communicate them safely. These situations make it difficult to ensure the safety of people with visual impairments.

[0370] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0371] In this invention, the server includes a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, a risk assessment means for determining the surrounding situation in real time, a signal recognition means for determining the status of traffic signals, and a notification means for notifying the user of the voice alert, thereby enabling the user to accurately grasp the surrounding situation in real time and receive a warning at an appropriate time.

[0372] A "camera device" is a device for acquiring video data.

[0373] The "analysis means" is a device or software that processes the video data acquired by the camera device and extracts specific information.

[0374] "Audio generating means" refers to a device or software that generates an audio alert based on the data obtained by the analyzing means.

[0375] "Risk assessment means" means a device or software that assesses the surrounding situation in real time, in particular by performing risk assessment based on recognized objects and traffic signals.

[0376] "Signal recognition means" refers to a device or software that recognizes the state of a traffic signal (red, yellow, green) based on acquired image data.

[0377] A "notification means" is a device or software for notifying a user of a generated audio alert.

[0378] This invention is a system that uses a device equipped with a camera to capture and analyze surrounding video data in real time, generating audio alerts, primarily for the purpose of ensuring the safety of people with visual impairments and those who are distracted.

[0379] Hardware and software used

[0380] Device: For example, audio glasses with a camera

[0381] Camera device: Camera built into the device

[0382] Analysis tools: High-performance processors and dedicated object recognition algorithms (e.g., YOLO, TensorFlow)

[0383] Voice Generation Method: Text-to-Speech (TTS) software (e.g., Google Cloud Text-to-Speech, Amazon Polly)

[0384] Hazard Assessment Tools: Risk Assessment Algorithms

[0385] Signal Recognition: An algorithm for identifying the color of traffic signals

[0386] Notification Method: Speaker or earphone to notify the user of audio alerts

[0387] System Configuration

[0388] 1. The terminal uses a camera device to capture video data of the user's surroundings in real time.

[0389] 2. The captured video data is sent to the device's built-in analysis means, which uses an object recognition algorithm (e.g., YOLO).

[0390] 3. The device uses analytical means to identify objects in the video (cars, pedestrians, traffic lights, etc.).

[0391] 4. Based on the identified object, the danger assessment means assesses the risk, for example, if an approaching car is at a dangerous distance, the information is assessed immediately.

[0392] 5. The device uses signal recognition means to identify traffic signals in the video, for example, by determining the color of the signal (red, yellow, green) and determining its status.

[0393] 6. Finally, the necessary audio alert is generated using the audio generation means based on the analysis results, and the generated audio alert is notified to the user through the notification means.

[0394] Specific examples

[0395] Consider a scenario where a user is walking in a busy downtown area. The device's camera captures video and the acquired data is sent to an analysis means (e.g., the YOLO algorithm). The object recognition means detects vehicles and pedestrians, and the risk assessment means immediately assesses the risk. If a pedestrian suddenly tries to cross the road, the device generates a voice alert saying, "There is a pedestrian ahead. Please be careful," and notifies the user through a notification means.

[0396] Next, when the user approaches an intersection with a traffic light, the traffic light recognition means determines the color of the traffic light, and when it recognizes a red light, it generates a voice alert saying, "Red light. Please wait." When the light turns green, it provides an alert saying, "Green light. Please proceed."

[0397] Prompt Sentence Examples

[0398] The generative AI model generates a voice alert by providing a prompt like this:

[0399] There is a pedestrian ahead. Please be careful.

[0400] Red light, please wait. Green light, please go ahead.

[0401] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0402] Step 1:

[0403] The user wears audio glasses with a camera.

[0404] Input: Audio glasses with camera

[0405] Output: System boot

[0406] Specific operation: When the user puts on the audio glasses and turns on the power switch, the system starts up.

[0407] Step 2:

[0408] The terminal activates the camera device and captures the video data.

[0409] Input: Powered camera device

[0410] Output: Real-time video data

[0411] What it does: The camera device begins capturing video of the surroundings in real time, for example, capturing a continuous 360-degree view.

[0412] Step 3:

[0413] The terminal transmits the captured video data to the analysis means.

[0414] Input: Real-time video data

[0415] Output: Video data sent to analysis means

[0416] Specific operation: The video data captured by the camera is sent to the internal high-performance processor, which allows it to proceed to the next stage of analysis.

[0417] Step 4:

[0418] The terminal uses object recognition means to identify objects in the video.

[0419] Input: Transmitted video data

[0420] Output: Recognized object data

[0421] How it works: Object recognition methods such as the YOLO algorithm analyze video data and identify objects such as cars and pedestrians. The results are updated for each frame.

[0422] Step 5:

[0423] The terminal performs a risk assessment using a risk assessment means.

[0424] Input: Recognized object data

[0425] Output: The result of the risk assessment

[0426] Specific operation: Based on data such as the speed and distance of the recognized object, the risk assessment method evaluates the possibility of contact. For example, it immediately determines whether an approaching vehicle is within a dangerous distance.

[0427] Step 6:

[0428] The terminal uses the signal recognition means to identify the traffic signal.

[0429] Input: Video data

[0430] Output: Traffic light status (color)

[0431] Specific operation: The signal recognition means detects traffic signals from the video data and distinguishes their colors (red, yellow, green). For example, it analyzes the traffic light and recognizes a red signal.

[0432] Step 7:

[0433] The terminal generates an audio alert using the audio generating means to notify the user.

[0434] Input: Risk assessment results, traffic signal status

[0435] Output: Audio alert

[0436] Specific operation: A voice generation means (e.g., Google Cloud Text-to-Speech) generates a voice message based on the results of risk assessment and traffic signal recognition. This voice alert is delivered to the user through a notification means. For example, if there is a high risk of an approaching vehicle, the user will be notified by a voice saying, "There is a vehicle ahead. Please be careful."

[0437] (Application example 1)

[0438] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0439] In recent years, there has been a demand for improved safety in autonomous vehicles, but collisions with people and obstacles remain frequent. Technology that can properly recognize the surrounding situation in real time and respond immediately is lacking, especially in complex situations such as busy streets and intersections. There is also a lack of effective means to notify drivers of emergency situations using voice alerts. This calls for new technologies to improve the safety of autonomous vehicles and prevent accidents.

[0440] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0441] In this invention, the server is equipped with a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, an object identification means for identifying objects from the video data in real time, a risk assessment means for assessing a risk within a specific distance based on the identified objects, and a means for generating a voice alert when the risk level exceeds an assessment standard. This enables the server to recognize the situation around the autonomous vehicle in real time and immediately provide a voice alert in an emergency, thereby improving safety.

[0442] A "camera device" is a device for acquiring video data.

[0443] The "analysis means" is a means having a function for processing the acquired video data and extracting specific information.

[0444] The "audio generating means" is a means having the function of generating an audio alert based on the analyzed data.

[0445] The "object identification means" is a means having a function for identifying an object from video data in real time.

[0446] A "danger assessment means" is a means having the function of assessing danger within a certain distance based on an identified object.

[0447] The "signal recognition means" is a means having a function of recognizing traffic signals based on image data acquired by a camera device.

[0448] "Real-time" refers to processing or reacting immediately, without any time delay.

[0449] "Audio alert" refers to a warning or guidance given to the user by voice.

[0450] "Evaluation criteria" refers to the numerical values ​​and conditions that serve as the criteria when the risk assessment means makes an assessment.

[0451] "Analyzed data" refers to data after it has been processed by an analytical means.

[0452] The present invention relates to a system for improving the safety of autonomous vehicles.

[0453] System Configuration

[0454] The system includes a camera device, an analysis means, a sound generation means, an object identification means, a risk assessment means, and a signal recognition means.

[0455] 1. Camera equipment

[0456] This camera device acquires surrounding video data in real time. In particular, a camera installed in an autonomous vehicle is used. A specific example of a camera is an in-vehicle camera system.

[0457] 2. Analysis method

[0458] The acquired video data is processed by the analysis means, which uses libraries such as OpenCV and TensorFlow. The analysis means works in conjunction with the object identification means to recognize specific objects such as pedestrians and other vehicles.

[0459] 3. Voice Generation Method

[0460] Based on the analyzed data, audio alerts are generated using gTTS (Google Text-to-Speech) and other technologies, which are then sent to the user via a speaker in real time.

[0461] 4. Object Identification Method

[0462] It uses deep learning models such as TensorFlow to identify objects in video data in real time, such as pedestrians, other vehicles, and road signs.

[0463] 5. Risk Assessment Tools

[0464] Based on the identified object, it evaluates the object's potential danger to the autonomous vehicle, based on criteria such as a specific distance, speed, direction, etc. If the danger exceeds the criteria, an audio alert is generated.

[0465] 6. Signal recognition means

[0466] The system recognizes traffic signals based on the video data acquired by the camera device. This recognition means also works in conjunction with the analysis means to recognize the traffic light status (red, yellow, green) and provide that information to the user.

[0467] Specific examples

[0468] As a specific example, consider a case where an autonomous vehicle approaches an intersection. A camera device captures video of the area around the intersection, and an object identification means identifies pedestrians and other vehicles. If a pedestrian is assessed as approaching the vehicle, the risk assessment means immediately determines this information and generates an audio alert saying, "Pedestrian approaching." If the traffic light is red, an audio alert saying, "Red light. Please wait" is also generated.

[0469] Prompt Sentence Examples

[0470] Below is an example of a prompt for a program in a system for autonomous vehicles that recognizes the surrounding situation in real time and provides a voice alert in the event of an emergency.

[0471] Prompt: Write a program for a system that uses dashcam footage to detect pedestrians near intersections in real time and provide audio alerts.

[0472] This allows the present invention to improve safety by providing real-time awareness of the situation around an autonomous vehicle and providing immediate audio alerts in the event of an emergency.

[0473] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0474] Step 1:

[0475] The device activates a camera device and captures video data of the surroundings in real time. The input is video data from the camera. The camera captures the surroundings at a high frame rate and continuously collects the video data. The output is video data for each frame.

[0476] Step 2:

[0477] The device sends the captured video data to the analysis means. The input is the video data obtained in step 1. The video data is preprocessed using the OpenCV library and converted into an appropriate format. This process allows the analysis means to process the data efficiently. The output is the preprocessed video data.

[0478] Step 3:

[0479] The analysis means uses the preprocessed video data to perform object identification using the object identification means. The preprocessed video data is used as input. Object identification is performed using deep learning using a generative AI model from TensorFlow. This process identifies specific objects such as pedestrians and vehicles from each frame of video. The output is information about the identified objects (position, type).

[0480] Step 4:

[0481] The terminal performs a risk assessment using the risk assessment means based on the object information identified by the object identification means. The identified object information is input. Based on this information, the distance to the object, relative speed, direction, etc. are evaluated. Here, an evaluation standard is set, and if the risk is high, a flag is raised as a high risk. The risk assessment result is obtained as the output.

[0482] Step 5:

[0483] The terminal uses the voice generation means to generate the necessary voice alert based on the result of the danger assessment means. The input is the danger assessment result. Using gTTS (Google Text-to-Speech), a voice message is generated according to the assessment result. For example, if a pedestrian is approaching, a message such as "Pedestrian approaching" is generated, and if a stop is required at a red light, a message such as "Red light. Please wait" is generated. The output is the generated voice alert (voice file).

[0484] Step 6:

[0485] The device notifies the user of the generated audio alert. It has as input the generated audio alert. It plays the audio file through the speaker to provide a real-time warning to the user. It has as output the audio alert delivered to the user.

[0486] This allows the device to monitor the autonomous vehicle's surroundings in real time and generate and provide instant voice alerts when necessary, improving driver safety.

[0487] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0488] The present invention relates to a system that includes a camera device, analyzes video data, and generates an audio alert. The present invention also relates to a system that includes emotion recognition means that recognizes a user's emotion and adjusts the content and tone of the audio alert accordingly. Hereinafter, an embodiment of the present invention will be described.

[0489] System Configuration

[0490] The device is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data, including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0491] System Operation

[0492] 1. When a user puts on the camera-equipped audio glasses, the device starts the camera device and starts capturing video data, including not only the surrounding scenery but also the user's face.

[0493] 2. The device sends the video data to the analysis means, which first uses the object recognition means to identify objects in the video (e.g., pedestrians, cars, and other obstacles). Based on this, the risk assessment means assesses the risk.

[0494] 3. The device then uses its signal recognition means to identify traffic signals in the video. Specifically, it recognizes the color of the signal (red, yellow, green) and determines its status.

[0495] 4. The device further uses emotion recognition means to recognize emotions from the user's facial expressions, including algorithms for identifying emotional states such as smiling, angry, surprised, etc.

[0496] 5. The device will comprehensively evaluate the analysis results from the object recognition means, signal recognition means, and emotion recognition means and generate the necessary audio alert. For example, if a dangerous object is detected and the user is startled, the tone of the audio alert will be adjusted to be softer.

[0497] 6. The device then generates and sends the generated audio alert to the user in real time, adjusting the tone and content of the audio notification based on the emotion recognition results.

[0498] Specific examples

[0499] Suppose a user is walking in a busy shopping district. The device's camera captures video, and the object recognition means detects vehicles and pedestrians in the video. If the risk assessment means detects a car suddenly approaching at an intersection, it determines this as "danger."

[0500] The device also analyzes the user's facial expression using emotion recognition means and recognizes that the user is nervous. In this case, the device generates a voice alert in a calm tone saying, "Please be careful." If the traffic light recognition means recognizes a red light, the device generates a voice alert saying, "The light is red, please wait," to notify the user.

[0501] In this way, the present invention takes into account the user's emotional state, recognizes surrounding hazards and traffic signals in real time, and provides tailored audio alerts to the user, supporting safe and secure behavior.

[0502] The processing flow will be explained below.

[0503] Step 1:

[0504] The device initializes the camera and starts capturing video data, capturing images of the surrounding scenery and the user's face in real time.

[0505] Step 2:

[0506] The terminal transmits the captured video frames to the analysis means, which receives them and uses the object recognition means to detect objects (e.g., cars, pedestrians, and obstacles) in the frames.

[0507] Step 3:

[0508] The terminal uses the risk assessment means to assess the risk based on the output of the object recognition means. For example, if an approaching vehicle is within a certain distance, it is judged to be "dangerous."

[0509] Step 4:

[0510] The device uses a signal recognition means to analyze traffic signals in the video frames, recognize the signal color (red, yellow, green) and determine its status. This information is used in the next step.

[0511] Step 5:

[0512] The device analyzes the user's facial expressions using an emotion recognition means. The emotion recognition means identifies emotions (e.g., tension, surprise, joy) from the user's facial expressions in the video. The device then determines the user's emotional state from the facial expressions.

[0513] Step 6:

[0514] The device combines analysis results from object recognition, signal recognition, and emotion recognition methods, so if the user is nervous, for example, it will soften the content and tone of the audio alerts.

[0515] Step 7:

[0516] The terminal generates an appropriate audio alert using the audio generation means based on the integration result. For example, if a dangerous object is detected and the user is startled, the terminal generates an audio alert saying "Be careful" in a calm tone.

[0517] Step 8:

[0518] The device generates audio alerts and notifies the user in real time, driving a speech synthesis engine to play audio with adjusted tone and content to inform the user of the situation.

[0519] Step 9:

[0520] Users can take appropriate action based on the audio alerts provided, such as stopping at a red light when necessary or being aware of hazards.

[0521] This step allows the device to analyze the surrounding video in real time and provide audio alerts that take into account the user's emotional state, with the aim of ensuring the user's safety and peace of mind.

[0522] Example 2

[0523] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0524] Conventional systems can analyze video data captured by cameras and generate audio alerts, but they are unable to generate appropriate alerts based on the user's emotions. This creates the risk of the user being surprised or making a wrong decision. Furthermore, comprehensive risk assessment is not adequately performed when recognizing traffic signals and surrounding objects.

[0525] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0526] In this invention, the server is equipped with a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, and an emotion recognition means for recognizing the user's emotion and adjusting the content and tone of the voice alert accordingly, thereby enabling the server to generate and notify an appropriate voice alert in real time according to the user's emotional state.

[0527] "Camera device" refers to a device for acquiring video data, capturing the user's surroundings and facial expressions.

[0528] "Analysis means" refers to a function or device for processing video data acquired by a camera device and performing object recognition, signal recognition, and emotion recognition.

[0529] "Voice generation means" refers to a function or device for generating a voice alert based on the analyzed data, and notifying the user of necessary information by voice.

[0530] "Emotion recognition means" refers to a function or device for analyzing the user's facial expression data and identifying the user's emotional state (e.g., smile, anger, surprise).

[0531] "Object recognition means" refers to a function or device for identifying surrounding objects (e.g., pedestrians, automobiles, and other obstacles) based on video data.

[0532] The "risk assessment means" refers to a function or device for conducting risk assessment and determining danger based on the object recognized by the object recognition means.

[0533] "Signal recognition means" refers to a function or device for recognizing traffic signals (e.g., red, yellow, green) based on video data.

[0534] MODE FOR CARRYING OUT THE INVENTION

[0535] The present invention relates to a system that includes a camera device, analyzes video data, and generates an audio alert. The present invention also relates to a system that includes emotion recognition means that recognizes a user's emotion and adjusts the content and tone of the audio alert accordingly. Hereinafter, embodiments of the present invention will be described in detail.

[0536] System Configuration

[0537] The device is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data, including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0538] System Operation

[0539] When a user puts on the camera-equipped audio glasses, the device activates the camera and begins capturing video data. This includes not only the surrounding scenery but also the user's face. The device then sends the video data to the analysis means. The analysis means first uses the object recognition means to identify objects in the video (e.g., pedestrians, cars, and other obstacles). Based on this, the risk assessment means evaluates the risk. Next, the signal recognition means identifies traffic signals in the video. Specifically, it recognizes the color of the signal (red, yellow, green) and determines its status. The emotion recognition means also recognizes emotions from the user's facial expressions. This includes algorithms for identifying emotional states such as smiling, anger, and surprise. The device comprehensively evaluates the analysis results from the object recognition means, signal recognition means, and emotion recognition means and generates the necessary audio alert. The generated audio alert is then notified to the user in real time. The tone and content of the audio alert are adjusted according to the emotion recognition results.

[0540] Specific hardware and software names used

[0541] 1. Camera device: A device that captures the user's surroundings and facial expressions in real time.

[0542] 2. Analysis method: A computer for analyzing video data, using object recognition algorithms such as TensorFlow and YOLOv3, image processing libraries such as OpenCV, and deep learning algorithms such as the Facial Emotion Recognition (FER) model.

[0543] 3. Voice generation means: Has the function of generating a voice alert based on the analysis results.

[0544] 4. Scene capture and facial recognition software: Software used to capture and analyze video data in real time.

[0545] Specific examples

[0546] Example 1:

[0547] Assume that a user is walking in a busy shopping district. At this time, the device's camera device captures video, and the object recognition means detects vehicles and pedestrians in the video. If the risk assessment means detects a car suddenly approaching at an intersection, it judges this to be "danger." The device also analyzes the user's facial expression using the emotion recognition means and recognizes that the user looks nervous. In this case, the device generates a voice alert in a calm tone saying, "Be careful." If the traffic light recognition means recognizes a red light, it generates a voice alert saying, "The light is red, please wait," to notify the user.

[0548] Example 2:

[0549] Assume that a user is walking in a park. The device's camera captures video, and the object recognition means detects children and dogs playing. If the danger assessment means determines that there is no particular danger, the device generates a voice alert in a normal tone saying, "Be careful of your surroundings." The device also recognizes through its emotion recognition means that the user is relaxed. In this case, the voice alert is provided in a calm tone while still raising caution.

[0550] Prompt Sentence Examples

[0551] "Generate an audio alert if a car approaches quickly while walking downtown. If the user is nervous, tone down the tone of the alert."

[0552] "Generate audio alerts to get your attention based on your surroundings while walking through the park. If the user is relaxed, notify them in a gentle tone."

[0553] The present invention supports safe and secure behavior by taking into account the user's emotional state in real time, recognizing surrounding dangers and traffic signal information, and providing appropriate audio alerts.

[0554] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0555] Step 1:

[0556] When a user puts on the camera-equipped audio glasses, the device activates the camera and starts capturing video data. The input is a video of the user's environment and face, and the output is real-time video data. This video data includes the user's facial expressions and the surrounding scenery. Specifically, the camera captures video at a rate of 30 frames per second (fps).

[0557] Step 2:

[0558] The terminal transmits the captured video data to the analysis means. The input is real-time video data, and the output is data transmission to the analysis means. The video data is temporarily stored in a buffer, and individual frames are analyzed sequentially. Specifically, the video data is stored in a buffer and prepared for analysis.

[0559] Step 3:

[0560] The device's analysis means uses an object recognition algorithm (e.g., YOLOv3) to identify objects in the video. The input is each frame of video data stored in the buffer, and the output is information about the recognized objects. Specifically, it detects objects such as cars and pedestrians and identifies their location and type. For example, a specific set of pixels in a frame is recognized as a "car," and that information is passed on to the next step.

[0561] Step 4:

[0562] The device's analysis method uses image processing libraries such as OpenCV to identify traffic signals in the video. The input is each frame of video data stored in the buffer, and the output is the recognized signal color (red, yellow, green). Specifically, it detects the location of the traffic signal and analyzes its color from pixel values. For example, if a "red light" is detected, that information influences the next step.

[0563] Step 5:

[0564] The device's analysis means analyzes the user's facial expression data using an emotion recognition algorithm (e.g., Facial Emotion Recognition). The input is the video data of the user's face stored in a buffer, and the output is the recognized emotion (e.g., smile, anger, surprise). Specifically, the emotion of "surprise" is recognized from the movement of the user's eyebrows and the shape of their mouth.

[0565] Step 6:

[0566] The device comprehensively evaluates the analysis results obtained from the object recognition means, signal recognition means, and emotion recognition means, and generates the necessary voice alert. The input is the analysis result from each recognition means, and the output is the generated voice alert. For example, if a dangerous object is detected and the user is startled, a gentle voice alert such as "It's dangerous, please be careful" is generated.

[0567] Step 7:

[0568] The terminal notifies the user of the generated audio alert in real time. The input is the generated audio alert, and the output is a notification to the user as an audio alert. In specific operations, the audio generating means generates an audio alert with a predetermined tone, which is notified through the user's audio glasses.

[0569] (Application example 2)

[0570] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0571] Conventional camera devices and video analysis systems can analyze captured video data and provide audio alerts to warn users of danger and other dangers. However, these systems are often limited in their effectiveness because they do not take into account the user's emotional state. Furthermore, detecting suspicious individuals or dangerous objects through simple object recognition alone may not be sufficient. Therefore, there is a need for systems that can recognize the user's emotional state in real time and appropriately adjust the tone and content of alerts based on that information.

[0572] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server is equipped with a camera device and includes analysis means for analyzing video data acquired by the camera device, audio generation means for generating an audio alert based on the analyzed data, emotion recognition means for recognizing the user's emotion by the analysis means, and means for adjusting the tone and content of the audio alert based on the emotional state recognized by the emotion recognition means. This enables appropriate and effective audio alerts to be generated in accordance with the user's emotional state.

[0573] A "camera device" is an optical device for acquiring video data.

[0574] "Video data" refers to a series of image information captured by a camera device.

[0575] "Analysis means" refers to equipment or software that processes acquired video data and extracts and analyzes specific information.

[0576] "Audio generating means" refers to a device or software that generates an audio alert based on the data obtained by the analyzing means.

[0577] The "emotion recognition means" is a device or software for analyzing the facial expressions and movements of the user in the video data and determining the user's emotional state.

[0578] "Tone" refers to the manner of delivery and tone of voice of the audio alert, which is adjusted according to emotion and urgency.

[0579] "Content" refers to the specific message or information in the audio alert.

[0580] "Object recognition means" refers to a device or software for identifying objects in video data and extracting information about them.

[0581] The "risk assessment means" is a device or software for assessing potential danger based on the information obtained by the object recognition means and determining the risk level.

[0582] A "signal recognition means" is a device or software for identifying a traffic signal within the video data and determining whether it is in a particular state.

[0583] The present invention relates to a system that includes a camera device, analyzes captured video data, and generates an audio alert, and further relates to a system that recognizes a user's emotions and adjusts the content and tone of the audio alert according to the emotions.

[0584] System Configuration

[0585] The terminal is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a danger assessment means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0586] Hardware and software used

[0587] Hardware:

[0588] Camera device (e.g. webcam)

[0589] Microphone and speaker unit

[0590] software:

[0591] TensorFlow (object recognition and emotion recognition models)

[0592] OpenCV (camera image capture and processing)

[0593] gTTS (Google Text-to-Speech, generating audio alerts)

[0594] Data processing and calculation

[0595] The angle is initially input from the video data captured by the camera. The video data is analyzed by the analysis means in the following steps:

[0596] 1. Object recognition:

[0597] OpenCV is used to detect objects in the environment, and a TensorFlow model is used to identify specific objects.

[0598] 2. Risk Assessment:

[0599] The system evaluates the risk level based on the objects identified by the object recognition means. For example, if it detects suspicious movements, it will determine that the risk is high.

[0600] 3. Signal recognition:

[0601] A signal recognition means identifies traffic signals in the video data and identifies their status (red, yellow, green).

[0602] 4. Emotion recognition:

[0603] Using a TensorFlow model, we recognize the user's emotional state from the captured facial expressions, thereby determining whether the user is surprised or relaxed.

[0604] 5. Audio alert generation:

[0605] Based on the analysis results and the user's emotional state, gTTS is used to generate an audio alert with the optimal tone and content.

[0606] Specific examples

[0607] As a concrete example of how the system works, consider the case where a suspicious individual breaks into a room while the user is inside. The camera captures the scene, and the object recognition means identifies the presence of the suspicious individual. The danger assessment means determines that the risk is high, and the emotion recognition means detects the user's surprised facial expression. Based on the results of this analysis, gTTS generates a calm-toned voice alert, such as "Please be careful, a suspicious individual has been detected," to notify the user.

[0608] Prompt Sentence Examples

[0609] If the user is startled, create a program that will alert them in a calm tone, saying "Caution, a suspicious person has been detected."

[0610] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0611] Step 1:

[0612] The device activates the camera device and captures video data in real time. The input includes the surrounding scenery and the user's face. The output is the captured video data.

[0613] Step 2:

[0614] The device receives video data captured using OpenCV and performs object recognition. The input video data is analyzed using an object recognition model (TensorFlow). The output is information about the identified objects. Specifically, it detects moving objects and distinctive shapes in the video and classifies them as objects.

[0615] Step 3:

[0616] The terminal uses the output of object recognition to perform risk assessment. The input is the result of object recognition, and the risk is assessed by the risk assessment means. The output is the risk level assessment result. Specifically, if suspicious movement is detected, the situation is judged to be "high risk."

[0617] Step 4:

[0618] The device uses analysis means to extract the user's facial expressions from the video data and perform emotion recognition. The input is the captured video data, which is analyzed by an emotion recognition model (TensorFlow). The output is the user's emotional state. Specifically, it identifies emotions such as whether the user is surprised, nervous, or relaxed.

[0619] Step 5:

[0620] The terminal uses a signal recognition means to identify traffic signals in the video data. The input is the captured video data, and the analysis means determines the color and state of the signal. The output is the signal state (e.g., red, yellow, green). Specifically, if a red signal is displayed, it is determined to be a "stop" signal.

[0621] Step 6:

[0622] The device integrates the results of object recognition, danger assessment, emotion recognition, and signal recognition, and generates a voice alert using a voice generation means. The input is the analysis results, and gTTS is used to create a voice alert with an appropriate tone and content. The output is the generated voice alert. Specifically, if the user is surprised, it generates an alert with a calm tone such as "Be careful, a suspicious person has been detected."

[0623] Step 7:

[0624] The device notifies the user of the generated audio alert. The input is the audio alert data, and the output is a warning message that is audibly provided to the user. Specifically, the audio alert is played using a speaker.

[0625] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0626] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0627] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0628] [Third embodiment]

[0629] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0630] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0631] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0632] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0633] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0635] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0636] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0637] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0638] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0639] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0640] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0641] The present invention relates to a system that includes a camera device, analyzes video data, and generates a voice alert. Hereinafter, an embodiment of the present invention will be described.

[0642] System Configuration

[0643] The terminal is equipped with a camera device that captures images of the surroundings in real time. The captured image data is processed by an analysis means to extract specific information (e.g., object identification and traffic signal recognition). Based on the analysis results, an audio generation means generates an audio alert to notify the user.

[0644] System Operation

[0645] 1. When a user wears the camera-equipped audio glasses, the device activates the camera and starts capturing video data continuously.

[0646] 2. The device sends the captured video data to the analysis means, and the object recognition means identifies objects in the video, for example, using a specific algorithm to detect cars, pedestrians, and other obstacles.

[0647] 3. The terminal performs risk assessment using the danger assessment means based on the object recognized by the object recognition means, for example, if an approaching vehicle is within a dangerous distance, the terminal evaluates the information.

[0648] 4. The device uses a signal recognition means to identify traffic signals in the video. Specifically, it identifies the color of the signal (red, yellow, green) and determines its status.

[0649] 5. The device uses the voice generation means to generate necessary voice alerts based on the analysis results, for example, when a dangerous object is detected or a red light is recognized, it generates a corresponding voice message to notify the user.

[0650] Specific examples

[0651] Suppose a user is walking down a busy street. The device's camera captures video and its object recognition system detects vehicles and pedestrians in the video. If a pedestrian suddenly tries to cross the road, the device immediately assesses this risk and generates a voice alert saying, "There is a pedestrian ahead. Please be careful."

[0652] Next, when the user approaches an intersection and the traffic light recognition means recognizes that the traffic light is red, the device provides a voice alert saying, "Red light. Please wait." When the light turns green, the device generates a voice alert saying, "Green light. Please proceed."

[0653] The system provides users with real-time audio information about their surroundings, helping to ensure the safety of people with visual impairments and those who are distracted.

[0654] The processing flow will be explained below.

[0655] Step 1:

[0656] The device initializes the camera and starts capturing video data, which specifically means configuring the camera device and starting continuous frame capture.

[0657] Step 2:

[0658] The device sends the captured video frames to the analysis means. The analysis means receives the video data and detects objects in the frames using the object recognition means. For example, it applies a specific algorithm (e.g., YOLO, Haar Cascades, etc.) to identify objects (pedestrians, cars, obstacles, etc.).

[0659] Step 3:

[0660] The terminal evaluates the risk using the danger evaluation means based on the output of the object recognition means. If a certain distance or speed is exceeded, it can be judged as "dangerous." The evaluation result is passed to the next processing step.

[0661] Step 4:

[0662] The device uses a signal recognition means to identify traffic signals from video frames captured by the camera. Specifically, it recognizes the color of the signal and determines its status (red, yellow, green). For example, it uses color region extraction and pattern recognition to identify the color of the signal.

[0663] Step 5:

[0664] The device generates an appropriate voice alert using the voice generation means based on the analysis results from the danger assessment means and the signal recognition means. For example, when a red light is recognized or a dangerous object is detected, it generates a voice message such as "Red light" or "Dangerous object present," respectively.

[0665] Step 6:

[0666] The device notifies the user of the generated voice alert by activating a speech synthesis engine and providing a voice alert to the user in real time, allowing the user to take appropriate action.

[0667] Step 7:

[0668] Based on the audio alerts, the user can take appropriate action depending on the surrounding situation, for example, stopping when hearing the audio alert for a red light, or crossing the road when hearing the audio alert for a green light.

[0669] This series of steps allows the system to recognize surrounding hazards and traffic signals in real time and provide appropriate audio alerts to the user.

[0670] Example 1

[0671] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0672] In modern society, people with visual impairments and those who are distracted need to be able to travel safely. However, current safety assistance devices often have difficulty accurately recognizing their surroundings in real time and providing timely warnings. They also lack the ability to accurately recognize traffic signal status and approaching dangerous objects and communicate them safely. These situations make it difficult to ensure the safety of people with visual impairments.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0674] In this invention, the server includes a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, a risk assessment means for determining the surrounding situation in real time, a signal recognition means for determining the status of traffic signals, and a notification means for notifying the user of the voice alert, thereby enabling the user to accurately grasp the surrounding situation in real time and receive a warning at an appropriate time.

[0675] A "camera device" is a device for acquiring video data.

[0676] The "analysis means" is a device or software that processes the video data acquired by the camera device and extracts specific information.

[0677] "Audio generating means" refers to a device or software that generates an audio alert based on the data obtained by the analyzing means.

[0678] "Risk assessment means" means a device or software that assesses the surrounding situation in real time, in particular by performing risk assessment based on recognized objects and traffic signals.

[0679] "Signal recognition means" refers to a device or software that recognizes the state of a traffic signal (red, yellow, green) based on acquired image data.

[0680] A "notification means" is a device or software for notifying a user of a generated audio alert.

[0681] This invention is a system that uses a device equipped with a camera to capture and analyze surrounding video data in real time, generating audio alerts, primarily for the purpose of ensuring the safety of people with visual impairments and those who are distracted.

[0682] Hardware and software used

[0683] Device: For example, audio glasses with a camera

[0684] Camera device: Camera built into the device

[0685] Analysis tools: High-performance processors and dedicated object recognition algorithms (e.g., YOLO, TensorFlow)

[0686] Voice Generation Method: Text-to-Speech (TTS) software (e.g., Google Cloud Text-to-Speech, Amazon Polly)

[0687] Hazard Assessment Tools: Risk Assessment Algorithms

[0688] Signal Recognition: An algorithm for identifying the color of traffic signals

[0689] Notification Method: Speaker or earphone to notify the user of audio alerts

[0690] System Configuration

[0691] 1. The terminal uses a camera device to capture video data of the user's surroundings in real time.

[0692] 2. The captured video data is sent to the device's built-in analysis means, which uses an object recognition algorithm (e.g., YOLO).

[0693] 3. The device uses analytical means to identify objects in the video (cars, pedestrians, traffic lights, etc.).

[0694] 4. Based on the identified object, the danger assessment means assesses the risk, for example, if an approaching car is at a dangerous distance, the information is assessed immediately.

[0695] 5. The device uses signal recognition means to identify traffic signals in the video, for example, by determining the color of the signal (red, yellow, green) and determining its status.

[0696] 6. Finally, the necessary audio alert is generated using the audio generation means based on the analysis results, and the generated audio alert is notified to the user through the notification means.

[0697] Specific examples

[0698] Consider a scenario where a user is walking in a busy downtown area. The device's camera captures video and the acquired data is sent to an analysis means (e.g., the YOLO algorithm). The object recognition means detects vehicles and pedestrians, and the risk assessment means immediately assesses the risk. If a pedestrian suddenly tries to cross the road, the device generates a voice alert saying, "There is a pedestrian ahead. Please be careful," and notifies the user through a notification means.

[0699] Next, when the user approaches an intersection with a traffic light, the traffic light recognition means determines the color of the traffic light, and when it recognizes a red light, it generates a voice alert saying, "Red light. Please wait." When the light turns green, it provides an alert saying, "Green light. Please proceed."

[0700] Prompt Sentence Examples

[0701] The generative AI model generates a voice alert by providing a prompt like this:

[0702] There is a pedestrian ahead. Please be careful.

[0703] Red light, please wait. Green light, please go ahead.

[0704] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0705] Step 1:

[0706] The user wears audio glasses with a camera.

[0707] Input: Audio glasses with camera

[0708] Output: System boot

[0709] Specific operation: When the user puts on the audio glasses and turns on the power switch, the system starts up.

[0710] Step 2:

[0711] The terminal activates the camera device and captures the video data.

[0712] Input: Powered camera device

[0713] Output: Real-time video data

[0714] What it does: The camera device begins capturing video of the surroundings in real time, for example, capturing a continuous 360-degree view.

[0715] Step 3:

[0716] The terminal transmits the captured video data to the analysis means.

[0717] Input: Real-time video data

[0718] Output: Video data sent to analysis means

[0719] Specific operation: The video data captured by the camera is sent to the internal high-performance processor, which allows it to proceed to the next stage of analysis.

[0720] Step 4:

[0721] The terminal uses object recognition means to identify objects in the video.

[0722] Input: Transmitted video data

[0723] Output: Recognized object data

[0724] How it works: Object recognition methods such as the YOLO algorithm analyze video data and identify objects such as cars and pedestrians. The results are updated for each frame.

[0725] Step 5:

[0726] The terminal performs a risk assessment using a risk assessment means.

[0727] Input: Recognized object data

[0728] Output: The result of the risk assessment

[0729] Specific operation: Based on data such as the speed and distance of the recognized object, the risk assessment method evaluates the possibility of contact. For example, it immediately determines whether an approaching vehicle is within a dangerous distance.

[0730] Step 6:

[0731] The terminal uses the signal recognition means to identify the traffic signal.

[0732] Input: Video data

[0733] Output: Traffic light status (color)

[0734] Specific operation: The signal recognition means detects traffic signals from the video data and distinguishes their colors (red, yellow, green). For example, it analyzes the traffic light and recognizes a red signal.

[0735] Step 7:

[0736] The terminal generates an audio alert using the audio generating means to notify the user.

[0737] Input: Risk assessment results, traffic signal status

[0738] Output: Audio alert

[0739] Specific operation: A voice generation means (e.g., Google Cloud Text-to-Speech) generates a voice message based on the results of risk assessment and traffic signal recognition. This voice alert is delivered to the user through a notification means. For example, if there is a high risk of an approaching vehicle, the user will be notified by a voice saying, "There is a vehicle ahead. Please be careful."

[0740] (Application example 1)

[0741] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0742] In recent years, there has been a demand for improved safety in autonomous vehicles, but collisions with people and obstacles remain frequent. Technology that can properly recognize the surrounding situation in real time and respond immediately is lacking, especially in complex situations such as busy streets and intersections. There is also a lack of effective means to notify drivers of emergency situations using voice alerts. This calls for new technologies to improve the safety of autonomous vehicles and prevent accidents.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0744] In this invention, the server is equipped with a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, an object identification means for identifying objects from the video data in real time, a risk assessment means for assessing a risk within a specific distance based on the identified objects, and a means for generating a voice alert when the risk level exceeds an assessment standard. This enables the server to recognize the situation around the autonomous vehicle in real time and immediately provide a voice alert in an emergency, thereby improving safety.

[0745] A "camera device" is a device for acquiring video data.

[0746] The "analysis means" is a means having a function for processing the acquired video data and extracting specific information.

[0747] The "audio generating means" is a means having the function of generating an audio alert based on the analyzed data.

[0748] The "object identification means" is a means having a function for identifying an object from video data in real time.

[0749] A "danger assessment means" is a means having the function of assessing danger within a certain distance based on an identified object.

[0750] The "signal recognition means" is a means having a function of recognizing traffic signals based on image data acquired by a camera device.

[0751] "Real-time" refers to processing or reacting immediately, without any time delay.

[0752] "Audio alert" refers to a warning or guidance given to the user by voice.

[0753] "Evaluation criteria" refers to the numerical values ​​and conditions that serve as the criteria when the risk assessment means makes an assessment.

[0754] "Analyzed data" refers to data after it has been processed by an analytical means.

[0755] The present invention relates to a system for improving the safety of autonomous vehicles.

[0756] System Configuration

[0757] The system includes a camera device, an analysis means, a sound generation means, an object identification means, a risk assessment means, and a signal recognition means.

[0758] 1. Camera equipment

[0759] This camera device acquires surrounding video data in real time. In particular, a camera installed in an autonomous vehicle is used. A specific example of a camera is an in-vehicle camera system.

[0760] 2. Analysis method

[0761] The acquired video data is processed by the analysis means, which uses libraries such as OpenCV and TensorFlow. The analysis means works in conjunction with the object identification means to recognize specific objects such as pedestrians and other vehicles.

[0762] 3. Voice Generation Method

[0763] Based on the analyzed data, audio alerts are generated using gTTS (Google Text-to-Speech) and other technologies, which are then sent to the user via a speaker in real time.

[0764] 4. Object Identification Method

[0765] It uses deep learning models such as TensorFlow to identify objects in video data in real time, such as pedestrians, other vehicles, and road signs.

[0766] 5. Risk Assessment Tools

[0767] Based on the identified object, it evaluates the object's potential danger to the autonomous vehicle, based on criteria such as a specific distance, speed, direction, etc. If the danger exceeds the criteria, an audio alert is generated.

[0768] 6. Signal recognition means

[0769] The system recognizes traffic signals based on the video data acquired by the camera device. This recognition means also works in conjunction with the analysis means to recognize the traffic light status (red, yellow, green) and provide that information to the user.

[0770] Specific examples

[0771] As a specific example, consider a case where an autonomous vehicle approaches an intersection. A camera device captures video of the area around the intersection, and an object identification means identifies pedestrians and other vehicles. If a pedestrian is assessed as approaching the vehicle, the risk assessment means immediately determines this information and generates an audio alert saying, "Pedestrian approaching." If the traffic light is red, an audio alert saying, "Red light. Please wait" is also generated.

[0772] Prompt Sentence Examples

[0773] Below is an example of a prompt for a program in a system for autonomous vehicles that recognizes the surrounding situation in real time and provides a voice alert in the event of an emergency.

[0774] Prompt: Write a program for a system that uses dashcam footage to detect pedestrians near intersections in real time and provide audio alerts.

[0775] This allows the present invention to improve safety by providing real-time awareness of the situation around an autonomous vehicle and providing immediate audio alerts in the event of an emergency.

[0776] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0777] Step 1:

[0778] The device activates a camera device and captures video data of the surroundings in real time. The input is video data from the camera. The camera captures the surroundings at a high frame rate and continuously collects the video data. The output is video data for each frame.

[0779] Step 2:

[0780] The device sends the captured video data to the analysis means. The input is the video data obtained in step 1. The video data is preprocessed using the OpenCV library and converted into an appropriate format. This process allows the analysis means to process the data efficiently. The output is the preprocessed video data.

[0781] Step 3:

[0782] The analysis means uses the preprocessed video data to perform object identification using the object identification means. The preprocessed video data is used as input. Object identification is performed using deep learning using a generative AI model from TensorFlow. This process identifies specific objects such as pedestrians and vehicles from each frame of video. The output is information about the identified objects (position, type).

[0783] Step 4:

[0784] The terminal performs a risk assessment using the risk assessment means based on the object information identified by the object identification means. The identified object information is input. Based on this information, the distance to the object, relative speed, direction, etc. are evaluated. Here, an evaluation standard is set, and if the risk is high, a flag is raised as a high risk. The risk assessment result is obtained as the output.

[0785] Step 5:

[0786] The terminal uses the voice generation means to generate the necessary voice alert based on the result of the danger assessment means. The input is the danger assessment result. Using gTTS (Google Text-to-Speech), a voice message is generated according to the assessment result. For example, if a pedestrian is approaching, a message such as "Pedestrian approaching" is generated, and if a stop is required at a red light, a message such as "Red light. Please wait" is generated. The output is the generated voice alert (voice file).

[0787] Step 6:

[0788] The device notifies the user of the generated audio alert. It has as input the generated audio alert. It plays the audio file through the speaker to provide a real-time warning to the user. It has as output the audio alert delivered to the user.

[0789] This allows the device to monitor the autonomous vehicle's surroundings in real time and generate and provide instant voice alerts when necessary, improving driver safety.

[0790] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0791] The present invention relates to a system that includes a camera device, analyzes video data, and generates an audio alert. The present invention also relates to a system that includes emotion recognition means that recognizes a user's emotion and adjusts the content and tone of the audio alert accordingly. Hereinafter, an embodiment of the present invention will be described.

[0792] System Configuration

[0793] The device is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data, including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0794] System Operation

[0795] 1. When a user puts on the camera-equipped audio glasses, the device starts the camera device and starts capturing video data, including not only the surrounding scenery but also the user's face.

[0796] 2. The device sends the video data to the analysis means, which first uses the object recognition means to identify objects in the video (e.g., pedestrians, cars, and other obstacles). Based on this, the risk assessment means assesses the risk.

[0797] 3. The device then uses its signal recognition means to identify traffic signals in the video. Specifically, it recognizes the color of the signal (red, yellow, green) and determines its status.

[0798] 4. The device further uses emotion recognition means to recognize emotions from the user's facial expressions, including algorithms for identifying emotional states such as smiling, angry, surprised, etc.

[0799] 5. The device will comprehensively evaluate the analysis results from the object recognition means, signal recognition means, and emotion recognition means and generate the necessary audio alert. For example, if a dangerous object is detected and the user is startled, the tone of the audio alert will be adjusted to be softer.

[0800] 6. The device then generates and sends the generated audio alert to the user in real time, adjusting the tone and content of the audio notification based on the emotion recognition results.

[0801] Specific examples

[0802] Suppose a user is walking in a busy shopping district. The device's camera captures video, and the object recognition means detects vehicles and pedestrians in the video. If the risk assessment means detects a car suddenly approaching at an intersection, it determines this as "danger."

[0803] The device also analyzes the user's facial expression using emotion recognition means and recognizes that the user is nervous. In this case, the device generates a voice alert in a calm tone saying, "Please be careful." If the traffic light recognition means recognizes a red light, the device generates a voice alert saying, "The light is red, please wait," to notify the user.

[0804] In this way, the present invention takes into account the user's emotional state, recognizes surrounding hazards and traffic signals in real time, and provides tailored audio alerts to the user, supporting safe and secure behavior.

[0805] The processing flow will be explained below.

[0806] Step 1:

[0807] The device initializes the camera and starts capturing video data, capturing images of the surrounding scenery and the user's face in real time.

[0808] Step 2:

[0809] The terminal transmits the captured video frames to the analysis means, which receives them and uses the object recognition means to detect objects (e.g., cars, pedestrians, and obstacles) in the frames.

[0810] Step 3:

[0811] The terminal uses the risk assessment means to assess the risk based on the output of the object recognition means. For example, if an approaching vehicle is within a certain distance, it is judged to be "dangerous."

[0812] Step 4:

[0813] The device uses a signal recognition means to analyze traffic signals in the video frames, recognize the signal color (red, yellow, green) and determine its status. This information is used in the next step.

[0814] Step 5:

[0815] The device analyzes the user's facial expressions using an emotion recognition means. The emotion recognition means identifies emotions (e.g., tension, surprise, joy) from the user's facial expressions in the video. The device then determines the user's emotional state from the facial expressions.

[0816] Step 6:

[0817] The device combines analysis results from object recognition, signal recognition, and emotion recognition methods, so if the user is nervous, for example, it will soften the content and tone of the audio alerts.

[0818] Step 7:

[0819] The terminal generates an appropriate audio alert using the audio generation means based on the integration result. For example, if a dangerous object is detected and the user is startled, the terminal generates an audio alert saying "Be careful" in a calm tone.

[0820] Step 8:

[0821] The device generates audio alerts and notifies the user in real time, driving a speech synthesis engine to play audio with adjusted tone and content to inform the user of the situation.

[0822] Step 9:

[0823] Users can take appropriate action based on the audio alerts provided, such as stopping at a red light when necessary or being aware of hazards.

[0824] This step allows the device to analyze the surrounding video in real time and provide audio alerts that take into account the user's emotional state, with the aim of ensuring the user's safety and peace of mind.

[0825] Example 2

[0826] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0827] Conventional systems can analyze video data captured by cameras and generate audio alerts, but they are unable to generate appropriate alerts based on the user's emotions. This creates the risk of the user being surprised or making a wrong decision. Furthermore, comprehensive risk assessment is not adequately performed when recognizing traffic signals and surrounding objects.

[0828] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0829] In this invention, the server is equipped with a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, and an emotion recognition means for recognizing the user's emotion and adjusting the content and tone of the voice alert accordingly, thereby enabling the server to generate and notify an appropriate voice alert in real time according to the user's emotional state.

[0830] "Camera device" refers to a device for acquiring video data, capturing the user's surroundings and facial expressions.

[0831] "Analysis means" refers to a function or device for processing video data acquired by a camera device and performing object recognition, signal recognition, and emotion recognition.

[0832] "Voice generation means" refers to a function or device for generating a voice alert based on the analyzed data, and notifying the user of necessary information by voice.

[0833] "Emotion recognition means" refers to a function or device for analyzing the user's facial expression data and identifying the user's emotional state (e.g., smile, anger, surprise).

[0834] "Object recognition means" refers to a function or device for identifying surrounding objects (e.g., pedestrians, automobiles, and other obstacles) based on video data.

[0835] The "risk assessment means" refers to a function or device for conducting risk assessment and determining danger based on the object recognized by the object recognition means.

[0836] "Signal recognition means" refers to a function or device for recognizing traffic signals (e.g., red, yellow, green) based on video data.

[0837] MODE FOR CARRYING OUT THE INVENTION

[0838] The present invention relates to a system that includes a camera device, analyzes video data, and generates an audio alert. The present invention also relates to a system that includes emotion recognition means that recognizes a user's emotion and adjusts the content and tone of the audio alert accordingly. Hereinafter, embodiments of the present invention will be described in detail.

[0839] System Configuration

[0840] The device is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data, including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0841] System Operation

[0842] When a user puts on the camera-equipped audio glasses, the device activates the camera and begins capturing video data. This includes not only the surrounding scenery but also the user's face. The device then sends the video data to the analysis means. The analysis means first uses the object recognition means to identify objects in the video (e.g., pedestrians, cars, and other obstacles). Based on this, the risk assessment means evaluates the risk. Next, the signal recognition means identifies traffic signals in the video. Specifically, it recognizes the color of the signal (red, yellow, green) and determines its status. The emotion recognition means also recognizes emotions from the user's facial expressions. This includes algorithms for identifying emotional states such as smiling, anger, and surprise. The device comprehensively evaluates the analysis results from the object recognition means, signal recognition means, and emotion recognition means and generates the necessary audio alert. The generated audio alert is then notified to the user in real time. The tone and content of the audio alert are adjusted according to the emotion recognition results.

[0843] Specific hardware and software names used

[0844] 1. Camera device: A device that captures the user's surroundings and facial expressions in real time.

[0845] 2. Analysis method: A computer for analyzing video data, using object recognition algorithms such as TensorFlow and YOLOv3, image processing libraries such as OpenCV, and deep learning algorithms such as the Facial Emotion Recognition (FER) model.

[0846] 3. Voice generation means: Has the function of generating a voice alert based on the analysis results.

[0847] 4. Scene capture and facial recognition software: Software used to capture and analyze video data in real time.

[0848] Specific examples

[0849] Example 1:

[0850] Assume that a user is walking in a busy shopping district. At this time, the device's camera device captures video, and the object recognition means detects vehicles and pedestrians in the video. If the risk assessment means detects a car suddenly approaching at an intersection, it judges this to be "danger." The device also analyzes the user's facial expression using the emotion recognition means and recognizes that the user looks nervous. In this case, the device generates a voice alert in a calm tone saying, "Be careful." If the traffic light recognition means recognizes a red light, it generates a voice alert saying, "The light is red, please wait," to notify the user.

[0851] Example 2:

[0852] Assume that a user is walking in a park. The device's camera captures video, and the object recognition means detects children and dogs playing. If the danger assessment means determines that there is no particular danger, the device generates a voice alert in a normal tone saying, "Be careful of your surroundings." The device also recognizes through its emotion recognition means that the user is relaxed. In this case, the voice alert is provided in a calm tone while still raising caution.

[0853] Prompt Sentence Examples

[0854] "Generate an audio alert if a car approaches quickly while walking downtown. If the user is nervous, tone down the tone of the alert."

[0855] "Generate audio alerts to get your attention based on your surroundings while walking through the park. If the user is relaxed, notify them in a gentle tone."

[0856] The present invention supports safe and secure behavior by taking into account the user's emotional state in real time, recognizing surrounding dangers and traffic signal information, and providing appropriate audio alerts.

[0857] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0858] Step 1:

[0859] When a user puts on the camera-equipped audio glasses, the device activates the camera and starts capturing video data. The input is a video of the user's environment and face, and the output is real-time video data. This video data includes the user's facial expressions and the surrounding scenery. Specifically, the camera captures video at a rate of 30 frames per second (fps).

[0860] Step 2:

[0861] The terminal transmits the captured video data to the analysis means. The input is real-time video data, and the output is data transmission to the analysis means. The video data is temporarily stored in a buffer, and individual frames are analyzed sequentially. Specifically, the video data is stored in a buffer and prepared for analysis.

[0862] Step 3:

[0863] The device's analysis means uses an object recognition algorithm (e.g., YOLOv3) to identify objects in the video. The input is each frame of video data stored in the buffer, and the output is information about the recognized objects. Specifically, it detects objects such as cars and pedestrians and identifies their location and type. For example, a specific set of pixels in a frame is recognized as a "car," and that information is passed on to the next step.

[0864] Step 4:

[0865] The device's analysis method uses image processing libraries such as OpenCV to identify traffic signals in the video. The input is each frame of video data stored in the buffer, and the output is the recognized signal color (red, yellow, green). Specifically, it detects the location of the traffic signal and analyzes its color from pixel values. For example, if a "red light" is detected, that information influences the next step.

[0866] Step 5:

[0867] The device's analysis means analyzes the user's facial expression data using an emotion recognition algorithm (e.g., Facial Emotion Recognition). The input is the video data of the user's face stored in a buffer, and the output is the recognized emotion (e.g., smile, anger, surprise). Specifically, the emotion of "surprise" is recognized from the movement of the user's eyebrows and the shape of their mouth.

[0868] Step 6:

[0869] The device comprehensively evaluates the analysis results obtained from the object recognition means, signal recognition means, and emotion recognition means, and generates the necessary voice alert. The input is the analysis result from each recognition means, and the output is the generated voice alert. For example, if a dangerous object is detected and the user is startled, a gentle voice alert such as "It's dangerous, please be careful" is generated.

[0870] Step 7:

[0871] The terminal notifies the user of the generated audio alert in real time. The input is the generated audio alert, and the output is a notification to the user as an audio alert. In specific operations, the audio generating means generates an audio alert with a predetermined tone, which is notified through the user's audio glasses.

[0872] (Application example 2)

[0873] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0874] Conventional camera devices and video analysis systems can analyze captured video data and provide audio alerts to warn users of danger and other dangers. However, these systems are often limited in their effectiveness because they do not take into account the user's emotional state. Furthermore, detecting suspicious individuals or dangerous objects through simple object recognition alone may not be sufficient. Therefore, there is a need for systems that can recognize the user's emotional state in real time and appropriately adjust the tone and content of alerts based on that information.

[0875] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server is equipped with a camera device and includes analysis means for analyzing video data acquired by the camera device, audio generation means for generating an audio alert based on the analyzed data, emotion recognition means for recognizing the user's emotion by the analysis means, and means for adjusting the tone and content of the audio alert based on the emotional state recognized by the emotion recognition means. This enables appropriate and effective audio alerts to be generated in accordance with the user's emotional state.

[0876] A "camera device" is an optical device for acquiring video data.

[0877] "Video data" refers to a series of image information captured by a camera device.

[0878] "Analysis means" refers to equipment or software that processes acquired video data and extracts and analyzes specific information.

[0879] "Audio generating means" refers to a device or software that generates an audio alert based on the data obtained by the analyzing means.

[0880] The "emotion recognition means" is a device or software for analyzing the facial expressions and movements of the user in the video data and determining the user's emotional state.

[0881] "Tone" refers to the manner of delivery and tone of voice of the audio alert, which is adjusted according to emotion and urgency.

[0882] "Content" refers to the specific message or information in the audio alert.

[0883] "Object recognition means" refers to a device or software for identifying objects in video data and extracting information about them.

[0884] The "risk assessment means" is a device or software for assessing potential danger based on the information obtained by the object recognition means and determining the risk level.

[0885] A "signal recognition means" is a device or software for identifying a traffic signal within the video data and determining whether it is in a particular state.

[0886] The present invention relates to a system that includes a camera device, analyzes captured video data, and generates an audio alert, and further relates to a system that recognizes a user's emotions and adjusts the content and tone of the audio alert according to the emotions.

[0887] System Configuration

[0888] The terminal is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a danger assessment means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[0889] Hardware and software used

[0890] Hardware:

[0891] Camera device (e.g. webcam)

[0892] Microphone and speaker unit

[0893] software:

[0894] TensorFlow (object recognition and emotion recognition models)

[0895] OpenCV (camera image capture and processing)

[0896] gTTS (Google Text-to-Speech, generating audio alerts)

[0897] Data processing and calculation

[0898] The angle is initially input from the video data captured by the camera. The video data is analyzed by the analysis means in the following steps:

[0899] 1. Object recognition:

[0900] OpenCV is used to detect objects in the environment, and a TensorFlow model is used to identify specific objects.

[0901] 2. Risk Assessment:

[0902] The system evaluates the risk level based on the objects identified by the object recognition means. For example, if it detects suspicious movements, it will determine that the risk is high.

[0903] 3. Signal recognition:

[0904] A signal recognition means identifies traffic signals in the video data and identifies their status (red, yellow, green).

[0905] 4. Emotion recognition:

[0906] Using a TensorFlow model, we recognize the user's emotional state from the captured facial expressions, thereby determining whether the user is surprised or relaxed.

[0907] 5. Audio alert generation:

[0908] Based on the analysis results and the user's emotional state, gTTS is used to generate an audio alert with the optimal tone and content.

[0909] Specific examples

[0910] As a concrete example of how the system works, consider the case where a suspicious individual breaks into a room while the user is inside. The camera captures the scene, and the object recognition means identifies the presence of the suspicious individual. The danger assessment means determines that the risk is high, and the emotion recognition means detects the user's surprised facial expression. Based on the results of this analysis, gTTS generates a calm-toned voice alert, such as "Please be careful, a suspicious individual has been detected," to notify the user.

[0911] Prompt Sentence Examples

[0912] If the user is startled, create a program that will alert them in a calm tone, saying "Caution, a suspicious person has been detected."

[0913] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0914] Step 1:

[0915] The device activates the camera device and captures video data in real time. The input includes the surrounding scenery and the user's face. The output is the captured video data.

[0916] Step 2:

[0917] The device receives video data captured using OpenCV and performs object recognition. The input video data is analyzed using an object recognition model (TensorFlow). The output is information about the identified objects. Specifically, it detects moving objects and distinctive shapes in the video and classifies them as objects.

[0918] Step 3:

[0919] The terminal uses the output of object recognition to perform risk assessment. The input is the result of object recognition, and the risk is assessed by the risk assessment means. The output is the risk level assessment result. Specifically, if suspicious movement is detected, the situation is judged to be "high risk."

[0920] Step 4:

[0921] The device uses analysis means to extract the user's facial expressions from the video data and perform emotion recognition. The input is the captured video data, which is analyzed by an emotion recognition model (TensorFlow). The output is the user's emotional state. Specifically, it identifies emotions such as whether the user is surprised, nervous, or relaxed.

[0922] Step 5:

[0923] The terminal uses a signal recognition means to identify traffic signals in the video data. The input is the captured video data, and the analysis means determines the color and state of the signal. The output is the signal state (e.g., red, yellow, green). Specifically, if a red signal is displayed, it is determined to be a "stop" signal.

[0924] Step 6:

[0925] The device integrates the results of object recognition, danger assessment, emotion recognition, and signal recognition, and generates a voice alert using a voice generation means. The input is the analysis results, and gTTS is used to create a voice alert with an appropriate tone and content. The output is the generated voice alert. Specifically, if the user is surprised, it generates an alert with a calm tone such as "Be careful, a suspicious person has been detected."

[0926] Step 7:

[0927] The device notifies the user of the generated audio alert. The input is the audio alert data, and the output is a warning message that is audibly provided to the user. Specifically, the audio alert is played using a speaker.

[0928] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0929] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0930] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0931] [Fourth embodiment]

[0932] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0933] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0934] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0935] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0936] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0937] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0938] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0939] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0940] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0941] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0942] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0943] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0944] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0945] The present invention relates to a system that includes a camera device, analyzes video data, and generates a voice alert. Hereinafter, an embodiment of the present invention will be described.

[0946] System Configuration

[0947] The terminal is equipped with a camera device that captures images of the surroundings in real time. The captured image data is processed by an analysis means to extract specific information (e.g., object identification and traffic signal recognition). Based on the analysis results, an audio generation means generates an audio alert to notify the user.

[0948] System Operation

[0949] 1. When a user wears the camera-equipped audio glasses, the device activates the camera and starts capturing video data continuously.

[0950] 2. The device sends the captured video data to the analysis means, and the object recognition means identifies objects in the video, for example, using a specific algorithm to detect cars, pedestrians, and other obstacles.

[0951] 3. The terminal performs risk assessment using the danger assessment means based on the object recognized by the object recognition means, for example, if an approaching vehicle is within a dangerous distance, the terminal evaluates the information.

[0952] 4. The device uses a signal recognition means to identify traffic signals in the video. Specifically, it identifies the color of the signal (red, yellow, green) and determines its status.

[0953] 5. The device uses the voice generation means to generate necessary voice alerts based on the analysis results, for example, when a dangerous object is detected or a red light is recognized, it generates a corresponding voice message to notify the user.

[0954] Specific examples

[0955] Suppose a user is walking down a busy street. The device's camera captures video and its object recognition system detects vehicles and pedestrians in the video. If a pedestrian suddenly tries to cross the road, the device immediately assesses this risk and generates a voice alert saying, "There is a pedestrian ahead. Please be careful."

[0956] Next, when the user approaches an intersection and the traffic light recognition means recognizes that the traffic light is red, the device provides a voice alert saying, "Red light. Please wait." When the light turns green, the device generates a voice alert saying, "Green light. Please proceed."

[0957] The system provides users with real-time audio information about their surroundings, helping to ensure the safety of people with visual impairments and those who are distracted.

[0958] The processing flow will be explained below.

[0959] Step 1:

[0960] The device initializes the camera and starts capturing video data, which specifically means configuring the camera device and starting continuous frame capture.

[0961] Step 2:

[0962] The device sends the captured video frames to the analysis means. The analysis means receives the video data and detects objects in the frames using the object recognition means. For example, it applies a specific algorithm (e.g., YOLO, Haar Cascades, etc.) to identify objects (pedestrians, cars, obstacles, etc.).

[0963] Step 3:

[0964] The terminal evaluates the risk using the danger evaluation means based on the output of the object recognition means. If a certain distance or speed is exceeded, it can be judged as "dangerous." The evaluation result is passed to the next processing step.

[0965] Step 4:

[0966] The device uses a signal recognition means to identify traffic signals from video frames captured by the camera. Specifically, it recognizes the color of the signal and determines its status (red, yellow, green). For example, it uses color region extraction and pattern recognition to identify the color of the signal.

[0967] Step 5:

[0968] The device generates an appropriate voice alert using the voice generation means based on the analysis results from the danger assessment means and the signal recognition means. For example, when a red light is recognized or a dangerous object is detected, it generates a voice message such as "Red light" or "Dangerous object present," respectively.

[0969] Step 6:

[0970] The device notifies the user of the generated voice alert by activating a speech synthesis engine and providing a voice alert to the user in real time, allowing the user to take appropriate action.

[0971] Step 7:

[0972] Based on the audio alerts, the user can take appropriate action depending on the surrounding situation, for example, stopping when hearing the audio alert for a red light, or crossing the road when hearing the audio alert for a green light.

[0973] This series of steps allows the system to recognize surrounding hazards and traffic signals in real time and provide appropriate audio alerts to the user.

[0974] Example 1

[0975] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0976] In modern society, people with visual impairments and those who are distracted need to be able to travel safely. However, current safety assistance devices often have difficulty accurately recognizing their surroundings in real time and providing timely warnings. They also lack the ability to accurately recognize traffic signal status and approaching dangerous objects and communicate them safely. These situations make it difficult to ensure the safety of people with visual impairments.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0978] In this invention, the server includes a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, a risk assessment means for determining the surrounding situation in real time, a signal recognition means for determining the status of traffic signals, and a notification means for notifying the user of the voice alert, thereby enabling the user to accurately grasp the surrounding situation in real time and receive a warning at an appropriate time.

[0979] A "camera device" is a device for acquiring video data.

[0980] The "analysis means" is a device or software that processes the video data acquired by the camera device and extracts specific information.

[0981] "Audio generating means" refers to a device or software that generates an audio alert based on the data obtained by the analyzing means.

[0982] "Risk assessment means" means a device or software that assesses the surrounding situation in real time, in particular by performing risk assessment based on recognized objects and traffic signals.

[0983] "Signal recognition means" refers to a device or software that recognizes the state of a traffic signal (red, yellow, green) based on acquired image data.

[0984] A "notification means" is a device or software for notifying a user of a generated audio alert.

[0985] This invention is a system that uses a device equipped with a camera to capture and analyze surrounding video data in real time, generating audio alerts, primarily for the purpose of ensuring the safety of people with visual impairments and those who are distracted.

[0986] Hardware and software used

[0987] Device: For example, audio glasses with a camera

[0988] Camera device: Camera built into the device

[0989] Analysis tools: High-performance processors and dedicated object recognition algorithms (e.g., YOLO, TensorFlow)

[0990] Voice Generation Method: Text-to-Speech (TTS) software (e.g., Google Cloud Text-to-Speech, Amazon Polly)

[0991] Hazard Assessment Tools: Risk Assessment Algorithms

[0992] Signal Recognition: An algorithm for identifying the color of traffic signals

[0993] Notification Method: Speaker or earphone to notify the user of audio alerts

[0994] System Configuration

[0995] 1. The terminal uses a camera device to capture video data of the user's surroundings in real time.

[0996] 2. The captured video data is sent to the device's built-in analysis means, which uses an object recognition algorithm (e.g., YOLO).

[0997] 3. The device uses analytical means to identify objects in the video (cars, pedestrians, traffic lights, etc.).

[0998] 4. Based on the identified object, the danger assessment means assesses the risk, for example, if an approaching car is at a dangerous distance, the information is assessed immediately.

[0999] 5. The device uses signal recognition means to identify traffic signals in the video, for example, by determining the color of the signal (red, yellow, green) and determining its status.

[1000] 6. Finally, the necessary audio alert is generated using the audio generation means based on the analysis results, and the generated audio alert is notified to the user through the notification means.

[1001] Specific examples

[1002] Consider a scenario where a user is walking in a busy downtown area. The device's camera captures video and the acquired data is sent to an analysis means (e.g., the YOLO algorithm). The object recognition means detects vehicles and pedestrians, and the risk assessment means immediately assesses the risk. If a pedestrian suddenly tries to cross the road, the device generates a voice alert saying, "There is a pedestrian ahead. Please be careful," and notifies the user through a notification means.

[1003] Next, when the user approaches an intersection with a traffic light, the traffic light recognition means determines the color of the traffic light, and when it recognizes a red light, it generates a voice alert saying, "Red light. Please wait." When the light turns green, it provides an alert saying, "Green light. Please proceed."

[1004] Prompt Sentence Examples

[1005] The generative AI model generates a voice alert by providing a prompt like this:

[1006] There is a pedestrian ahead. Please be careful.

[1007] Red light, please wait. Green light, please go ahead.

[1008] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1009] Step 1:

[1010] The user wears audio glasses with a camera.

[1011] Input: Audio glasses with camera

[1012] Output: System boot

[1013] Specific operation: When the user puts on the audio glasses and turns on the power switch, the system starts up.

[1014] Step 2:

[1015] The terminal activates the camera device and captures the video data.

[1016] Input: Powered camera device

[1017] Output: Real-time video data

[1018] What it does: The camera device begins capturing video of the surroundings in real time, for example, capturing a continuous 360-degree view.

[1019] Step 3:

[1020] The terminal transmits the captured video data to the analysis means.

[1021] Input: Real-time video data

[1022] Output: Video data sent to analysis means

[1023] Specific operation: The video data captured by the camera is sent to the internal high-performance processor, which allows it to proceed to the next stage of analysis.

[1024] Step 4:

[1025] The terminal uses object recognition means to identify objects in the video.

[1026] Input: Transmitted video data

[1027] Output: Recognized object data

[1028] How it works: Object recognition methods such as the YOLO algorithm analyze video data and identify objects such as cars and pedestrians. The results are updated for each frame.

[1029] Step 5:

[1030] The terminal performs a risk assessment using a risk assessment means.

[1031] Input: Recognized object data

[1032] Output: The result of the risk assessment

[1033] Specific operation: Based on data such as the speed and distance of the recognized object, the risk assessment method evaluates the possibility of contact. For example, it immediately determines whether an approaching vehicle is within a dangerous distance.

[1034] Step 6:

[1035] The terminal uses the signal recognition means to identify the traffic signal.

[1036] Input: Video data

[1037] Output: Traffic light status (color)

[1038] Specific operation: The signal recognition means detects traffic signals from the video data and distinguishes their colors (red, yellow, green). For example, it analyzes the traffic light and recognizes a red signal.

[1039] Step 7:

[1040] The terminal generates an audio alert using the audio generating means to notify the user.

[1041] Input: Risk assessment results, traffic signal status

[1042] Output: Audio alert

[1043] Specific operation: A voice generation means (e.g., Google Cloud Text-to-Speech) generates a voice message based on the results of risk assessment and traffic signal recognition. This voice alert is delivered to the user through a notification means. For example, if there is a high risk of an approaching vehicle, the user will be notified by a voice saying, "There is a vehicle ahead. Please be careful."

[1044] (Application example 1)

[1045] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1046] In recent years, there has been a demand for improved safety in autonomous vehicles, but collisions with people and obstacles remain frequent. Technology that can properly recognize the surrounding situation in real time and respond immediately is lacking, especially in complex situations such as busy streets and intersections. There is also a lack of effective means to notify drivers of emergency situations using voice alerts. This calls for new technologies to improve the safety of autonomous vehicles and prevent accidents.

[1047] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1048] In this invention, the server is equipped with a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, an object identification means for identifying objects from the video data in real time, a risk assessment means for assessing a risk within a specific distance based on the identified objects, and a means for generating a voice alert when the risk level exceeds an assessment standard. This enables the server to recognize the situation around the autonomous vehicle in real time and immediately provide a voice alert in an emergency, thereby improving safety.

[1049] A "camera device" is a device for acquiring video data.

[1050] The "analysis means" is a means having a function for processing the acquired video data and extracting specific information.

[1051] The "audio generating means" is a means having the function of generating an audio alert based on the analyzed data.

[1052] The "object identification means" is a means having a function for identifying an object from video data in real time.

[1053] A "danger assessment means" is a means having the function of assessing danger within a certain distance based on an identified object.

[1054] The "signal recognition means" is a means having a function of recognizing traffic signals based on image data acquired by a camera device.

[1055] "Real-time" refers to processing or reacting immediately, without any time delay.

[1056] "Audio alert" refers to a warning or guidance given to the user by voice.

[1057] "Evaluation criteria" refers to the numerical values ​​and conditions that serve as the criteria when the risk assessment means makes an assessment.

[1058] "Analyzed data" refers to data after it has been processed by an analytical means.

[1059] The present invention relates to a system for improving the safety of autonomous vehicles.

[1060] System Configuration

[1061] The system includes a camera device, an analysis means, a sound generation means, an object identification means, a risk assessment means, and a signal recognition means.

[1062] 1. Camera equipment

[1063] This camera device acquires surrounding video data in real time. In particular, a camera installed in an autonomous vehicle is used. A specific example of a camera is an in-vehicle camera system.

[1064] 2. Analysis method

[1065] The acquired video data is processed by the analysis means, which uses libraries such as OpenCV and TensorFlow. The analysis means works in conjunction with the object identification means to recognize specific objects such as pedestrians and other vehicles.

[1066] 3. Voice Generation Method

[1067] Based on the analyzed data, audio alerts are generated using gTTS (Google Text-to-Speech) and other technologies, which are then sent to the user via a speaker in real time.

[1068] 4. Object Identification Method

[1069] It uses deep learning models such as TensorFlow to identify objects in video data in real time, such as pedestrians, other vehicles, and road signs.

[1070] 5. Risk Assessment Tools

[1071] Based on the identified object, it evaluates the object's potential danger to the autonomous vehicle, based on criteria such as a specific distance, speed, direction, etc. If the danger exceeds the criteria, an audio alert is generated.

[1072] 6. Signal recognition means

[1073] The system recognizes traffic signals based on the video data acquired by the camera device. This recognition means also works in conjunction with the analysis means to recognize the traffic light status (red, yellow, green) and provide that information to the user.

[1074] Specific examples

[1075] As a specific example, consider a case where an autonomous vehicle approaches an intersection. A camera device captures video of the area around the intersection, and an object identification means identifies pedestrians and other vehicles. If a pedestrian is assessed as approaching the vehicle, the risk assessment means immediately determines this information and generates an audio alert saying, "Pedestrian approaching." If the traffic light is red, an audio alert saying, "Red light. Please wait" is also generated.

[1076] Prompt Sentence Examples

[1077] Below is an example of a prompt for a program in a system for autonomous vehicles that recognizes the surrounding situation in real time and provides a voice alert in the event of an emergency.

[1078] Prompt: Write a program for a system that uses dashcam footage to detect pedestrians near intersections in real time and provide audio alerts.

[1079] This allows the present invention to improve safety by providing real-time awareness of the situation around an autonomous vehicle and providing immediate audio alerts in the event of an emergency.

[1080] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1081] Step 1:

[1082] The device activates a camera device and captures video data of the surroundings in real time. The input is video data from the camera. The camera captures the surroundings at a high frame rate and continuously collects the video data. The output is video data for each frame.

[1083] Step 2:

[1084] The device sends the captured video data to the analysis means. The input is the video data obtained in step 1. The video data is preprocessed using the OpenCV library and converted into an appropriate format. This process allows the analysis means to process the data efficiently. The output is the preprocessed video data.

[1085] Step 3:

[1086] The analysis means uses the preprocessed video data to perform object identification using the object identification means. The preprocessed video data is used as input. Object identification is performed using deep learning using a generative AI model from TensorFlow. This process identifies specific objects such as pedestrians and vehicles from each frame of video. The output is information about the identified objects (position, type).

[1087] Step 4:

[1088] The terminal performs a risk assessment using the risk assessment means based on the object information identified by the object identification means. The identified object information is input. Based on this information, the distance to the object, relative speed, direction, etc. are evaluated. Here, an evaluation standard is set, and if the risk is high, a flag is raised as a high risk. The risk assessment result is obtained as the output.

[1089] Step 5:

[1090] The terminal uses the voice generation means to generate the necessary voice alert based on the result of the danger assessment means. The input is the danger assessment result. Using gTTS (Google Text-to-Speech), a voice message is generated according to the assessment result. For example, if a pedestrian is approaching, a message such as "Pedestrian approaching" is generated, and if a stop is required at a red light, a message such as "Red light. Please wait" is generated. The output is the generated voice alert (voice file).

[1091] Step 6:

[1092] The device notifies the user of the generated audio alert. It has as input the generated audio alert. It plays the audio file through the speaker to provide a real-time warning to the user. It has as output the audio alert delivered to the user.

[1093] This allows the device to monitor the autonomous vehicle's surroundings in real time and generate and provide instant voice alerts when necessary, improving driver safety.

[1094] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1095] The present invention relates to a system that includes a camera device, analyzes video data, and generates an audio alert. The present invention also relates to a system that includes emotion recognition means that recognizes a user's emotion and adjusts the content and tone of the audio alert accordingly. Hereinafter, an embodiment of the present invention will be described.

[1096] System Configuration

[1097] The device is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data, including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[1098] System Operation

[1099] 1. When a user puts on the camera-equipped audio glasses, the device starts the camera device and starts capturing video data, including not only the surrounding scenery but also the user's face.

[1100] 2. The device sends the video data to the analysis means, which first uses the object recognition means to identify objects in the video (e.g., pedestrians, cars, and other obstacles). Based on this, the risk assessment means assesses the risk.

[1101] 3. The device then uses its signal recognition means to identify traffic signals in the video. Specifically, it recognizes the color of the signal (red, yellow, green) and determines its status.

[1102] 4. The device further uses emotion recognition means to recognize emotions from the user's facial expressions, including algorithms for identifying emotional states such as smiling, angry, surprised, etc.

[1103] 5. The device will comprehensively evaluate the analysis results from the object recognition means, signal recognition means, and emotion recognition means and generate the necessary audio alert. For example, if a dangerous object is detected and the user is startled, the tone of the audio alert will be adjusted to be softer.

[1104] 6. The device then generates and sends the generated audio alert to the user in real time, adjusting the tone and content of the audio notification based on the emotion recognition results.

[1105] Specific examples

[1106] Suppose a user is walking in a busy shopping district. The device's camera captures video, and the object recognition means detects vehicles and pedestrians in the video. If the risk assessment means detects a car suddenly approaching at an intersection, it determines this as "danger."

[1107] The device also analyzes the user's facial expression using emotion recognition means and recognizes that the user is nervous. In this case, the device generates a voice alert in a calm tone saying, "Please be careful." If the traffic light recognition means recognizes a red light, the device generates a voice alert saying, "The light is red, please wait," to notify the user.

[1108] In this way, the present invention takes into account the user's emotional state, recognizes surrounding hazards and traffic signals in real time, and provides tailored audio alerts to the user, supporting safe and secure behavior.

[1109] The processing flow will be explained below.

[1110] Step 1:

[1111] The device initializes the camera and starts capturing video data, capturing images of the surrounding scenery and the user's face in real time.

[1112] Step 2:

[1113] The terminal transmits the captured video frames to the analysis means, which receives them and uses the object recognition means to detect objects (e.g., cars, pedestrians, and obstacles) in the frames.

[1114] Step 3:

[1115] The terminal uses the risk assessment means to assess the risk based on the output of the object recognition means. For example, if an approaching vehicle is within a certain distance, it is judged to be "dangerous."

[1116] Step 4:

[1117] The device uses a signal recognition means to analyze traffic signals in the video frames, recognize the signal color (red, yellow, green) and determine its status. This information is used in the next step.

[1118] Step 5:

[1119] The device analyzes the user's facial expressions using an emotion recognition means. The emotion recognition means identifies emotions (e.g., tension, surprise, joy) from the user's facial expressions in the video. The device then determines the user's emotional state from the facial expressions.

[1120] Step 6:

[1121] The device combines analysis results from object recognition, signal recognition, and emotion recognition methods, so if the user is nervous, for example, it will soften the content and tone of the audio alerts.

[1122] Step 7:

[1123] The terminal generates an appropriate audio alert using the audio generation means based on the integration result. For example, if a dangerous object is detected and the user is startled, the terminal generates an audio alert saying "Be careful" in a calm tone.

[1124] Step 8:

[1125] The device generates audio alerts and notifies the user in real time, driving a speech synthesis engine to play audio with adjusted tone and content to inform the user of the situation.

[1126] Step 9:

[1127] Users can take appropriate action based on the audio alerts provided, such as stopping at a red light when necessary or being aware of hazards.

[1128] This step allows the device to analyze the surrounding video in real time and provide audio alerts that take into account the user's emotional state, with the aim of ensuring the user's safety and peace of mind.

[1129] Example 2

[1130] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1131] Conventional systems can analyze video data captured by cameras and generate audio alerts, but they are unable to generate appropriate alerts based on the user's emotions. This creates the risk of the user being surprised or making a wrong decision. Furthermore, comprehensive risk assessment is not adequately performed when recognizing traffic signals and surrounding objects.

[1132] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1133] In this invention, the server is equipped with a camera device and includes an analysis means for analyzing video data acquired by the camera device, a voice generation means for generating a voice alert based on the analyzed data, and an emotion recognition means for recognizing the user's emotion and adjusting the content and tone of the voice alert accordingly, thereby enabling the server to generate and notify an appropriate voice alert in real time according to the user's emotional state.

[1134] "Camera device" refers to a device for acquiring video data, capturing the user's surroundings and facial expressions.

[1135] "Analysis means" refers to a function or device for processing video data acquired by a camera device and performing object recognition, signal recognition, and emotion recognition.

[1136] "Voice generation means" refers to a function or device for generating a voice alert based on the analyzed data, and notifying the user of necessary information by voice.

[1137] "Emotion recognition means" refers to a function or device for analyzing the user's facial expression data and identifying the user's emotional state (e.g., smile, anger, surprise).

[1138] "Object recognition means" refers to a function or device for identifying surrounding objects (e.g., pedestrians, automobiles, and other obstacles) based on video data.

[1139] The "risk assessment means" refers to a function or device for conducting risk assessment and determining danger based on the object recognized by the object recognition means.

[1140] "Signal recognition means" refers to a function or device for recognizing traffic signals (e.g., red, yellow, green) based on video data.

[1141] MODE FOR CARRYING OUT THE INVENTION

[1142] The present invention relates to a system that includes a camera device, analyzes video data, and generates an audio alert. The present invention also relates to a system that includes emotion recognition means that recognizes a user's emotion and adjusts the content and tone of the audio alert accordingly. Hereinafter, embodiments of the present invention will be described in detail.

[1143] System Configuration

[1144] The device is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data, including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[1145] System Operation

[1146] When a user puts on the camera-equipped audio glasses, the device activates the camera and begins capturing video data. This includes not only the surrounding scenery but also the user's face. The device then sends the video data to the analysis means. The analysis means first uses the object recognition means to identify objects in the video (e.g., pedestrians, cars, and other obstacles). Based on this, the risk assessment means evaluates the risk. Next, the signal recognition means identifies traffic signals in the video. Specifically, it recognizes the color of the signal (red, yellow, green) and determines its status. The emotion recognition means also recognizes emotions from the user's facial expressions. This includes algorithms for identifying emotional states such as smiling, anger, and surprise. The device comprehensively evaluates the analysis results from the object recognition means, signal recognition means, and emotion recognition means and generates the necessary audio alert. The generated audio alert is then notified to the user in real time. The tone and content of the audio alert are adjusted according to the emotion recognition results.

[1147] Specific hardware and software names used

[1148] 1. Camera device: A device that captures the user's surroundings and facial expressions in real time.

[1149] 2. Analysis method: A computer for analyzing video data, using object recognition algorithms such as TensorFlow and YOLOv3, image processing libraries such as OpenCV, and deep learning algorithms such as the Facial Emotion Recognition (FER) model.

[1150] 3. Voice generation means: Has the function of generating a voice alert based on the analysis results.

[1151] 4. Scene capture and facial recognition software: Software used to capture and analyze video data in real time.

[1152] Specific examples

[1153] Example 1:

[1154] Assume that a user is walking in a busy shopping district. At this time, the device's camera device captures video, and the object recognition means detects vehicles and pedestrians in the video. If the risk assessment means detects a car suddenly approaching at an intersection, it judges this to be "danger." The device also analyzes the user's facial expression using the emotion recognition means and recognizes that the user looks nervous. In this case, the device generates a voice alert in a calm tone saying, "Be careful." If the traffic light recognition means recognizes a red light, it generates a voice alert saying, "The light is red, please wait," to notify the user.

[1155] Example 2:

[1156] Assume that a user is walking in a park. The device's camera captures video, and the object recognition means detects children and dogs playing. If the danger assessment means determines that there is no particular danger, the device generates a voice alert in a normal tone saying, "Be careful of your surroundings." The device also recognizes through its emotion recognition means that the user is relaxed. In this case, the voice alert is provided in a calm tone while still raising caution.

[1157] Prompt Sentence Examples

[1158] "Generate an audio alert if a car approaches quickly while walking downtown. If the user is nervous, tone down the tone of the alert."

[1159] "Generate audio alerts to get your attention based on your surroundings while walking through the park. If the user is relaxed, notify them in a gentle tone."

[1160] The present invention supports safe and secure behavior by taking into account the user's emotional state in real time, recognizing surrounding dangers and traffic signal information, and providing appropriate audio alerts.

[1161] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1162] Step 1:

[1163] When a user puts on the camera-equipped audio glasses, the device activates the camera and starts capturing video data. The input is a video of the user's environment and face, and the output is real-time video data. This video data includes the user's facial expressions and the surrounding scenery. Specifically, the camera captures video at a rate of 30 frames per second (fps).

[1164] Step 2:

[1165] The terminal transmits the captured video data to the analysis means. The input is real-time video data, and the output is data transmission to the analysis means. The video data is temporarily stored in a buffer, and individual frames are analyzed sequentially. Specifically, the video data is stored in a buffer and prepared for analysis.

[1166] Step 3:

[1167] The device's analysis means uses an object recognition algorithm (e.g., YOLOv3) to identify objects in the video. The input is each frame of video data stored in the buffer, and the output is information about the recognized objects. Specifically, it detects objects such as cars and pedestrians and identifies their location and type. For example, a specific set of pixels in a frame is recognized as a "car," and that information is passed on to the next step.

[1168] Step 4:

[1169] The device's analysis method uses image processing libraries such as OpenCV to identify traffic signals in the video. The input is each frame of video data stored in the buffer, and the output is the recognized signal color (red, yellow, green). Specifically, it detects the location of the traffic signal and analyzes its color from pixel values. For example, if a "red light" is detected, that information influences the next step.

[1170] Step 5:

[1171] The device's analysis means analyzes the user's facial expression data using an emotion recognition algorithm (e.g., Facial Emotion Recognition). The input is the video data of the user's face stored in a buffer, and the output is the recognized emotion (e.g., smile, anger, surprise). Specifically, the emotion of "surprise" is recognized from the movement of the user's eyebrows and the shape of their mouth.

[1172] Step 6:

[1173] The device comprehensively evaluates the analysis results obtained from the object recognition means, signal recognition means, and emotion recognition means, and generates the necessary voice alert. The input is the analysis result from each recognition means, and the output is the generated voice alert. For example, if a dangerous object is detected and the user is startled, a gentle voice alert such as "It's dangerous, please be careful" is generated.

[1174] Step 7:

[1175] The terminal notifies the user of the generated audio alert in real time. The input is the generated audio alert, and the output is a notification to the user as an audio alert. In specific operations, the audio generating means generates an audio alert with a predetermined tone, which is notified through the user's audio glasses.

[1176] (Application example 2)

[1177] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1178] Conventional camera devices and video analysis systems can analyze captured video data and provide audio alerts to warn users of danger and other dangers. However, these systems are often limited in their effectiveness because they do not take into account the user's emotional state. Furthermore, detecting suspicious individuals or dangerous objects through simple object recognition alone may not be sufficient. Therefore, there is a need for systems that can recognize the user's emotional state in real time and appropriately adjust the tone and content of alerts based on that information.

[1179] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server is equipped with a camera device and includes analysis means for analyzing video data acquired by the camera device, audio generation means for generating an audio alert based on the analyzed data, emotion recognition means for recognizing the user's emotion by the analysis means, and means for adjusting the tone and content of the audio alert based on the emotional state recognized by the emotion recognition means. This enables appropriate and effective audio alerts to be generated in accordance with the user's emotional state.

[1180] A "camera device" is an optical device for acquiring video data.

[1181] "Video data" refers to a series of image information captured by a camera device.

[1182] "Analysis means" refers to equipment or software that processes acquired video data and extracts and analyzes specific information.

[1183] "Audio generating means" refers to a device or software that generates an audio alert based on the data obtained by the analyzing means.

[1184] The "emotion recognition means" is a device or software for analyzing the facial expressions and movements of the user in the video data and determining the user's emotional state.

[1185] "Tone" refers to the manner of delivery and tone of voice of the audio alert, which is adjusted according to emotion and urgency.

[1186] "Content" refers to the specific message or information in the audio alert.

[1187] "Object recognition means" refers to a device or software for identifying objects in video data and extracting information about them.

[1188] The "risk assessment means" is a device or software for assessing potential danger based on the information obtained by the object recognition means and determining the risk level.

[1189] A "signal recognition means" is a device or software for identifying a traffic signal within the video data and determining whether it is in a particular state.

[1190] The present invention relates to a system that includes a camera device, analyzes captured video data, and generates an audio alert, and further relates to a system that recognizes a user's emotions and adjusts the content and tone of the audio alert according to the emotions.

[1191] System Configuration

[1192] The terminal is equipped with a camera device that captures images of the surroundings in real time. This camera device acquires video data including the user's facial expressions. The acquired video data is processed by an analysis means, and specific information is extracted and analyzed by an object recognition means, a danger assessment means, a signal recognition means, and an emotion recognition means. Based on the analysis results, a voice generation means generates a voice alert to notify the user.

[1193] Hardware and software used

[1194] Hardware:

[1195] Camera device (e.g. webcam)

[1196] Microphone and speaker unit

[1197] software:

[1198] TensorFlow (object recognition and emotion recognition models)

[1199] OpenCV (camera image capture and processing)

[1200] gTTS (Google Text-to-Speech, generating audio alerts)

[1201] Data processing and calculation

[1202] The angle is initially input from the video data captured by the camera. The video data is analyzed by the analysis means in the following steps:

[1203] 1. Object recognition:

[1204] OpenCV is used to detect objects in the environment, and a TensorFlow model is used to identify specific objects.

[1205] 2. Risk Assessment:

[1206] The system evaluates the risk level based on the objects identified by the object recognition means. For example, if it detects suspicious movements, it will determine that the risk is high.

[1207] 3. Signal recognition:

[1208] A signal recognition means identifies traffic signals in the video data and identifies their status (red, yellow, green).

[1209] 4. Emotion recognition:

[1210] Using a TensorFlow model, we recognize the user's emotional state from the captured facial expressions, thereby determining whether the user is surprised or relaxed.

[1211] 5. Audio alert generation:

[1212] Based on the analysis results and the user's emotional state, gTTS is used to generate an audio alert with the optimal tone and content.

[1213] Specific examples

[1214] As a concrete example of how the system works, consider the case where a suspicious individual breaks into a room while the user is inside. The camera captures the scene, and the object recognition means identifies the presence of the suspicious individual. The danger assessment means determines that the risk is high, and the emotion recognition means detects the user's surprised facial expression. Based on the results of this analysis, gTTS generates a calm-toned voice alert, such as "Please be careful, a suspicious individual has been detected," to notify the user.

[1215] Prompt Sentence Examples

[1216] If the user is startled, create a program that will alert them in a calm tone, saying "Caution, a suspicious person has been detected."

[1217] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1218] Step 1:

[1219] The device activates the camera device and captures video data in real time. The input includes the surrounding scenery and the user's face. The output is the captured video data.

[1220] Step 2:

[1221] The device receives video data captured using OpenCV and performs object recognition. The input video data is analyzed using an object recognition model (TensorFlow). The output is information about the identified objects. Specifically, it detects moving objects and distinctive shapes in the video and classifies them as objects.

[1222] Step 3:

[1223] The terminal uses the output of object recognition to perform risk assessment. The input is the result of object recognition, and the risk is assessed by the risk assessment means. The output is the risk level assessment result. Specifically, if suspicious movement is detected, the situation is judged to be "high risk."

[1224] Step 4:

[1225] The device uses analysis means to extract the user's facial expressions from the video data and perform emotion recognition. The input is the captured video data, which is analyzed by an emotion recognition model (TensorFlow). The output is the user's emotional state. Specifically, it identifies emotions such as whether the user is surprised, nervous, or relaxed.

[1226] Step 5:

[1227] The terminal uses a signal recognition means to identify traffic signals in the video data. The input is the captured video data, and the analysis means determines the color and state of the signal. The output is the signal state (e.g., red, yellow, green). Specifically, if a red signal is displayed, it is determined to be a "stop" signal.

[1228] Step 6:

[1229] The device integrates the results of object recognition, danger assessment, emotion recognition, and signal recognition, and generates a voice alert using a voice generation means. The input is the analysis results, and gTTS is used to create a voice alert with an appropriate tone and content. The output is the generated voice alert. Specifically, if the user is surprised, it generates an alert with a calm tone such as "Be careful, a suspicious person has been detected."

[1230] Step 7:

[1231] The device notifies the user of the generated audio alert. The input is the audio alert data, and the output is a warning message that is audibly provided to the user. Specifically, the audio alert is played using a speaker.

[1232] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1233] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1234] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1235] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1236] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1237] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1238] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1239] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1240] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1241] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1242] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1243] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1244] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1245] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1246] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1247] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1248] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1249] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1250] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1251] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1252] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1253] The following is further disclosed regarding the above embodiment.

[1254] (Claim 1)

[1255] Equipped with a camera device,

[1256] analysis means for analyzing video data acquired by the camera device;

[1257] an audio generating means for generating an audio alert based on the analyzed data;

[1258] A system including:

[1259] (Claim 2)

[1260] an object recognition means for recognizing surrounding objects based on video data acquired by the camera device;

[1261] a risk assessment means for assessing risk based on the object recognized by the object recognition means;

[1262] The system of claim 1, wherein an audio alert is generated based on the assessment result of the risk assessment means.

[1263] (Claim 3)

[1264] a signal recognition means for recognizing a traffic signal based on video data acquired by the camera device;

[1265] 10. The system of claim 1, wherein the system generates an audio alert when a recognized traffic signal is in a particular state.

[1266] "Example 1"

[1267] (Claim 1)

[1268] Equipped with a camera device,

[1269] analysis means for analyzing video data acquired by the camera device;

[1270] an audio generating means for generating an audio alert based on the analyzed data;

[1271] A risk assessment means for determining the surrounding situation in real time;

[1272] a signal recognition means for determining the state of a traffic signal;

[1273] a notification means for notifying a user of an audio alert;

[1274] A system including:

[1275] (Claim 2)

[1276] an object recognition means for recognizing surrounding objects based on video data acquired by the camera device;

[1277] a risk assessment means for assessing risk based on the object recognized by the object recognition means;

[1278] Generate audio alerts based on the results of the risk assessment measures

[1279] 10. The system of claim 1.

[1280] (Claim 3)

[1281] a signal recognition means for recognizing a traffic signal based on video data acquired by the camera device;

[1282] Generate audio alerts when recognized traffic signals are in a specific state

[1283] 10. The system of claim 1.

[1284] "Application Example 1"

[1285] (Claim 1)

[1286] Equipped with a camera device,

[1287] analysis means for analyzing video data acquired by the camera device;

[1288] an audio generating means for generating an audio alert based on the analyzed data;

[1289] an object identification means for identifying an object from video data in real time;

[1290] a risk assessment means for assessing risk within a certain distance based on the identified object;

[1291] means for generating an audio alert when the risk level exceeds the assessment criteria;

[1292] A system including:

[1293] (Claim 2)

[1294] an object recognition means for recognizing surrounding objects based on video data acquired by the camera device;

[1295] a risk assessment means for assessing risk based on the object recognized by the object recognition means;

[1296] 10. The system of claim 1, further comprising: generating an audio alert based on the evaluation result.

[1297] (Claim 3)

[1298] a signal recognition means for recognizing a traffic signal based on video data acquired by the camera device;

[1299] 10. The system of claim 1, wherein the system generates an audio alert when a recognized traffic signal is in a particular state.

[1300] "Example 2: Combining Emotion Engines"

[1301] (Claim 1)

[1302] Equipped with a camera device,

[1303] analysis means for analyzing video data acquired by the camera device;

[1304] an audio generating means for generating an audio alert based on the analyzed data;

[1305] emotion recognition means for recognizing a user's emotion and adjusting the content and tone of the audio alert accordingly;

[1306] A system including:

[1307] (Claim 2)

[1308] an object recognition means for recognizing surrounding objects based on video data acquired by the camera device;

[1309] a risk assessment means for assessing risk based on the object recognized by the object recognition means;

[1310] The system of claim 1, wherein an audio alert is generated based on the assessment result of the risk assessment means.

[1311] (Claim 3)

[1312] a signal recognition means for recognizing a traffic signal based on video data acquired by the camera device;

[1313] 10. The system of claim 1, wherein the system generates an audio alert when a recognized traffic signal is in a particular state.

[1314] "Application example 2 when combining emotion engines"

[1315] (Claim 1)

[1316] Equipped with a camera device,

[1317] analysis means for analyzing video data acquired by the camera device;

[1318] an audio generating means for generating an audio alert based on the analyzed data;

[1319] emotion recognition means for recognizing an emotion of a user by analysis means;

[1320] means for adjusting the tone and content of the audio alert based on the emotional state recognized by the emotion recognition means;

[1321] A system including:

[1322] (Claim 2)

[1323] an object recognition means for recognizing surrounding objects based on video data acquired by the camera device;

[1324] a risk assessment means for assessing risk based on the object recognized by the object recognition means;

[1325] The system according to claim 1, wherein an audio alert is generated based on the assessment result of the danger assessment means and the recognition result of the emotion recognition means.

[1326] (Claim 3)

[1327] a signal recognition means for recognizing a traffic signal based on video data acquired by the camera device;

[1328] 2. The system of claim 1, wherein the system generates a tailored audio alert based on the recognition result of the emotion recognition means when the recognized traffic signal is in a particular state. [Explanation of symbols]

[1329] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. Equipped with a camera device, analysis means for analyzing video data acquired by the camera device; an audio generating means for generating an audio alert based on the analyzed data; A system including:

2. an object recognition means for recognizing surrounding objects based on video data acquired by the camera device; a risk assessment means for assessing risk based on the object recognized by the object recognition means, The system of claim 1, wherein an audio alert is generated based on the assessment result of the risk assessment means.

3. a signal recognition means for recognizing a traffic signal based on the video data acquired by the camera device; The system of claim 1 , wherein the system generates an audio alert when a recognized traffic signal is in a particular state.

Images